HOW-TO:Help with quality assurance testing - Revision history

Karellen at 10:45, 20 July 2020

2020-07-20T10:45:55Z

← Older revision		Revision as of 10:45, 20 July 2020
Line 145:		Line 145:
	[[Category:How-to]]		[[Category:How-to]]
	[[Category:Development]]		[[Category:Development]]
	~~[[Category:Trac]]~~

Davu at 21:00, 9 October 2018

2018-10-09T21:00:31Z

← Older revision		Revision as of 21:00, 9 October 2018
Line 10:		Line 10:

	=== Specifically, what the QA team does is ===		=== Specifically, what the QA team does is ===
	# Step in as soon as a problem is reported on the forums or on our trac tracking-system, and attempt to help the user fix the problem, or confirm whether the problem is a bug by reproducing it. Once confirmed, your role is to ensure it gets properly reported in ~~trac~~, either by the original user or by doing it yourself.		# Step in as soon as a problem is reported on the forums or on our trac tracking-system, and attempt to help the user fix the problem, or confirm whether the problem is a bug by reproducing it. Once confirmed, your role is to ensure it gets properly reported in [https://github.com/xbmc/xbmc/issues GitHub], either by the original user or by doing it yourself.
	# Ensure all relevant information has been reported on ~~trac~~ tickets, and that it is reproducible. You should close any reports in which the user doesn't respond with needed information in a timely manner. You should then ensure the bug gets assigned to the appropriate developer, and track its progress so that it doesn't slip through the cracks. The report is considered in your hands until it has been confirmed as fixed by either yourself or the original user. When the developer needs more information, the QA team makes sure they get it.		# Ensure all relevant information has been reported on GitHub tickets, and that it is reproducible. You should close any reports in which the user doesn't respond with needed information in a timely manner. You should then ensure the bug gets assigned to the appropriate developer, and track its progress so that it doesn't slip through the cracks. The report is considered in your hands until it has been confirmed as fixed by either yourself or the original user. When the developer needs more information, the QA team makes sure they get it.
	# Perform regression testing. You should have a media library of various formats to test against our product, (a media library which of course could be a shared team resource), that can be used to regression test.		# Perform regression testing. You should have a media library of various formats to test against our product, (a media library which of course could be a shared team resource), that can be used to regression test.
	# Testing new and changed features or functions in our products, and the main application for consistency. You should be willing to test things that you might not normally use (as well as things you will of course be using all the time) as they are developed to ensure that our products stay at the leading edge in terms of both features and stability.		# Testing new and changed features or functions in our products, and the main application for consistency. You should be willing to test things that you might not normally use (as well as things you will of course be using all the time) as they are developed to ensure that our products stay at the leading edge in terms of both features and stability.
Line 19:		Line 19:

	== Eight Quick Steps to Triage Nirvana ==		== Eight Quick Steps to Triage Nirvana ==
	The main job of Quality Assurance is to triage bugs to our ~~trac~~ tracking-system or to report issues they have found their selves.		The main job of Quality Assurance is to triage bugs to our [https://github.com/xbmc/xbmc/issues GitHub] tracking-system or to report issues they have found their selves.

	=== Triaging a bug involves ===		=== Triaging a bug involves ===
Line 78:		Line 78:

	=== Add yourself to the CC list ===		=== Add yourself to the CC list ===
	By adding yourself to the CC list of bugs you change, you will receive e-mails with comments and changes others do to the individual report. You learn what further investigations others make and learn more rapidly about triaging bugs in the ~~trac~~ tracking-system.		By adding yourself to the CC list of bugs you change, you will receive e-mails with comments and changes others do to the individual report. You learn what further investigations others make and learn more rapidly about triaging bugs in the tracking-system.

Koder at 10:32, 7 March 2017

2017-03-07T10:32:27Z

← Older revision		Revision as of 10:32, 7 March 2017
Line 44:		Line 44:

	=== Does it make sense? ===		=== Does it make sense? ===
	Does it have enough information? Common sense rules here - "this sucks" or "this crashed" should be be asked for more information ~~polite message as~~ change the resolution to "''Need more information''", as should other things that are too ambiguous to be useful. It is not appropriate for bug volunteers to mark a ticket as "'''Will never be fixed/added''", that is up to the maintainers. The exception is crasher bugs that have a debug log, crash log or any other extend information that could help. See below for details; in general you should not change a bug to "''Need more information''" if it's got a good debug log (and stack trace if Linux). If you mark a bug "''Need more information''", "''Invalid''" or "''Incomplete''", it is useful to add yourself to the CC field; if the reporter adds more information, you can then reopen the bug and re-triage it. If a bug report has been set to "''Need more information''" state and if there has been no response for 4 weeks, feel free to close the report as "''Incomplete''" by using the "incomplete" stock response.		Does it have enough information? Common sense rules here - "this sucks" or "this crashed" should be asked for more information with the appropriate stock response and change the resolution to "''Need more information''", as should other things that are too ambiguous to be useful. It is not appropriate for bug volunteers to mark a ticket as "'''Will never be fixed/added''", that is up to the maintainers. The exception is crasher bugs that have a debug log, crash log or any other extend information that could help. See below for details; in general you should not change a bug to "''Need more information''" if it's got a good debug log (and stack trace if Linux). If you mark a bug "''Need more information''", "''Invalid''" or "''Incomplete''", it is useful to add yourself to the CC field; if the reporter adds more information, you can then reopen the bug and re-triage it. If a bug report has been set to "''Need more information''" state and if there has been no response for 4 weeks, feel free to close the report as "''Incomplete''" by using the "incomplete" stock response.

	=== Is the bug report a duplicate? ===		=== Is the bug report a duplicate? ===

Ned Scott at 23:24, 5 May 2015

2015-05-05T23:24:28Z

Martijn: /* Encouragement from the XB
2014-08-30T19:15:24Z

Encouragement from the XB<C Foundation

← Older revision Revision as of 19:15, 30 August 2014
Line 140: Line 140:

# Look for bugs which look interesting. Contact the maintainer to let him/her know you would like to work on it and verify that no one else is working on it. Then fix it.
# Look for bugs which look interesting. Contact the maintainer to let him/her know you would like to work on it and verify that no one else is working on it. Then fix it.

= Encouragement from the XB<C Foundation=
= Encouragement from the XBMC Foundation=

Remember that this is only the first step. At least a couple other people will read and have the opportunity to correct your mistakes. That isn't to say "''make lots of mistakes''" but it is to say "''don't be afraid to make them''", especially once you get used to it, your work will make the lives of the foundation developers- and as a result the project itself- a heck of a lot better.
Remember that this is only the first step. At least a couple other people will read and have the opportunity to correct your mistakes. That isn't to say "''make lots of mistakes''" but it is to say "''don't be afraid to make them''", especially once you get used to it, your work will make the lives of the foundation developers- and as a result the project itself- a heck of a lot better.

Martijn at 19:12, 30 August 2014

2014-08-30T19:12:38Z

Show changes

Ned Scott: {{todo}}

2014-08-26T01:58:57Z

← Older revision		Revision as of 01:58, 26 August 2014
Line 1:		Line 1:
	{{mininav\|[[Development]]}}		{{mininav\|[[Development]]}}
			{{todo}}
	{{cleanup}}		{{cleanup}}
	XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.		XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.

Martijn at 20:15, 4 September 2013

2013-09-04T20:15:20Z

← Older revision		Revision as of 20:15, 4 September 2013
Line 1:		Line 1:
			{{mininav\|[[Development]]}}

	{{cleanup}}		{{cleanup}}
	XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.		XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.

>UNiversal: /* Stock responses for incomplete bugs */past tense should be present tense in context of this sentence.

2012-11-25T19:32:11Z

Stock responses for incomplete bugs: past tense should be present tense in context of this sentence.

← Older revision		Revision as of 19:32, 25 November 2012
Line 85:		Line 85:

	* No XBMC debug log is provided		* No XBMC debug log is provided
	** ''Please ~~collected~~ and ~~attached~~ a XBMC debug log that shows the problem, see http://wiki.xbmc.org/?title=HOW-TO_Submit_a_Proper_Bug_Report''		** ''Please '''collect and attach a XBMC debug log that shows the problem''', see http://wiki.xbmc.org/?title=HOW-TO_Submit_a_Proper_Bug_Report''

	* If the bug is a crasher and no stack trace is provided		* If the bug is a crasher and no stack trace is provided

>UNiversal at 12:42, 25 November 2012

2012-11-25T12:42:47Z

← Older revision		Revision as of 12:42, 25 November 2012
Line 2:		Line 2:
	XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.		XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.

	QA is regarded as an essential part of making XBMC into a better software and people helping with QA is in a sense just as important as developers to ensuring the continued success of the project. Taking a QA role is to help take ownership and responsibility of the bug reporting and tracking process, ensure that projects developers have as much time available for fixing things, and to ensure that users get prompt replies to their queries, and that things are fixed in a timely manner.		QA is regarded as an essential part of making XBMC into a better software and people helping with QA is in a sense just as important as developers to ensuring the continued success of the project. Taking a QA role is to help take ownership and responsibility of the bug reporting and tracking process, ensure that projects developers have as much time available for fixing things, and to ensure that users get prompt replies to their queries, and that things are fixed in a timely manner.<br />


	== What a Quality Assurance Specialist expected to do ==		== What a Quality Assurance Specialist expected to do ==

← Older revision		Revision as of 23:24, 5 May 2015
Line 41:		Line 41:

	=== Is it part of one of the XBMC Foundation projects? ===		=== Is it part of one of the XBMC Foundation projects? ===
	Some pieces of add-ons (like third-party skins, plugins, scripts, and more) that users think of as part of one of the XBMC Foundation projects, have their bug tracking systems elsewhere. If the bug relates to one of these components, please comment on the bug with the resolution as "''Not part of ~~XBMC~~''" and ask the reporter to report it to the correct tracking-system. Note that there is a stock response for this situation.		Some pieces of add-ons (like third-party skins, plugins, scripts, and more) that users think of as part of one of the XBMC Foundation projects, have their bug tracking systems elsewhere. If the bug relates to one of these components, please comment on the bug with the resolution as "''Not part of Kodi''" and ask the reporter to report it to the correct tracking-system. Note that there is a stock response for this situation.

	=== Does it make sense? ===		=== Does it make sense? ===
Line 59:		Line 59:
	* Crasher bugs		* Crasher bugs
	** Should have Severity=Critical, Priority=High		** Should have Severity=Critical, Priority=High
	*** Mark them "''Need more information''" if they don't have a debug log (or stack trace if Linux), and ask the reporter to get a ~~XBMC~~ debug log (and stack trace if Linux). Otherwise, do not mark them "''Need more information''" since that is the first step to them being closed as "''Incomplete''" and the stack trace often has enough information for a developer to fix the bug. However, if there is no information about how it is triggered you should ask the original reporter how to reproduce the bug.		*** Mark them "''Need more information''" if they don't have a debug log (or stack trace if Linux), and ask the reporter to get a Kodi debug log (and stack trace if Linux). Otherwise, do not mark them "''Need more information''" since that is the first step to them being closed as "''Incomplete''" and the stack trace often has enough information for a developer to fix the bug. However, if there is no information about how it is triggered you should ask the original reporter how to reproduce the bug.
	* Something not working as expected		* Something not working as expected
	** Severity=Minor/Normal/Major depending on how much it breaks the application (see the guidelines). If our product would suck badly if we shipped another version with this bug, set Priority=High; otherwise Priority=Normal.		** Severity=Minor/Normal/Major depending on how much it breaks the application (see the guidelines). If our product would suck badly if we shipped another version with this bug, set Priority=High; otherwise Priority=Normal.
Line 83:		Line 83:

	= Stock responses for incomplete bugs =		= Stock responses for incomplete bugs =
	Here are some stock responses to cut and paste into ~~XBMC~~ tracking-system for common situations.		Here are some stock responses to cut and paste into Kodi tracking-system for common situations.

	* No debug log is provided		* No debug log is provided
	** ''Please '''collect and attach a debug log that shows the problem''', see ~~http://wiki.xbmc.org/?title=~~HOW-~~TO_Submit_a_Proper_Bug_Report~~''		** ''Please '''collect and attach a debug log that shows the problem''', see '''[[HOW-TO:Submit a bug report]]'''.

	* If the bug is a crasher and no stack trace is provided		* If the bug is a crasher and no stack trace is provided
Line 92:		Line 92:

	* If the bug is not described well		* If the bug is not described well
	** ''Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read ~~http://wiki.xbmc.org/index.php?title=~~HOW-~~TO_Submit_a_Proper_Bug_Report and add~~ a ~~more useful description to this~~ bug report.''		** ''Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read '''[[HOW-TO:Submit a bug report]]'''.

	* If the bug is not described well and it's a crasher		* If the bug is not described well and it's a crasher
	** ''Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read ~~http://wiki.xbmc.org/index.php?title=~~HOW-~~TO_Submit_a_Proper_Bug_Report~~ and add a more useful description to this bug report. Also note that without a stack trace (also known as a backtrace) from the crash it's very hard to determine what caused it. Thanks in advance!''		** ''Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read '''[[HOW-TO:Submit a bug report]]''' and add a more useful description to this bug report. Also note that without a stack trace (also known as a backtrace) from the crash it's very hard to determine what caused it. Thanks in advance!''

	* If the stack trace is incomplete or without debugging symbols		* If the stack trace is incomplete or without debugging symbols
Line 120:		Line 120:
	* Why should I join/help the QA testing team?		* Why should I join/help the QA testing team?
	** There are several reasons:		** There are several reasons:
	*** Maintainers will not have to spend as much time reading bug reports and will thus be able to concentrate on fixing bugs, making ~~XBMC~~ better for you.		*** Maintainers will not have to spend as much time reading bug reports and will thus be able to concentrate on fixing bugs, making Kodi better for you.
	*** You will learn more about the capabilities of our products (e.g. when you look at a bug that someone files about a specific feature in the Music or Video Library Mode which you did not know about...).		*** You will learn more about the capabilities of our products (e.g. when you look at a bug that someone files about a specific feature in the Music or Video Library Mode which you did not know about...).
	*** You will get a better feel for the different areas of our products and will potentially be able to find other areas that you can also help (e.g. documentation, graphics, translation, web design, user/human interface work, assisting with developing certain applications, etc.).		*** You will get a better feel for the different areas of our products and will potentially be able to find other areas that you can also help (e.g. documentation, graphics, translation, web design, user/human interface work, assisting with developing certain applications, etc.).

← Older revision		Revision as of 19:15, 30 August 2014
Line 140:		Line 140:
	# Look for bugs which look interesting. Contact the maintainer to let him/her know you would like to work on it and verify that no one else is working on it. Then fix it.		# Look for bugs which look interesting. Contact the maintainer to let him/her know you would like to work on it and verify that no one else is working on it. Then fix it.

	= Encouragement from the ~~XB<C~~ Foundation=		= Encouragement from the XBMC Foundation=
	Remember that this is only the first step. At least a couple other people will read and have the opportunity to correct your mistakes. That isn't to say "''make lots of mistakes''" but it is to say "''don't be afraid to make them''", especially once you get used to it, your work will make the lives of the foundation developers- and as a result the project itself- a heck of a lot better.		Remember that this is only the first step. At least a couple other people will read and have the opportunity to correct your mistakes. That isn't to say "''make lots of mistakes''" but it is to say "''don't be afraid to make them''", especially once you get used to it, your work will make the lives of the foundation developers- and as a result the project itself- a heck of a lot better.