HOW-TO:Help with quality assurance testing
XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.
QA is regarded as an essential part of making XBMC into a better software and people helping with QA is in a sense just as important as developers to ensuring the continued success of the project. Taking a QA role is to help take ownership and responsibility of the bug reporting and tracking process, ensure that projects developers have as much time available for fixing things, and to ensure that users get prompt replies to their queries, and that things are fixed in a timely manner.
- 1 What a Quality Assurance Specialist expected to do
- 2 Eight Quick Steps to Triage Nirvana
- 3 Steps of Triaging
- 4 Add yourself to the CC list
- 5 Encouragement from Team-XBMC
1 What a Quality Assurance Specialist expected to do
Those people that show an enthusiastic contribution to the project and the community on the forums might be a good candidate for the official XBMC QA Team that is a part of Team-XBMC. Invitations are sent out to community members who stand out and show that they could be willing to take on the role as an official QA Specialist on Team-XBMC.
1.1 Specifically, the QA team is expected to
- Step in as soon as a problem is reported on the forums or on our trac tracking-system, and attempt to help the user fix the problem, or confirm whether the problem is a bug by reproducing it. Once confirmed, your role is to ensure it gets properly reported in trac, either by the original user or by doing it yourself.
- Ensure all relevant information has been reported on trac tickets, and that it is reproducible. You should close any reports in which the user doesn't respond with needed information in a timely manner. You should then ensure the bug gets assigned to the appropriate developer, and track its progress so that it doesn't slip through the cracks. The report is considered in your hands until it has been confirmed as fixed by either yourself or the original user. When the developer needs more information, the QA team makes sure they get it.
- Perform regression testing. You should have a media library of various formats to test against XBMC, (a media library which of course could be a shared team resource), that can be used to regression test.
- Testing new and changed features or functions in XBMC, and the main application for consistency. You should be willing to test things that you might not normally use (as well as things you will of course be using all the time) as they are developed to ensure that XBMC stays at the leading edge in terms of both features and stability.
If you where to get invited to become an official QA Specialist on Team-XBMC then the expectation is that it will require an hour or so a day in checking up on the forums and trac, (so approximate 7 hours a week), possibly with additional time taken to verify bugs, and identify ways to reproduce them. It will be important to also either know or be willing to learn how to compile your own builds of XBMC from SVN, for one or more platforms.
2 Eight Quick Steps to Triage Nirvana
The main job of Quality Assurance is to triage bugs to our trac tracking-system.
2.1 Triaging a bug involves
- Making sure the bug has not already been reported before
- Making sure the bug has enough information for the developers and makes sense
- Making sure the bug is filed in the correct place
- Making sure the bug has sensible "Severity" and "Priority" fields
- Making sure the bug is versioned correctly
2.1.1 First steps for newbies
- Register an account on XBMC Community Forum.
- Read the Stock Responses further down this article. These stock responses are general examples of the kinds of comments that triagers should add to bug reports. There are even a surprising number of bug reports for which you can simply use these responses.
- Pick a bug to triage. Any bug report will do. There are also some tips on finding bugs to triage. Once you have picked a bug report, read it carefully.
- Find what should be changed. Read the Steps of Triaging further down this article to determine what should be changed (if anything) and how.
- Ask someone to verify your changes.
- Make the changes. After someone verifies your changes (and likely refines them and/or points out other changes), either make those changes yourself or make sure that the reviewer makes them.
- Repeat steps 2-6.
3 Steps of Triaging
This guide should make triaging a bug as easy and simple as possible. Remember, if in doubt, ask! Also, remember that you may not be able to find anything to change for some bugs since not all bugs are filed incorrectly or incompletely.
3.1 Is it part of XBMC Media Center software from Team-XBMC?
Some pieces of add-ons (like third-party skins, plugins, scripts, and more) that users think of as part of the XBMC Media Center software from Team-XBMC have their bug tracking systems elsewhere. If the bug relates to one of these components, please close the bug with the resolution as "Not part of XBMC" and ask the reporter to report it to the correct tracking-system. Note that there is a stock response for this situation.
3.2 Does it make sense?
Does it have enough information? Common sense rules here - "this sucks" or "this crashed" should be be asked for more information polite message as change the resolution to "Need more information", as should other things that are too ambiguous to be useful. It is not appropriate for bug volunteers to mark a ticket as "'Will never be fixed/added", that is up to the maintainers. The exception is crasher bugs that have a XBMC debug log (and stack trace if Linux). See below for details; in general you should not change a bug to "Need more information" if it's got a good XBMC debug log (and stack trace if Linux). If you mark a bug "Need more information", "Invalid" or "Incomplete", it is useful to add yourself to the CC field; if the reporter adds more information, you can then reopen the bug and retriage it. If a bug report has been set to "Need more information" state and if there has been no response for 4 weeks, feel free to close the report as "Incomplete" by using the "incomplete" stock response.
3.3 Is the bug report a duplicate?
Many bugs ported onto our tracking-system have been reported before, and finding them is more of an art than a science. If the bug is a duplicate, change it as a duplicate in the resolution box and use the "Duplicate" stock response for this situation. If the original report (i.e. not the one that you currently triage) is in "Need more information" state, please let the reporter know by asking him/her to answer the questions in the original, "Need more information" bug report. You don't need to carry on triaging it if it's a duplicate; press the Submit Changes button and go onto your next bug!
3.4 What platform and version of XBMC?
If the bug is in a version of XBMC prior to 8.10, ask the reporter to upgrade and close it as "Obsolete". There is a stock response for this situation. Otherwise, select the appropriate version that the bug appears in for the XBMC Version field.
- There are two exceptions. If fixing the bug requires breaking string or UI freeze, you can mark the bug with the version beyond the one that will be released next. If the bug is a feature request the you can change it to say so.
3.5 What severity/priority should be set?
Finding the correct severity and priority takes experience, and you will get a feel for it after triaging several bugs. Here are some guidelines for the most commonly reported bugs. You should read the severity and priority guidelines at least once.
Please note that only changing severity and/or priority slightly may do more fuss than good. It causes an extra email to the maintainers and consumes their precious time and attention. If other changes are done as well then there is only little additional impact when changing severity and/or priority.
- Crasher bugs
- Should have Severity=Critical, Priority=High
- Mark them "Need more information" if they don't have a XBMC debug log (or stack trace if Linux), and ask the reporter to get a XBMC debug log (and stack trace if Linux). Otherwise, do not mark them "Need more information" since that is the first step to them being closed as "Incomplete" and the stack trace often has enough information for a developer to fix the bug. However, if there is no information about how it is triggered you should ask the original reporter how to reproduce the bug.
- Should have Severity=Critical, Priority=High
- Something not working as expected
- Severity=Minor/Normal/Major depending on how much it breaks the application (see the guidelines). If GNOME would suck badly if we shipped another version with this bug, set Priority=High; otherwise Priority=Normal.
- Usability issues
- Severity=Minor, Priority=Normal.
- Add the "usability" keyword.
- Accessibility issues
- Severity=Minor or Normal depending on the extent of the bug. Priority=Normal.
- Add the accessibility keyword, and possibly the keynav keyword if it's a keyboard navigation bug.
- These really require expert assessment to determine their severity, so the accessibility keyword is the most important part of our triage.
- Incorrect strings, typos, etc.
- Severity=Trivial, Priority=Normal or High depending on how user-visible it is.
- Set the string keyword.
- Feature requests
- If it really is a new feature, and not an improvement to something already in the product, set Severity=Enhancement, Priority=Normal.
If you're unsure about your Severity/Priority triage, add a comment in the bug explaining why you triaged it as you did. You can also add yourself to the Cc field, so if someone retriages it differently you can find out why.
4 Add yourself to the CC list
By adding yourself to the CC list of bugs you change, you will receive e-mails with comments and changes others do to the individual report. You learn what further investigations others make and learn more rapidly about triaging bugs in the XBMC tracking-system.
5 Encouragement from Team-XBMC
Remember that this is only the first step. At least a couple other people will read and have the opportunity to correct your mistakes. That isn't to say "make lots of mistakes" but it is to say "don't be afraid to make them", especially once you get used to it, your work will make the lives of the XBMC developers- and as a result XBMC itself- a heck of a lot better.