HOW-TO:Help with quality assurance testing

From Official Kodi Wiki
Jump to navigation Jump to search
Home icon grey.png   ▶ Development ▶ HOW-TO:Help with quality assurance testing

XBMC Foundation projects are FOSS (Free and Open Source Software) projects, driven by the community of users, and you and anyone else who uses these are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC Foundation and Quality Assurance (QA) is one of them.

QA is regarded as an essential part of making our products into better software and people helping with QA is in a sense, just as important as developers to ensuring the continued success of the project. Taking a QA role is to help test our products and investigate possible problems that might arise during this process. By bug reporting and tracking process, ensure that projects developers have as much time available for fixing things, and to ensure that users get prompt replies to their queries, and that things are fixed in a timely manner.


What a Quality Assurance tester expected to do

Those people that show an enthusiastic contribution to the project and the community on the forums might be a good candidate for participating in the QA Team (also known as the "Bug Hunter Squad"). This can be any one who is willing to invest his time to test our products and investigate possible causes should an issue arise.

Specifically, what the QA team does is

  1. Step in as soon as a problem is reported on the forums or on our trac tracking-system, and attempt to help the user fix the problem, or confirm whether the problem is a bug by reproducing it. Once confirmed, your role is to ensure it gets properly reported in GitHub, either by the original user or by doing it yourself.
  2. Ensure all relevant information has been reported on GitHub tickets, and that it is reproducible. You should close any reports in which the user doesn't respond with needed information in a timely manner. You should then ensure the bug gets assigned to the appropriate developer, and track its progress so that it doesn't slip through the cracks. The report is considered in your hands until it has been confirmed as fixed by either yourself or the original user. When the developer needs more information, the QA team makes sure they get it.
  3. Perform regression testing. You should have a media library of various formats to test against our product, (a media library which of course could be a shared team resource), that can be used to regression test.
  4. Testing new and changed features or functions in our products, and the main application for consistency. You should be willing to test things that you might not normally use (as well as things you will of course be using all the time) as they are developed to ensure that our products stay at the leading edge in terms of both features and stability.

How can i join?

Being part of the QA is entirely at voluntary basis. You can spend as much time as you wish helping the XBMC Foundation and their project. It will be important to also either know or be willing to learn how to compile your own builds from git, for one or more platforms. Official Team members can of course also provide build for your specific platform to test should this be needed.

Eight Quick Steps to Triage Nirvana

The main job of Quality Assurance is to triage bugs to our GitHub tracking-system or to report issues they have found their selves.

Triaging a bug involves

  1. Making sure the bug has not already been reported before
  2. Making sure the bug has enough information for the developers and makes sense
  3. Making sure the bug is filed in the correct place
  4. Making sure the bug has sensible "Severity" and "Priority" fields
  5. Making sure the bug has the correct version of the project in question

First steps for newbies

  1. Register an account on our official community forum.
  2. Read the Stock Responses further down this article. These stock responses are general examples of the kinds of comments that triager's should add to bug reports. There are even a surprising number of bug reports for which you can simply use these responses.
  3. Pick a bug to triage. Any bug report will do. There are also some tips on finding bugs to triage. Once you have picked a bug report, read it carefully.
  4. Find what should be changed. Read the Steps of Triaging further down this article to determine what should be changed (if anything) and how.
  5. Ask someone to verify your changes.
  6. Make the changes. After someone verifies your changes (and likely refines them and/or points out other changes), either make those changes yourself or make sure that the reviewer makes them.
  7. Repeat steps 2-6.

Steps of Triaging

This guide should make triaging a bug as easy and simple as possible. Remember, if in doubt, ask! Also, remember that you may not be able to find anything to change for some bugs since not all bugs are filed incorrectly or incompletely.

Is it part of one of the XBMC Foundation projects?

Some pieces of add-ons (like third-party skins, plugins, scripts, and more) that users think of as part of one of the XBMC Foundation projects, have their bug tracking systems elsewhere. If the bug relates to one of these components, please comment on the bug with the resolution as "Not part of Kodi" and ask the reporter to report it to the correct tracking-system. Note that there is a stock response for this situation.

Does it make sense?

Does it have enough information? Common sense rules here - "this sucks" or "this crashed" should be asked for more information with the appropriate stock response and change the resolution to "Need more information", as should other things that are too ambiguous to be useful. It is not appropriate for bug volunteers to mark a ticket as "'Will never be fixed/added", that is up to the maintainers. The exception is crasher bugs that have a debug log, crash log or any other extend information that could help. See below for details; in general you should not change a bug to "Need more information" if it's got a good debug log (and stack trace if Linux). If you mark a bug "Need more information", "Invalid" or "Incomplete", it is useful to add yourself to the CC field; if the reporter adds more information, you can then reopen the bug and re-triage it. If a bug report has been set to "Need more information" state and if there has been no response for 4 weeks, feel free to close the report as "Incomplete" by using the "incomplete" stock response.

Is the bug report a duplicate?

Many bugs ported onto our tracking-system have been reported before, and finding them is more of an art than a science. If the bug is a duplicate, change it as a duplicate in the resolution box and use the "Duplicate" stock response for this situation. If the original report (i.e. not the one that you currently triage) is in "Need more information" state, please let the reporter know by asking him/her to answer the questions in the original, "Need more information" bug report. You don't need to carry on triaging it if it's a duplicate; press the Submit Changes button and go onto your next bug!

What platform and version?

If the bug is in a older version than the current stable release, ask the reporter to upgrade and test again. It is up to the users to respond within an a reasonable time, to either confirm it has been fixed or provide additional information should the issue persist.

What severity/priority should be set?

Finding the correct severity and priority takes experience, and you will get a feel for it after triaging several bugs. Here are some guidelines for the most commonly reported bugs. You should read the severity and priority guidelines at least once.

Please note that only changing severity and/or priority slightly may do more fuss than good. It causes an extra email to the maintainers and consumes their precious time and attention. If other changes are done as well then there is only little additional impact when changing severity and/or priority.

  • Crasher bugs
    • Should have Severity=Critical, Priority=High
      • Mark them "Need more information" if they don't have a debug log (or stack trace if Linux), and ask the reporter to get a Kodi debug log (and stack trace if Linux). Otherwise, do not mark them "Need more information" since that is the first step to them being closed as "Incomplete" and the stack trace often has enough information for a developer to fix the bug. However, if there is no information about how it is triggered you should ask the original reporter how to reproduce the bug.
  • Something not working as expected
    • Severity=Minor/Normal/Major depending on how much it breaks the application (see the guidelines). If our product would suck badly if we shipped another version with this bug, set Priority=High; otherwise Priority=Normal.
  • Usability issues
    • Severity=Minor, Priority=Normal.
    • Add the "usability" keyword.
  • Accessibility issues
    • Severity=Minor or Normal depending on the extent of the bug. Priority=Normal.
    • Add the accessibility keyword, and possibly the keynav keyword if it's a keyboard navigation bug.
    • These really require expert assessment to determine their severity, so the accessibility keyword is the most important part of our triage.
  • Incorrect strings, typos, etc.
    • Severity=Trivial, Priority=Normal or High depending on how user-visible it is.
    • Set the string keyword.
  • Feature requests
    • If it really is a new feature, and not an improvement to something already in the product, set Severity=Enhancement, Priority=Normal.

If you're unsure about your Severity/Priority triage, add a comment in the bug explaining why you triaged it as you did. You can also add yourself to the Cc field, so if someone re-triages it differently you can find out why.

Add yourself to the CC list

By adding yourself to the CC list of bugs you change, you will receive e-mails with comments and changes others do to the individual report. You learn what further investigations others make and learn more rapidly about triaging bugs in the tracking-system.


Stock responses for incomplete bugs

Here are some stock responses to cut and paste into Kodi tracking-system for common situations.

  • If the bug is a crasher and no stack trace is provided
    • Thanks for taking the time to report this bug. Without a stack trace (also known as a backtrace) from the crash it's very hard to determine what caused it. Thanks in advance!
  • If the bug is not described well
    • Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read HOW-TO:Submit a bug report.
  • If the bug is not described well and it's a crasher
    • Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read HOW-TO:Submit a bug report and add a more useful description to this bug report. Also note that without a stack trace (also known as a backtrace) from the crash it's very hard to determine what caused it. Thanks in advance!
  • If the stack trace is incomplete or without debugging symbols
    • Thanks for taking the time to report this bug. Unfortunately, that stack trace is missing some elements that will help a lot to solve the problem, so it will be hard for the developers to fix that crash. Can you get us a stack trace with debugging symbols? Thanks in advance!
  • If the bug is in an obsolete version of the application
    • Thanks for taking the time to report this bug. However, you are using a version that is too old and not supported anymore. The developers are no longer working on that version, so unfortunately there will not be any bug fixes for the version that you use. By upgrading to a newer version you could receive bug fixes and new functionality. You may need to upgrade your Linux distribution to obtain a newer version. Please feel free to reopen this bug if the problem still occurs with a newer version.
  • If the bug is not part of the main software
    • Thanks for taking the time to report this bug. However, this third-party skin/script/plugin/software does not track its bugs in the tracker-system. We kindly ask you to report the bug to the third-party skin/script/plugin/software authors. If the affected third-party skin/script/plugin/softwarehas a bug tracking-system you should investigate whether a bug for the reported issue is already filed in this system. If it has not been filed yet please do so. Also ensure that both bug reports contain a link to each other. Thanks in advance!
  • If the bug is a duplicate
    • Thanks for the bug report. This particular bug has already been reported into our bug tracking system, but please feel free to report any further bugs you find. or Thanks for taking the time to report this bug. This particular bug has already been reported into our bug tracking system, but the maintainers need more information to fix the bug. Could you please answer the questions in the other report in order to help the developers?

Triage FAQ

  • What are Quality Assurance testers?
    • The QA testing team is the Quality Assurance (QA) team for XBMC Foundation projects. We keep track of current bugs in the variety of software offered and try to make sure that major bugs do not go unnoticed by developers. It is not our job to fix the bugs, although there is nothing to stop you from submitting patches if you have hacking experience. It is a great way to return something to the project's community whether you can program or not.
  • What skills are needed in order to help triage bugs?
    • Common sense and a bit of pragmatism is all that is needed to join the QA testing team. Coding skills might help you evaluate or fix particular bugs, but triaging most bugs will not require knowledge of programming. Bug triaging is a great way for people who do not know anything about programming to return something to the project's community.
  • What software is needed in order to assist in triaging bugs?
    • All you need is to be running the most recent version of our product(s) yourself.
  • How long does it take to learn how to triage bugs?
    • Triaging bugs is a great way to help out with our projects if you do not have much time. It does not take very long to get started, and on average, the simpler bugs can be triaged in anywhere from 2-10 minutes.
  • Why should I join/help the QA testing team?
    • There are several reasons:
      • Maintainers will not have to spend as much time reading bug reports and will thus be able to concentrate on fixing bugs, making Kodi better for you.
      • You will learn more about the capabilities of our products (e.g. when you look at a bug that someone files about a specific feature in the Music or Video Library Mode which you did not know about...).
      • You will get a better feel for the different areas of our products and will potentially be able to find other areas that you can also help (e.g. documentation, graphics, translation, web design, user/human interface work, assisting with developing certain applications, etc.).
  • What if I can not find any changes to make to a bug? What if all the fields look like they are set correctly?
    • Then simply move on to a different bug.
  • What if a certain bugs look like a duplicate of many different bugs?
    • Try to find the "original" one and mark your bug as a duplicate of that one. (i.e. if bug A has been reported and bugs B & C have been marked as a duplicate of A, then try to mark future bugs as a duplicate of A and not as duplicates of B or C).
  • I can not reproduce a certain bug. Can I close it?
    • You have to take into account why you can't reproduce it. Many bugs happen only under certain rare circumstances, and you might not be able to reproduce it due to a different system configuration, or you might be running Linux instead of Windows, Android or Mac.
    • If the bug is sufficiently general, is filed against an old version and looks as though it is now fixed, you can close it. If you're not sure, ask in forum and IRC channels if someone else can replicate it. You can also add a comment saying that you can not reproduce the bug (along with any relevant information about your system) if you think it will help.

Advanced

  • I am a programmer. Are there more advanced things I can do related to triaging?
    • Yes, there are several things you could do that someone without programming skills might not be as adept at or would not be able to do at all. Bug triaging is a great way to find a small part of our projects to start on while making your first XBMC Foundation baby steps. Here are a few of the things you can do:
  1. Obtain easy directions to duplicate bugs (sometimes it is easier for those with a programming background to find these)
  2. For bugs that are reported but do not contain debugging symbols, try to obtain debugging symbols for them. Some stack traces can help a developer find a problem even without debugging symbols, but often times they are not very helpful.
  3. Find and mark duplicates of crasher bugs (those without a programming background may take longer to get used to dealing with stack traces).
  4. Look for bugs with the "love" keyword. Fix them.
  5. Look for bugs which look interesting. Contact the maintainer to let him/her know you would like to work on it and verify that no one else is working on it. Then fix it.

Encouragement from the XBMC Foundation

Remember that this is only the first step. At least a couple other people will read and have the opportunity to correct your mistakes. That isn't to say "make lots of mistakes" but it is to say "don't be afraid to make them", especially once you get used to it, your work will make the lives of the foundation developers- and as a result the project itself- a heck of a lot better.