Delphi/C++ QA, Quality Central and Bug prioritization

By: Chris Pattinson

Abstract: An under the hood look at thoughts behind Quality Central, and our process and criteria to classify defects.

    Delphi/C++ Team, Quality Central and Bug Prioritization

By Chris Pattinson, QA Manager

June 13, 2007

Overview: Quality Central is a great tool that we want to use more effectively. This document describes the internal QA process and thoughts in regards to bugs in general and areas we plan to work on to improve our use of Quality Central.

The Delphi/C++ QA team uses an internal bug tracking tool affectionately called RAID to manage the defects identified during testing and development. We also provide Quality Central (http://dn.codegear.com/article/33885 ) to our customers as a vehicle to log issues, and have the community self promote bugs, feature requests, and general issues in the products.

Quality Central was an initiative originally formed with the concept of community self support. This meant Sysops (veteran developers, field test marshals, Team B members and also those who just volunteered to help) review and browse public reports, then promote to the internal system. To become a Sysop, you’d be identified as a constructive and methodical community member who had a talent to reproduce and investigate a report. The QA team has been using Quality Central a lot more recently, though our main bug database is still internal, and I’m seeing that we are now in a state where it appears a number of Quality Central reports are ‘neglected’.

Quality Central itself is a success, with several thousand entries in a reported state, and a couple thousand in an open state. Now the reports themselves need to be processed.

We’ll work to clean this up in the future, and plan to promote additional Sysops and spend more QA time in Quality Central to make that happen. In addition we’ll assign a new status (I just added ‘Automated Report’ to our internal tool for this use) specifically for Automated Incident Reports. We data mine AIRs. They are typically not used as a source of unit tests/use cases since they do not provide the steps required to create tests.

One example of how we use them, is when we see that a number of users report a crash in an area, all indicating they were doing some similar action. This has let us identify difficult to track down and reproduce issues from a different angle. However since Automated Incident Reports are rarely commented or acted on individually, it makes sense to categorize them into their own status for queries. Right now, they make it more difficult to track and review the latest manually entered quality central reports.

We already spend a large amount of effort on the internal database maintenance, so it may also make sense to combine the two databases to prevent the impression we’re neglecting the product. Several thousand bug fixes to Delphi alone have been made in the past year, and these have not been clearly reflected in Quality Central.

On that note, if you would like to volunteer to be a Sysop, please send me an email ( cpattinson@codegear.com ) indicating your history with the product, amount of time you can dedicate to the task, what areas of the product you want to focus on and a few of your own QC reports as example of a ‘good’ report.

This next section of this document talks about bugs and prioritization as well as how the QA team classifies a defect during development. We have an active field test and quality assurance team who reports thousands of ‘issues’ a year – some are very serious, some are cosmetic, and some involve usability or preference of how a feature works. It’s critical that the serious issues are correctly classified so R&D works on them in order of priority.

We also use QC and our internal tool to track feature requests, however we’ve been moving most of that work into XPlanner and on team Wiki in the form of design specs.

'Defects' in their purest form can include minor irritations or critical failures. A strict criteria needs to be defined to ensure that defects are prioritized by potential impact and severity to the customer.

Here is some insight into the thought process and classification process. It’s very helpful when you log a Quality Central issues to set the Type and Severity correctly. When we do field tests, we have a group of “Field Test Marshals” who help review Quality Central Bugs and promote the ones they can reproduce.

There is a strong relationship between Quality Central reports that have code samples and reproducible steps, and how quickly the defect is fixed in the product. If a serious issue is raised that has a code/project sample the team can reproduce rapidly, often we’ve had a fix to that issue within a day. A report stating there was a crash, but no information on how to reproduce, is very difficult to identify and fix. The better you can provide information to help the engineering team reproduce it, the better the chance is of it getting fixed.

Here is a section of our internal reference on setting the Type, Severity and QA Recommend of a defect. The intent is to provide clear window into our guidelines. The QA Recommend is currently used internally and not visible in Quality Central. Feel free to discuss/debate them in a constructive manner.

In general the goal is to work on the most important bugs first before releasing the product. Feature requests are generally outside the scope of ‘bug fixing’, though in some cases there is overlap if a feature implementation needs to be redone to fix a feature’s behavior.

    Type field

This drop down consists of a small list of values used to indicate the type of report being entered. Use this field to set the description of the type of defect based on the following:

  • A – Crash / Data loss / Total Failure

The user will be kicked out of the program, their system locks up, data is corrupted, lost or cannot be saved, or system crash occurs.

  • B – Basic Functionality Failure

The feature does not work or does not behave per documentation. There is no work around or the work around is tedious and doesn’t give the desired result.

  • C – Minor Failure / Design Problem

The user interface has a cosmetic problem, there are duplicate pick letters, there is a spelling or painting error. There is a simple work around that gives the desired results.

  • D – Documentation Problem

All problems related to documentation. This includes missing content, broken links, incorrect information, and spelling errors. Invalid Topic or other context help problems are of this type. In general, the documentation team queries on type 'D' bugs to determine their workload.

  • E – Suggestion / Enhancement Request

The current feature would be improved by this suggestion.

  • F – New Feature Request

The feature does not exist in the current product.

    Severity field

Select the severity of the problem that best matches the defect.

  • 0 – Critical / Blocking

This problem causes the feature to be completely unusable or blocks major portions of automation for this feature. This problem causes the feature to be completely unusable or blocks major portions of automation for this feature. QA will use ‘0’ to indicate a bug is preventing them from testing a products feature(s). This severity should be accompanied by meeting R&D in person, phone and/or email to resolve rapidly.

  • 1 – Serious / Highly visible problem

This problem causes the feature to fail in most cases or is something that users see constantly while using the product. This problem should be fixed as soon as possible.

  • 2 – Commonly encountered problem

This problem or lack of feature causes the product to be more difficult to use. This problem occurs regularly. It should be fixed in a reasonable amount of time.

  • 3 – Infrequently encountered problem

This problem is encountered occasionally and causes minor annoyances.

  • 4 – Extreme Corner Case

This problem is rarely encountered or occurs due to extreme circumstances. Example would be "Only happens on a non-supported system" or "Only happens when you add 10,000 elements".

    How is CodeGear QA Recommend Set?

Internally, we set a ‘QA Recommend’ to determine our internal prioritization of a reported defect. This section is to allow you some insight into how the engineer team prioritizes defects that come in via Quality Central. This is a field you will not see in Quality Central (possible enhancement for the future…)

Recommend field

The Recommend field is for QA to set their recommendation of how soon the defect should be corrected. The default is "Not Rev." which means that the assigned QA Engineer has not evaluated the problem. Defects promoted from Quality Central into RAID default to “Not Rev.” If you are a sysop send a friendly email to a QA engineer, especially for serious issues. This will help ensure it gets looked at rapidly.

The Recommend field is also used by Bug Council and the Core Team as a way of measuring what needs to be accomplished at each milestone. R&D then reviews the QA recommend and sets an appropriate Action field. The Bug Council reviews any bugs that have a difference in QA Recommend and R&D Action.

  • FT-Milestone: This problem must be addressed as soon as possible as it breaks fundamental product areas, development or testing. It should be corrected before the next milestone. When conducting Field Tests, this recommendation is used to identify bugs QA expected fixed before the next field test release. They are usually of very high severity to a number of customers and block the testing of new features we wish the field tests to look at.
  • Must Fix: This problem must be addressed as soon as possible. It should be corrected before product release. This issue has a major negative impact on quality.
  • High: This problem should be addressed as soon as possible. This issue has a negative impact on quality.
  • Med: This problem is not as severe, but it would be nice if it was addressed soon. It does not necessarily have to be corrected for this release. This issue has a small negative impact on product quality.
  • Low: This problem doesn’t necessarily have to be addressed for this release. It has little to no impact on product quality.
  • Tester Inv: The extent of this problem is not completely understood by the assigned QA Engineer or more information is needed. The QA Engineer is investigating further.
  • Don’t Fix: This problem should not be addressed.
  • Readme: This problem should be addressed via documentation.
  • WUpdate: This problem should be addressed in the next Update to be created for this product. This is often reserved for bugs that have a large negative impact on quality but require significant development time and testing to address that would add too much risk to the current project.

Examples

In general, functional bugs that affect a number of customers with severe impact are Must Fix. The factors that add up to determining Must Fix include:

  • Visibility - if the IDE has a package load error on startup, even if cosmetic, it makes the product look broken and therefore should be fixed. A duplicate pick letter on a dialog embedded 4 levels down in a dialog structure is not very visible, and would not be Must Fix
  • Type - refer to the section above. In general, A and B bugs would be of a type making them Must Fix, this needs to also take into account Severity. Example: an 'A' bug of '3' severity is not always Must Fix.

In some cases a Documentation problem (such as in the README, License or Install) could have legal impact or visibility high enough to make them Must Fix. In those situations, make sure to contact the appropriate management and engineering. The action field from the developer should be marked 'Must Fix' for any bugs that are not A or B and QA considers Must Fix.

  • Severity - refer to Severity above. In general, 0, 1 and 2 bugs are severe enough to be Must Fix, however this must be balanced with type.

Majority of '3' bugs should not be Must Fix, and you'd want to discuss with R&D if you feel a '3' bug should be Must Fix. If they agree, action of the bug from R&D should be set to Must Fix. Example: a Type 'C' bug of '1' severity is often not Must Fix.

  • Legal requirements - if a defect is detected that has a legal requirement, a common example being licensing, this is severe enough to stop the shipping of a product and can be logged as Must Fix.
  • Block testing or development - if a product defect blocks automation testing or development, and has no acceptable workaround, then this defect could be marked as Must Fix. Again, these types of bugs should be communicated to the engineering group very rapidly - ideally in person or via phone.

The following criteria can be used as a guideline to determining Must Fix defects:

  • Crash bug
  • Support Escalations
  • Data loss
  • Performance
  • Runtime Memory Leak
  • Design time Memory Leak
  • Major Usability complaints
  • Basic Feature failure
  • Serious L10n blocker
  • Regressions (A 'High' bug could qualify as 'Must Fix' if it is a regression many users encounter)

The following criteria can be used to establish defects that are NOT Must Fix:

  • No Deprecated feature
  • If a simple workaround exists
  • Defect fix requires significant new feature work
  • Defect is a suggestion on how something that works should work in a different way

Comments on Resolution

  • Inactive was used to set a number of older and lesser priority bugs to 'inactive' so R&D and QA could focus on higher priority and more recent bugs reports first. This resolution was used by bug council, and in a batch process to 'clean up' RAID.
  • Won't Do is set by R&D or management to indicate that a defect or suggestion will never be implemented. This could be for a variety of reasons, and if disputed can be discussed with the Bug Council.

Additional Procedures

  • Version Field

The Version field in RAID should reflect the version of the product that a defect was found. It is not changed.

The exception would be if a defect was found, then fixed, then regressed in a later build/release. In this latter case a bug ‘comes back’, it is a better idea to log a new bug, since the causes and code path may be considerably different.

Server Response from: ETNASC04