Despite these difficulties, performing an attribute analysis on bug tracking systems is not a waste of time. In fact, it is (or may be) an extremely informative, valuable and necessary exercise. The analysis of attributes should only be applied with caution and with a certain focus. Analytically, this technique is a wonderful idea. But in practice, the technique can be difficult to execute judiciously. First, there is always the question of sample size. For attribute data, relatively large samples are required to be able to calculate percentages with relatively low confidence intervals. If an expert looks at 50 different error scenarios – twice – and the match rate is 96 percent (48 votes vs. 50), the 95 percent confidence interval ranges from 86.29% to 99.51 percent.
It is a fairly large margin of error, especially in terms of the challenge of choosing the scenarios, checking them in depth, making sure the value of the master is assigned, and then convincing the examiner to do the job – twice. If the number of scenarios is increased to 100, the 95 per cent confidence interval for a 96 per cent match rate will be reduced to a range of 90.1 to 98.9 per cent (Figure 2). First, the analyst should determine that there is indeed attribute data. One can assume that the assignment of a code – that is, the division of a code into a category – is a decision that characterizes the error with an attribute. Either a category is correctly assigned to an error, or it is not. Similarly, the appropriate source location is either attributed to the defect or not. These are “yes” or “no” and “correct allocation” or “wrong allocation” answers. This part is pretty simple. In this example, a repeatability assessment is used to illustrate the idea, and it also applies to reproducibility.
The fact is that many samples are needed to detect differences in an analysis of the attribute, and if the number of samples is doubled from 50 to 100, the test does not become much more sensitive. Of course, the difference that needs to be identified depends on the situation and the level of risk that the analyst is prepared to bear in the decision, but the reality is that in 50 scenarios, it is difficult for an analyst to think that there is a statistical difference in the reproducibility of two examiners with match rates of 96 percent and 86 percent. With 100 scenarios, the analyst will not be able to see any difference between 96% and 88%. Unlike a continuous measurement value, which cannot be accurate (on average), any lack of precision in an attribute measurement system inevitably leads to accuracy problems. If the error coder is not clear or undecided on how to encode a defect, different codes are assigned to several defects of the same type, making the database imprecise. In fact, the vagueness of an attribute measurement system is an important factor in inaccuracies. Second, the evaluation of the attribute agreement should be applied and the detailed results of the audit should provide a number of information that will help to understand how evaluation can be the best way to be organized. The attribute analysis study not only applies to Pass/Fail ratings, as used with Go/No-Go gauges, but can also be used to test the consistency of operators for whom they make evaluations on an evaluation scale. If the test is planned and designed effectively, it can reveal enough information about the causes of the accuracy problems to justify a decision not to use attribute analysis at all. In cases where the trial does not provide sufficient information, the analysis of the attribute agreement allows for a more detailed review to inform the introduction of training changes and error correction in the measurement system.