Dynamic Analysis Offers a Better MSA Management Alternative

Just like process tools, a measurement system that can demonstrate statistically stable output should not have to undergo routine characterization. Instead, a dynamic MSA can be achieved with at-hand SPC data.

Phillip H. Williams, Freescale Semiconductor Inc., Chandler, Ariz. -- Semiconductor International, 2/1/2008

One question that always seems to come up in manufacturing and quality system circles is, "How often must we repeat our measurement system analysis?" Ask 10 different engineers this question, and you may get 10 different answers. The reason for this apparent confusion is understandable: Quality standards like ISO9001 and TS16949 do not mandate a time interval. They merely imply (quite heavily) that convincing documented evidence must exist that a measurement system is producing valid data — particularly if the characteristic being measured is used to directly drive or demonstrate the quality of the product (sometimes referred to as a key characteristic).

The answer to this "How often?" question is "Seldom, if ever" — but only if proper statistical process control (SPC) of the measurement system is in place and ongoing. This answer may not be so surprising if you consider not a measurement tool, but instead a process tool. Before a process tool is used for the first time, it undergoes thorough characterization and qualification. Then, once its performance is shown to be acceptable, process control is implemented and maintained. As long as the process remains statistically stable and is not intentionally changed, there is not a compelling reason to perform a new characterization and qualification.

The same should be true of a measurement system. A measurement system is really just a process that generates measurements as the output. As long as we can demonstrate that this output is statistically stable and that the measurement process has not been intentionally changed, there is not a compelling reason to perform a new Sources of Variation (SoV) analysis (i.e., characterization and qualification) of the measurement system.

The real question, then, is how to demonstrate the stability and performance of a measurement system at any time using the at-hand SPC data for the measurement system. In a sense, how do we achieve a dynamic measurement system analysis (MSA) by being able to fully evaluate the performance of a measurement system at any time? As mentioned already, the key is having proper SPC in place for the measurement system, just as you would for any other manufacturing process.

The concept of incorporating a dynamic MSA into the overall strategy of maintaining measurement system integrity is illustrated in Figure 1. Not only can it help relieve the stress of deciding when to update which MSA (and hoping you didn't miss any), but it provides several advantages. First, existing and up-to-date data is utilized. A new SoV study does not need to be performed unless the dynamic MSA indicates that one is needed (Fig. 1). Second, the information obtained represents a longer, more relevant time frame vs. a relatively isolated SoV experiment. Finally, dynamic MSA drives the implementation of proper SPC for measurement systems. This needs to be in place to detect special cause variation in the measurement system, regardless of whether we use it to perform the actions described in this article. Again, this is the same reason we apply SPC in the monitoring of our processes and products.

Measurement system requirements

1. Incorporating a dynamic MSA into the overall strategy of maintaining measurement system integrity can help relieve the stress of deciding when to update which MSA, as well as other advantages.

We will begin by briefly going over the basic requirements of an MSA. We frequently hear of the five-part MSA or the full MSA. These five parts consist of discrimination, bias, linearity, precision and stability.

Discrimination, sometimes called resolution, is typically not an issue. Essentially, the smallest increment of measure should be no greater than one-tenth of the control width for the characteristic that is being monitored. It can be checked with the following simple relationship:

Smallest increment of measure > 6s_total/10

s_total is simply the historical standard deviation of the monitored characteristic, taken from its control chart. Assuming the control limits for the chart have been statistically determined, the numerator 6s_total can be adequately approximated by the upper control limit (UCL) minus the lower control limit (LCL). Usually, the discrimination requirement is met by orders of magnitude — but not always, particularly if the measurement tool is old or out-of-date.

Bias refers to the average difference between the observed or measured value of a standard or reference, and the "known" value of the standard or reference. It is sometimes referred to as accuracy. However, accuracy implies the standard is NIST-traceable, which is not always the case.

Linearity refers to how differently the measurement system behaves across its normal range of measurement. Bias linearity refers to how the bias may vary over the range of measurement; this is the aspect that is usually implied when the term "linearity" is used. The other aspect of linearity is precision linearity, or linearity of spread, which refers to how the precision may vary over the measurement range. The precision aspect of linearity, while often overlooked, may actually be the more critical of the two. This is because bias linearity can often be minimized through a straightforward adjustment. Issues with precision linearity, on the other hand, are usually not dealt with as simply.

Precision, historically referred to as Gage R&R or simply R&R, is really the central core of an MSA. R&R stands for repeatability and reproducibility, which are the two overall components of precision. Repeatability is the inherent variability of the measurement tool. Reproducibility may be comprised of several subcomponents, including possible interactions with aspects not associated directly with the measurement system. All of these components and subcomponents can only be properly evaluated using Analysis of Variance (ANOVA). By definition, precision is a standard deviation, not a variance.

Stability, put quite simply, describes the constancy over time of the bias, linearity and precision. Discrimination is really more an aspect of the tool design and should not change over time, unless there is a large shift in the variability of the process or product characteristic being monitored, or the tool is in need of significant maintenance or repair. In the latter case, some or all of the other measurement aspects (i.e., bias, linearity, precision, stability) will be clearly impacted.

Measurement system SPC

As mentioned earlier, the key to performing a dynamic MSA is to have SPC properly implemented for the subject measurement system. In practice, this should be done regardless of MSA intent — to guarantee the stability of the measurement system. Proper SPC implementation for a measurement system implies the following:

Reference pieces have been established, and backups to these reference pieces have also been established and safely put away. These reference pieces — or rather, each set of reference pieces, including the backups — span the entire range of measurements for which the measurement system will be used. We want to guarantee the measurement system is stable over its entire range, just like we want to guarantee that a process or product characteristic is stable over its entire range.
Statistical control limits have been established for the reference pieces. This is not possible at the very beginning, of course, but after 30 or more independent and consistently variable (i.e., no obvious special causes present) data points have been collected, a valid calculation can be performed.

2. The three control charts shown correspond to the small, medium and large CD reference sizes. The charts show that during the last seven months for each nominal reference value, the variability of the measurements are within the control limits.

If these criteria are met, we have the ability to perform a full MSA at any time. We will discuss each aspect of the MSA, while at the same time illustrating the dynamic MSA methodology with a real-life example.

Performing the dynamic MSA

As an example, let's look at an optical measurement system for evaluating critical dimensions (CDs). Three reference geometries — small (s), medium (m) and large (l) — have been set up to represent the range of dimensions that the system is used to monitor. Because we are measuring the same "parts" time after time, the variability of the observed values reflects only the measurement system and not the process.

If the system is not stable, none of the other measurement system attributes are really meaningful. So it is logical to evaluate stability first. The most appropriate way to do this is to look at the control charts. Figure 2 contains control charts corresponding to the three different CD reference sizes.

From the charts, it can be seen that during the past seven months for each nominal reference value, the variability of the measurements are within the control limits. Therefore, we can conclude that the stability criterion has been met.

Next, we can evaluate the bias. By definition, the bias will be the average of the observed measurements minus the corresponding reference value. So, using the control charts, this will be the process mean minus the target value. If we do this for each chart, we have effectively determined how the bias changes over the range of measurement, which will allow us to evaluate the bias linearity as well.

But shouldn't we expect the magnitude of the bias to increase as the magnitude of the measured value increases? Yes, although this may not always be true, it is typical of many measurement systems. To take this normal behavior into account so that we can better evaluate the "abnormal" bias, we can normalize our measurements to the corresponding target values. This is demonstrated in Table 1.

For each reference part, we can see that the bias is &0.1%. Therefore, any difference in bias across the range of measurement (i.e., the bias linearity) is unlikely to be significant.

Finally, we can assess the precision of the measurement system, s_MS. If we do this for each chart, we have effectively determined how the precision changes over the range of measurement, which will allow us to evaluate the precision linearity as well. s_MS for each chart is simply the chart's standard deviation. Then, if we know — or have an estimate of — the process standard deviation associated with each range of measurement s_total, we can calculate the %R&R according to the formula:

%R&R = 100*s_MS/s_total

Bear in mind that s_total can also be estimated from the process control limits, assuming they have been statistically derived, as the UCL minus the LCL, divided by 6.

Alternatively, if we want to calculate the precision-to-tolerance ratio instead of the %R&R, the formula will be as follows:

%P/T = 100*6*s_MS/(USL-LSL)

where the denominator is the process tolerance, or upper specification limit (USL) minus the lower specification limit (LSL).

Keep in mind that s_MS from the measurement system SPC chart incorporates the main precision components repeatability and reproducibility, and includes all the subcomponents of the latter.

To enable the calculation of these metrics for our CD example, information about the processes being monitored is given in Table 2. With the process information in Table 2, the measurement system control chart information (Fig. 2) and the above formulas, the precision metrics can now be generated (Table 3).

Because the %R&R is &10% across the entire measurement range, we can say—under most circumstances—that we have met the precision criterion for our measurement system. We can also say that the precision appears to worsen with CD size, which is a qualitative assessment of precision linearity. However, we can also say the difference across the range of measurement is not >0.5%. This should be an acceptable precision linearity in most cases.

But what if the %R&R is considered to be unacceptably high? If there is not an assignable cause that can be remedied, then we will need to design and perform a new SoV analysis for the measurement system as indicated in Figure 1. This will allow us to determine the variance components that need to be reduced. Once this has been successfully done, we can put the measurement system back in SPC mode. This will enable an ongoing, dynamic MSA that can be fully validated at any time as described above (Fig. 1).

Statistical vs. practical significance

An MSA is based on ANOVA. When performing ANOVA to determine the variance components, or sources of variation, we are typically engaging in a pseudo-quantitative evaluation. In other words, we are not making probability-driven statistical comparisons of the derived variance components. It is usually enough, for instance, to say things like "Day-to-day is the dominant source of variation," or "The bias linearity is very small relative to the operating range." These types of statements are based on graphical analyses (e.g., a multi-vari or SPC chart) and/or the computed percentages of the variance component contributions.

In contrast, to make statistical statements like "The day-to-day variation is significantly greater than the repeatability," or "Tool A, with a %R&R of 8%, is better than Tool B, with a %R&R of 12%," would not be appropriate without first validating the basic assumptions of normality and independence of the data sets. In sources of variation studies, these assumptions are often violated because of the nested and auto-correlated nature of the data.

To illustrate taking an MSA to "the next level" statistically, we will continue with our current example. Let's begin by making a comparison box plot of the previously shown control chart data (Fig. 2). This is shown in Figure 3.

3. This comparison box plot of the control chart data shown in Figure 2 enables the qualitative assessment of all the bias and precision aspects of the measurement system.

Each measured value has been "normalized" (not to be confused with normally distributed) by dividing it by the corresponding reference value. From this chart, all the bias and precision aspects of the measurement system can be qualitatively assessed. It is clear that precision non-linearity is present, with the best precision, or tightest spread, occurring for the large values (l_ratio), and the worst precision, or highest scatter, occurring for the small values (s_ratio). But even for the latter, the entire spread is &2.5% of the nominal value (1.015-0.990).

We can also see that for each group of measurements, the mean (indicated by the circle with a cross inside) is very close to 1.000, thus not only indicating a negligible bias, but a negligible bias non-linearity as well.

For these three data sets, the data characteristics are amenable to statistical comparison (which is often not the case). Table 4 summarizes the probability values (p-values) associated with the tests for normality (Anderson-Darling) and independence (Runs test). If we assume an alpha-risk of 0.05, then each of the data sets may be considered both normal and independent.

Given that each data set is normal and independent, we can apply the appropriate tests. Application of Bartlett's test yields a p-value of &0.0001, affirming that the variances of the three data sets are not all equal. This establishes the presence of a statistically significant precision non-linearity. And since the variances are not equal, Welch's test, rather than One-Way ANOVA, should be used to compare the means of each data set. When this is done, a p-value of 0.625 is obtained. This high p-value affirms that the difference in means is not statistically significant. Hence, there is not a significant bias non-linearity present in the measurement system.

Finally, when we use a one-sample t-test to compare the mean of each data set to 1.000, we generate p-values of 0.451, 0.130 and 0.005 for the s_ratio, m_ratio and l_ratio sets of data, respectively. Again, if we assume an alpha-risk of 0.05, then a statistically significant bias is only present in the large-CD range. Is this worth worrying about? Most likely it is not. Referring back to Table 1, the large-CD bias, although statistically significant, represents &0.02% of the nominal value.

What about the statistically significant precision non-linearity? Keep in mind that just because something is statistically significant, that does not mean it is practically significant (the reverse, however, is not true). Good engineering judgment must ultimately rule the day, based on the data. And the best people to make the judgment are the ones who are most familiar with the measurement system and the process or product that it is monitoring.

In this example, the worst-case precision is 0.5% of the reference value, which corresponds to 8.0% R&R or 4.7% P/T (Table 3). This is not likely to be a concern, but if it is, then actions should be put in place to improve the precision linearity.

Summary

Keeping track of which MSAs should be repeated for which tools for a large manufacturing area can be problematic. In addition, repeating an SoV study for a measurement system represents a relatively short experimental time frame. Implementing a dynamic MSA methodology not only drives the proper implementation of SPC for each measurement system, but it enables the calculation of full MSA metrics at any time, for any tool, using all of the available data. If the metrics are acceptable, a new SoV study is not required.

Author Information

Phillip H. Williams is a member of the technical staff at Freescale Semiconductor, with responsibilities in quality, consulting and formal instruction in the areas of statistics and Six Sigma techniques. His 22 years of experience include process and quality engineering in the microelectronics and flat panel industries, where he has held positions in manufacturing, R&D, process integration and start-up environments. He holds a B.S. and M.S. in chemical engineering, both from Arizona State University.

Talkback

We would love your feedback!

No articles available

Featured Company

WL Gore & Associates

The company began in 1958, when Bill and Vieve Gore set out to explore opportunities for fluorocarbon polymers, especially polytetrafluoroethylene (PTFE). Within the first twelve years, Gore had wire and cables on the moon and operations... more