Text preview for : Too Much Calibration_ - White Paper 5991-1311EN c20140529 [15].pdf part of Agilent Too Much Calibration - White Paper 5991-1311EN c20140529 [15] Agilent Too Much Calibration_ - White Paper 5991-1311EN c20140529 [15].pdf
Back to : Too Much Calibration_ - W | Home
Keysight Technologies
Too Much Calibration?
White Paper
Abstract
"Hey Wendle, our cost of sales is increasing, and my boss is riding me
hard to reduce it! That calibration budget you submitted is "killing" me.
It is just too much calibration. Get back to me tomorrow with a better
proposal will you? " "Oh, and by the way could you look at our warranty
failures and get that under control? We are up 5% from a week ago." As
with another common euphemism, calibration, confidence, and expense
all rolls down hill.
Introduction
High confidence in the test system directly translates to high confidence in the DUT
test results and "high" costs buying that confidence. The opposite is true as well.
Low confidence in the test system is a guarantee of low confidence in the DUT test
results, but a great way to trim costs. Between the "too much" and the "not enough"
is the elusive balance point between cost and confidence. This paper will explain the
variables that are at work in the world of calibration, how they can be used to find
that balance point, and ultimately answer those most important questions. "Will the
product work?" and "How much is it going to cost me to know?" It will also provide a
common denominator in the languages used by the metrologists and manufacturing.
Webster's Dictionary has always been a favorite of mine. He worked night and day to
define our language to a "T", no pun intended. Then we create nuances around each
word to match our needs in verbal communication. One would think that any words
connected with the world of metrology would be meticulously defined and universally
used and understood. That however is not the case. Here is some metrological and
manufacturing vocabulary as described in Webster's.
Author: Todd B. Wendle
Keysight Technologies, Inc.
Accuracy: "Noun, freedom from mistake or error : correctness 2. conformity
to truth or to a standard or model : exactness b : degree of conformity of a
measure to a standard or a true value."
Calibrate: "Verb, to standardize (as a measuring instrument) by determining
the deviation from a standard so as to ascertain the proper correction
factors."
Now we look at a couple of words from the ma nufacturing test side of the
fence.
Adequate: "Adjective sufficient for a specific requirement, especially: barely
sufficient or satisfactory."
Confidence: "The quality or state of being certain: certitude."
So, right there in Webster's is the answer to the question "Too much calibra-
tion?" The question changes to a statement:
"Compare to a known standard a measuring instrument with barely suffi-
cient accuracy as to have adequate confidence that the product being tested
will perform as specified in the marketplace all at a reasonable cost."
Filled with subjectivity, this definition is doomed. The words mean different
things to different people. My accuracy may not be near as accurate as yours.
Your adequate may not meet my needs. And a consumer's confidence may not
match with that of the manufacturer.
In order to gain that consumer confidence, the device performance has to be
measured in such a manner as to provide "proof" (there is another one of those
words) the results of the testing are accurate enough. "Accurate enough" has
a lot of implications. It implies the device has to work correctly, is ready for
use, will meet customer expectations, and will be at the lowest possible cost.
And to make it more interesting, there is always the metrology request for 95%
confidence levels. It just sounds good. And that is the basis for this paper.
"Accurate enough" is a very difficult unit of measure. 95% confidence levels are
sometimes expensive. How on earth can a metrologist survive where there is no
"accurate enough" measurement device at a guaranteed 95% confidence? The
real answer lies hidden within the performance characteristics of the testing
equipment and the use of a unit of measure everyone understands, money.
How DOES the manufacturer "know" the testing equipment is providing the
degree of certainty needed to have the device work correctly and meet, or hope-
fully, exceed customer expectations all at a reasonable cost?
Performance characteristics of a test instrument are defined by two primary
variables. The two are measurement uncertainty of the test system and the
measurement error as the measurement results drift over time. Note there are
also uncertainties associated with the Device Under Test (DUT). While these
uncertainties are important, they are out of scope for this paper.
3
In order to establish the contributions of both variables, they must be separated
and defined. For the purposes of this paper the first variable is defined to be the
inherent uncertainties defined by the individual components of the test system.
Also involved in the calculation of uncertainties are the connections between
test instruments, fixtures used to hold the DUT and combination effects. Every
piece of test equipment and test system has some degree of uncertainty. This
uncertainty is dependent on many factors. As you look at the following graphic,
the measurement error component has NOT been introduced. This is done later
in the paper.
Gaussian
distribution
curve
Measurement mean
Lower Upper
test limit test limit
Environment Phone Fixture Connector Test engine Connector Fixture Phone Environment
and cabling and cabling
True measurement
Uncertainty band
Figure 1. Measurement uncertainty
Figure 1 is an example of the composite uncertainty band in the testing of cel-
lular phones. In this example, as in most test environments, the test is only run
once and the single result determines pass or fail. If the DUT were to be tested
more than once, the result has a high probability it will not be the same the
second time. Nor would it be the same the third time. This variation in results is
caused by measurement uncertainty and is illustrated with the colored band.
When that colored band of uncertainty crosses a test limit, See Figure 2., the
uncertainty of the measurement result begins to be important. As the test
result moves closer to the test limit, the importance of uncertainty continues to
increase until there is a 50% chance the DUT is a true pass and a 50% chance
that it is a true failure. Once the test result and its uncertainty band are beyond
the test limit, the probability of being a true failure is 100%. Or is it?
This limited test determination creates two very important DUT populations.
Those are false passes and false fails. An entire industry has been built around
these two populations. These populations are at the center of the need for
reduction of measurement uncertainty and correction of measurement error.
With measurement uncertainty defined, let's move on to the second variable,
measurement error. Test instrument results can and do drift over time. The
test result provided a year ago may very well be different if run today. In some
instances drift can happen in a matter of minutes. Take the sides off of the test
rack, and the ambient temperature drops over 10C in a few minutes, causing
the spectrum analyzer to drift by over 1 dB. For the purposes of this paper, this
drift is defined as measurement error and the measurement of that error as
calibration.
To best define measurement error, one more diagram is needed to further the
definition and effects of measurement uncertainty.
4
Multiple DUT distribution, Test line limit
single test
Single DUT distribution,
multiple tests
Measurement uncertainty band
Figure 2. Test limit crossing multiple DUT
Pass Fail
single test normal distribution
Figure 2. is an illustration of a best-case situation where the test instrument
output has been calibrated and then the results adjusted to the midpoint
between its OEM specifications. The uncertainty band is the measurement
uncertainty of the test instrument and other possible factors as described in
Figure 1. With this situation, there is not a measurement error component in the
uncertainty band. The green dot is a single DUT result measured once. The pink
distribution curve is the distribution curve that is defined by running that single
DUT multiple times. That distribution defines the measurement uncertainty
associated with that particular test. Each single DUT test result has this mea-
surement uncertainty distribution.
When there are DUT failures, the multiple DUTs single test distribution (this is
the heavy black line in Fig. 2) falls across one or both of the specification limits
for that test. Around the specification limit are four populations; true passes,
true failures, false passes, and false failures. In the purest sense, false passes
are really true failures and false failures are really true passes. See Figure 3.
(Warranty returns and
customer dissatisfaction)
Pass True pass False pass
Fail
True fail False fail
(No trouble found)
True measurement False measurement
Figure 3. False passes and false fails
5
In that range of measurement uncertainty and measurement error, there is an
ever-changing probability ranging from almost zero to 50% that the test result
indicated is not correct. And if it is not correct, finding out which is correct will
cost money. That is ONE thing that does have a 100% probability of occurring.
Figure 4 illustrates the test results of a single test where the test instrument's
output for the particular test has drifted (over time) adding a measurement error
component to the original uncertainty band. The single test result in Figure 2
had a 50-50 chance of being a true pass or a true fail. After instrument drift, the
result will be a fail. This increase in total fails will have an increased population
of false fails. The costs associated with this false fail population (all false fails
attributable to the measurement error) create the demand for calibration.
Measurement Measurement error or
distribution after drift drift over time
Test line limit
Pass Fail
Figure 4. False fail drift
Now let's drift the other way. Figure 5 illustrates this event. In this case the
measurement error drifts such that all the DUTs are now passing. As this is
occurring, and since it is a gradual drift, production would be happy in that all
their efforts to reduce failures is "working". Bosses all the way up the chain are
happy, and back patting is available for all. But as with all "good" things, this
comes to an unhappy end. These false passes are getting to the customer and
they are sending them back under warranty for replacement. And maybe even
more important is that repair operation databases are not directly coupled with
warranty return databases. This means that it will take a long time, if ever, to
recognize any connection between the two at an operational level. The failures
are still "zero", but the warranty costs from this instrument continue to mount.
Now you have a case where these mounting false pass costs create additional
demand for calibration.
Measurement error or Measurement
drift over time distribution after drift
Test line limit
Pass Fail
Figure 5. False pass drift
6
Instrument drift can have no effect, create additional false fails, and can cre-
ate additional false passes. As the drift moves the DUT single measurement
distribution past a test limit, the effects will show up as unexplained yield
changes, good or bad, and after some period of time unexplained increases in
warranty costs. Management sets the expectations for what are reasonable
costs (yet another one of those words). In the manufacturing world, those costs
levels are the result of the confidence level set for the test system. This can be
compounded by multiple test systems on a single production line. The costs are
created from true failures, false failures, and false passes. The confidence level
of a test system will be improved for only one reason. That reason is money.
Either make more revenue or cut more costs with expense reductions. There are
no other reasons.
Confidence levels can be improved in three ways. One way is to reduce mea-
surement error with calibration. The second way is to buy more expensive (read
lower uncertainties) test instruments.
And the third way is to statistically account for the uncertainties by setting
calculated "test limits" that are tighter than the actual test limits. This is
commonly called guard banding. These guard bands "artificially" decrease the
uncertainty band as illustrated in the previous Figures. As mentioned earlier,
these second two ways will not be discussed in this paper.
Any confidence level improvement proposal will have an associated cost. To
management, this cost must be justified with some type of return on investment
analysis with a defensible and acceptable level of return. Reducing measure-
ment error with calibration is no exception. With a robust calibration program,
the calibrations will corrects problems before they ever occur thus making
them difficult to quantify an ROI for calibration expenses. That does not mean
it cannot be done however. As mentioned earlier, the heart of the matter is
the bottom line. Answering the following three questions begins the process
of knowing when there is "too much calibration" and too much impact to that
bottom line.