Method Verification Within a Performance-Based System Framework: Pilot Study using Chemical Oxygen Demand (COD) Methods

Version 4.4.3

Authors: J. Diamond (1), A. Eaton (2), R. Dunn (3), C. Annis (4), R. Vitale (5), and H. Brass (6)

(1) Tetra Tech, Inc., Owings Mills, MD; (2) MWH Laboratories , Pasadena, CA; (3) Hach Company, Loveland, CO; (4) Merck & Co., Inc., West Point, PA; (5) Environmental Standards, Valley Forge, PA; (6) USEPA, Cincinnati, OH

ABSTRACT

A performance-based method or measurement systems (PBMS) approach has been suggested as a useful means for ensuring improved water information in a variety of monitoring programs in the U.S. One challenge that currently impedes the implementation of a PBMS is identifying agreeable and reasonable requirements for a laboratory to demonstrate that it can perform a new or modified method adequately and that the method is appropriate for the intended use. To address this issue, seven laboratories, including federal, state, and private entities, generated data using both the currently approved chemical oxygen demand (COD) method and a new COD method that does not include the use of hazardous chemicals. Results were subjected to two common PBMS approaches: (a) a measurement quality objectives (MQO) approach and (b) a “ reference method ” approach. Results of this pilot indicated that most labs successfully completed the initial demonstration of capability using laboratory reagent water but performance was poorer for both methods using actual matrix samples. Several labs could meet the target MQOs using the approved (reference) method but only one lab met the MQOs using the new method. Different labs had acceptable results depending on the type of PBMS examined. Furthermore, the reference method approach could yield a false sense of acceptable method performance. Our results indicate that shortcomings in the new COD method are also inherent to some degree in the reference COD method. Accordingly, the use of sample matrix spikes is critical to achieving accurate data, regardless of the type of PBMS, and that laboratory verification of a method can be achieved with reasonable effort.

Keywords: chemical oxygen demand, quality control, performance-based system, water quality, comparability, matrix spike

INTRODUCTION

The reliance on prescriptive methods, without appropriate method performance documentation, has had significant consequences on data quality for both ambient water monitoring (e.g., Total Maximum Daily Load evaluations) and compliance monitoring (e.g., wastewater effluent compliance under the Clean Water Act) in the U.S. [1, 2]. The National Methods and Data Comparability Board (Board), an interagency panel chartered to promote and coordinate the collection of comparable, known quality, water resource data [3-5], endorses the use of a performance-based method or measurement system (PBMS) as an important means to address this issue [6, 7]. A performance-based approach permits the use of any scientifically appropriate method that demonstrates the ability to meet established method performance criteria ( e.g., accuracy, sensitivity, bias, precision) and complies with specified data quality needs or requirements [2]. There are several challenges to implementing a performance-based method or measurement system (PBMS) approach, particularly in regulatory compliance programs. A key challenge is defining laboratory procedures that rigorously verify the appropriateness of a new or modified method for a given monitoring objective, while not being too excessive and costly for the laboratories conducting the analyses.

To address this challenge, we designed and coordinated a pilot study that evaluated the implementation of a PBMS using a new chemical oxygen demand (COD) method (Hach Company Method 10125, Hach Co., Loveland, CO) that does not use hazardous chemicals. The currently approved method (Hach Method 8000) requires the use of mercury and hexavalent chromium as reagents, both of which present potential safety hazards for the analyst and are expensive to dispose.

Two prominent PBMS approaches were evaluated in this study: (1) measurement quality objectives (MQO), and (2) “ reference method ”. The reference method approach requires demonstrating that the new method yields equivalent or superior data quality relative to the approved reference method [9]. The MQO approach to PBMS requires demonstrating that the new method meets the MQOs established for a given project [10]. The reference method approach is the PBMS currently recognized by the US Environmental Protection Agency (EPA), Office of Water for method-defined parameters such as COD [9].

This paper summarizes attempts to develop, test, and evaluate an analytical protocol that reasonably and efficiently: (a) demonstrates laboratory competence with a method, (b) demonstrates that new method performance is appropriate for the matrix of interest, and (c) demonstrates acceptable laboratory performance.

METHODS

Results from seven laboratories are reported including those from federal agencies (USEPA and US Geological Survey), states laboratories (Oregon Department of Environmental Quality), Municipal (City of Phoenix and Hampton Roads Sanitation District), and private sectors (United Sewerage, and Montgomery Watson Laboratories). As this was not designed to be an inter-laboratory study, and PBMS is implemented ultimately on a laboratory-specific basis [2], results from individual labs were analyzed separately. Having multiple participating laboratories in this study, however, provided an enhanced overview of the challenges and constraints inherent in the laboratory method verification process.

The methods tested in this study were the current USEPA-approved method for analyzing chemical oxygen demand (COD) (Hach Method 8000 [13]) and a new method (Method 10125) developed by Hach Co. (Loveland, CO). Method 8000 is a dichromate oxidation procedure that is widely used for wastewater testing and uses reagents containing mercury and hexavalent chromium, both of which are potentially hazardous to analysts and result in high disposal or recycling costs. The applicable concentration range for Method 8000 used in this study is 20 – 1500 mg/L. Method 10125 uses a digestion reagent containing trivalent manganese, which is a strong chemical oxidant that changes quantitatively from purple (Mn III) to colorless (Mn II) when it reacts with organic matter. Organic content is determined as mg/L COD by spectrophotometrically measuring the absorbance of (Mn III) in the reacted sample solution. As Method 10125 does not include the use of hazardous or carcinogenic reagents, this new method is potentially advantageous and a more environmentally sound practice. The applicable concentration range for Method 10125 is 50 – 1000 mg/L according to the method developer.

Initial Demonstration of Capability

Each laboratory conducted an initial demonstration of capability (IDC) with both methods using laboratory reagent water (ASTM Type II or equivalent [13]) and samples were spiked with Potassium Acid Phthalate ( KHP) , similar to what was used in originally validating the approved COD Method 8000. A total of five COD concentrations ranging between 20 and 1000 mg/L, plus a reagent water blank, were analyzed by each laboratory using both COD methods. Analysis of covariance was used to test whether there was a difference in bias (at a 95% confidence level) between methods in a given laboratory as a function of COD concentration. Precision data for each concentration and method in a given laboratory were compared with suggested method precision limits ( + 10% relative standard deviation [RSD]) and with each other using an F-ratio test (using 95% confidence) to determine if method-specific differences in precision were observed. A greater variance in the new method ( i.e., F-statistic is > 1), would indicate that precision of the new method is less than that of the reference method.

Matrix Analyses

Following the IDC, each laboratory conducted COD analyses, using both methods, on 12 different samples. A subsample of each sample was spiked with a known concentration of KHP and analyses were performed on both spiked and unspiked samples. Each laboratory also performed analyses on two matrix samples, randomly chosen, spiked with 800-1000 mg/L chloride, which is known to be a potential interference to both COD methods. All samples were preserved with H 2 SO 4 to a pH < 2 in the field and the sample holding time did not exceed 28 days prior to analysis [13].

PBMS Statistical Analyses

Two sets of statistical analyses were performed on the laboratory data, one using the reference method approach and one using the MQO approach to PBMS. All statistical analyses were performed for each laboratory individually because in a PBMS framework, it is important to evaluate whether a given laboratory can use a new or modified method with confidence. In the reference method approach, bias and precision were compared between the two methods using the reference method value as the true value. We assumed that the error of the reference method and new method were independent and that the two error variances were similar, if not equal. To compare bias, the absolute value of the differences between method values was computed for each sample and these difference values were analyzed using a Wilcoxon matched pairs analysis (at a 95% confidence level). Precision was determined by comparing the standard deviation in spike recoveries between the two methods for the 12 samples analyzed by each laboratory. Since the reference method is assumed to yield the true value in this set of analyses, we tested whether the variation in recovery using the new method was equal to or lower than the variation observed using the reference method. An F-ratio test (using 95% confidence) was used to compare the variances exhibited between methods.

For the MQO approach, both methods were compared to MQOs which were considered achievable by the method developer. The MQOs for precision and bias were a relative percent difference (RPD) £ 10% among replicate analyses and spike recoveries between 90 and 110%, respectively. Bias for each method was calculated by comparing spiked sample values to the predicted value based on the unspiked sample value and the spike concentration added. A percent recovery for each sample was calculated, yielding a total of 12 recovery values for each method. Mean bias of each method was evaluated by calculating the 95% confidence interval of the spike recoveries. If the results were entirely within the recovery range, then the method was considered to have met the MQO for bias for that laboratory. Precision for each method was evaluated by calculating the 95% confidence interval of the relative standard deviation of the recovery values for each method and then comparing the confidence interval with the MQO of RSD ± 10%. If the upper limit of the confidence interval for the RSD was < 10%, the MQO for method precision was considered to be met.

RESULTS

Initial Demonstration of Capability

Results of the IDC indicated that most labs adequately completed the initial demonstration of capability with the new method using laboratory reagent water (Table 1). For all participants, the blank and MDL levels were closer to 0.0 using the approved method than the new method. Recoveries of spiked COD in reagent water were generally acceptable using both methods, however, they were more variable using the new method, particularly in the lower (50 mg/L) COD concentration. Thus more bias was generally observed at the low end of the respective applicable concentration range using the new method. Precision of both methods was generally high (relative standard deviation of blanks was <10% in all cases, Table 1). Thus, most of the participating laboratories were capable of achieving similarly precise data using both COD methods in spiked reagent water.

Reference Method PBMS

Using the approved Method 8000 results as the “ true ” value, most laboratories reported a difference in COD using the new method. Mean absolute differences between the two methods ranged between 40.2 and 149.9 mg/L COD (Table 2). Wilcoxon matched pairs tests indicated a significant difference (p < 0.05) between sample results using the two methods for Laboratories 3, 4, and 5 (Table 2). The standard deviation of the difference values varied between 26.2 and 197.2 indicating inconsistent data among samples for a given laboratory. Closer inspection of the data indicated that neither the absolute nor relative difference (difference divided by the Method 8000 value for each sample) were related to COD concentration (r < 0.2, p > 0.05). Furthermore, there was no consistent relationship within a laboratory between the two methods for the 12 samples analyzed; for some samples Method 10125 yielded a higher COD than Method 8000 while other samples within the same laboratory yielded the opposite result.

Method precision was generally higher for Method 8000 than Method 10125 for most laboratories, as indicated by a lower standard deviation for the former method (Table 3). Standard deviations ranged between 2.6 and 54.3 for Method 8000 and 11.3 and 75.6 for Method 10125. For Laboratories 1, 2, 3, 5, and 6, Method 10125 was significantly less precise than Method 8000 (F = 3.89, p < 0.05; Table 3). Thus for those laboratories, Method 10125 did not perform as consistently as Method 8000.

MQO PBMS Approach

Many of the laboratories could not meet the initial MQOs for precision or bias using either COD method when testing actual matrix samples. Average percent recoveries ranged between 65.3 (Laboratory 5) and 96.3% (Laboratory 3) for Method 8000 and between 49.3 (Laboratory 5) and 118.1% (Laboratory 6) for Method 10125 (Table 4). Most of the laboratories obtained average recoveries within the specified MQO for bias. However, the 95% confidence intervals for recoveries indicated that, for many of the laboratories, neither method met the specified MQO of 90-110% recovery. Only Laboratory 3 met the MQO for bias using Method 8000 and none of the laboratories met this MQO using Method 10125 (Table 4). If the MQO had been defined as 80 – 120% recovery (a fairly common bias objective used in wastewater and other regulatory programs [13]), four of the seven laboratories would have met this objective using Method 8000 and three laboratories would have met the MQO using Method 10125.

Closer examination of each laboratory ' s data indicated that some samples yielded values below the associated laboratory method reporting level. This was an issue for Laboratories 1 (5 samples), 2 (6 samples), and 3 (1 sample). In a few other cases, samples had very high COD values such that the spike amount was £ 5% of the total COD in the sample (Laboratory 1 [3 samples]; Laboratory 4 [2 samples]). In an attempt to address this issue, sample values that were either below the laboratory detection limit or that were > 10 times the spike concentration were censored . Re-analysis of each laboratory 's censored data yielded similar bias results in general (Table 4) indicating that there was little effect of sample COD concentration on observed bias in this study. In fact, for Method 10125, the 95% confidence interval increased using censored data for Laboratories 1-3, probably due to the smaller sample sizes available for analysis.

For Laboratories 1, 2, 3, 5, and 7, precision was higher for Method 8000 as evidenced by lower RSD values (p < 0.05; Table 5). For Laboratory 6, the RSD value was higher using Method 8000. Analyzing RSD based on censored data, as described above for bias, did not substantially alter precision results for a given laboratory or method (Table 5). Laboratories 2, 3, and 7 met the MQO for precision using Method 8000. None of the laboratories met this MQO for Method 10125. If the MQO for precision had been defined as ± 20% RSD, a fairly common criterion used in regulatory programs [13], then Laboratories 2-5, and 7 demonstrated acceptable precision for Method 8000 and Laboratories 3, 4, 6, and 7 met this criterion for Method 10125 (Table 5).

Chloride-spiked Matrix Samples

For most of the participating laboratories, samples spiked with 800-1000 mg/L chloride yielded similar percent recoveries using either COD method (Table 6). Laboratories 1, 3, and 7 showed no significant difference in COD recoveries using either method. Except for Laboratory 6, percent COD recoveries were similar for chloride-spiked and unspiked samples using both methods (see Table 4). Thus, it appeared that both methods generally performed similarly to a chloride challenge in actual sample matrices.

DISCUSSION

Results of analyses using the two different PBMS approaches demonstrate the inherent challenges in implementing a PBMS approach, particularly for a method-dependent parameter such as COD. Using the reference method approach, one laboratory (Laboratory 7) obtained similar precision and sample values using both methods (8000 and 10125). However, the MQO approach indicated that only Laboratory 3 met the precision and bias criteria for the new method and that three laboratories could meet the MQOs using the reference method (Table 7). The fact that different laboratories demonstrated appropriate use of the new method, depending on which PBMS approach was used, suggests that the two approaches can yield very different outcomes. This has important ramifications for implementing a PBMS. For example, a laboratory that performs the reference method imprecisely or inaccurately may not acknowledge a truly better method using a reference method approach.

In implementing a reference method PBMS approach, it may be critical for laboratories to demonstrate that they can achieve the stated performance criteria of the reference method. This recommendation, however, is reasonable only if the reference method performance criteria were subject to appropriate method validation (e.g., inter-laboratory comparison studies) and are achievable. Similarly, MQOs need to be realistically formulated such that a method is capable of consistently meeting them [2, 8]. We do not advocate one PBMS approach over another, as each approach may be valid. A critical evaluation of the reason or need to consider alternative methods should help define which approach is more prudent and reasonable in a given situation.

We observed that many laboratories could not meet the MQOs using Method 8000. It is not clear whether this is a laboratory, method, or an MQO issue. If an MQO is set unrealistically, it may not be meetable, regardless of the performance of the method or laboratory. EPA, voluntary consensus standards bodies (e.g., Standard Methods, ASTM, AOAC) have used inter-laboratory method validation studies to establish method performance. Because methods are empirical, arbitrarily selecting an MQO of 90-110% or without a knowledge of method performance may lead to unrealistic expectations of performance for a method or laboratory. Alternatively, a new method may be rejected for use when, in fact, even the approved methods do not meet the MQOs for the matrices of interest. Had only the MQO approach been examined in this study, there would have been no reason to perform analyses using the currently approved method, since interest is in performance of the new method only. This study demonstrates that a laboratory needs to confirm that the method currently used meets the desired MQOs on a consistent basis.

This Study identified two major issues that must be considered in designing or implementing method verification studies within a PBMS. First, as the quality of the data is paramount, it is critical that most samples analyzed have detectable levels of the targeted analyte using an approved reference method. If analyte concentrations are at or below the laboratory practical quantitation limit, it will not be feasible to differentiate method performance from random error due to analytical “ noise ” . This was an issue for several laboratories in this study and may have influenced the observed performance of both COD methods.

Second, it is important to analyze enough samples that cover the range of matrices typically encountered by the user laboratory, and that provide sufficient statistical power to determine method performance, without being too onerous an effort. While more replication can provide greater statistical certainty of results, such replication usually entails higher costs and effort, which could prohibit laboratory implementation of any form of PBMS. Documenting laboratory verification of a new method should require substantially less effort than that used to validate a new method . It should be noted that a user laboratory would not need to analyze samples using both versions of PBMS as was done here. Typically, only one version of PBMS would be used which would substantially reduce the cost and effort of a method verification exercise. The use of 12 paired samples in matrix analyses in this study appeared to be sufficient to determine whether the new method yielded results consistent with MQOs and/or with the approved reference method. Although paired matrix spike samples were used in our study, they are not required in the reference method PBMS approach.

Spiked samples provided an important measure with which to document method performance using an MQO approach. Therefore, the selected spike concentration needs to be at a concentration that can be detected relative to the concentration of the original samples, given the precision limits of the method. In a few cases in this study, samples had very high concentrations of COD (> 2000 mg/L), relative to the targeted spiking concentration (250-280 mg/L). The effect was that spiking, prior to diluting the sample to within the range of the method, resulted in dilution of the spike to such an extent that it was too small to be detected accurately using either method. In future studies, it would be useful to measure the background concentration of the analyte first, and then determine the spike concentration (i.e., 1 to 5 times the background), similar to what is required in EPA ' s Wastewater Alternate Test Procedure [9].

Results using actual matrices suggested that for many of the laboratories involved, the new COD method was more variable (less precise) and more biased than the approved referenced method, both using a reference method and a MQO approach to PBMS. Reasons for this could include: (1) inexperience of lab personnel with the new method, (2) certain aspects of the new procedure were susceptible to laboratory error, and (3) certain matrix interferences may occur that are not observed using KHP spikes and laboratory reagent water. The differences observed in method performance between the IDC and the matrix analyses for most laboratories clearly demonstrates that a complete picture of method performance requires the inclusion of sample matrices [9]. Programs that validate new or modified methods should consider this finding carefully. Our results suggest that, at least for laboratory method verification (and perhaps validation by the method developer as well), matrix sample analyses could be more informative than an IDC using reagent water. The importance of testing real-world matrices was also demonstrated in a recent validation study examining a different COD method [14, 15].

We recognize several important conditions which must be met to successfully implement a PBMS [2] including : (1) DQOs and MQOs must realistically define the quality of data needed, (2) validated methods must be available that meet those DQOs and MQOs, (3) qualified operators are also capable of meeting those DQOs and MQOs using the selected methods, (4) reference materials in relevant matrices must be available for analysis, and (5) method ruggedness is demonstrated. This study demonstrates the critical role that each of these conditions plays in implementing a PBMS. MQOs were defined in different ways in this study depending on the data quality needs, which resulted in different decisions concerning a laboratory ' s ability to use either COD method with confidence. The approved reference method did not always meet the initial MQOs in this study, indicating the importance of documenting that the validated method meets the MQOs identified. The reference material used in this study was a labile, easily measured form of COD that may or may not adequately challenge the performance of a COD method. Use of more realistic COD reference materials, in relevant matrices, would improve the overall confidence in our results. Finally, ruggedness of either COD method could not be determined. However, in general, the currently approved method yielded more reliable results over a broader range of matrices and COD concentrations than the new method. Training in the new method is believed to be one important factor affecting method ruggedness in this case.

Not all matrices are known or defined beforehand in a PBMS study, nor is it desirable to evaluate the performance of a method in all possible matrices. Therefore, for laboratories that endeavor to verify a method, on-going quality control analyses are critical to the successful implementation of a PBMS, and to good laboratory practice in general [2]. Based on our results, duplicate matrix spikes are one quality control check recommended for implementing a PBMS and for routine quality control. Failure to meet the specified DQOs and MQOs for duplicate matrix spikes should result in further investigations to determine whether problems are due to a lack of ruggedness of the method or due to interferences from particular matrices.

Acknowledgements

The authors wish to thank Robin Costas, Randy Gottler, Deborah Kay, Patty Lee, Tom Reppert, Penny Bristol, Jackie Davis, Mark Cree , Glenda Brown, Lisa Ramirez and Chuck Wibby for their assistance in providing laboratory data for this study. Chris Frebis provided review of statistical approaches used in this study. Many members of the Methods Board provided constructive comments and suggestions at various points in the design and execution of the study including Charlie Peters, Larry Keith, Bill Ingersoll, and Katherine Alben.

Literature Cited

(1) ITFM. 1995a. The Strategy for Improving Water Quality Monitoring in U.S. Geological Survey Open-File Report 95-742, Reston, VA.

(2) NWQMC. 2001. Towards a definition of a performance-based approach to laboratory methods. Methods and Data Comparability Board, National Water Quality Monitoring Council, Technical Report 01-02, US Geological Survey, Water Information Office, Reston, VA.

(3) MDCB. 1998. The National Methods and Data Comparability Board: Collaboration and Comparability. Methods and Data Comparability Board, http://wi.water.usgs.gov/methods

(4) Brass, H.J., H. Ardourel, J. M. Diamond, A. Eaton, L. H. Keith, and C. A. Peters, Activities of the Interagency Methods and Data Comparability Board. Proceedings of the American Water Works Association, Water Quality Technology Conference, Salt Lake City, Utah, November 2000.

(5) Peters, C.A., H.J. Brass, J. Diamond. 2000. United States Water Quality Methods and Data Comparability Board: Creating a framework for collaboration and comparability, in proceedings of Monitoring Tailor-made III, September 25 - 28, 2000, Nunspeet, the Netherlands.

(6) ITFM. 1995b. Performance-based approach to water quality monitoring. In : Strategy for Improving Water Quality Monitoring in the U.S., Appendix M, U.S. Geological Survey Open-File Report 95-742, Reston, VA.

(7) Eaton, A. and J. Diamond. 1999. Reservoir dogs and performance-based systems. Envir. Testing and Analysis 8: 18-19.

(8) ELAB. 1998. Recommendations for Implementation of PBMS. Draft. Environmental Laboratory Advisory Board, Washington, DC

(9) USEPA. 1997. Streamlining EPA's Test Methods Approval Program. EPA-821-F-97-001 Environmental Protection Agency, Office of Water, Washington, D.C.

(10) Federal Register. 1998. Federal participation in the development and use of voluntary consensus standards and in conformity assessment activities. OMB Circular A-119, Federal Register 63(33) p. 8546, Washington, D.C.

(11) Federal Register. 1997. Guidelines establishing test procedures for analysis of pollutants and national primary drinking water regulations; flexibility in existing test procedures and streamlined proposal of new test procedures. Vol. 62:14975-15049, Washington, DC.

(12) Cantillo, A. and G. Lauenstein. 1998. Performance-Based Quality Assurance — The NOAA National Status and Trends Program Experience. In: Proceedings of the NWQMC National Monitoring Conference. USEPA, Washington, DC. Pp. 63-74.

(13) USEPA. 1994. Guidelines establishing test procedures for the analysis of pollutants under the Clean Water Act. Final Rule. 40 CFR Part 136, Federal Register. 59:20 4504.

(14) USEPA. 1999. Protocol for EPA approval of Alternate Test Procedures for Organic and Inorganic Analytes in Wastewater and drinking water. EPA-821-B-98-002. Office of Water, Washington, D.C.

(15) Miller, D.G., S. V. Brayton, and W.T. Boyles. 2000. Chemical oxygen demand analysis of wastewater using trivalent manganese oxidant with chloride removal by sodium bismuthate pretreatment. Water Environ. Res. 73:63-71.


Table 1. Summary of results of initial demonstration of capability using two COD methods (8000 and 10125) and reagent-grade water, by participating laboratory. MDL = method detection limit.

Lab

Blank*

8000/10125 mg/L

MDL

8000/10125 mg/L

Spike Recovery for all test concentrations

(Range - %)

8000/10125

Spike Recovery (%) for the 50 mg/L concentration

8000/10125

Precision of fortified blanks

(% RSD)

8000/10125

1

1,2/6,9

6.7/19.9

100-102/93.9-106

102/106

0/1.4

2

2,4/-11, - 2

4.3/8.5

100-101/95-102

100/100

0/0.7

3

0,2/0,0

3.4/5.0

103-104/101-104

104/104

0.3/0.3

4

0,0/3,10

4.4/21.5

94-100/72-91

94/72

0.7/3.1

5

0,0/-25, - 9

4.0/16.7

98-102/72-101

98/72

0/2.6

6

0.5,2.2/18,35

3.4/41

100-102/96-140

100/140

2.6/7.1

7

0,0/0,0

9.1/18.6

99-110/97-118

110/118

4.1/0.7

*Results for both blanks are shown for each method.

Table 2. Results (as mg/L COD) of Wilcoxon matched pairs test of unspiked sample data (unspiked) to evaluate method bias using a Reference Method approach. N=12 for each lab except Laboratory #6 where N=11.

 

Statistics

Lab

Mean Difference (Absolute Value)

Relative Difference (Mean Difference/Mean COD Value using Method 8000)

Standard Deviation

T

Z

p

1

149.9

0.13

197.2

31

0.18

0.86

2

43.7

0.21

47.1

16

1.80

0.07

3

40.2

1.37

26.2

0

3.06

0.002

4

92.7

0.05

228.4

1.0

2.98

0.002

5

78.3

0.30

52.2

0

3.06

0.002

6

68.8

0.37

62.8

24

0.80

0.42

7

41.3

0.09

24.6

36

0.19

0.844

Table 3. Results (as mg/L COD) of F-tests on spiked sample data to evaluate method precision using a Reference Method approach. N=12 for each laboratory except Laboratory 6 where N=11.

 

Recovery Standard Deviation

 

Lab

Method 8000

Method 10125

F Statistic

P Value

1

38.3

75.6

3.89

0.03

2

4.2

29.4

49.95

< 0.001

3

2.6

12.8

23.76

< 0.001

4

16.1

15.8

1.04

0.94

5

10.6

25.9

5.91

0.006

6

54.3

17.9

9.21

0.016

7

8.6

11.3

1.72

0.38

Table 4. Results (as percent relative standard deviation) of statistical analyses of spiked samples for each laboratory to evaluate method bias using a MQO approach.

 

Method 8000

Method 10125

Lab

Mean

95% C.I.

95% C.I.*

Mean

95% C.I.

95% C.I.*

1

95.1

70.8 - 119.5

85.0 – 98.6

106.8

58.8 – 154.8

29.2 – 212.8

2

91.2

88.5 – 93.8

88.0 – 93.1

91.8

73.1 – 110.5

70.9 – 112.4

3

96.3

94.7 – 98.0

96.6 – 99.1

91.1

82.9 – 99.2

73.6 – 97.9

4

92.8

82.6 – 103.0

82.6 – 103.0

91.7

81.7 – 101.7

81.7 – 101.7

5

65.3

58.6 – 72.1

58.6 – 72.1

49.3

32.8 – 65.7

32.8 – 65.7

6

87.3

50.8 – 123.8

50.8 – 123.8

118.1

106.1 – 130.1

106.1 – 130.1

7

94.8

89.4 – 100.3

89.4 – 100.3

91.2

84.1 – 98.4

84.1 – 98.4

* 95% C.I.* = censored data < MDL

Table 5. Results of statistical analysis of spiked sample data for each laboratory to evaluate method precision using a MQO approach.

 

Method 8000

Method 10125

Lab

RSD %

RSD* %

RSD %

RSD* %

1

40.3

51.7

70.8

81.8

2

4.6

4.2

31.6

33.7

3

2.7

1.5

14.0

8.9

4

17.3

17.3

17.2

17.2

5

16.2

16.2

61.2

61.2

6

62.2

62.2

15.2

15.2

7

9.1

9.1

12.3

12.3

RSD* = Data removed for samples with COD that were either < the laboratory MRL using each method or that were too high relative to spike level.

Table 6. Percent recoveries of COD in chloride-spiked matrix samples using both COD methods.

 

Method 8000

Method 10125

Lab

Mean

SD

Mean

SD

1

92.8

1.9

92.8

8.0

2

88.0

4.4

68.1

9.9

3

89.1

6.4

85.8

8.9

4

86.1

0.7

78.8

1.7

5

73.9

5.7

Data not available

6

21.6**

0.1

89.2

4.0

7

102.9

4.8

98.1

2.0

** Data suspect – may have been reporting error.

Table 7. Summary of results using the new COD Method 10125 for each laboratory, using either a reference method (Ref Method) or measurement quality objectives (MQO) approach to performance based systems. No = data did not meet criterion or specification; Yes = data met criterion or specification.

 

Precision

Bias

Laboratory

Ref Method

10% RSD

20% RSD

Ref Method

90-110%

80-120%

1

No

No

No

No

No

No

2

No

No

No

No

No

No

3

No

No

Yes

No

No

Yes

4

Yes

No

Yes

No

No

Yes

5

No

No

No

No

No

No

6

No

No

Yes

No

No

No

7

Yes

No

Yes

Yes

No

80