This web site was copied prior to January 20, 2005. It is now a Federal record managed by the National Archives and Records Administration. External links, forms, and search boxes may not function within this collection. Learn more.   [hide]

Executive Summary

Final Minutes from Meeting

Methods and Data Comparability Board

April 16-17, Cincinnati, OH

A total of 25 participants attended the meeting either by phone or in person. Federal agencies represented included EPA, USGS, ACOE, and USDA/NRCS. States represented included Virginia, Arizona, Kentucky, Delaware River Basin Commission (New Jersey), and New York. Other monitoring interests represented were ASTM, Standard Methods, East Bay Municipal Utility District, Merck and Company/Chemical Manufacturers Association, Hach Company, Quanterra Environmental Services, and Tetra Tech, Inc.

Announcements and General Comments

USGS will soon appoint an Executive Secretary for the Methods Board who will be responsible for Board communications, minutes, and meeting logistics, as well as coordinating some technical projects.

Co-chairs Herb Brass and Merle Shockey are co-authoring a paper on the Methods Board for the AWWA National Water Quality Technology Conference this fall.

Jerry Diamond will give a poster presentation on the Board at the ASTM meetings April 21-22.

Comment: Priorities for the Board should include specific short-term projects as well as long-term objectives.

Comment: Need to recognize use of data for public health.

Clean Water Action Plan

Chuck Spooner (EPA) summarized Vice President Al Gore’s Clean Water Action Plan (CWAP) and discussed many action items pertaining to the National Council and the Methods Board. Chuck stressed the importance of how and what the Board produces in the interim to help ensure funding of the CWAP and to help the Board achieve authority and stature.

Cooperative Research and Development Agreements

Larry Fradkin (EPA) presented an overview of Cooperative Research and Development Agreements (CRADAs) and how they can relate to Board activities/objectives. Bill Battaglin (USGS) discussed the USGS-DuPont CRADA indicating that the Board will be asked to review technical papers and data produced from the project.


A discussion of Board priorities indicated that PBMS issues, a methods compendium (including minimum data elements and quality control criteria), public health issues, and public outreach are all high priority areas, especially in light of several specific action items in the CWAP. Workgroups have already been established for PBMS and public outreach. A workgroup was established to address the methods compendium and metadata requirements. This group will build on efforts already started by Larry Keith and Rob Henry on the Board. Metadata requirements and minimum data elements for several databases or programs including STORET, SDWIS, NAWQA, EPA’s proposed PBMS, and EMAP, will be reviewed by this Workgroup and the Board.

Public Outreach

The Board decided to move forward with the following: develop a simple slide show for upcoming presentations about the Board (e.g., Reno meeting); finalize the Fact Sheet (i.e., incorporate laboratory photos provided by USGS) and get it printed; set-up an internal restricted access Web page for Board business and communications; and finalize and distribute press releases. The Fact Sheet and other outreach materials will be distributed at the Norfolk meeting and the NELAC and Solid Waste Conferences. A list is being compiled of suitable outlets (e.g., journals, newsletters, conferences) for press releases and other outreach materials.

National Contaminant Occurrence Database

EPA’s National Contaminant Occurrence Data Base (NCOD) structure was reviewed by members of the Board. A description of the modernized STORET system, upon which part of the NCOD relies, will be presented at the next Board meeting along with a presentation on EPA’s drinking water data base, SDWIS. Several issues need Board input concerning appropriate meta data requirements and the handling of non-detect data.

Performance-based Systems

EPA’s performance-based measurement system approach was presented by David Friedman (EPA-EMMC), and Bill Telliard (EPA, Office of Water) presented EPA’s streamlining proposal. Andy Eaton (Standard Methods) reported for the PBMS Workgroup which resolved to examine other agencies’ or program experiences with PBMS-type frameworks (e.g., NOAA, FDA, NASA). The Board indicated that field and biological methods are to be included in the review of PBMS approaches. Definitions of PBMS and related terms (e.g., reference method) will also be examined. The Board recognized the importance of defining the information that needs to be associated with data for comparability demonstrations and data validation. Herb indicated that the Board needs to discuss the type of products that may come out of the PBMS workgroup and the Board. A scoping document needs to be available from the PBMS workgroup that the Board could look at during the next meeting.

State/Tribal Participation on the Board

Efforts are on-going to solicit more state and tribal participation on the Board. Some potential contact people or organizations were identified that will be pursued.

Methods and Data Comparability Board

Final Meeting Minutes

16 April 98 Cincinnati, Ohio


Herb Brass, Merle Shockey, Ann Strong, Gary Cottrell, Cliff Annis, Jr., Kitty Kono (for Bob Held), Barbara Erickson, Katherine Alben, Ed Santoro, Howard Hankin, Michal Harthill, Larry Penfold, Richard Ayers, Andy Eaton, Bob Berger, Jerry Diamond, David Friedman, Giles Miller, Chuck Spooner, Dave Gustafson, Larry Fradkin, Dennis McChesney, Bill Battaglin, John Klein, Bill Telliard, Dick Reding

The Agenda is included in Attachment 1.


There will be conference call meetings on 5/20 and 6/17, 11:00 AM EDT

The next meeting is 12-14 August 98 in Denver. It may run 2-2 ½ days. An August 12 PBMS meeting will be followed by a 13-14 August meeting of the full Board.

USGS should be appointing an Executive Secretary for the Board soon.

An internal web page will be constructed to address e-mail limitations/difficulties in conducting Board business.

Herb and Merle are co-authoring a paper on an overview of activities of the Board that has been accepted for the AWWA National Water Quality Technology Conference this fall.

The perception has been voiced that field methods may not be addressed by the Board to the same extent as laboratory methods. Herb indicated that this was never the intention of the Board, and that there is a lot of interest and expertise in field methods represented on the Board. This was further addressed in the Priorities discussion later in the meeting.

Herb is encouraging people to attend the next National Council Meeting on 12-13 May 98 in Reston, VA. The ITFM strategy and recommendations, including recommendations made by the ITFM Methods and Data Comparability Task Group, will be reviewed and discussed.

The Clean Water Action Plan (CWAP) was publicized in October 1997 and a Final Action Plan was issued February 19, 1998. The National Council is named in the Plan and issues (specifically PBMS and QA/QC) are specified. Many items in the CWAP reference the Council and the Board.

Merle mentioned that USGS is developing a new draft method for deriving minimum detection levels (MDL) for laboratory analyses of water. He is interested in having the Board review this protocol. Cliff and Richard are interested in reviewing the USGS MDL draft method.

Perspectives of Board Participants

Herb and Merle asked Board participants for their perspectives/comments concerning the Board’s mission and objectives.

Michal: PBMS is requisite. The issue of low level detection of endocrine disruptors and carcinogenic organics is critical. The USGS Columbia, MO lab activities are relevant. Perhaps for the next meeting, time can be allotted concerning Environmental and Contaminants Research Center (ECRC) activities.

Katherine: Discussion of PBMS is good but we need to keep our focus on different classes of compounds. The Board should undertake some smaller projects in addition to major ones, so as to ensure identifiable products are generated.

Bob: We need to think more broadly than chemical parameters and methods; biological, pathogen, and microbial methods are also important.

David F.: There seems to be a need to get back to the objectives of monitoring and monitoring questions. However, once data are collected, the database often gets used for other purposes.

Dave G.: More thought needs to be put into the front end of monitoring in terms of data quality objectives and appropriate methods.

Cliff: We don’t need to test simply for the sake of testing; there needs to be a reason for testing. His superiors are very interested in the definition of PBMS and how it will be implemented.

Barbara: Public health data need to be more useful. Someone from the Center for Disease Control (CDC) should be represented on the Board. Barbara will give Herb and Merle names of contacts.

Ed and Andy indicated that there are many general questions pertaining to the development and handling of MDL’s in assessment work.

Andy said that defining MDL is a very complex issue and that ASTM had a large group discussing this issue.

Richard: The Board’s mission needs to be clarified as to whether it is concerned with specific projects or large scale conceptual, programmatic issues.

Andy: What is the Board’s authority and responsibility with regards to USEPA and USGS programs?

Howard: He is impressed by the number of action items in the CWAP related to the National Council and Methods Board activities.

Ann: There is a need to deal with data base issues.

Chuck: The Board needs to identify and resolve methods-specific comparability issues and develop a methods compendium with performance measures. Subsequent discussion suggested that the methods compendium would include grouping and prioritizing methods. The Board can assist EPA on the re-engineered STORET database, especially related to meta data requirements.

The Council and Clean Water Action Plan (CWAP)

Chuck Spooner gave an overview of the CWAP. A copy of his presentation slides is included in Attachment 2. He indicated that an amended Clean Water Act is politically unlikely even though it is old and needs updating, especially in light of recent water issues such as Pfeisteria, atmospheric nitrogen, and nonpoint source pollution. The CWAP appears to be a politically feasible approach to reaching the intended goals of the Clean Water Act.

Chuck stressed that there needs to be a unified watershed assessment approach in which states take the lead in water monitoring and assessment. The CWAP is primarily concerned with restoration programs rather than preventative actions. However, a part of the Plan includes identification of any pristine watersheds on federal lands.

The major agencies involved in the Plan are Department of Agriculture and EPA although COE, NOAA, and DOI are also involved. The first step is to develop a unified assessment approach, making due with available data (305(b), 303(d), IWI). Other facets of the Plan include:

a "Healthy People" initiative that will deal with issues such as endocrine disruptors and public beach safety

natural resource protection and restoration including wetland and coastal water protection, encouragement of private land stewardship, and restoration of riparian buffers

reducing nutrient-caused water resource degradation through the development and implementation of nutrient criteria, reductions in atmospheric nitrogen, improvements in subsurface sewage disposal and septic systems, strengthening of EPA’s 319 program to control polluted runoff nutrient enrichment, and expand EPA’s permit controls on industry, agriculture, and feedlots.

Chuck indicated that there are 111 Action Items in the CWAP. He presented the following key action items relating to the Methods Board:

Compare sampling and laboratory methods and protocols leading to performance-based acceptable methods.

Establish reference parameters for specific monitoring purposes.

Develop guidelines on quality assurance and quality control.

He pointed out that the Steering Committee for the Plan has been coordinating with the White House Committee on Environmental and Natural Resources (CENR). Chuck indicated that the CENR has no staff or independent money. The National Council appears to be the major intergovernmental body that will be charged with implementing the CWAP.

USEPA has a current initiative called Reinventing Environmental Information that goes beyond water media and is under the requirements of Governmental Performance Act. This Act is taken very seriously by many agencies, including EPA’s Office of Water, and will utilize ITFM indicators and other Office of Water goals to assess their performance.

Chuck concluded by saying that, in the interim, the National Council and the Methods Board need to be creative about what products are put out and how they are distributed. By focusing on high quality products the Council and the Board will eventually gain stature, acceptance, and authority. He noted that the political atmosphere is currently good for acceptance of the Council and the Board.


Larry Fradkin gave a brief overview of the purpose and structure of cooperative research and development agreements (CRADAs) and contrasted that with grants. He noted that for a grant there is no direct financial benefit to the Federal agency. In a CRADA, though, associations can involve private, for-profit, and non-profit organizations and money can come to EPA or other organizations for their own use. The Board, for example, could use a CRADA to promote new methods, QA/QC, and funding private companies interested in refining existing or developing new methods. Also, funds can come to the Board from a CRADA. CRADAs are not binding and easy to get out of if necessary. Larry indicated that CRADAs cannot be used if a project is not involved in research and development or further evaluations of current methods. However, he noted that research and development is loosely defined in the Federal Technology Transfer Act. Therefore, CRADAs could include testing as well as projects defined by EPA as research and development. He added that data from CRADAs can be exempt from Freedom of Information Act requirements for 5 years and treated as confidential. Larry stated that USDA, DOE, DOD, FDA, and NIH all have CRADAs in place.

Larry indicated that the scope of work for CRADAs are reviewed by experts or parties internally by the organizations involved. There is no other peer-review required. A difficulty with peer-review of CRADAs is maintaining confidentiality and proprietary requirements for new technology.

Review and Discussion of Board Priorities

Jerry lead a discussion of Board priorities which included a brief summary of the ITFM recommended priorities (termed "Objectives" in the Board Charter) and a summary of the priorities submitted by Board participants over the past 3-4 months. Jerry then opened the discussion to new ideas, or priorities not previously voiced, or modifications to those already listed. The following list summarizes ideas offered at the meeting:

After some discussion, the Board agreed that the following activities were highest priority:

1). Define a usable PBMS; develop guidelines for data quality under a PMBS and necessary QA/QC elements (metadata) to determine comparability of data for different methods.

2). Identify and resolve method-specific comparability issues.

3). Develop a compendium of methods with performance measures.

A more complete list of activities suggested by Board members follows:

• Develop a set of guidelines for data quality under PBMS.

• There is a need for more resources within programs to collect and use metadata for later use as well as a need for a uniform data format.

• There is a need for facilitating the use of appropriate data elements for better data sharing and a need to collaborate with NCOD.

• Develop and refine guidance on QA/QC within the CWAP; this might be subsumed in the PBMS priority already raised by the Board. Develop guidelines for usable data for different levels or uses (e.g., legal, trend, etc.).

• Standards need to reflect dimensions of the environment (i.e., methods need to reflect environmental conditions).

• Need to define benchmarks for assessing data comparability - look at Mid-Atlantic Integrated Assessment (MAIA) - Landscape Atlas.

• PBMS needs to include sampling and other field aspects as well.

• Develop guidelines to ensure that data collected now will be useful in the future.

• Need to focus on public health monitoring and assessment and collect information for which it will be used.

A PMBS workgroup has already been formed and is addressing the first priority. A second workgroup was identified to handle priorities (2) and (3) above. Tentatively, this "methods" workgroup consists of Larry Keith, Rob Henry, Michal Harthill, and Chris Yoder. Others are free to join this workgroup if interested. Larry Keith has a database of methods that can be put on the internet. Rob Henry is producing a matrix that needs to be examined.

The Workgroup will need to define its mission in consultation with the Board co-chairs and the National Council. It was suggested that this Workgroup meets soon and possibly also in conjunction with the August Board meeting.

Review of DuPont CRADA

Cliff, Howard, and Richard reviewed a draft Fact Sheet and this years’ workplan provided by Bill Battaglin. Bill noted that the DuPont CRADA was reviewed by at least twenty people. Board members responded favorably to the workplan and the objectives of the CRADA. Cliff commented that for an occurrence study, more than 2 samples per site are needed. Bill indicated that due to laboratory limitations on the number of samples they could handle, the initial focus of the CRADA has been on a spatial analysis of the herbicides rather than a temporal analysis. However, additional follow-up work may be undertaken as an add-on effort.

Bill mentioned that quarterly progress reports will be prepared by USGS. The Board will get these progress reports but there is no need for it to review them. The Board will be needed to review more formal data publications stemming from the CRADA work.

Howard mentioned that a similar type of study was published by USGS for a west Tennessee stream system. Howard gave a copy of the paper to Gary who will forward to Bill. Jerry also has a copy of the paper for those interested.

Public Outreach

Bob Berger and Jerry Diamond lead a discussion of public outreach objectives and alternatives. Jerry distributed the latest version of the Fact Sheet which is in color and includes all edits and format changes required by USGS. Gary will send Tetra Tech digital versions of photographs from USGS’s lab that could be incorporated into the Fact Sheet in addition to some of the field photographs presently included.

Jerry also displayed a poster prepared by Tetra Tech on the Methods Board for presentation and discussion at the ASTM meetings in Atlanta April 20-22.

The following outreach items were recommended by the Board:

Tetra Tech will set up a restricted access web site or list server for Board internal business; Lockheed-Martin will work with Tetra Tech on the design.

Draft press releases prepared by Tetra Tech need to be distributed to all Board members for review. Jerry will get suggestions from Board participants on organizations, conferences, journals, etc. that should be considered for press releases or short articles.

A simple slide show should be developed by Tetra Tech for Board presentations. It can be used in conjunction with the upcoming NWQC in Reno in July.

An Executive Summary of the minutes could be distributed elsewhere as an outreach vehicle.

Outreach information needs to target managers of groups that do monitoring. Items should include tutorials, and guidance with links to Council activities. The Estuary Program is a good venue in this regard as there are many types of groups working in this area (SOLEC, Great Lakes, Canada).

Informative articles would be good in journals such as Environmental Science and Technology or Environmental Management. The Board Fact Sheet and introductory letter will be given to all attending the Norfolk conference.

David suggested a poster at the NELAC and maybe other meetings.

A mailing list for the fact sheet needs to be developed.


Herb announced that he will send out a reminder to all delegates verifying to whom a copy of the official letter should go. Participants suggested that Herb add to the letter that the Board was named in Vice President Al Gore’s CWAP.

Herb and Merle noted that the Board is still actively soliciting state delegates and alternates to the Board. It was noted that the National Environmental Laboratory Program (NELAP) may be able to provide contacts for the Methods Board. Herb will contact Jeannie Mourrain (NELAP Director).

David mentioned that EPA regions meet periodically with their states on monitoring issues and usually senior state people (lab directors) are present. It might be useful to coordinate a Board meeting during one of those regional meetings to get more state participation at least within that region.

Next Board Meeting

The next Board meeting will be in Denver, August 12, 13 & 14. Details will follow later.

National Contaminant Occurrence Database (NCOD) Review

Dennis McChesney, Bob Berger, and Andy Eaton reviewed the database framework for EPA’s National Contaminant Occurrence Database and provided comments (previously distributed to the Board). A question was raised by Chuck Spooner regarding certain comments made concerning STORET. Chuck indicated that a strength of modernizing STORET is that owners can withdraw data, if bad, and replace it. The architecture of STORET will be greatly improved according to Chuck and the new version will be out in May. However, "who" enters the data will still be a significant issue because only state or federal agency staff, not private organizations, are able to input data to STORET. Chuck noted that SDWIS (drinking water database) is working in a parallel track with STORET but is somewhat behind in the degree of progress. The question was raised as to what data in the Information Collection Rule will be included in NCOD? There need to be linkages so that the full ICR data set can be accessed.

Chuck indicated that he will bring a demonstration of the new STORET, as well as metadata considered for inclusion in STORET, to the next Board meeting. Dennis indicated that the same should be done for SDWIS. Data qualifiers need to be similar in the two databases. The Board also needs an update on NCOD of the next meeting.

Chuck needs to know if there are aspects relating to STORET and SDWIS they would like to see stressed or focused for his presentation at the next Board meeting.

Bob indicated that the Board should have input on handling of non-detect data in NCOD and method detection limits.

EPA’s PBMS Activities

David Friedman discussed EMMC’s perspective on PBMS. A copy of his presentation is included in Attachment 3. He indicated that each EPA program is implementing PBMS in its own way and according to its own schedule but the goal is to have as much consistency as possible. The will be a need to establish performance criteria and change regulations to remove method-specific restrictions. This will facilitate the development and use of new technologies. EPA needs to codify documentation requirements for methods and develop a training program to help implement PBMS.

Issues regarding the implementation of a PBMS:

How will method accuracy be determined?

What QC and documentation data should be required of the labs and the regulated community?

How can PBMS be made program-specific yet consistent across the Agency?

David indicated several specific areas in which EPA would like the Board’s assistance:

Review EPA draft implementation plans

Provide peer-review in the development of training programs

Provide recommendations on method validation standards

Andy raised the question about EPA wanting recommendations on implementation plans, yet these plans will be program specific according to David. How would the Board’s recommendations be handled? David indicated that the Board’s recommendations would go to the PBMS workgroup within EMMC which has representation from all EPA programs.

Cliff raised the concern that a PBMS developed by EPA will not work if states do not buy into the program. Herb asked to what extent does state buy-in affect EPA’s progress on PBMS? David indicated that state buy-in won’t affect EPA’s progress but for the program to be successful the states will also need to adopt the PBMS approach. He said that the states will need to be convinced by showing them PBMS works. David also noted that outreach programs are planned in all EPA regions that will involve regional management, states and laboratories, and program management (e.g., EMMC staff will be meeting with Region 7 and their states in April). PBMS discussions will also be held at Bill’s Norfolk meeting at the June NELAC meeting in San Antonio, and at the July WTQA meeting in Washington). David also noted that there are Agency-wide workgroups developing communications and training strategies.

Office of Water Streamlining Effort and Discussion

Bill Telliard summarized EPA’s Office of Water streamlining proposal. Bill indicated that streamlining was proposed as an option rather than a rule because otherwise, it would require extensive cost analysis and take much longer to implement. EPA also wanted to use this proposal as a "trial balloon" and the Office of Water is trying to get a few states on board to try to implement it.

The objectives of the streamlining effort are:

implement a new performance based approach

foster approval of new techniques

provide flexibility in methods used

increase confidence in methods and data

decrease time and Agency resources required.

Bill noted that reference methods are the cornerstone of the streamlining framework. Reference methods will have documented QC specifications and acceptance criteria, and the types of appropriate matrices will be specified.

Currently, the streamlining proposal offers no flexibility in modifications allowed in sample holding time and sample preservation/storage. Bill also noted that Office of Water’s Compliance and Enforcement Division is still unsure as to how far EPA can proceed with PBMS or streamlining.

Bill indicated that the Office is considering screening methods for certain types of compliance monitoring. The Norfolk conference will include discussions on this topic. He noted that the Drinking Water program is actively discussing the incorporation of screening methods.

Bill summarized in the following table the differences between EPA’s current framework, streamlining, and EPA’s PBMS discussed previously by David.







no flexibility

controlled flexibility

total flexibility

reference methods


QC acceptance


no methods required

standardized QC

no standardized QC

Comments on the Streamlining Proposal

Dick Reding summarized the comments received by EPA. He noted that most comments came from state and local authorities although industry also commented. Approximately 1000 comments or issues were raised. State and local concerns were mainly focused on:

training and resources,

the need for clarification of legal standing of modifications,

updating reference methods and performance criteria, and

developing a system to track method modifications.

Industry comments focused on;

legal weight of modified methods, and

consistent implementation across states

Laboratories and vendors were concerned that the rule be clear as to whether certain sections are guidance or requirements.

Report to Board from PBMS Workgroup

Andy summarized the results of the PMBS workgroup meeting in Cincinnati (see Attachment 4 for a summary of those minutes). The Workgroup decided that the Board needs to examine and evaluate experiences other agencies have had with PBMS approaches. Herb indicated that the Board needs to discuss the type of products that may come out of the PBMS workgroup and the Board. A scoping document needs to be available from the PBMS workgroup that the Board could look at during the next meeting.

The Board agreed that it would be helpful to review and collate PBMS experiences from other programs and agencies. David will contact the EPA GLP program, Larry P. will contact FDA, and Dennis will get information on NASA’s PBMS experience. Adriana will be contacted by Jerry concerning NOAA’s status and trends PBMS; David will follow up on USFS PBMS efforts.

Jerry will follow-up with Workgroup people and obtain relevant information for review by the PBMS workgroup.

David suggested the American Chemical Society PBMS study supported by EPA/ORD might be augmented with audits performed by regional compliance staff to examine some of the PBMS concepts. Dennis knew of a similar type of audit recently in Denver. He will send the specifics to David.

Barbara suggested that we need to focus on what needs to be changed: methods, regulations, or something else? Richard pointed out that the Board needs to evaluate biological field methods as well as laboratory methods. Following up on Richard’s statement, Herb suggested that it may be useful to have Jerry again present the pre-laboratory PBMS framework developed by ITFM. Dennis reminded everyone that the Board’s mission is to recommend guidance on obtaining consistency among programs and among PBMS frameworks. He also suggested that non-regulatory monitoring programs need to be incorporated in the PBMS discussions and not just regulatory programs. He stressed that data comparability issues are especially important for ambient monitoring data.

David asked whether we are concerned with data usability or comparability? Herb and others answered that for ambient monitoring data, comparability is usability. This may not be the case for regulatory programs.

Chuck questioned whether the science of DQO derivation is understood enough so that comparability demonstrations are possible. He added he would like the Board to consider developing a "pull-down" menu for deriving DQO’s depending on the type of study (biological, sediment, etc.).

David suggested that the Board needs to review metadata requirements for DQO documentation. DQO requirements may also be necessary for comparability demonstrations of metadata requirements.

Chuck stated that the Board needs to foster collaborative monitoring efforts to help the stature and visibility of the Council and Board; this should be a high priority. John Klein added that the Board needs to provide guidance on collaborative studies and encourage regional and inter-organizational efforts.

David will get his list of minimum data elements for EPA’s PBMS to Jerry for distribution. Merle will check NAWQA minimum data elements. Andy and Jerry will check streamlining list of elements. Andy will set-up a conference call for the PBMS workgroup after getting some of the information together from Jerry.


Minutes from PBMS Workgroup Meeting (4/16/98), Cincinnati, OH

Participants: Andy Eaton (Chair), Cliff Annis, Jr., Ann Strong, Chuck Spooner, Michal Harthill, Merle Shockey, Katherine Alben, Barbara Erickson, Howard Hankin, Ed Santoro, Herb Brass, Jerry Diamond, Kitti Kono, David Friedman, Larry Penfold

The Workgroup meeting began by discussing definitions of different PBMS systems. David made the distinction between a measurement versus a method system. The latter assumes a reference method with which the performance of other methods is compared. A performance-based measurement system, which is what EPA’s EMMC and the Agency as a whole is advocating, specifies data and measurement quality objectives and any method that meets those objectives can be used. A reference method is not specified in a measurement system. However, there must be a method available that can meet specified data quality criteria. EPA’s streamlining effort requires a reference method while EPA’s performance-based measurement system does not.

The question was raised as to where ITFM’s definition (termed a performance-based method system in the ITFM Final Report) fits in with these other two definitions. After much discussion, and after hearing Bill Telliard’s use of the term reference method, it became clear that there are some issues regarding semantics and definitions that need to be resolved. It was not clear that Bill Telliard’s description of EPA’s PBMS was very different from ITFM’s definition.

David sought to highlight similarities and differences among the three definitions (and also determine the relative ease of using each framework) by making the following table.








reference method?

yes - EPA


yes MDCB





how is DQO set?

as good as reference method

meet study need

as good or better than reference method





ease of implementation




quality of data




After much discussion among Workgroup participants, it was clear that there was disagreement about some of the table entries and there were several unknowns. Because firm definitions appeared to be lacking for the moment, the Workgroup decided a more fruitful course of action would be to discuss what is necessary to make any performance-based system useful, not overly burdensome on the data generator, ensures data of known quality, and enhances the determination of data comparability among methods. All participants were then asked to voice their concerns/issues/questions relating to finding the most appropriate performance-based system.

What is the minimum information needed to ensure and generate high data quality?

What is required to validate the method and who reviews the results? What is the peer review process?

How is EPA going to implement this testing program, in many laboratories or one laboratory?

Who is establishing DQOs?

What about field methods?

Comparability is a key aspect of PBMS. Which is more important, consistency or accuracy?

Issue of expectations in field versus expectations in laboratory.

Screening methods may do an excellent job satisfying certain data quality objectives and addressing environmental or human health questions. The doctor testing analogy was presented in which screening methods are very informative. There is no accurate lab data. We must find the most useful methods and obtain the most informative data.

It may be too simplistic to have a single system that works well in all situations. We need some experiences to draw from - specific projects.

There is not enough money and time to validate methods for every single matrix. We need to provide information that gives customer/user confidence in data quality.

The Board is charged to develop guidance on determining method comparability.

Validation costs would be extensive given the measurement system described by David. Laboratories could do PBMS as the Board has defined it but availability of adequate reference materials is still a problem.

Permittees are regulated by states and if the states don’t accept EPA’s PBMS, then the system isn’t going to make a difference. Data quality may still be an issue because of the legal questions surrounding the implementation of a PBMS.

Measurement systems do not reduce the cost of monitoring. ITFM’s definition of data of known quality is preferred. Reference methods need to be available.

EPA needs to change the way it does business and add more flexibility in the methods used.

There is a question as to how a PBMS should be implemented. Should it be implemented in stages or should the ultimate desired system be introduced all at once. The Board is a stakeholder group and there is concern about implementation.

If one has two validated methods, who makes the decision on which is the correct reference method, especially if they give different results (e.g., phenol and pH).

As a result of hearing everyone’s questions and issues, the Workgroup decided that the Board should look at EPA’s implementation plans for a PBMS and that the Workgroup should also examine attempts by other agencies and programs (FDA, NOAA, etc.) to develop and/or implement a PBMS framework. In this way, we may avoid some pitfalls experienced by other programs and develop a truly useful strategy and approach.


HomeReturn to the Minutes Page