PERFORMANCE DATA COMPLETENESS AND RELIABILITY
Performance measurement is dependent on the availability of useful data that will indicate level of performance and helps progress toward organizational goals. Because all data are imperfect in some fashion, pursuing “perfect” data may consume public resources without creating appreciable value. For this reason, there must be an approach that provides sufficient accuracy and timeliness but at a reasonable cost. This section of the Performance and Accountability Report provides information on how DOT uses performance data, assesses limitations of the data, and plans to improve DOT’s data.
In an attempt to bring consistency and quality to its performance reporting, DOT has implemented some general rules regarding the data it uses and how it is evaluated.
Annual Data – Whenever available, the data in this document are reported on a Federal Government fiscal year basis. However, there are instances where this is not possible so calendar year data are used instead. This often occurs when data are collected and reported to DOT by external sources and a calendar year reporting requirement is specified in the implementing regulation.
Completeness of Data for Annual Results – If available, the results for the most recent year in the Report are listed as “Actual” in the shaded box for each performance measure. However, given the December deadline for submission of the Performance and Accountability Report, quite often data have not been reported, compiled and finalized for the entire year. When this occurs and an actual value is not available for the current year, either a projection or a preliminary estimate is provided instead. For most of the measures, projections are based on partial year data that are extrapolated to cover a full 12-month period. Given the partial year data, historical trend information, and program expertise, the remaining portion of the year’s performance is estimated. The result will be identified as a “preliminary estimate” in the Report. If partial year data are not available, then past trend information will be analyzed and supplemented by program knowledge, to develop a projected value for the annual performance measure. The result will be identified as a “projection” in the Report. As data are finalized, the projections and preliminary estimates will be replaced by actual results, with resulting changes denoted by an “(r)”. Results are also amended as errors and omissions are identified in the data verification process, as updated information is provided by the reporting sources, or because of legal or other action that changes a previously reported value.
Reliability of Measurement Data – DOT performance data are generally reliable (useful to program managers and policy makers). But because performance results in a given year are influenced by multiple factors, some of which are beyond DOT’s control, and some of which are due to random chance, there may be considerable variation from year to year. A better “picture” of performance may be gained by looking at results over time to determine if there is a trend.
Virtually all data have errors. We have compiled Source and Accuracy Statements for each of the DOT data programs used in this report, which can be found at www.bts.gov/statpol/SAcompendium.html. The Source and Accuracy Statements give more detail on the methods used to collect the data, sources of variation and bias in the data, and methods used to verify and validate the data.
Assessing and, where possible, eliminating sources of error in DOT data collection programs has always been an important task for data program managers. As part of their ongoing work, managers of departmental data programs use quality control techniques, such as flowcharting the data collection process, to identify where errors can be introduced into the data collection system. Program managers also use computerized edit checks and range checks to minimize errors that may be introduced into the data of their respective programs. In addition, quality measurement techniques are employed to measure the effects of unanticipated errors. These include verification of data collection and coding, as well as coverage, response and non-response error studies to measure the extent of human error affecting the data. As sources of error are identified, data collection is improved.
The data used in measuring performance come from a wide variety of sources. Much of it originates from sources outside the Department and, therefore, outside the direct control of the Department. The data often come from administrative records or from sample surveys. While DOT may not have a strong voice in improving the quality of outside data, the Department takes all available information about the limitations and known biases in outside data into account when using the data.
To help the Operating Administrations address these issues, the Bureau of Transportation Statistics (BTS) is developing a statistical policy framework where the Operating Administrations will work together to identify and implement the current statistical “best practices” in all aspects of their data collection programs. This project is consistent with the data capacity discussions found in the DOT Strategic Plan.
In 2003, a DOT intermodal working group addressing DOT data quality issues continued to:
▪ update the DOT Information Dissemination Quality Guidelines;
▪ implement those guidelines in DOT agencies;
▪ develop a detailed guide for transportation data collection;
▪ develop comprehensive data quality assessment practices; and;
▪ develop common definitions for data across transportation modes.
See Exhibit I for detailed explanations of completeness and reliability for each performance measure.
DOT Data Source Limitations – Timeliness is the most significant limitation for DOT performance measurement data. Some DOT data are not collected annually. For example, the National Household Travel Survey and the Commodity Flow Survey each collect data every five years. Data that are collected each year (or more frequently) require time to analyze, confirm and report results. For example, Highway Performance Monitoring System vehicle-miles traveled (VMT) data require several months of post-collection processing, making final results unavailable for this performance report.
Other performance measurement data limitations are located in the previously mentioned Source and Accuracy Statements for DOT data programs. These statements contain descriptions of data collection program design, estimates of sampling errors (if applicable), and discussions of non-sampling errors. Non-sampling errors include under-coverage, item and unit non-response, interviewer and respondent response errors, processing errors, and errors made in data analysis.
Projection Techniques used – As discussed under completeness, most of the 2003 measures have to be projected from either partial year data or from historical trends. The projections that use partial year data are more likely to reflect changes occurring in 2003 due to recent DOT treatment effects. The measures projected from 2002 and prior historical trend will reflect a continuing trend from ongoing programs, but will not reflect a change in trend due to 2003 effects.
External Data Source Limitations – Timeliness is also a significant limitation for external or third party data. In some cases, DOT has replaced external data, where little is known about the quality of the data, with internal data. For example, DOT has used estimates of person-miles traveled (PMT) from private organizations, absent any better estimate. The 1995 Nationwide Personal Transportation Survey and American Travel Survey give DOT data with known error properties that allow a better estimate of PMT.