Most of the observational data at NCEP are stored in WMO BUFR format. This format is an international standard and provides an efficient means for transferring data. In addition it allows for great flexibility for adding new observation elements.

There are a series of tables associated with BUFR.  “Table A” defines the data category associated with a particular BUFR message containing report data.  “Table B” classifies and defines data elements, or descriptors, according to scale, reference value, number of bits and units.   “Table C” defines data description operators.  “Table D” defines the list of common sequences.  In addition, there are BUFR code and flag tables as well as code tables common to BUFR and other binary and alphanumeric codes.  The need for external tables can make the process for BUFR data encoding and decoding quite cumbersome for a typical user.

As such, a special application has been designed at NCEP which provides user-friendly access to the BUFR files through a series of FORTRAN subroutines in a machine independent BUFR library (called BUFRLIB).  These routines allow one to encode or decode data into BUFR using mnemonics to represent the data.  The mnemonics are associated with BUFR descriptors in a special version of the Tables A, B, C and D.  When a BUFR file is created, the mnemonic table is read in from an external location and is itself encoded into BUFR messages at the top of the output file.  These messages have Table A data category (message type) 11 (BUFR tables).  This allows each BUFR file to be “self defined”.  No external tables are needed to decode data out of the file.

NCEP Central Operations has written a BUFRLIB software user guide which provides a detailed explanation of the NCEP BUFRLIB subroutines along with other useful information on BUFR as it is used at NCEP.  (http://www.ncep.noaa.gov/NCO/DMQAB/Decoders/BUFRLIB/)

Next is a brief outline on the current method for processing observations that arrive at NCEP. Its main function is to provide links to web pages which discuss each item in detail.
 

1. NCEP receives the majority of its data from the Global Telecommunications System (GTS) and the National Environmental Satellite, Data, and Information Service (NESDIS).

    a. The GTS and aviation circuit bulletins are transferred from the NWS Telecommunications Gateway (NWSTG) to NCEP's Central Operations (NCO) and networked to one of two IBM workstations.  These data are then encoded into WMO BUFR format and transferred directly to the IBM SP via the Distributive Brokered Network (DBNet).  (http://www.ncep.noaa.gov/NCO/DMQAB/Decoders/)

    b. Most of the satellite data are processed in batch mode as they become available from the various NESDIS servers.  Regularly scheduled jobs on the IBM SP transfer "new" files from these servers and encode the data into WMO BUFR format. (http://www.emc.ncep.noaa.gov/mmb/papers/keyser/satellite_ingest.doc/document.htm)

    c. NCEP receives radial wind data from 154 NEXRAD radar stations via the radar multicast (NIDS format).  At eight minutes past each hour, a script runs to gather up the most recent radar data into eight pairs of files containing the raw data and a station list.  At thirty minutes past each hour these raw data are superobed and encoded into BUFR format.

All of the encoded BUFR data are then appended to the appropriate files in the data base.  The files are organized by the WMO BUFR type and local subtype and contain information in 24 hour blocks (based on report time).  Observational files remain on-line for up to 10 days before migration to offline cartridges. While online, there is open access to them for accumulating late arriving observations and for research and study.
 

2. The various NCEP networks access the observational data base at a set time each day (i.e., the data cutoff time) and perform a time-windowed dump of requested observations.  Observations of a similar type (e.g., satellite-derived winds, surface land reports) are dumped into individual BUFR files which maintain the original structure of  reports, although some interactive quality control is applied, duplicate reports are removed, and upper-air report "parts" are merged.  (http://www.emc.ncep.noaa.gov/mmb/papers/keyser/data_dumping.doc/document.htm)
 

3. A series of programs read in the observations from the various dump files, add background and observation error information, perform automated quality control, and finally output the observations in a  monolithic BUFR file known as "PREPBUFR". (http://www.emc.ncep.noaa.gov/mmb/papers/keyser/prepbufr.doc/document.htm)
 

4. The PREPBUFR file is read by the analysis codes. (Global SSI analysis system documentation can be found at:  http://sgi62.wwb.noaa.gov:8080/RTPUB/research/jhtml/ssi3.html.)
 

Here are some other links to my web sites devoted to data processing:

PREPBUFR Report Types used by Global/SSI (Aviation and Final) systems

PREPBUFR Report Types used by Global CDAS/reanalysis systems

PREPBUFR Report Types used by ETA/3DVAR (EDAS, Eta and NGM) systems

PREPBUFR Report Types used by upper-air Rapid Update Cycle (RUC-2 A and B) systems

Summary of Virtual Temperature Processing in PREPBUFR

RAOB/PIBAL Balloon Drift Processing in PREPBUFR

Redesign of the NOAA/NCEP Wind Profiler Quality Control Module

Sample program to decode reports from PREPBUFR file

Summary of Changes to the PREPDATA Program

**NEW** User Guide to Interpreting Data Dump Counts in Data Dump Status Files

BUFR Table B Descriptors and Mnemonics in NCEP Observational Database

Satellite Historical Documentation

NMC Office Note 29

NMC Office Note 124