NSF Award Abstract - #0241020| NSF Award Abstract - #0241020 |
AWSFL008-DS3 |
| NSF Org | SES |
| Latest Amendment Date | March 18, 2003 |
| Award Number | 0241020 |
| Award Instrument | Standard Grant |
| Program Manager |
Cheryl L. Eavey SES DIVN OF SOCIAL AND ECONOMIC SCIENCES SBE DIRECT FOR SOCIAL, BEHAV & ECONOMIC SCIE |
| Start Date | March 15, 2003 |
| Expires | February 28, 2006 (Estimated) |
| Expected Total Amount | $180000 (Estimated) |
| Investigator | Hua-Hua Chang hua.chang@mail.utexas.edu (Principal Investigator current) |
| Sponsor |
U of Texas Austin P.O Box 7726 Austin, TX 787137726 512/471-6424 |
| NSF Program | 1333 METHOD, MEASURE & STATS |
| Field Application | |
| Program Reference Code | 0000,OTHR, |
While the implementation of computerized adaptive testing (CAT) has many advantages, many issues related to CATs are not well understood. This project will study three specific issues in development and implementation of CAT: (1) compatibility between CAT and "paper and pencil" (P&P;) tests, (2) test security and item pool usage, and (3) how to calibrate test items in large quantities efficiently and economically. With respect to compatibility between CAT and "paper and pencil" (P&P;) tests, it has been widely reported that some students get much lower scores than they would if an alternative P&P; version were given. However, examinees currently required to take Graduate Record Examination (GRE) in the United States, for instance, are not given a choice between the standard P&P; version of the tests and the CAT versions. Without effective remedial measures, the credibility of CAT could be significantly undermined. This project proposes to modify the statistical procedure used for CAT item selection by incorporating some advanced analytic techniques. It is expected that the analytic and simulation results will show that weighting likelihood score may alleviate the problem of underestimation. With respect to test security and item pool usage, in current operational CATs, computers tend to select certain types of items too frequently, making item exposure rates quite uneven. This project will show that test security and the underestimation problem discussed in the research on CAT and P&P; tests are closely related. It is also expected that the project will show that the alpha-stratified approach proposed by Chang and Ying in 1999 tends to improve both the underestimation and test security. With respect to calibrating test items in large quantities efficiently and economically, administration of CATs requires very large item pools. Fortunately, CAT provides great potential to large-scale calibration during on-line testing. This project will explore the development of on-line calibration in CAT.CAT has become a popular mode of educational assessment in the United States. Examples of large scale CATs include the Graduate Record Examination (GRE), the Graduate Management Admission Test (GMAT), the National Council of State Boards of Nursing, and the Armed Services Vocational Aptitude Battery (ASVAB). Findings from this research project may speed up the process of improvement over current item selection algorithms. Because many CATs are high-stakes examinations, improving their test reliabilities will greatly benefit society.
