A Toolkit for Evaluating Public R&D Investment Models, Methods, and Findings from ATP's First Decade Rosalie Ruegg Irwin Feller July 2003

Contents
Download in Adobe PDF front section, Part I, Part II and Part III.
For paper copies, e-mail your request to atp-eao@nist.gov.

Note from Project Manager  [View PDF]

About the Authors [View PDF]

Acknowledgments [View PDF]

Abstract [View PDF]

Executive Summary [View PDF]

• Evaluation Underpinnings
• Multi-Faceted Methodological Approach
• An Emerging Body of Findings
• Firm/Industry Effects
• Collaboration Effects
• Spillover Effects
• Interfaces with State Programs and Comparison with Counterpart Programs Abroad
• Overall ATP Performance
• Recommendations for Future Directions

Acronyms and Abbreviations [View PDF]

Introduction [View PDF]

• The Political Economy of the Advanced Technology Program
• The Role of Evaluation

PART I: EVALUATION FRAMEWORK [View PDF]

Chapter 1. Evaluation Fundamentals

• Objectives of Evaluation
• Mapping Evaluation to Mission and Stages of Implementation: A Generic Evaluation Logic Model
• Evaluation Steps
• Best Practices

Chapter 2. Choosing Methods of Evaluation

• Analytical/Conceptual Methods for Modeling and Informing Underlying Program Theory
• Survey Method
• Case Study: Descriptive
• Case Study: Economic Estimation
• Sociometric/Social Network Analysis
• Bibliometrics: Counting, Citing, and Analyzing Content of Documents
• Historical Tracing
• Expert Judgment
• Suggested Readings on Evaluation Methods

Chapter 3. ATP’s Evaluation Program

• Background: Evaluation Drivers
• ATP’s Evaluation Logic Model
• Conceptual Tests of ATP’s Success
• ATP’s Approach to Evaluation

PART II: APPLYING EVALUATION METHODS TO ATP [View PDF]

Chapter 4. Modeling and Informing Underlying Program Theory

• Concepts, Models, Metrics, and Paths Connecting Program Activities to Intended Impacts
• Conditioning Expectations through Studies of Private-Sector Behavior
• Conditioning Expectations through Studies of Other Public Sector Programs
• Testing Dominant Paradigms
• Summary of Research Informing Underlying Program Theory

Chapter 5. Survey Method

• Gaining Early and Broad Perspective of a Program’s Effects
• Extending and Deepening Survey Data on Program Effects
• Establishing Routine Project Reporting by Electronic Survey
• Soliciting Feedback by Survey on Customer Satisfaction and Marketing Issues
• Using Survey for Case Study and to Address Research Questions
• Summary of ATP’s Use of the Survey Method

Chapter 6. Case Study Method

• Economic Case Study of Individual Projects
• Project and Portfolio Assessment Using Multiple Cases Studies with Uniform Collection of Key Indicator Data
• Explicating Program Features and Exploring Program Dynamics
• Summary of ATP’s Use of the Case Study Method

Chapter 7. Econometric/Statistical Method

• Testing ATP’s Leveraging Effects on Advanced Technology Development
• Modeling Impacts of Public-Private Partnerships on Firm Productivity
• Spillovers, Appropriability, and Firm Productivity
• Analyzing the Role of Universities in Public-Private Partnerships
• Modeling the Impact of Publishing by Industry Scientists on the Quality of Innovative Output
• Investigating Characteristics and Impacts of Joint Ventures
• Summary of Econometric/Statistical Methods

Chapter 8. Other Evaluation Methods Used by ATP

• Expert Judgment
• Bibliometrics
• Emerging Methods: Using Social Network Analysis to Identify Knowledge Spillovers
• Emerging Method: Using the Cost Index Method to Estimate Social Benefits
• Emerging Method: Using a Composite Performance Rating System Constructed from Case Study-Derived “Indicator” Data
• Summary of Other Evaluation Methods

PART III: AN EMERGING BODY OF KNOWLEDGE [View PDF]

Chapter 9. A Crosscutting Look at Study Findings

• Direct Private Firm Effects
• Collaboration Effects
• Spillover Effects
• State and International Technology Programs
• Measures of Overall ATP Performance
• Summary of Crosscutting Findings

Chapter 10. Conclusions and Recommendations

• Program Evaluation: An Essential Tool for ATP
• Opportunities and Future Directions
• Closing Note

References

About the Advanced Technology Program (inside back cover)

Figures

• Generic Evaluation Logic Model
• ATP’s Evolving Use of Methods Over its First Decade
• Highlights of Main Topics
• Figure 1–1. Evaluation Logic Model
• Figure 3–1. ATP’s Evaluation Logic Model
• Figure 3–2. Intensity of ATP’s Use of Evaluation Methods
• Figure 4–1. Private and Social Returns to R&D: Pure Market Spillover, Plus Pure Knowledge Spillover, Plus Interaction of the Two
• Figure 4–2. Three Dimensions of ATP and the “Journey to Q”
• Figure 4–3. Direct and Indirect Paths to ATP’s Impacts
• Figure 4–4. Social Benefits from Product Innovation that Reduces Costs of Industries Using It
• Figure 4–5. Reducing Inaccuracies in Profitability Estimates Over Time
• Figure 4–6. Roadmap of Financing Options
• Figure 4–7. Strategic Mind Map for a Rich Technology Platform
• Figure 4–8. Conceptual Time Path for ATP’s Technology Diffusion and Expected Impacts
• Figure 4–9. A Social Interaction View of Innovation
• Figure 6–1. Elements Determining Social Return on Public Investment and Social Return on Investment
• Figure 6–2. Valuing Per-Patient Changes in Health Outcomes
• Figure 6–3. Historical Adoption Rates of Technologies Enhancing Fuel Efficiency, and Implementation Rates used by Ehlen in the Case Study
• Figure 6–4. Market Drivers for the Further-Processed Food Industry
• Figure 6–5. Distribution of 27 Completed Projects at Small, Single Applicant Companies by Percentage of Employment Change
• Figure 7–1. Relationships across Industry R&D Intensity, Intra-Industry R&D Information Flows, and Appropriability Mechanism Effectiveness
• Figure 7–2. Patenting and Publishing in the Scientific Literature, Gene Therapy: Patents and Publications
• Figure 8–1. The Classic Bass Diffusion Curve
• Figure 8–2. Interview Guide Used by RTI Researchers to Compile Judgments from Physicians Who Are Expert in Treating the Relevant Diseases
• Figure 8–3. Distribution of Projects by Number of Publications and Presentations
• Figure 8–4. Distribution of Completed Projects by Number of Patents Filed
• Figure 8–5. Illustrative Patent Tree for ATP
• Figure 8–6. Conceptual Framework of the R&D Network
• Figure 8–7. All Citations are not Equal: A One-Way Flow Illustration
• Figure 8–8. Top Twenty Optical Waveguide Regions
• Figure 8–9. Derived Demand for New Technologies: Illustration of Net Surplus Change
• Figure 9–1. Change in the Nature of Industry R&D
• Figure 9–2. Importance of Timing
• Figure 9–3. Employment Change at 31 Small Companies Receiving a Single-Company Award
• Figure 9–4. Key Factors Influencing Collaborative Success
• Figure 9–5. Illustration of Two Project Structures with Alternative Collaborative Relationships
• Figure 9–6. Costs Attributed to Collaboration
• Figure 9–7. Comparison of the Focus of ATP and State Technology Development Programs
• Figure 9–8. Distribution of Terminated Projects by Reason for Termination
• Figure 9–9. Distribution of Projects by Overall Performance Score
• Figure 9–10. Distribution of Projects by Overall Performance Score and Technology Area
• Figure 10–1. ATP’s Evolving Use of Methods Over its First Decade

Tables

• Table 1–1. Why Evaluate?
• Table 1–2. Steps in Evaluation
• Table 2–1. Overview of Evaluation Methods
• Table 2–2. Steps in Survey-Based Statistical Studies
• Table 3–1. ATP’s Mission, Mechanisms, and Features
• Table 3–2. Select ATP Studies Commissioned and Completed, 1991–1995
• Table 3–3. Select ATP Studies Commissioned and Completed, 1996–1999
• Table 3–4. Select ATP Studies Commissioned and Completed, or Delivered in Draft, 2000
• Table 4–1. Twenty-Two Studies and Papers Modeling and Informing Underlying Program Theory
• Table 4–2. Classification of Spillovers
• Table 4–3. Factors Increasing the Likelihood of Spillovers
• Table 4–4. Recommended Metrics for ATP and Timing of Data Collection
• Table 4–5. Interview Questions Investigating Determinants of Success of Joint Ventures
• Table 4–6. Volume of Venture Capital Activity
• Table 4–7. Categorizing State Technology Assistance Program by Type of Challenge and Stage of Technology Development and Commercialization
• Table 4–8. ATP Strategies Designed to Promote Technology Diffusion
• Table 5–1. Eight of Eleven Studies Using Survey Method Represented*
• Table 5–2. Program Effects Treated in the First Survey of ATP Projects
• Table 5–3. Illustrative Survey Questions on Commercial Progress
• Table 5–4. Illustrative Survey Questions Relating to Knowledge Spillovers
• Table 5–5. Illustrative Survey Questions about Collaborative Partners
• Table 5–6. Components of ATP’s Business Reporting System
• Table 5–7. Examples of Variables for Comparing Small and Larger Firms Using BRS Data
• Table 5–8. Competitive Position of Member Companies in the World PWB Market
• Table 5–9. Survey Questions Probing the Counterfactual: With and Without ATP
• Table 6–1. Ten of Sixteen Studies Featuring Case Study Represented
• Table 6–2. Summary of Survey Findings on Partial Early-Stage Economic Impacts
• Table 6–3. Overview of ATP Projects Included in this Study
• Table 6–4. Impact of ATP Funding on the Development of Medical Technologies for Seven Tissue Engineering Projects
• Table 6–5. Comparison of QALY Utility-Weights for Different Health States
• Table 6–6. Social Return on Investment and Social Return on Public Investment: ATP Projects in Tissue Engineering for a Single Preliminary Application
• Table 6–7. Composite Private Returns: ATP Projects in Tissue Engineering for a Single Preliminary Application
• Table 6–8. Annual Impact on U.S. Macroeconomy of Near-Term, Five-Year Implementation Path: Year 2004
• Table 6–9. Secondary Market Opportunities for CCAR Technology
• Table 6–10. Base Case Cash Flows from Improved Quality, Yield, and Production Rates and from Reduced Refrigeration Costs from Application of the CCAR Technology for Food Processing
• Table 6–11. Base Case Net Present Value, Internal Rate of Return, and Benefit Cost Ratio (Calculated from the Cash Flows in Table 6–10) for the CCAR Technology
• Table 6–12. Additional Qualitative Benefits from Using the CCAR Technology
• Table 6–13. ATP’s Role
• Table 6–14. Status Report Data
• Table 7–1. Eight of Ten Studies Using Econometric/Statistical Methods Represented*
• Table 8–1. Ten Studies Using Expert Judgment, Bibliometrics, and Emerging Methods, including Cost Index, Social Network Analysis/Fuzzy Logic, and Composite Performance Rating System
• Table 8–2. Data Sample Collected from Expert Physicians and Used to Estimate the Bass Technology Diffusion Model
• Table 8–3. Dissemination of Non-Proprietary Information from ATP-Funded Projects
• Table 8–4. Structured Interview Guide for Collecting Data Needed to Exercise the Model
• Table 9–1. Studies Informing the Impacts of ATP on Private Firms
• Table 9–2. Illustrative Metrics Showing ATP’s Impact on Firms’ Innovation Goals
• Table 9–3. The Extent to Which Non-Winners Pursue the Proposed R&D Project Without ATP Funding
• Table 9–4. Estimates of Economic Value of a One-Year Reduction in Applied Research Cycle Time, in Order of Decreasing Value: $5 Million to $6 Million Median Value
• Table 9–5. ATP Effects that Helped Interviewees to Reduce Cycle Time
• Table 9–6. Carryover of Cycle-Time Improvements to Other Projects
• Table 9–7. Effect of ATP Funding on Completed Projects
• Table 9–8. A Comparison of Small and Larger Firms in ATP
• Table 9–9. Progress of Participating Companies in Commercializing New Technologies
• Table 9–10. Examples of Products and Processes from the First 50 Completed ATP Projects
• Table 9–11. Impact on Industries of Near-Term, Five-Year Implementation Path for Flow-Control Machining Technology
• Table 9–12. Studies Extending Knowledge about Collaboration Activities
• Table 9–13. Summary of Study Findings on Frequency of Collaboration
• Table 9–14. Propensity to Collaborate with Other Organizations and Form New Partnerships
• Table 9–15. Collaborative Activity of the First 50 Completed Projects
• Table 9–16. Membership Changes in the Printed Wiring Board Research Joint Venture, 1992–1996
• Table 9–17. Specific Benefits of Collaborations
• Table 9–18. Studies Informing Spillover Effects
• Table 9–19. Characteristics of R&D Networks Generating Pre-Competitive, Enabling Technologies
• Table 9–20. Spillovers Imputed by Comparing Composite Social Returns, Public Returns, and Composite Private Returns on Seven Tissue Engineering Projects
• Table 9–21. Studies Comparing State and International Programs
• Table 9–22. Comparative Features of ATP and Its Analogues
• Table 9–23. Studies Providing Broader Assessments of Program Performance
• Table 9–24. Measures of Economic Benefit from Six Detailed Case Studies

Date created: February 9, 2004
Last updated: September 7, 2004