Inside IES Research

Just over five years ago the Institute of Education Sciences (IES) and the National Science Foundation (NSF) released the Common Guidelines for Education Research and Development. The Guidelines provided the expected purposes, justifications, and contributions of various types of research aimed at improving our knowledge of interventions and strategies for improving teaching and learning.  Since 2013, there has been increased attention to replication and reproducibility studies and their role in building the evidence base. In response to this interest and the importance of this work, the two organizations jointly issued the new Companion Guidelines on Replication and Reproducibility in Education Research to supplement the Common Guidelines for Education Research and Development. The companion document provides guidance on the steps researchers can take to promote corroboration, ensure the integrity of research, and extend the evidence base.

The Companion Guidelines identify principles to help education stakeholders design and report reproducibility and replication studies. These principles are consistent with and draw from guidelines provided by scientific and professional organizations, advisory committees, and have emerged in consultation with the field (e.g., Dettmer, Taylor, and Chhin, 2017; Subcommittee on Replicability and Science, 2015). The principles address three main areas – (1) replication and reproducibility at the proposal stage, (2) promoting transparency and openness in designing studies, and (3) considerations in the reporting of results. 

Although the importance of reproducibility and replication studies for advancing scientific knowledge has been widely acknowledged, there are several challenges for researchers in our field, including actual or perceived disincentives (e.g., publication bias; reputation and career advancement norms; emphases on novel, potentially transformative lines of inquiry), implementation difficulties (especially for direct replications), and complexities of interpreting results (e.g., lack of consensus on what it means to “replicate” findings, low statistical power for replications). Grant funding agencies such as IES and NSF as well as education researchers have a role to play in addressing these challenges, promoting reproducibility and replication studies, and ultimately moving the field forward.

Why focus on replication and reproducibility?

The original Common Guidelines document did not substantively address issues pertaining to replication and reproducibility of research.  Given the interest in and importance of this work, IES and NSF are providing additional clarity to the field in terms of common definitions and principles around replication and reproducibility.

Who is the audience for the Companion Guidelines on Replication and Reproducibility? 

The primary audience for this document is education researchers; however, education research funding agencies and reviewers of grant applications are additional audiences for this document.

How should this document be used by researchers intending to apply for grants to conduct a reproducibility or replication study?

This document is meant to highlight the importance of replication and reproducibility studies and to offer guidelines to education stakeholders for thinking about and promoting reproducibility and replication in education research. It does not supersede the guidance provided in the requests for applications provided by IES and NSF. 

What are the guiding principles for proposing replication and reproducibility studies?

The overarching principles at the grant proposal stage are as follows:

1. Clarify how reproducibility or replication studies would build on prior studies and contribute to the knowledge base.
2. Clearly specify any variations from prior studies and the rationale for such variations.
3. Ensure objectivity (e.g., by conducting an independent investigation, or by putting safeguards in place if the original investigator(s) is involved).

In addition to these principles, the document also lays out principles for promoting transparency, open science, and reporting results.

Read the full Companion Guidelines here.

November 8^th, 2018 is National STEM Day! Today is a great day to talk to learners of all ages and abilities about Science, Technology, Engineering and Mathematics (STEM). The Institute of Education Sciences (IES) has some great resources for exploring STEM learning - visit our new STEM Topic page to learn more. Through research grants from the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER), and innovations developed as part of the Small Business Innovation Research (SBIR) program, IES has supported the development and testing of many programs, practices, and policies to improve student outcomes in STEM. 

Below, we provide links to a few projects and activities for instructors and learners to explore on National STEM Day, but remember, STEM Day can be every day!

• NumberShire is a mobile and desktop game-based mathematics intervention funded with several grants from NCSER and SBIR that  builds understanding of whole number concepts among early elementary students with or at risk for learning disabilities (video demonstration).
• Improving Children's Understanding of Equivalence (ICUE) supplements teachers’ existing mathematics instruction and helps students develop understanding of mathematical equivalence. Developed with support from NCER, ICUE is currently being evaluated in second grade classrooms. ICUE includes teacher manuals, student workbooks, manipulatives, assessment items, and a 2-hour professional development workshop to provide teachers with information about how to implement the intervention (video demonstration).
• Two innovative education technology products developed with funding from the SBIR program are intended to transform chemistry instruction and learning. Happy Atoms is a physical hand-held magnetic molecular modeling set with a companion digital app that can recognize student created models and provide feedback and information to enrich learning.  HoloLab Champions uses an immersive virtual reality (VR) game environment within which high school students perform chemistry experiments.
• Combined Cognitive and Motivational Supports for STEM Learning is a supplemental Blackboard module for postsecondary introductory biology courses developed with support from NCER. This module leverages short cognitive and motivational interventions that show promise for engaging students and improving outcomes, and is available from IDEALS.

Christina Chhin is the program officer for the Science, Technology, Engineering, and Mathematics (STEM) Education research topic within the National Center for Education Research, Sarah Braisel is the program officer for the Science, Technology, Engineering, and Mathematics (STEM) Education research topic within the National Center for Special Education Research, and Ed Metz is the program officer for both our Small Business Innovation Research program as well as our Education Technology research topic within the National Center for Education Research.

The Registry of Efficacy and Effectiveness Studies (REES) is now ready for use! REES is a registry of causal impact studies in education developed with a grant from the National Center for Education Research (NCER). REES will increase transparency, improve the replicability of studies, and provide easy access to information about completed and ongoing studies.

The release of REES aligns with recent IES efforts to promote study registration. In the FY 2019 Requests for Applications (RFAs) for the Education Research and Special Education Research Grants Programs, IES recommended that applicants describe a plan for pre-registering their studies in both the project narrative (as part of the research) as well as the data management plan. IES is also developing the Standards for Excellence in Education Research (SEER) and has identified study registration as an important dimension of high value education research.

We asked the REES team to tell us more about how the registry works.

How can researchers access REES? REES can be accessed through the SREE website at www.sreereg.org. Over the next year, REES will transition to a permanent home at the Inter-university Consortium for Political and Social Research (ICPSR) at the University of Michigan, but it will still be accessible through the SREE link.

What kinds of studies can be registered? REES is a registry of causal impact studies. It accommodates a range of study designs including randomized controlled trials, quasi-experimental designs, regression discontinuity designs, and single case designs.

What information should be included in a study entry? A REES entry includes basic study information and a pre-analysis plan. The checklist of required information for a registry entry provides detailed information for each of the different design options. All of the information for a REES entry should be easily found in a grant application.

How long does it take to register a study? For a study with a complete grant application, completing a REES entry should be straightforward and take approximately one hour.

What if a study entry needs to be changed? Principal investigators (PI) or other authorized research team members should update a REES entry as changes occur. All updates to an entry will be time-stamped. Original entries and updated entries will be publically available.

Are registered studies searchable by the public? Yes! When a PI or authorized research team member is ready to make the study available in the public domain, they click on the publish option. This will time stamp the entry and make it publically available. REES entries that are published are available on the search page. A pdf of individual entries can be downloaded from the search page or an Excel file of multiple entries can be exported.

What will happen to studies that were entered in the pilot phase of REES? A REES entry that was started and/or completed during the pilot phase is a part of the REES database. To make the study publically available and a part of the searchable database, the PI or other authorized research team member needs to click on the publish option for the entry.

Over the next two years, the REES team will be working to ensure the sustainability and visibility of REES with a grant from the National Center for Special Education Research (NCSER). To do this, the team will transfer REES to its permanent location on the ICPSR website and disseminate information about REES within the education research community, as well as with funders, publishers, and users of education research, through meetings, conferences, websites, social media, and targeted outreach.

So, what are you waiting for? Go check it out!

If you have questions about REES, please email contact@sreereg.org.

Career and Technical Education (CTE) has been evolving and expanding at a rapid pace in recent years as industry and education leaders focus on students’ readiness for college and careers. While some studies have shown positive effects of CTE on students, the evidence base is thin. To learn more about the research needs of the CTE field, the National Center for Special Education Research (NCSER) and the National Center for Education Research (NCER) at the Institute of Education Sciences (IES) convened a group of experts in policy, practice and research related to CTE.  The discussion held by the Technical Working Group (TWG) led both NCER and NCSER to increase their investments in CTE for fiscal year 2019: NCSER included a CTE special topic, and NCER changed its CTE topic from a special topic to a standing topic. Applications to both are due August 23, 2018.  Both research centers hope to fund more studies that will help us better understand this growing aspect of education.

The TWG focused on the following four questions:

1. Who is served by CTE and who is left behind? From national CTE statistics, we know that 82% of all public high schools offer CTE. And, 85% of students earn at least one credit in CTE with the average high school student earning 2.5 CTE credits. However, TWG members noted that research is lacking on specific subpopulations in CTE, such as students from various demographic backgrounds and students with disabilities. Disaggregated data on these dimensions are needed to better understand the CTE experiences of the range of students being served. Such data may help educators improve equity of access to high quality programs for all students.
2. What do we know―and need to know―about CTE policies, programs, and practices at the secondary and postsecondary levels?  TWG experts discussed the need to know more about industry-recognized credentials and about business and industry engagement in CTE at the secondary level. They argued that we do not know if credentials align with industry requirements, nor do we understand the impact of different types of credentials on student outcomes and wage trajectories. TWG members also noted that the higher the perceived quality or prestige of the CTE program, the more exclusive it becomes, and the more difficult it is for disadvantaged students to obtain access. TWG members also expressed concerns about CTE teacher training, particularly for experts who are recruited from industry without prior teacher preparation. As the experts discussed postsecondary CTE, they suggested that the field would be best served by framing the conversation about secondary to postsecondary pathways as a continuum that enables transparent and sequential transitions from secondary to 2-year and then to 4-year programs or to training or employment, with guidance for students to understand possible sequences.
3. What are the critical methodological issues in CTE?  TWG members noted that, with a few notable exceptions (e.g.,  a 2008  MDRC study on career academies in New York and a recent study of CTE high schools in Massachusetts), few causal studies on CTE have been conducted. There is an urgent need for more high quality, causal research on CTE policies and programs. In addition, the experts noted that there is almost no research on students with disabilities in CTE. TWG members concluded that the field needs to re-conceptualize CTE research – including better defining CTE students, instructors, programs, and measures – and identify the critical research questions in order to encourage more research in this field.
4. What is needed to advance CTE research?  State CTE administrators want to know how to identify quality CTE programs so they know how to spend their dollars most effectively on programs that best meet the needs of students. Policymakers also want to know what “works” and what the benefits are of such investments. The TWG members encouraged studies that examine the educational benefits of particular instructional approaches. They also highlighted the importance of collaborative cross-institutional and cross-agency efforts to advance CTE research.

Readers are invited to read the summary of the TWG discussion.

By Corinne Alfeld (NCER program officer) and Kimberley Sprague (former NCSER program officer)

The IES Goal Structure was created to support a continuum of education research that divides the research process into stages for both theoretical and practical purposes. Individually, the five goals – Exploration (Goal 1), Development and Innovation (Goal 2), Efficacy and Replication (Goal 3), Effectiveness (Goal 4), and Measurement (Goal 5) – were intended to help focus the work of researchers, while collectively they were intended to cover the range of activities needed to build evidence-based solutions to the most pressing education problems in our nation. Implicit in the goal structure is the idea that over time, researchers will identify possible strategies to improve student outcomes (Goal 1), develop and pilot-test interventions (Goal 2), and evaluate the effects of interventions with increasing rigor (Goals 3 and 4).

Over the years, IES has received many applications and funded a large number of projects under Goals 1-3.  In contrast, IES has received relatively few applications and awarded only a small number of grants under Goal 4. To find out why – and to see if there were steps IES could take to move more intervention studies through the evaluation pipeline – IES hosted a Technical Working Group (TWG) meeting in 2016 to hear views from experts on what should come after an efficacy study (see the relevant summary and blog post). IES also issued a request for public comment on this question in July 2017 (see summary).

The feedback we received was wide-ranging, but there was general agreement that IES could do more to encourage high-quality replications of interventions that show prior evidence of efficacy. One recommendation was to place more emphasis on understanding “what works for whom” under various conditions.  Another comment was that IES could provide support for a continuum of replication studies.  In particular, some commenters felt that the requirements in Goal 4 to use an independent evaluator and to carry out an evaluation under routine conditions may not be practical or feasible in all cases, and may discourage some researchers from going beyond Goal 3.   

In response to this feedback, IES revised its FY 2019 RFAs for Education Research Grants (84.305A) and Special Education Research Grants (84.324A) to make clear its interest in building more and better evidence on the efficacy and effectiveness of interventions. Among the major changes are the following:

• Starting in FY 2019, Goal 3 will continue to support initial efficacy evaluations of interventions that have not been rigorously tested before, in addition to follow-up and retrospective studies.
• Goal 4 will now support all replication studies of interventions that show prior evidence of efficacy, including but not limited to effectiveness studies.
• The maximum amount of funding that may be requested under Goal 4 is higher to support more in-depth work on implementation and analysis of factors that moderate or mediate program effects.

The table below summarizes the major changes. We strongly encourage potential applicants to carefully read the RFAs (Education Research, 84.305A and Special Education Research, 84.324A) for more details and guidance, and to contact the relevant program officers with questions (contact information is in the RFA).

Applications are due August 23, 2018 by 4:30:00 pm Washington DC time.

By Thomas Brock (NCER Commissioner) and Joan McLaughlin (NCSER Commissioner)