Inside IES Research

In recent months, several federal reports and resources related to English learner (EL) learning and education related to science, technology, engineering, and mathematics (STEM) have been released.

First, the Office of English Language Acquisition (OELA) released its third “data story” about ELs in US schools. This story, which builds on two previously released stories about the characteristics and educational experiences of ELs, focuses specifically on ELs’ NAEP performance and high school graduation rates. Through interactive infographics (many of which are built on data from the National Center for Education Statistics), the story shows that higher percentages of ELs are proficient in math than in reading, but that nearly half of all states experienced declines in the number of ELs who scored proficient in math between 2009 and 2017. The story also shows that graduation rates for ELs improved by 10 percentage points between 2010-11 and 2015-16 (from 57 percent to 67 percent), but still fall well below the rates for non-ELs (84 percent). While interesting and informative, the data story also underscores the necessity of research and development to produce better resources and information to support EL learning.

In that vein, the National Academies of Sciences, Engineering, and Medicine released English Learners in STEM Subjects: Transforming Classrooms, Schools, and Lives. This report examines what we know about ELs’ learning, teaching, and assessment in STEM subjects and provides guidance on how to improve STEM learning outcomes for these students. It reflects the consensus of a committee of EL experts that was chaired by NCER and NCSER grantee Dr. David Francis and included past grantees Dr. Okhee Lee and Dr. Mary Schleppegrell alongside a dozen other experts in EL education, STEM education, and teaching. One of the report’s central conclusions is that ELs develop proficiency in both STEM subjects and language when their classroom teachers provide them with opportunities for meaningful interaction and actively support both content and language learning. Given that many STEM teachers do not receive preparation to teach in this way, the report provides several recommendations to improve pre-service and in-service training. It also includes recommendations for how developers and publishers might produce better instructional materials and assessments to help both teachers and EL students. 

Efforts of both types – instructional preparation and development of new materials – may be further supported by two new toolkits released by the Office of Education Technology. The toolkits are designed for educators and developers, and each is organized around five specific guiding principles to help the targeted group approach education technology with ELs’ unique needs in mind. The principles for developers emphasize the importance of thinking ahead about EL needs for those who wish to make products for this population. Meanwhile, the educator principles center on issues of awareness, and encourage teachers to learn more about the features, platforms, and resources that are available for ELs in the world of education technology. The principles also complement one another – for example, developers are encouraged to offer instruction-focused professional development, and educators are encouraged to seek out the same.

Brought together, these resources provide a snapshot of ELs’ mathematics achievement, a summary of research evidence about learning and instruction for ELs in STEM, and a set of principles to guide instruction and development efforts in the technology space moving forward. They also make a clear case for continued investment in R&D efforts to support STEM learning for both EL students and their teachers. Since 2010, the National Center for Education Research has invested nearly $20 million across 13 research and researcher-practioner partnership grants that have focused on STEM learning and ELs. Several such grants are coming to a close in the 2019 fiscal year; watch this space for future blog posts about the products and findings from these projects.

Just over five years ago the Institute of Education Sciences (IES) and the National Science Foundation (NSF) released the Common Guidelines for Education Research and Development. The Guidelines provided the expected purposes, justifications, and contributions of various types of research aimed at improving our knowledge of interventions and strategies for improving teaching and learning.  Since 2013, there has been increased attention to replication and reproducibility studies and their role in building the evidence base. In response to this interest and the importance of this work, the two organizations jointly issued the new Companion Guidelines on Replication and Reproducibility in Education Research to supplement the Common Guidelines for Education Research and Development. The companion document provides guidance on the steps researchers can take to promote corroboration, ensure the integrity of research, and extend the evidence base.

The Companion Guidelines identify principles to help education stakeholders design and report reproducibility and replication studies. These principles are consistent with and draw from guidelines provided by scientific and professional organizations, advisory committees, and have emerged in consultation with the field (e.g., Dettmer, Taylor, and Chhin, 2017; Subcommittee on Replicability and Science, 2015). The principles address three main areas – (1) replication and reproducibility at the proposal stage, (2) promoting transparency and openness in designing studies, and (3) considerations in the reporting of results. 

Although the importance of reproducibility and replication studies for advancing scientific knowledge has been widely acknowledged, there are several challenges for researchers in our field, including actual or perceived disincentives (e.g., publication bias; reputation and career advancement norms; emphases on novel, potentially transformative lines of inquiry), implementation difficulties (especially for direct replications), and complexities of interpreting results (e.g., lack of consensus on what it means to “replicate” findings, low statistical power for replications). Grant funding agencies such as IES and NSF as well as education researchers have a role to play in addressing these challenges, promoting reproducibility and replication studies, and ultimately moving the field forward.

Why focus on replication and reproducibility?

The original Common Guidelines document did not substantively address issues pertaining to replication and reproducibility of research.  Given the interest in and importance of this work, IES and NSF are providing additional clarity to the field in terms of common definitions and principles around replication and reproducibility.

Who is the audience for the Companion Guidelines on Replication and Reproducibility? 

The primary audience for this document is education researchers; however, education research funding agencies and reviewers of grant applications are additional audiences for this document.

How should this document be used by researchers intending to apply for grants to conduct a reproducibility or replication study?

This document is meant to highlight the importance of replication and reproducibility studies and to offer guidelines to education stakeholders for thinking about and promoting reproducibility and replication in education research. It does not supersede the guidance provided in the requests for applications provided by IES and NSF. 

What are the guiding principles for proposing replication and reproducibility studies?

The overarching principles at the grant proposal stage are as follows:

1. Clarify how reproducibility or replication studies would build on prior studies and contribute to the knowledge base.
2. Clearly specify any variations from prior studies and the rationale for such variations.
3. Ensure objectivity (e.g., by conducting an independent investigation, or by putting safeguards in place if the original investigator(s) is involved).

In addition to these principles, the document also lays out principles for promoting transparency, open science, and reporting results.

Read the full Companion Guidelines here.

The IES Goal Structure was created to support a continuum of education research that divides the research process into stages for both theoretical and practical purposes. Individually, the five goals – Exploration (Goal 1), Development and Innovation (Goal 2), Efficacy and Replication (Goal 3), Effectiveness (Goal 4), and Measurement (Goal 5) – were intended to help focus the work of researchers, while collectively they were intended to cover the range of activities needed to build evidence-based solutions to the most pressing education problems in our nation. Implicit in the goal structure is the idea that over time, researchers will identify possible strategies to improve student outcomes (Goal 1), develop and pilot-test interventions (Goal 2), and evaluate the effects of interventions with increasing rigor (Goals 3 and 4).

Over the years, IES has received many applications and funded a large number of projects under Goals 1-3.  In contrast, IES has received relatively few applications and awarded only a small number of grants under Goal 4. To find out why – and to see if there were steps IES could take to move more intervention studies through the evaluation pipeline – IES hosted a Technical Working Group (TWG) meeting in 2016 to hear views from experts on what should come after an efficacy study (see the relevant summary and blog post). IES also issued a request for public comment on this question in July 2017 (see summary).

The feedback we received was wide-ranging, but there was general agreement that IES could do more to encourage high-quality replications of interventions that show prior evidence of efficacy. One recommendation was to place more emphasis on understanding “what works for whom” under various conditions.  Another comment was that IES could provide support for a continuum of replication studies.  In particular, some commenters felt that the requirements in Goal 4 to use an independent evaluator and to carry out an evaluation under routine conditions may not be practical or feasible in all cases, and may discourage some researchers from going beyond Goal 3.   

In response to this feedback, IES revised its FY 2019 RFAs for Education Research Grants (84.305A) and Special Education Research Grants (84.324A) to make clear its interest in building more and better evidence on the efficacy and effectiveness of interventions. Among the major changes are the following:

• Starting in FY 2019, Goal 3 will continue to support initial efficacy evaluations of interventions that have not been rigorously tested before, in addition to follow-up and retrospective studies.
• Goal 4 will now support all replication studies of interventions that show prior evidence of efficacy, including but not limited to effectiveness studies.
• The maximum amount of funding that may be requested under Goal 4 is higher to support more in-depth work on implementation and analysis of factors that moderate or mediate program effects.

The table below summarizes the major changes. We strongly encourage potential applicants to carefully read the RFAs (Education Research, 84.305A and Special Education Research, 84.324A) for more details and guidance, and to contact the relevant program officers with questions (contact information is in the RFA).

Applications are due August 23, 2018 by 4:30:00 pm Washington DC time.

By Thomas Brock (NCER Commissioner) and Joan McLaughlin (NCSER Commissioner)

Welcome to Career and Technical Education (CTE) month!

Over the course of the next few weeks, we will be blogging about IES projects and resources relevant to CTE. We will be highlighting grant competitions, including our newest competition, Expanding the Evidence Base for Career and Technical Education, which aims to increase research on the impact of CTE programs and policies on student outcomes and support training of new CTE researchers. And we will showcase work conducted by IES and our grantees.

For this first blog, we wanted to share our working definition of CTE, along with links to information and resources.

CTE aims to help students enter into and succeed in specific occupational fields such as health science, information technology, and business administration. Students in secondary, postsecondary, and adult education may pursue CTE activities as part of their other education requirements (e.g., courses for high school graduation, classes to prepare for General Equivalency Development or General Equivalency Diploma (GED) test) or as a program to earn an occupational certificate at the subbaccalaureate level.

Over the past decade, interest has been growing in CTE and career pathway models across public and private arenas. For example, at the federal level, interest in CTE is reflected in the legislations that authorize these education and training activities, namely the Carl D. Perkins Career and Technical Education Act of 2006  and the Workforce Innovation and Opportunity Act of 2014.

At IES, our goal is to identify the needs of CTE students and expand our understanding of effective CTE practices. For example, the National Center for Education Statistics (NCES) assists in monitoring the status of CTE by providing national information on student participation in CTE at the secondary and postsecondary levels and on adults’ preparation for work. This information is available on the CTE Statistics website.

The National Center for Education Evaluation and Regional Assistance (NCEE) also supports work on CTE. For example, NCEE’s Regional Education Laboratory Programs work with policymakers and practitioners on career and college readiness issues, including CTE. Some regions have groups that focus on CTE directly, such as REL Appalachia's West Virginia Workforce Readiness Partnership and REL Mid-Atlantic's Readiness for Career Entry and Success Research Alliance.

The two grant-awarding centers, the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER) support field-initiated research in CTE primarily through the Career and Technical Education, Postsecondary and Adult Education, and Transition Outcomes for Secondary Students with Disabilities topic areas. In September 2017, NCER and NCSER sponsored a technical working group meeting to gain insights from CTE practitioners and researchers. On January 22, NCER released the request for applications for the new research network mentioned above, Expanding the Evidence Base for Career and Technical Education.

We look forward to sharing more information about our CTE research and statistics. Come back throughout the month to hear from IES staff and grantees about this work!

By Meredith Larson (NCER)

The American Educational Research Association (AERA) will hold its annual meeting April 8 through April 12 in Washington, D.C.—the largest educational research gathering in the nation. This will be a special meeting for AERA, as it is celebrating 100 years of advocating for the development and use of research in education. The program includes hundreds of sessions, including opportunities to learn about cutting edge education research and opportunities to broaden and deepen the field. 

About 30 sessions will feature staff from the Institute of Education Sciences (IES) discussing IES-funded research, evaluation, and statistics, as well as training and funding opportunities.

On Saturday, April 9, at 10:35 a.m., attendees will have a chance to meet the Institute’s leadership and hear about the areas of work that IES will be focusing on in the coming year. Speakers include Ruth Curran Neild, IES’ delegated director, and the leaders of the four centers in IES: Thomas Brock, commissioner of the National Center for Education Research (NCER); Peggy Carr, acting commissioner of the National Center for Educational Statistics (NCES); Joy Lesnick, acting commissioner of the National Center for Education Evaluation and Regional Assistance (NCEE), and Joan McLaughlin, commissioner of the National Center for Special Education Research (NCSER).

On Monday, April 11, at 9:45 a.m., attendees can speak to one of several IES staffers who will be available at the Research Funding Opportunities—Meet Your Program Officers session. Program officers from NCER, NCSER, and NCEE will be on hand to answer questions about programs and grant funding opportunities. Several IES representatives will also be on hand Monday afternoon, at 4:15 p.m. for the Federally Funded Data Resources: Opportunities for Research session to discuss the myriad datasets and resources that are available to researchers.

NCES staff will lead sessions and present on a variety of topics, from The Role of School Finance in the Pursuit of Equity (Saturday, 12:25 p.m.) to Understanding Federal Education Policies and Data about English Learners (Sunday, April 10, 8:15 a.m.) and what we can learn from the results of PIAAC, a survey of adult skills (also Sunday, 8:15 a.m.). Dr. Carr will be a part of several sessions, including one on Sunday morning (10:35 a.m.) about future directions for NCES longitudinal studies and another on Monday morning (10 a.m.) entitled Issues and Challenges in the Fair and Valid Assessment of Diverse Populations in the 21^st Century. 

On Monday, at 11:45 a.m., you can also learn about an IES-supported tool, called RCT-YES, that is designed to reduce barriers to rigorous impact studies by simplifying estimation and reporting of study results (Dr. Lesnick will be among those presenting). And a team from the IES research centers (NCER/NCSER) will present Sunday morning (10:35 a.m.) on communication strategies for disseminating education research (which includes this blog!).

IES staff will also participate in a number of other roundtables and poster sessions. For instance, on Tuesday, April 12, at 8:15 a.m., grab a cup of coffee and attend the structured poster session with the Institute’s 10 Regional Educational Laboratories (RELs). This session will focus on building partnerships to improve data use in education.  REL work will also be featured at several other AERA sessions.  

Did you know that the National Library of Education (NLE) is a component of IES? On Friday and Monday afternoon, attendees will have a unique opportunity to go on a site visit to the library. You’ll learn about the library’s current and historical resources – including its collection of more than 20,000 textbooks dating from the mid-19^th century. The Library offers information, statistical, and referral services to the Department of Education and other government agencies and institutions, and to the public.

If you are going to AERA, follow us on Twitter to learn more about our sessions and our work.  And if you are tweeting during one of our sessions, please include @IESResearch in your tweet. 

By Dana Tofig, Communications Director, IES