Co-Principal Investigator: Ritter, Steve; Fancsali, Stephen E.

Purpose: In this project, the research team will iteratively develop and study MARACAS (Math and Reading Acquisition Co-Adaptive System), a set of individualized reading supports for students embedded within an adaptive mathematics learning system (MATHia) and associated teacher application (LiveLab). Teachers face significant barriers in supporting the needs of middle school students with reading difficulties (RD) when learning to solve mathematics problems. To address this need, MARACAS will be developed to estimate the extent to which students' reading challenges are impacting their math learning based on ongoing, platform-embedded measures of student behaviors and interactions. Heuristics will determine when reading supports or scaffolds should be provided or recommended to students. In addition, adaptive supports for teachers will alert them when students are likely exhibiting reading challenges and provide recommendations for intervention. The goal of this project is to develop and evaluate reading supports that can be embedded into a variety of digital and/or adaptive math tools to decrease reading challenges and thus increase students' ability to engage effectively with math.

Project Activities: Investigators will iteratively co-design and pilot test adaptive supports for struggling readers embedded with Carnegie Learning's MATHia student software and for teachers within the LiveLab teacher app for real-time classroom monitoring. In Years 1–2 of the project, investigators will work with teachers and middle school students from three suburban and urban public school districts. The research team will conduct a pilot study in Year 3 using a cluster randomized controlled trial with teachers and middle school students from five suburban and urban districts. Teachers will be randomly assigned to either use MATHia and LiveLab with MARACAS supports or without these developed supports, blocked by school-district and school. The final sixth months of the project are devoted to analyzing data from the pilot study and dissemination activities.

Products: This project will result in fully developed MARACAS reading support features embedded in Carnegie Learning's commercially available adaptive mathematics student software MATHia as well as the LiveLab teacher application and related professional development materials to decrease reading challenges. The project will result in peer-reviewed publications and presentations as well as additional dissemination products that reach education stakeholders, such as practitioners and policymakers.  To support use of project findings, generated technical resources, such as design assets and heuristics, will be Creative Commons licensed and made available through GitHub.

Structured Abstract

Setting: The research will take place in middle schools in six public school districts across urban and suburban settings in California, Florida, Massachusetts, and Pennsylvania.

Sample: During Years 1–2, the research team will work with middle school students (grades 6–8) with reading difficulties (RD) and their math teachers in inclusive classrooms. The pilot study in Year 3 will include approximately 10,000 students from 79 middle schools across five school districts, largely representative of diverse urban and suburban districts across the country. Students with reading difficulties will be defined as students who score below the 25th percentile on their state's English Language Arts Exam in the 2022–23 school year.

Intervention/Factors/Assessment: Phase 1 will develop a fully functional MARACAS prototype, a version of MATHia with five real-time supports for students: highlighting conceptual relationships, contextualizing/decontextualizing problems, adjusting text readability, advanced read aloud and synchronized highlighting, and personalized interest areas and names. The project will also include the iterative development of a version of LiveLab with alerts for teachers and a professional learning program.

Research Design and Methods:  In Phase 1 (Year 1–2), MARACAS will be developed through an iterative, design-based implementation research process with students and teachers. Minimally viable components will be rapidly prototyped and evaluated through continuous cycles of design, implementation, analysis, and redesign. Observations, surveys, focus groups, informal interviews, usage log data, and cognitive labs will be used to evaluate the clarity, accessibility, use, value, and promise of MARACAS. During the Phase 2 (Year 3) pilot study, a cluster randomized control trial, with assignment at the teacher level (blocked by district and school), will be used to determine the extent to which MARACAS increases mathematics learning gains of students with RD compared to the control condition and whether such gains are mediated by MATHia knowledge components that are related to reading comprehension.

Control Condition: Students with RD will use a "business-as-usual" version of MATHia, without MARACAS components. Their teachers will use a version of LiveLab without MARACAS components and will receive professional learning without additional MARACAS Materials.

Key Measures: Student outcome measures include each state's standardized end-of-year exams for mathematics, students' mathematics course pass rates, and the rate at which students complete MATHia mathematics content. Data collected from observations, surveys, focus groups, informal interviews, cognitive labs, and usage data from MATHia and LiveLab will be used to understand use, usability, acceptance, the promise of supports, and any areas where those supports can be improved.

Data Analytic Strategy: Researchers will use multilevel modeling to test for the main treatment effect of MARACAS and moderator effects of various teacher and student characteristics controlling for covariates. The research team will also analyze individual- and group-level mediation effects. They will conduct an instrumental variable analysis to evaluate the relationship between implementation fidelity and student outcomes.

Cost Analysis:  The proposed project includes a plan for a cost analysis based on data collected during the Phase 2 pilot using the ingredients method to understand the costs associated with implementing MARACAS beyond what schools typically spend.  Cost analysis will consider both national and local costs as well as total, average, and incremental costs. Researchers will also explore the cost-effectiveness of MATHia and LiveLab with MARACAS as compared to MATHia and LiveLab implemented without MARACAS using cost-effectiveness ratios.

Related IES Projects: Enhancing Student Learning with an Orchestration Tool for Personalized Teacher-Student Interactions in Classrooms Using Intelligent Tutoring Software (R305A180301)