Replicability, Generalizability, and the ManyClasses Approach to Open Science

Event Date & Time

  • June 21, 2023
    11:00 am - 12:00 pm

Event Description

As in many fields, applied research in education is facing serious questions about the replicability of its findings — and further, some ponder whether findings should be expected to replicate in the first place. In education, a particularly vexing challenge is the variability of learners and learning settings, and it is a legitimate question whether any one finding might be expected to generalize across classes. To answer this question, the obvious solution is to conduct research that spans and incorporates this variability. Rather than embedding an experiment in one class, one discipline, or one domain, researchers should distribute an experiment across many such situations, customizing treatments and outcome measures to authentic local contexts, so that they are representative of a range of materials and implementations. This talk will present ManyClasses, a research model that achieves this need, and it will also describe how the principles of open science are fundamental to the ManyClasses approach. The presenter’s work, which aims to test the generalizability of practical interventions, relies heavily on transparency both in our methods and in our findings.

Please note that this session is being offered as part of POSE, the Program for Open Scholarship and Education, but is open to all.


Dr. Ben Motz is Assistant Professor at Indiana University’s Department of Psychological and Brain Sciences. He received his BS from Indiana University, MS from University of California San Diego, and his PhD at Indiana University, all in Cognitive Science. His research is at the intersection of cognitive psychology, technology, and education, characterized by large-scale experiments and analyses on students from real education settings that test theoretical predictions from the psychological science of learning. He also develops research infrastructure for this work: he currently runs Terracotta (, a platform that plugs into the Canvas LMS, and that makes it possible to run controlled experiments in Canvas course sites, and that aims to facilitate open practices by exporting deidentified data in a standard format. He is also the Associate Director of the eLearning Research and Practice Lab, a unit in IU’s information technology division that enables responsible use of institutional student data for research purposes.

About POSE

The Program for Open Scholarship and Education is a four-month flexible and blended program jointly presented by the UBC Open Working Group, the Centre for Teaching, Learning and Technology and UBC Library. The program will help educators gain foundational knowledge of open scholarship, open education, and open research.  Register for POSE here.

Please note that this session is being offered as part of POSE, but is open to all.