What Does It Mean Today to Be Evidence-Based in Teaching? Part 1

Most modern “data-driven” teaching tools sideline the very people who matter most: teachers. The promise often sounds like this: “Use our app, and we’ll algorithmically feed your students what they need.”

But here’s the problem: Most students don’t learn difficult things because an app gave them something to do. In several impressive studies of highly used math programs (Khan Academy, DreamBox, i-Ready), there are huge effect sizes for those "students who used the program as recommended." The problem was that only about 5% use the program as recommended. It's impressive both how effective ed tech is and how little it is used by the students who need it most (Holt, 2024). 

So how do the other 95% learn? Our hypothesis is that they learn because of teachers - teachers who notice what matters, connect ideas to students’ lives, and keep curiosity alive when things get hard. Teachers create a classroom environment to engage those that are the least engaged. 

At CourseKata, we believe being evidence-based means giving teachers the right kinds of tools - not replacing those teachers. Our goal is to help teachers measure and improve what matters most: deep learning, transfer, and curiosity.

The Performance Assessment: Measuring What Really Matters

Most tests focus on whether students can recall definitions or crank through calculations. But in the real world (e.g., on the job, in research projects, or in everyday decision-making) what matters is whether students can use data to answer real questions.

That’s why, together with teachers, we developed a performance assessment that takes less than an hour and can be given as a unit test (after chapter 9 of CourseKata's ABC textbook).

Instead of multiple choice, students:

• Explore variation with data visualization
• Build & evaluate models
• Communicate what their results mean

Why does creative thinking about new methods of assessments matter, especially in the age of AI? Because assessments send powerful signals to students about what’s valued in a class. If the test is just about memorizing formulas and delivering the answer, they’ll think that’s all statistics is. If the test requires curiosity, interpretation, and transfer, they’ll understand that those are the real goals.

Here’s what our performance assessment is designed to communicate:

• Transfer: Applying learning to messy, unfamiliar contexts.
• Authenticity: Engaging in tasks that look like the ones they'll face in jobs or during research.
• Reasoning over recall: Making choices, interpretinging outputs, and justifyinging conclusions, not just plug in formulas.

It’s not about getting the right answer. It’s about making sense of data, making decisions about what they should do next, and making meaning - the very skills that stick long after the test is over.

Want to See the Assessment?

This performance assessment is just one example of how CourseKata is rethinking what it means to be evidence-based in teaching.

We’re happy to share sample tasks and scoring guides with anyone curious about bringing these ideas into their own classrooms. Interested? Reach out to claudia@coursekata.org for more information.

Stay tuned! This is the first in a short series on how CourseKata supports teachers with:

• Performance assessments
• Overview notebooks for enhancing in-class learning
• CKHub that make professional data science tools great for teaching

Reference

Holt, L. (2024). The 5 percent problem. Education Next.