Khan Academy Classic v. Khanmigo's Version
An essential thought exercise about math tutoring and generative AI.
Sean Geraghty and Mike Goldstein will make you much smarter about tutoring and generative AI with their recent article for Fordham. Right now you can throw a rock at LinkedIn and hit six different people with “generative AI consultant” in their bio making twelve different gauzy and maximalist predictions of impending transformation to teaching and learning. It is refreshing, then, to see Geraghty and Goldstein make a concrete prediction about GAI and justify it, suggesting the impact of GAI will depend to a large degree on a student’s motivation.
It’s a tool for two very different situations. Motivated learners will increasingly substitute AI for human tutors and teachers. Unmotivated learners will rarely (of their own accord) do this.
They anchor their prediction in a 2018 study of Khan Academy Classic—just video explanations and autograded exercises—in which:
11% of the study population used Khan Academy in the recommended dosage of 30 minutes per week.
The study population saw an effect size difference of .20 when compared to students outside the study population.1
Geraghty and Goldstein then ask the question:
What do you think the results of the study would look like today if we re-ran it using Khan Academy (Khanmigo’s Version)?
They describe the landscape of tutoring interventions and the three kinds of motivation you’ll find there. They then bet that Khanmigo will increase the population taking the recommended dosage by roughly 100% and the effect size of the intervention by 50%.
Three thoughts about their predictions:
One, it is so weird to see anyone making any concrete predictions about generative AI in education like this. I took my own shot months ago, but it simply isn’t done. Please place your bets, say what you think will happen on a five-year timeline and why, or make room at the table for people who will.
Two, my guess is Geraghty and Goldstein are right about a 50% increase in effect size. Khanmigo has been engineered about as well as I could hope to seek to understand a student’s thinking, and especially to seek to understand their wrong answers. Where Khan Academy Classic encounters wrong answers and says, “Want to see how I would do it?” Khanmigo tells students “I see where you’re coming from” and tries to explain from that vantage point. Those are significant differences.
Three, their prediction of a 100% increase in the number of students enticed into the recommended dosage is way too high IMO. I will take the under.
Generative AI is often described as something like an “infinitely patient” on-demand tutor. But “infinitely patient” and “on-demand” are not virtues that describe great teachers.
Great teachers are often quite impatient. They do not wait for the demand for their teaching to arise naturally in a student. They see it as their job to create demand, oftentimes quite impatiently. They step into a student’s existing understanding of the world, an understanding that the student is often quite happy with and does not wish to disrupt, and they disrupt it, seeking to make the student unhappy with the limits of their existing knowledge and eager to work collaboratively to extend those limits.
Many students—11% of them sounds about right—will do well with an infinitely patient on-demand tutor. The other 89% of students need an impatient demand generator.
Odds & Ends
This is what I’m talking about. Jenna Laib took a bunch of students who were quite comfortable in their understanding that 0 is not a number and proceeded to make them uncomfortable in that understanding.
Matthew Rascoff, the vice provost of digital education at Stanford University, gave an interesting talk in Hong Kong called Empowering Students with AI and referenced some of my recent posting activity. He affirms much of it, but offers the caution that much of what I’m describing in my critiques of personalized learning depend on “heroic individual teachers” like Liz Clark-Garvey. This is a concern that worries me too and I’m hoping to find some time to write about how we can ensure that teachers like Liz are made, rather than born.
My latest episode of Math Teacher Lounge includes an episode with classroom teacher Lauren Carr, who describes how she develops math fluency and attends to some pretty lofty social and interpersonal ideals simultaneously. Check it out.
The Atlantic has a special edition on AI in society. Testimony from students and teachers begins on page 82 and it was all I read. High school senior Jedidah Worrell writes, “The big promise of AI is that it’ll make learning more efficient, but we shouldn’t be striving for efficiency. We should be thinking about how to make students curious and driven instead. A lot of my classmates don’t feel motivated or encouraged to learn and explore in their education lately ….”
“Is My Toddler a Stochastic Parrot?” by Angie Wang is an illustrated meditation on generative AI, parenting, and the nature of humanity, that had me completely gutted from start to finish.
To my knowledge, Sal Khan hasn’t made any concrete predictions for education in the age of Khanmigo, but the Harvard Crimson summarizes an event in which he predicts the future of … the motion picture industry? Khan said that “in the next five years, you, the screenplayer, are going to be in a position where you can produce a whole movie for maybe a budget of $10,000 as opposed to $100 million.”
There isn’t a lot in the policy landscape that excites me as much as automatic enrollment triggers for advanced and honors classes. One prominent agenda for equity involves educating teachers across the US, helping them reflect on the biases that lead them to restrict resources from racial minorities, girls, etc. Another agenda—this one—says, “Those resources are too important to leave to the whims of individuals at all. Let’s do policy instead.”
Equatio has an AI test item generator. Related: Imagine Learning surveyed teachers and found that “assessment/test creation” is the most common use case for generative AI (p. 12).
AI might disrupt math and computer science classes — in a good way. Claire Bryan. At what point should these headlines start reading in the present tense rather than the future tense. Related: Where Are the AI Skepticism Stories?
Why I Believed Edtech Could Save My School — and How It Failed Me. Michael Paul Ida.
Warming up prior knowledge. Peps Mccrea.
I’m not sure how to process the possible selection bias here. Even if none exists, this is like inventing a cancer pill that has a good effect on the people in the treatment group who took it but which 89% of the treatment group for some reason decided not to take. Really makes you wonder what’s going on with that cancer pill!
Man, there's a lot to chew on here (as per usual), but I'll confine myself to the one piece I know a little something about: "Much of what I’m describing in my critiques of personalized learning depend on 'heroic individual teachers' like Liz Clark-Garvey. This is a concern that worries me too and I’m hoping to find some time to write about how we can ensure that teachers like Liz are made, rather than born."
I founded an organization dedicated toward this goal (Deans for Impact) that was/is premised on two basic ideas: (1) We will have more great teaching if teachers have a better understanding of the science of how students learn. (2) Initial teacher education is a good place to focus effort on ensuring (1).
Both of those propositions are debatable, though I'm prepared to defend them still. But I am also happy to *complicate* them with some additional contextual claims borne of lived experience:
(3) Understanding the science of learning means a lot more than just "read about basic ideas from cognitive science." The key is in the *application* of that understanding in real classroom settings with kids. Learning science doesn't tell teachers what to do, but rather, gives them a "mental model" that helps inform the many thousands of pedagogical decisions that they'll have to make.
(4) Most -- I'm tempted to say all -- programs that prepare teachers aren't designed with (3) at their core. Modeling effective teaching is haphazard, feedback is unstructured, and end goals are uncertain. This is a design flaw, and I'm not sure it can be solved by trying to reform existing institutions that prepare teachers, though I am glad Deans for Impact is still trying (I no longer work there).
Interestingly, Dan, when you wrote recently about the blinkered view that ed-tech funders have about "Terra Mathematica" I found myself thinking everything you argued there applies with equal force to "Terra Teaching." A few years back there was a well-funded effort to create a wholly new teacher-training program that would be "competency" based (High Meadows Graduate School, for the curious). It failed spectacularly, and I'd argue in part because thinking about teaching as a series of discrete competencies is as misguided as thinking about mathematics as a set of discrete skills.
That said...the idea that we might design an entirely new and different program to foster great teaching holds great appeal. I've come to believe this has to be done from the ground up -- grafting it into existing institutions is a hard slog. What would this new Institute of Inspiring Teaching consist of? Well, instead of a disparate set of unconnected courses and basically random student teaching experiences, this new organization would be designed with near-constant opportunities to practice teaching (including lots of approximations of teaching); have a faculty comprised of cognitive scientists, practice-focused researchers, and expert veteran teachers; would provide ongoing support to novice teachers in their first few years of practice; and would make use of all the tools that learning science indicates are effective for learning.
Dare to dream. Also sorry for the essay/pitch in the guise of a comment.
Love your point "Great teachers are often quite impatient. They do not wait for the demand for their teaching to arise naturally in a student. They see it as their job to create demand, oftentimes quite impatiently."