For what's it worth Dan, roughly 100% of the people I spoke to who heard your presentation said it was fantastic, so your nimble ad-libbing worked! I've run into the same issue, people find it SO compelling to watch chatbots do their thing in real time, but from a presentation standpoint you might find the product has changed since you last played with it.
When I first started working in the classroom, if a student shared an idea with the whole class that was not what I expected, I felt the need to close that idea so we could focus on the "correct" one. I wanted to be gentle, so I would say things like "not quite" or "good idea, but..." My teaching was predicated on "no, but", which makes for terrible improvisation.
Within a few months, though, I noticed a change in my questioning. Instead of "no; good try" I would say things like "I like that you're thinking about multiplying." It became easier, with practice, to extract something productive from every student contribution.
Sometimes this meant pulling on one piece of their thinking that led towards my idea. Other times, though, it meant recognizing that the student was approaching the problem in a new way that I hadn't anticipated, and following their thinking that way instead. (These were my favorite, because it allowed me to be genuinely excited about a new idea that I was genuinely unsure about. When you're walking the tight rope without a net, students know it's real and it's so much more fun.)
This was difficult at first but became natural over time. It would be great if LLMs could do the same. I'm hopeful that if Khan Academy keeps listening to this feedback and improving their prompting (or maybe their training data), they might be able to achieve a chat bot that can extract something from the student's thinking that leads toward Khanmigo's idea of what a good solution looks like. I'm much less optimistic that LLMs, as they exist today, can get excited about a new method they haven't seen before in their training data. That kind of thinking requires genuine mathematical reasoning, not just pattern matching, which LLMs are (currently) notoriously bad at.
“I’m breathing into a bag.” Hopefully someone who works there is reading this. It is good feedback and it seems like there are achievable things they could do to make it less bad, though the underlying philosophy of addressing bugs in students would probably require a complete overhaul to change. I also get a little fed up with the structure of so many little lessons. A great resource yes, but mostly for autodidacts who can locate the specification lessons they’re looking for and brush off the flaws and philosophy (like the ones you’ve described in Khanmigo) instead of being turned off by them and running the other direction.
What Chatbot sees the frustration, the hopelessness, or the anxiety of a student? What Chatbot knows the lack of support at home, the fear of failure exhibited in class and the lack of peer support for learning? I am thankful that someone of your stature in the world of Math Ed has been willing to challenge the profit minded Ed tech industry who often have the eye of overburdened administrators who make life challenging for teachers and students.
You have focused on Khanmigo for several posts. Curious as to your take on Edia. In my conversations with them, I have shared many of the exact same reflections as you have had here. There is a big push in our state (VA) to find some type of tech-solution for tutoring our way out of math-ed deficiencies that are still blamed on Covid, but were actually only un-hidden during Covid and present much earlier. Edia, Zearn, and all the rest--is there anyone out there a little bit closer to doing it right? I have not seen it yet.
For what's it worth Dan, roughly 100% of the people I spoke to who heard your presentation said it was fantastic, so your nimble ad-libbing worked! I've run into the same issue, people find it SO compelling to watch chatbots do their thing in real time, but from a presentation standpoint you might find the product has changed since you last played with it.
"'What did the student do wrong?' the prompt asks, which is fundamentally and tellingly different from 'What did the student do right?'"
This flip was one of the biggest wins I felt from the beginning of this school year (when I felt like I was drowning https://danmeyer.substack.com/p/can-we-get-this-new-teacher-a-quick) to the end of my internship five days ago.
When I first started working in the classroom, if a student shared an idea with the whole class that was not what I expected, I felt the need to close that idea so we could focus on the "correct" one. I wanted to be gentle, so I would say things like "not quite" or "good idea, but..." My teaching was predicated on "no, but", which makes for terrible improvisation.
Within a few months, though, I noticed a change in my questioning. Instead of "no; good try" I would say things like "I like that you're thinking about multiplying." It became easier, with practice, to extract something productive from every student contribution.
Sometimes this meant pulling on one piece of their thinking that led towards my idea. Other times, though, it meant recognizing that the student was approaching the problem in a new way that I hadn't anticipated, and following their thinking that way instead. (These were my favorite, because it allowed me to be genuinely excited about a new idea that I was genuinely unsure about. When you're walking the tight rope without a net, students know it's real and it's so much more fun.)
This was difficult at first but became natural over time. It would be great if LLMs could do the same. I'm hopeful that if Khan Academy keeps listening to this feedback and improving their prompting (or maybe their training data), they might be able to achieve a chat bot that can extract something from the student's thinking that leads toward Khanmigo's idea of what a good solution looks like. I'm much less optimistic that LLMs, as they exist today, can get excited about a new method they haven't seen before in their training data. That kind of thinking requires genuine mathematical reasoning, not just pattern matching, which LLMs are (currently) notoriously bad at.
“I’m breathing into a bag.” Hopefully someone who works there is reading this. It is good feedback and it seems like there are achievable things they could do to make it less bad, though the underlying philosophy of addressing bugs in students would probably require a complete overhaul to change. I also get a little fed up with the structure of so many little lessons. A great resource yes, but mostly for autodidacts who can locate the specification lessons they’re looking for and brush off the flaws and philosophy (like the ones you’ve described in Khanmigo) instead of being turned off by them and running the other direction.
I couln't love this more! Thank you so much Dan.
Wouldn't it be great to see Dan Meyer and Sal Kahn talk about Khanmigo together on a podcast? Just saying....
What Chatbot sees the frustration, the hopelessness, or the anxiety of a student? What Chatbot knows the lack of support at home, the fear of failure exhibited in class and the lack of peer support for learning? I am thankful that someone of your stature in the world of Math Ed has been willing to challenge the profit minded Ed tech industry who often have the eye of overburdened administrators who make life challenging for teachers and students.
You have focused on Khanmigo for several posts. Curious as to your take on Edia. In my conversations with them, I have shared many of the exact same reflections as you have had here. There is a big push in our state (VA) to find some type of tech-solution for tutoring our way out of math-ed deficiencies that are still blamed on Covid, but were actually only un-hidden during Covid and present much earlier. Edia, Zearn, and all the rest--is there anyone out there a little bit closer to doing it right? I have not seen it yet.