Teachers Are Doing UNBELIEVABLE Things With AI!

Not even the TEACHERS believe it!

School is back in session here in the United States. Generative AI is still the buzziest consumer technology and people are rightly wondering, “What can this technology do to support learners?”

A gallery of edtech founders, consultants, and coaches have stepped in to answer that question, saying, “Here’s what.” They are giving speeches, writing position papers, and authoring guidance documents at the local, state, and national level all describing the power of generative AI to support learners.

Their ideas, generally, have two common features:

1. They are often written in the present and future tense—here is what students can and will be able to do—rather than the past tense, even though generative AI is almost two years old. Why, I wonder, must they draw from their imagination rather than from what I assume are copious examples of students using generative AI in transformative ways over the last two years?

2. They are often completely unbelievable.

For example, a prominent edtech founder recently suggested¹:

A class of 30 students could break out into five groups each around an “Alexa-type device” formatted with AI that facilitates discussion.

You need to understand that this is an unbelievable idea. It will not go well. Every bit of it is unbelievable—from the idea that an Alexa could do this, to the idea that a class would want the Alexa to do this, to the group size itself. Of the hundreds of thousands of K-12 teachers who have facilitated group work this week, you will find “six” among the least common group sizes.

Yet these unbelievable suggestions are unbelievably common. Recently, I collected several common AI lesson ideas, along with several non-AI lesson ideas, and asked my social networks, “Do you think this will go well?” where I defined “well” as “a productive and engaging experience for middle school students.” 

I received responses from 189 people who identified as a “Teacher” and 144 other responses from people across a range of other roles including “Coach”, “Industry,” “Researcher,” etc. Let’s see what they said!

Teachers do not think these AI ideas will go well. 

I included Scenario #3 as a check to make sure this measure would scale all the way to the floor. Beyond that throwaway item, the AI items all underperformed the non-AI items. What were the items?

Item Analysis

Here are all ten scenarios in descending order of agreement that it’ll work well.

---

Scenario 1

The teacher shows four related objects on the projector screen and asks the students to tell a neighbor which one is not like the others and why.

Agreement: 87%.

This is the “Which One Doesn’t Belong” routine you can find in our curriculum, and many others, right now.

---

Scenario 2

The teacher asks the students to write a response to a question, then compare responses with a neighbor, then write a second draft that's stronger and clearer.

Agreement: 48%.

This is the “Stronger & Clearer Every Time” routine. (Also prominent in our curriculum.)

---

Scenario 4

The teacher follows a sequence of lessons on a particular topic in a standards-aligned curriculum and gives students a quiz on the material at the end of the week.

Agreement: 34%.

This is the standard operating procedure in the majority of US classrooms right now. Teachers are clearly not overwhelmed by its efficacy. Yet they still think it will work better than any of the ideas ahead.

---

🤖 Scenario 9

The teacher assigns a presentation topic to students and asks the students to get feedback from an AI chatbot on their evolving thinking in a brainstorming session.

Agreement: 28%.

If I were building an AI-forward product right now, I would try to figure out why this AI lesson plan outperformed the others.

---

🤖 Scenario 6

The teacher tells the class they're going to learn about an idea by having a conversation with an AI chatbot that has been trained to impersonate one of the idea's original scholars. Then the teacher gives students a quiz on the material at the end of the week.

Agreement: 14%.

Education technologists have promoted this use case relentlessly for the last two years. There is a startup that raised $1.5M four months ago and this is all they do. I confess that I still find it pretty impressive as a magic trick. Very few surveyed teachers think it will work for students as a lesson, however. 

From the comments of the survey:

I have particular reservations about programs I have been shown where the chatbot impersonates a literary character and answers questions. This was obviously designed by someone who does not understand the nature of literature. Giving students foreclosed answers on character motivations or on the work as a whole effectively shuts down what literature is and what it’s for. 

---

🤖 Scenario 10

The teacher asks students to use AI to carry out virtual science experiments, hypothesizing, and testing outcomes without physical lab equipment.

Agreement: 14%.

I found this one in an AI guidance document prepared by one of edtech’s most prolific consultancies for one of the ten largest cities in the United States. Instead of combining baking soda and vinegar and watching the explosion, apparently students are meant to ask the LLM, "What would happen if I combine baking soda and vinegar?" and then read about it. Do I have that right?

---

🤖 Scenario 8

The teacher creates groups of six students, gives each group an Alexa-style AI device, which facilitates a discussion among the six students.

Agreement: 13%.

Hear it from a teacher in the survey.

A group of 6 middle schoolers is a nightmare. Groups of 3-4 with assigned roles would work better.

---

🤖 Scenario 7

The teacher shows students an AI generated slide like this one, talks about the bullet points, and asks for questions.

Agreement: 5%.

I made that slide in a very popular teacher copilot tool which just announced this slide generator feature last week. Teachers don’t think the slides work, folks!

---

🤖 Scenario 5

The teacher gives students a laptop and tells them to learn a particular topic by asking an AI chatbot questions about it and then gives students a quiz on the material at the end of the week.

Agreement: 4%.

This idea takes up a full city block in the imagination of many, many people working in edtech right now. I also found it in the same big-city AI guidance document. But the surveyed teachers (and I suspect most others) know that the students who would learn from this arrangement have long ago crossed the threshold from novice to expert.

A coach wrote:

I will change all my answers to “strongly agree” if the task is to see how many times the student can get the AI chatbot to say the words “skibidi Ohio rizzler,” “what the sigma,” “gyatt.” 

Another coach wrote:

I know 1 on 1 with AI is quite painful for most students. Slowly draining life from the educational experience. A nightmare really. 

---

Scenario 3

Agreement: 1%.

The teacher gives each student a textbook, assigns 100 pages to read, and gives students a quiz on the material at the end of the week.

This one is chalk, just here to make sure the survey measure is working as intended.

Additionally, I calculated the average difference in agreement with these scenarios between teachers and respondents in other roles. One might call this an “Out of Touch” index.

Discussion

Does this survey prove that all AI lesson plans are unworkable? Absolutely not.

Does it prove that all AI lesson plans are less workable than all non-AI ideas? Absolutely not.

Does it depict an edtech industry that is out of touch with the experiences of end users like teachers and students? 

Does it make a person wonder if some of the most lauded minds in edtech have spent, collectively, twenty minutes in an average middle school classroom? 

Does it point to an epidemic of magical thinking sweeping through edtech, the thinking that generative AI would be workable if only our brains had magically evolved differently over the last several million years, if only we had different goals for schooling, different assessments, different teachers, and different students?

Absolutely yes.

Featured Comments

A teacher in the survey:

The timing of this made me laugh! I had "AI" training yesterday to learn about more AI tools that students are using (school survey indicates that 80% are using AI to help with school work). I learned more tools esp. "text to image" LOL! Then we had workshop time to focus on our subject area. While I tried to find a way to use these tools to "save me time" for tasks I needed to do; I again found no helpful use for these AI tools.

SteveB, in last week’s post, points to the myriad ways generative AI could support his work:

Since I've spent the last couple of weeks setting up the Blackboard sites for my classes, and the past week fielding emails from students asking me why the "schedule" link goes to Spring 2022, here's a humble request for AI: Scan the whole damn site, check that every link works, alert me to any date references that don't fall within this prescribed range. Not glamorous, and it's not gonna make anyone a billionaire, but it sure would be helpful, and they all say they want to help, right?

Diana Laufenberg adds a +1 to AI scheduling software:

But yes, please AI folks deliver something that can... Show me a straight 7 bell schedule with these staffing parameters and these scheduling constraints. Now for a 4x4 or a year round AB block or straight 8 or year round AB block with a skinny day on Wednesday - with or without an advisory block 2 days a week or... or... or... some software can approximate some of these decisions.

James Cleveland-Tran adds a -1:

I'm gonna go with no on AI making a master schedule because, as with all things in education, making the schedule isn't just a mathematical operation but also an exercise in human empathy, which AI does not have.

Odds & Ends

¶ Related to last week’s post teaching versus surgery, here is a spicy one from Lee Shulman:

After some 30 years... I have concluded that classroom teaching is the most demanding, subtle, nuanced, and frightening activity that our species has ever invented... the only time medicine ever approaches the complexity of an average day for a classroom teacher is in an emergency room during a natural disaster.

¶ Michael Pershan reviewed Shalinee Sharma’s book Math Mind: The Simple Path to Loving Math. Pershan’s review is a gem and will do a lot to help you understand the limits of pure edtech, and even teaching itself, to support students in learning and loving math.

Educators know there’s only so much that great teaching can do. For one, math is tasked with educational gatekeeping, caught in what historian David Labaree describes as our desire to provide universal access to social advantage—an impossibility. Sharma likewise describes loving math as an “exclusive world” that every child could freely join. But given the role math plays socially, access to this sort of exclusivity is anything but simple.

¶ This study is making the rounds. It indicates that AI isn’t great at evaluating claims about text. Ethan Mollick says, well, they used older AI models. I’m only passing it along to make sure you know that I don’t care! I get no pleasure or pain from studies like this. I think it would be cool if AI could do cool things like evaluate claims about text really well. I could even imagine that happening! Just because I believe generative AI is vastly underpowered relative to the needs of teachers and students doesn’t mean I dislike it absolutely. It’s neat okay.

¶ Google’s Gemini now lets you ask questions directly of open source curriculum provider OpenStax. I can see that being helpful if your professor has assigned you an OpenStax curriculum or if you’re tired of seeing your chatbot hallucinating from Reddit in its answers to your questions about organic chemistry. But the problem with hallucinating chatbots in education isn’t just the “hallucinating.” It’s also the “chat” and the “bot.” The problem, especially in K-12 education, isn’t that you’re asking students to read incorrect text. It’s that you’re asking students to read text period, especially text absent any connection to its author. Making incorrect chatbots more correct, as Google is trying to do here, will fix only 1/3 of the problem with incorrect chatbots in K-12 education.

¶ Oprah Winfrey is hosting AI and the Future of Us later this week, a primetime television special in which various luminaries will tell you about the power of artificial intelligence. Should be fun. Can’t promise it won’t turn into next week’s post. Bill Gates will explain AI in education. It happens to be roughly eighteen months since he predicted that in eighteen months we’d be stunned by how much AI helps kids read. Maybe worth a follow-up, Oprah!

¶ I admire this daily graphing challenge more than I enjoy it. It presents options across several different domains, each of which you need to know pretty well in order to rule them out as answers. I’m adding it here because I’d love to see something like it that’s more accessible to more people.

1. I am not naming names throughout this piece because I have decided, this week, to try kindness. (This is me being kind!) We’ll see if I can maintain this posture long-term or if I will regress back to the (wait for it) mean.