Teachers: “These AI Resources Are Not Classroom-Ready.”

The "80-20 rule" is more accurately a "20-80 rule."

In my recent article about AI’s “delivery problem,” I claimed that AI-generated resources like lesson plans, assessments, and presentations, leave significant work to teachers, not just to customize them to their local context, but to make them even minimally viable for classroom use.

For example, new AI teacher “copilot” tools can easily produce paragraphs of text and call it a lesson plan, but students often benefit from images, videos, and interactives when they learn new ideas, all of which are absent from those lesson plans. Many ideas, like all of geometry, are inseparable from their visual presentation.

The resources I have generated in my testing generally conceive of teaching as the transmission of information and rarely detour from what seems like a Wikipedia-style bulleted summary of a topic.

An AI-generated lesson plan.

The 80-20 Rule

Many teacher copilot developers agree that the resources they generate are not completely classroom-ready. They frequently cite an “80-20 rule.” MagicSchool is a well-regarded teacher copilot tool, for example, and they write in their FAQ:

‍The 80-20 Approach: Use AI for initial work, but make sure to add your final touch, review for bias and accuracy, and contextualize appropriately for the last 20%.

As often as it is cited, I can find no empirical basis for the 80-20 rule. How was it determined? For example, did these copilot tools survey their users to ask them “How close is this resource to being ready for your use?”

I asked MagicSchool how they derived the 80-20 rule. They responded that “the 80/20 approach at MagicSchool advocates for a collaborative partnership between AI and educators, leveraging the efficiency of AI for initial content generation while recognizing the crucial role of human educators in refining, contextualizing, and ensuring the quality and appropriateness of educational materials.”

Their response describes their intent for human intervention but not how they measured that intervention at only 20% of the total versus 10% or 90%.

In the absence of any empirical basis for the 80-20 rule, I decided to survey educators on Twitter and LinkedIn and other places where educators spend time online. I shared with them four AI generated resources on the same idea—deciding whether two ratios are equivalent—and asked them, “How close is this to being ready for classroom use by a teacher?” with options for 0, 20, …, 100%.

I received 104 responses, 42 of which identified themselves as math teachers. The math teachers indicated these resources are quite far from 80% classroom-ready. 

With an overall readiness rating of 40%, it’d be more accurate to name the rule “20-80” than “80-20.”

Teachers described the inadequacy of the lesson plan:

This is a “telling lesson” not a “teaching lesson.” Suddenly during guided practice it tells me to encourage students to use cross products to determine if they are equivalent.  Where did that come from? Did I demonstrate that earlier in the lesson? Will 6th graders know this method? Have they learned about setting up proportions or solving for x yet? Why is this the preferred method for these students and this content?  

And the presentations:

I thought the presentation was really dry and would lose most of the class to boredom. It lacked the interactive examples a teacher could include using any type of manipulatives.

The presentation made me laugh.  It was a bulleted list.  No visuals, no animations, no basic clip art to even pretend that it wouldn’t be a nightmare to sit through this presentation. If reading a bulleted list was an effective way to teach students math, students wouldn’t struggle with this content, and my job wouldn’t be nearly as complex or rewarding.  

Discussion

No, of course this survey is not representative of every math teacher. It is possible that my respondents are more skeptical of AI than the median teacher, but I’m afraid the median teacher is still quite skeptical of AI’s utility. I suspect the respondents also have more experience and capacity than the median teacher, given they are participating in social networking and professional development experiences outside of their contract hours.

Perhaps novice teachers would find these resources more useful and closer to classroom-ready than veteran teachers. This would be in keeping with other research finding that generative AI benefitted lower-performing workers more than higher-performing ones.

Perhaps 40% is still better than 0%. If I’m walking along from point A to point B on a road, I would not turn down a ride 40% of the way just because it is not 100% or even 80%. But it seems equally possible that these teacher copilot tools are taking teachers 40% of the way towards point C, some alternate destination that is undesired by teachers or students. Two of the respondents spoke to this concern, saying these resources would create more work for them, not less.

The slides were utterly useless. It’s more work for me to extract something from those than to just create the slides on my own that would actually be engaging, have visual representations, and support understanding.

None of those AI components understand what it takes to "connect to students" and "engage students" in the lesson. I would still need to "craft" the lesson and the worksheets would need to be fixed for mistakes and clarity; therefore, not saving me any time in my planning. AI is great at creating documentation to check that box for lesson planning but thoughtful lesson planning is not about the “documentation.”

The needs in education are clear. Teachers clearly need help. Students clearly need effective resources. The public clearly needs a strong understanding of the sophistication of effective teaching.

I have conducted a survey here, one with a small sample and one that is likely unrepresentative of the population of teachers in important ways. But it is a unique survey, and contributes a uniquely useful insight. While most surveys about generative AI still ask teachers, “have you even used generative AI ever?” this survey has tried to understand how well generative AI is meeting those clear needs of teachers, students, and the public. The results indicate that generative AI is still leaving much of that need unmet.

Upcoming Presentations

Come hang out and learn together!

• Council of School Supervisors and Administrators 2024 Leadership Conference.

  • Solving the Math Problem

  • November 2, 2024. 8:55 AM.

  • New York City, NY.

• California Math Council - South Conference

  • The Teacher’s Superpower: How to Build and Put It to Work

  • November 8, 2024. 3:30 PM.

  • Palm Springs, CA.