Are You the Baby or the Bathwater?

Here is one way to know.

As parents scrutinize the quantity and quality of screen time in classrooms, edtech developers, funders, and commentators plead with them, “Please do not go too far! Please do not throw out the baby with the bathwater!”

Forming school policy around bad examples of edtech “is throwing the baby out with the bathwater,” says the CEO of the Digital Learning Collaborative in District Administration.

“Educators do not want to throw the baby out with the bathwater,” claims The Week.

“The conversation is missing nuance and it’s a little bit like throwing the baby out of the bathwater,” says Tony Wan of Reach Capital on Alexander Russo’s The Education Show podcast.

Well what’s the baby, then?

Edtech developers all seem to think that they are the baby, not bathwater, and I am very sorry but you cannot all be baby. Last week, Lina Eroh from the Overdeck Foundation made similar pleas for moderation in Education Week, and offered several questions districts can ask that she believes will help districts separate baby from bathwater:

Is there credible evidence that this tool is the best way to address the problem you’re hoping to solve? If so, what is the expected impact on student outcomes?

When implemented well, does the tool support, rather than replace, strong teaching?

Does the tool connect to and enhance current instructional approaches and curricula? Is it developmentally appropriate for the age band?

Once procured, is there a process in place for tracking and evaluating impact on student outcomes? What is the plan if impact is not detected?

Can educators clearly explain to families what the technology is, and is not, being used for?

Look: 99% of edtech companies know how to answer “yes” to all of those questions.

Everyone has an evidence page. Everyone talks a great game about supporting teachers, even the companies that would dearly like to replace them with a few hundred thousand output tokens from Anthropic. Everyone has a dashboard. Standards alignment. Etc. That includes the products that parents really hate right now.

Those questions don’t separate wheat from chaff, baby from bathwater. Here is a question that does:

After using this product for 30 minutes, do students feel more or less like they belong at this school?

Do kids feel more or less known by their teachers after using this product? By their classmates?

Belonging, for many students, predicts and is a prerequisite for learning. Out of a battery of motivation and belief measures studied by Barbieri and Miller-Cotto “sense of belonging to mathematics was the only significant predictor of learning.” Disbelonging plays a significant role in our crisis of chronic absenteeism. (Washington’s state education agency calls it “lack of connection to a caring adult.”) Parents also rate school climate among their top priorities in a school.

Most edtech companies can produce a PDF that is evidence-flavored. Far fewer can claim with any credibility that their products knit students and teachers together, especially within core academic instruction. In 2017, when RAND studied personalized learning models in schools—many of which plunked kids behind computers by themselves for extended periods of time, distilling all their ideas and brilliance into line graphs and bar charts—they found decreases of up to 10% on questions of safety and belonging compared to national averages.

If you lead a school and want to play with that kind of technology, you need to be sure—like ESSA Level 0 sure—that it is going to send some other key indicator to the moon because it is likely to drag other important metrics to the ground. It is likely to diminish your school’s climate, disconnect your students, and sour your parents.

Just visit a class, okay.

Ask edtech companies for evidence, yes, but know that you need more evidence than the kind you get in a PDF. The PDF is necessary but insufficient. Visit a class. Ask yourself what the product is doing to teachers and students? Do they know and appreciate one another more or less afterwards? Does the product waste or make good use of their gifts? Would you be proud to have a parent walk in and see this class? This is how you tell the bathwater from the baby.

Odds & Ends

¶ Alexander Russo had me on his Education Show to talk about the digital backlash (can’t bring myself to use “techlash” yet but I’m getting there) and how we got here.

¶ Russo asked Tony Wan this same question: “What is the baby then?” I’ll recommend Tony’s response because he basically listed off the last ten years of my resume. Thanks, Tony.

¶ I’ve been sitting with this study (pre-peer review) for a couple of weeks now—Generative AI Can Harm Teaching—and I think it’s important enough to share.

Halfway through the school year, a researcher gave a bunch of teachers (randomized at the school level) access to a localized ChatGPT wrapper for teacher support. Lesson planning. Homework support. Exam prep. Etc. Then the researcher surveyed the students on several questions like, “Compared to the fall, how enjoyable is [subject] class these days?”

Certain combinations of those questions, which the researcher called “intrinsic motivation” decreased by .11SD for the experimental group. USC professor Stephen Aguilar critiqued the arbitrary nature of those combinations and doesn’t think .11SD is all that much to get excited about. Me, I don’t want to see any combinations of those items decrease because of a technology we voluntarily introduced into schools, particularly one where the upside for learners really has yet to reveal itself.

Also: I don’t know what to make of the finding that the median depth of conversation between teacher and chatbot was two or three teacher messages (p A20) but I don’t like it.

You can’t one- or two- or three-shot your way to good teaching materials with AI. Maybe the teachers are modifying the AI output heavily afterwards, but that kind of observational research has yet to be conducted, to my knowledge.

(See Jill Barshay for more on this study.)

¶ The University of Chicago Law School is banning not just AI but laptops from first-year classes. I wish this quote from their press release was more deeply understood by edtech industry professionals:

The relationship between AI and school is very different from the relationship between AI and work, Hubbard said. “The idea that AI creates shortcuts, saves time, and avoids effort; these are all things that could be very beneficial in the professional context where you want to maximize efficiency. But they are very, very damaging in the educational context, when the whole point is to do things the hard way—because that’s how you learn.”

¶ An interesting sign of the edtech times and a new 5% problem for Khan Academy. Back in May, Michael Lee, the Majority Leader of the North Carolina Senate added a $10M no-bid contract to Khan Academy in the state budget. Following parent and legislator critique (Sen. Sophia Chitlik: “I’m struggling to find the evidence that shows that this is a better investment than, say, $10 million in instructional assistants.”) Lee’s final budget cut that proposed outlay by 95% to $500,000, making me think the market is starting to get soft for bathwater.