The only question you need to ask about education technology.

What happens to wrong answers?

Has anyone developed a checklist to review when choosing an online platform or app to recommend to teachers?

Over at the National Council of Teachers of Mathematics member forum, a teacher asks for help evaluating education technology. 

"How do I know if this edtech is any good?" is the sort of question that was hard to find time and space to answer inside the fog of virtual teaching but seems newly fresh on everyone's mind.

My own rubric for evaluating edtech is very simple. One question tells me most of what I need to know.

What happens to wrong answers?

If I have one minute with your product, if you lure me over to your booth at a conference with candy, if someone sends me a link to your website, I'll do the same thing every time:

I'll read the question you're asking and I'll answer it incorrectly.

That's because the vast majority of a student's time in a math class is spent producing wrong answers. Which is to say it is spent producing thoughts that are under formalized, loosely gathered, under construction, and which, if pressed with even minimal force, would not produce a correct answer. I want to know what your product does with all of that time and thinking.

Does edtech waste wrong answers or put them to work?

By this standard, you can put most education technology along an axis.

[Disclosure: I work at Desmos, a company that does quite well by this standard.]

Generally, a student writes an incorrect answer and the software tells the student they're incorrect. The student often gets another chance to answer. Sometimes they receive an encouraging message, a hint, or another resource. But the prime directive for most edtech is to let the student know they're wrong and get them to be not wrong as soon as possible.

I'm not saying we should protect students from knowing when they're wrong. I'm saying that every wrong answer is a resource and we shouldn't waste it.

Every wrong answer is the product of thinking that is correct and incorrect, formal and informal, developed and developing. The work of teaching is to help students understand and develop their thinking. Many of your most effective moments in a classroom, then, will be spent:

• Scrutinizing wrong answers, not dashing past them.

• Helping students identify the parts of their thinking that are correct, not writing it all off as incorrect.

• Helping students backtrack through their thoughts to identify the moment they started to over- or under-generalize an idea, not pushing them forward to the next exercise.

Here is a concrete example. Your product is trying to help students learn to plot points using numeric ordered pairs. Here is a student who is very wrong and very right. All of the numbers are correct and none of them are in the correct place. The student has reversed the x- and y-coordinates.

When the student presses "Check," the software tells them "Not quite," which is the same message it sends if the student types in anything—the number 420, the entire shooting script of Bee Movie, a cry for help—anything. "Not quite." 

The wrong answer is a very smart one and its brilliance is wasted on this edtech.

One approach that puts wrong answers to work is to simply interpret them and mirror them back to students in a form they can understand and learn from.

For example, in the activity Sand Dollar Search from the Desmos Math Curriculum, students try to locate sand dollars on a beach. If a student tries to locate the sand dollar at (3,4) and they type in (4,3), we don't tell them they're wrong. We don't even give them an encouraging message. We just take their thinking very seriously and make a crab pop up at (4,3).

That's all. The student experiences a moment of cognitive dissonance. "The crab did not pop up where I thought it would." Their mind works to resolve it. They try again.

On a later screen, we capture a huge range of student thinking by asking students to describe a new sand dollar in words, a sand dollar that's in a quadrant we have never seen before.

Three of my favorite student responses, all of them both wrong and right:

• It is 3 on the y, and -4 on the x

• It's 6 spaces away from the sand dollar

• The sand dollar is in the up left box, and it's on (4, 2)

In the previous example, our technology interpreted a student's answer automatically. Here, we need the teacher to do that. We hope here that the teacher wrings as much value as possible out of these answers.

The teacher can do that by taking each one and interpreting them so literally that the teacher winds up looking for the sand dollar in the wrong place. (For example, by looking “6 spaces away” in the wrong direction.)

In doing so, they'll communicate the value of the student's answer and the need for more precise mathematics. Telling the student "Not quite" communicates none of that.

Whether the technology we're talking about is a curriculum, a computer, or (stretching the definition somewhat) a teacher, we should hope that it makes room for different, interesting right and wrong ideas and helps the entire class find value in those ideas. The best technology makes something easy out of something hard, which is why I'll always give a second look to edtech that takes a student's wrong answers and puts them to work.

What Else?

• A team of researchers at George Mason University have developed a framework for thinking about technology centered in equity.

• Audrey Watters doesn’t play around with her “four key questions to ask of edtech.” (Scroll to the bottom.)

• If you’re tired of the usual professional learning offerings, tune into Elham Kazemi’s ideas about job-embedded professional learning on the most recent Math Teacher Lounge podcast, hosted by Bethany Lockhart Johnson and myself.

• Robert Talbert has a great post inviting you to re-consider points-based scoring.

• Over on Twitter, I asked people “what is the most wasteful sport?” which was a fantastic experience in mathematical modeling.

• The comments on my last post where I asked you to help me tune up a teaching moment were really tremendous. Just a high water mark for this newsletter. Can’t wait to get back into a class now.