I love your first/last mile analogy, but as a tutor, I don't think this Tutor CoPilot solves those problems for me. I couldn't quite tell from your example, but it seems like this is a chat that is only visible to the tutor, and the tutor is supposed to keep one eye on the chatbot's hints while also interacting with the student. I am not eager to bring this distraction into my tutoring sessions. And while these hints might help novice tutors in the short run, I wonder if they would hamper the tutors' longterm development, training them to seek answers from a bot rather than build their own teaching skills.
I'm also quite skeptical of the evidence that this had any effect (from the article you linked): "The study didn’t probe students’ overall math skills or directly tie the tutoring results to standardized test scores, but Rose E. Wang, the project’s lead researcher, said higher pass rates on the post-tutoring “mini tests” correlate strongly with better results on end-of-year tests like state math assessments." If the chatbot is prompting tutors to ask questions that are highly similar to those in the program's internal assessments, that would skew the test results while having little impact on standardized test performance (or the underlying skills they measure). There are many available retired standardized assessments available – I'd be curious to see how the students perform before/after the Tutor CoPilot intervention.
In general, I think non-tutors/teachers tend to place way too much value on identifying a student's misconception and providing good explanations. Question selection, rhythm, making the student "drive", and many other aspects are much more important.
The quote that stands out to me from the Tutor CoPilot article:
“But it is much better than what would have otherwise been there,” Wang said, “which was nothing.”
I feel like this is one of the lines that AI edtech advocates default to when they're compelled to acknowledge the problems with their products: The alternative is nothing - do you want the students to have nothing?
To me, this speaks to a remarkable narrowness of perception. It treats both students and teachers as tabulae rasae who bring nothing to the table that they are not explicitly trained to bring. In doing so, it neglects that there is always *something* that the use of edtech displaces, and that this displacement comes with costs that the actual (as opposed to advertised) benefits of edtech frequently fail to justify. And to connect with the critical point that Tracy Zager is making, it represents educational systems and institutions as immutable givens, such that the implementation of edtech is the only possible intervention.
Hi jwr! I'm the main author of this work. Thanks for your thoughts. I'm struggling to find the quote "But it is [...]" -- that quote neither appears in my paper nor in Dan's article.
The view that I'm promoting is that I believe in enhancing the human touch. I believe that both the student and educators are important interlocutors, and there are ways in which we can create more enjoyable interactions for both. I agree with you that both students and educators have important things to bring to the table.
However, in work, I also acknowledge the limitations in traditional ways of raising quality. Such as professional development programs that are expensive, inaccessible, and not immediately practical as they happen outside of teaching -- typically. I refer you to Section 2.1 of our work.
I'm open to hearing your ideas on alternative approaches. Reflecting on our answers, I believe we're on the same side of the table :)
Hi Rose, I appreciate the thoughtful response. The quote I'm referring to appears in the article from The 74 that Dan links to above. URL for reference:
I do feel that the quote that The 74 attributes to you echoes what I find to be a concerning theme in edtech advocacy. From what you say, though, it sounds like it doesn't adequately represent your views, and I appreciate that you recognize the importance of what students and teachers bring to the table.
On my side, I should acknowledge that I'm speaking a bit out of my field: I teach community college writing courses, not K12 math, though I do have a good bit of experience as a writing tutor. (I should also acknowledge that I've got a rather bad cold at the moment, which (a) is why I'm not teaching right now and (b) may have made my initial comment a bit grouchier than it needed to be.)
Looking through your study, and acknowledging again that I'm not an expert in math tutoring specifically: if I'm reading correctly, your study was done working with a virtual tutoring provider, not in-person tutoring. Moreover, it appears that the tutoring takes place through a chat system. (Please correct me if I'm wrong.)
While I recognize that online tutoring can have value in terms of expanding access, the fact is that chat-based tutoring is already a highly de-personalized experience. The interaction is stripped of tone of voice, physical presence, facial expression and emotional affect, gesture and body language, etc. (If I can put it this way, there are many ways of being silent, but they all look the same in a chat.)
The problem is that much of what we have been learning about learning tells us that we need to go in precisely the opposite direction: toward a more fully human and genuinely personal experience for learners and teachers. (Two books that have had a particular influence on my own thinking are In Search of Deeper Learning by Jal Mehta and Sarah Fine and Relationship-Rich Education by Peter Felten and Leo Lambert.) So, while I respect the fact that you're working to improve instruction within a specific modality, and I recognize that this work can be useful and important within a certain set of social and institutional constraints, I think that in the bigger picture we need to be setting a different baseline and taking a different approach.
Ah thanks both jwr and TT for clarifying. I hadnt caught that exact quote in the article previously - while I appreciate the news coverage, I may have edited parts of it to better reflect some nuances!
I also really appreciate the context you laid out. I agree with many things e.g. importance of speech and visual inputs; I point to the Limitations/Discussion section which I wanted to be thoughtful about on that topic!
I believe and hope that this work with Tutor CoPilot is the first step in meaningfully empowering those who do the on-the-ground work in creating meaningful interactions. Lots of alignment, and I have two new books on my reading list now!
On another note, I appreciate this approach and can see it's potential applicability for other domains e.g., for early grade literacy. It's often hard for tutors and even teachers less versed in the science of reading that I have observed in small groups and one-on-one to respond in the moment with productive feedback when children make mistakes reading. Add in some audio, which AI platforms now have available, and this approach may help when working with struggling readers.
Eye opening facts about Edtech startups using AI. Good deconstruction of why this maybe the case,especially the delivery problem. We will work on both sides! Thanks for the great article!
Hi Dan, I'm quite agree with your idea that AI can't solve the delivery problem from end to end. Copilot may be one of the solutions, but unlike program development, there is an unavoidable issue with copilot for teachers.
Copilot works well with vscode, because it's opensouced and has a large dev community, even though, the cursor team had to develop a separate vscode version to support the features they want.
Back to education, most LMS or teacher-used tools are very old, closed-source, and cannot support the extensions needed by copilot. So I think it may not be possible to develop a useful copilot in the short term, unless it is integrated with its own office suit like Microsoft.
I also have my own AI education platform. What I hope for is to benefit both teachers and students on the same platform, with course design, in-class learning, and post-class tutoring all in one place. Maybe it will be hard in the early stage, (this is exactly the problem that AI can solve, simplify the difficulty), but in the future, it may be able to go further. At least I hope so.
A common AI assistant that is used by the teacher and students will capture more of the learning process and lead to more effective EdTech tools regardless of how well the early AI assistants work (or don't work).
While I agree that Copilot for code is quite different from a teacher tool, I think the argument that most LMSs are closed source and old is really not the problem-most LMSs have dozens of add-on tools not manufactured by the LMS that can be integrated. There are standards that are adopted for tool interoperability- at IMSglobal. I strongly suspect that many of these AI assistants are not being marketed to teachers to support teaching but, instead, to get a large, useful, free dataset. Novice teachers may lean on these tools but their data will not be very helpful in training the tool with the knowledge Dan described in the opening of the Substack. Those special skills Dan described are what teachers spend a lifetime honing and the reason everyone wants that experienced teacher with that skillset.
If you have both tried to develop plugins for LMS or extensions for open source software, you will find that there is a very big difference. To create an active developer community, opensource and interfaces are essential.
Regarding the assistant part, I think the main limitation at present is the AI technology, because the most important factor for teachers is emotional connection, AI can easily surpass the level of teachers on knowledge, but can't replicate the human connection and empathy that teachers provide.
If one day AI has true visual and auditory capabilities and the ability to express emotions, it will be a revolution in the field of education.
I am working to make AI more emotionally connected, and at the same time make it easier for teachers and students to use these tools. This is a long way.
I love your first/last mile analogy, but as a tutor, I don't think this Tutor CoPilot solves those problems for me. I couldn't quite tell from your example, but it seems like this is a chat that is only visible to the tutor, and the tutor is supposed to keep one eye on the chatbot's hints while also interacting with the student. I am not eager to bring this distraction into my tutoring sessions. And while these hints might help novice tutors in the short run, I wonder if they would hamper the tutors' longterm development, training them to seek answers from a bot rather than build their own teaching skills.
I'm also quite skeptical of the evidence that this had any effect (from the article you linked): "The study didn’t probe students’ overall math skills or directly tie the tutoring results to standardized test scores, but Rose E. Wang, the project’s lead researcher, said higher pass rates on the post-tutoring “mini tests” correlate strongly with better results on end-of-year tests like state math assessments." If the chatbot is prompting tutors to ask questions that are highly similar to those in the program's internal assessments, that would skew the test results while having little impact on standardized test performance (or the underlying skills they measure). There are many available retired standardized assessments available – I'd be curious to see how the students perform before/after the Tutor CoPilot intervention.
In general, I think non-tutors/teachers tend to place way too much value on identifying a student's misconception and providing good explanations. Question selection, rhythm, making the student "drive", and many other aspects are much more important.
The quote that stands out to me from the Tutor CoPilot article:
“But it is much better than what would have otherwise been there,” Wang said, “which was nothing.”
I feel like this is one of the lines that AI edtech advocates default to when they're compelled to acknowledge the problems with their products: The alternative is nothing - do you want the students to have nothing?
To me, this speaks to a remarkable narrowness of perception. It treats both students and teachers as tabulae rasae who bring nothing to the table that they are not explicitly trained to bring. In doing so, it neglects that there is always *something* that the use of edtech displaces, and that this displacement comes with costs that the actual (as opposed to advertised) benefits of edtech frequently fail to justify. And to connect with the critical point that Tracy Zager is making, it represents educational systems and institutions as immutable givens, such that the implementation of edtech is the only possible intervention.
We need a different approach.
Hi jwr! I'm the main author of this work. Thanks for your thoughts. I'm struggling to find the quote "But it is [...]" -- that quote neither appears in my paper nor in Dan's article.
The view that I'm promoting is that I believe in enhancing the human touch. I believe that both the student and educators are important interlocutors, and there are ways in which we can create more enjoyable interactions for both. I agree with you that both students and educators have important things to bring to the table.
However, in work, I also acknowledge the limitations in traditional ways of raising quality. Such as professional development programs that are expensive, inaccessible, and not immediately practical as they happen outside of teaching -- typically. I refer you to Section 2.1 of our work.
I'm open to hearing your ideas on alternative approaches. Reflecting on our answers, I believe we're on the same side of the table :)
Hi Rose, I appreciate the thoughtful response. The quote I'm referring to appears in the article from The 74 that Dan links to above. URL for reference:
https://www.the74million.org/article/study-ai-assisted-tutoring-boosts-students-math-skills/
I do feel that the quote that The 74 attributes to you echoes what I find to be a concerning theme in edtech advocacy. From what you say, though, it sounds like it doesn't adequately represent your views, and I appreciate that you recognize the importance of what students and teachers bring to the table.
On my side, I should acknowledge that I'm speaking a bit out of my field: I teach community college writing courses, not K12 math, though I do have a good bit of experience as a writing tutor. (I should also acknowledge that I've got a rather bad cold at the moment, which (a) is why I'm not teaching right now and (b) may have made my initial comment a bit grouchier than it needed to be.)
Looking through your study, and acknowledging again that I'm not an expert in math tutoring specifically: if I'm reading correctly, your study was done working with a virtual tutoring provider, not in-person tutoring. Moreover, it appears that the tutoring takes place through a chat system. (Please correct me if I'm wrong.)
While I recognize that online tutoring can have value in terms of expanding access, the fact is that chat-based tutoring is already a highly de-personalized experience. The interaction is stripped of tone of voice, physical presence, facial expression and emotional affect, gesture and body language, etc. (If I can put it this way, there are many ways of being silent, but they all look the same in a chat.)
The problem is that much of what we have been learning about learning tells us that we need to go in precisely the opposite direction: toward a more fully human and genuinely personal experience for learners and teachers. (Two books that have had a particular influence on my own thinking are In Search of Deeper Learning by Jal Mehta and Sarah Fine and Relationship-Rich Education by Peter Felten and Leo Lambert.) So, while I respect the fact that you're working to improve instruction within a specific modality, and I recognize that this work can be useful and important within a certain set of social and institutional constraints, I think that in the bigger picture we need to be setting a different baseline and taking a different approach.
Ah thanks both jwr and TT for clarifying. I hadnt caught that exact quote in the article previously - while I appreciate the news coverage, I may have edited parts of it to better reflect some nuances!
I also really appreciate the context you laid out. I agree with many things e.g. importance of speech and visual inputs; I point to the Limitations/Discussion section which I wanted to be thoughtful about on that topic!
I believe and hope that this work with Tutor CoPilot is the first step in meaningfully empowering those who do the on-the-ground work in creating meaningful interactions. Lots of alignment, and I have two new books on my reading list now!
I believe he was referencing a quote in the article linked in Dan's blog. https://www.the74million.org/article/study-ai-assisted-tutoring-boosts-students-math-skills/?utm_source=substack&utm_medium=email
On another note, I appreciate this approach and can see it's potential applicability for other domains e.g., for early grade literacy. It's often hard for tutors and even teachers less versed in the science of reading that I have observed in small groups and one-on-one to respond in the moment with productive feedback when children make mistakes reading. Add in some audio, which AI platforms now have available, and this approach may help when working with struggling readers.
Eye opening facts about Edtech startups using AI. Good deconstruction of why this maybe the case,especially the delivery problem. We will work on both sides! Thanks for the great article!
Hi Dan, I'm quite agree with your idea that AI can't solve the delivery problem from end to end. Copilot may be one of the solutions, but unlike program development, there is an unavoidable issue with copilot for teachers.
Copilot works well with vscode, because it's opensouced and has a large dev community, even though, the cursor team had to develop a separate vscode version to support the features they want.
Back to education, most LMS or teacher-used tools are very old, closed-source, and cannot support the extensions needed by copilot. So I think it may not be possible to develop a useful copilot in the short term, unless it is integrated with its own office suit like Microsoft.
I also have my own AI education platform. What I hope for is to benefit both teachers and students on the same platform, with course design, in-class learning, and post-class tutoring all in one place. Maybe it will be hard in the early stage, (this is exactly the problem that AI can solve, simplify the difficulty), but in the future, it may be able to go further. At least I hope so.
A common AI assistant that is used by the teacher and students will capture more of the learning process and lead to more effective EdTech tools regardless of how well the early AI assistants work (or don't work).
While I agree that Copilot for code is quite different from a teacher tool, I think the argument that most LMSs are closed source and old is really not the problem-most LMSs have dozens of add-on tools not manufactured by the LMS that can be integrated. There are standards that are adopted for tool interoperability- at IMSglobal. I strongly suspect that many of these AI assistants are not being marketed to teachers to support teaching but, instead, to get a large, useful, free dataset. Novice teachers may lean on these tools but their data will not be very helpful in training the tool with the knowledge Dan described in the opening of the Substack. Those special skills Dan described are what teachers spend a lifetime honing and the reason everyone wants that experienced teacher with that skillset.
If you have both tried to develop plugins for LMS or extensions for open source software, you will find that there is a very big difference. To create an active developer community, opensource and interfaces are essential.
Regarding the assistant part, I think the main limitation at present is the AI technology, because the most important factor for teachers is emotional connection, AI can easily surpass the level of teachers on knowledge, but can't replicate the human connection and empathy that teachers provide.
If one day AI has true visual and auditory capabilities and the ability to express emotions, it will be a revolution in the field of education.
By the way, this is my website https://codekidz.ai/
I am working to make AI more emotionally connected, and at the same time make it easier for teachers and students to use these tools. This is a long way.