Thanks so much for sharing this analysis, Wess. Helping people who want to to be 'evidence-based' in their decisions better understand the evidence -- and the 'evidence' -- is often not a very glamorous task. When you do this, you don't get sexy publication credits or citations, but you do get admiration from many people who really care about separating fact from fiction -- and how to tell the difference. This is messy stuff, thanks for taking the time to sweep your broom in public like this!
Bless you for doing this work so I don't have to. You just provided my first stop when people push back on my claims that "evidence-based" approaches to AI in education are mostly bullshit, at least so far. Such claims only make sense if the evidence is based in research methods that inquire after truth. That's just not happening much, and you just provided some evidence. Thanks!
What one can notice from the table is that most "researchers" were very laconic concerning how exactly AI was used, or any training provided. I suspect not much happened there, leaving the door open to experiment with careful and purposeful integration of LLMs in instruction.
Bad Science is everywhere used as "PROOF" of whatever agenda is currently popular. Most studies are biased and lean heavily toward outcomes that funding bodies want to find. These studies are use to justified Education Department Policies and directives. The bottom line is like everything, AI used well is great, used poorly is crap. The TRUTH is, it is out there, it isn't a matter of it we should use it in EDUCATION, it already is being used in EDUCATION. It isn't a matter of whether it is used everyday by everyone in every social media app, cars and most fridges etc. It IS OUT THERE. So you better damn well learn to use it effectively, and if EDUCATION doesn't prepare kids to use it, then EDUCATORS are failing.
Check out my podcast DEEP DIVE Podcast - AI for GOOD or EVILon this topic.
Thanks so much for sharing this analysis, Wess. Helping people who want to to be 'evidence-based' in their decisions better understand the evidence -- and the 'evidence' -- is often not a very glamorous task. When you do this, you don't get sexy publication credits or citations, but you do get admiration from many people who really care about separating fact from fiction -- and how to tell the difference. This is messy stuff, thanks for taking the time to sweep your broom in public like this!
Thx Mike. While doing it I was indeed perplexed that none of the few lit reviews I included called any study out.
This is very well done and accords with what I'm seeing too (see link below). Thanks for doing the hard work of looking into all this shoddy research.
https://buildcognitiveresonance.substack.com/p/something-rotten-in-ai-research
right back at you!
Thank you Wess! It has taken me until now to totally absorb your work! Excellent helpful job!
Bless you for doing this work so I don't have to. You just provided my first stop when people push back on my claims that "evidence-based" approaches to AI in education are mostly bullshit, at least so far. Such claims only make sense if the evidence is based in research methods that inquire after truth. That's just not happening much, and you just provided some evidence. Thanks!
What one can notice from the table is that most "researchers" were very laconic concerning how exactly AI was used, or any training provided. I suspect not much happened there, leaving the door open to experiment with careful and purposeful integration of LLMs in instruction.
Bad Science is everywhere used as "PROOF" of whatever agenda is currently popular. Most studies are biased and lean heavily toward outcomes that funding bodies want to find. These studies are use to justified Education Department Policies and directives. The bottom line is like everything, AI used well is great, used poorly is crap. The TRUTH is, it is out there, it isn't a matter of it we should use it in EDUCATION, it already is being used in EDUCATION. It isn't a matter of whether it is used everyday by everyone in every social media app, cars and most fridges etc. It IS OUT THERE. So you better damn well learn to use it effectively, and if EDUCATION doesn't prepare kids to use it, then EDUCATORS are failing.
Check out my podcast DEEP DIVE Podcast - AI for GOOD or EVILon this topic.
https://youtu.be/-OlwnCwmiHg?si=AX-cTnKY8lapJZv8