The Generative AI Learning Penalty: Evidence from Chinese Secondary Education
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning. The data combine monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. We exploit staggered AI adoption in a difference-in-differences design. AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years. The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. The learning losses are concentrated among roughly 80% of AI users whose behavior is consistent with homework outsourcing, as indicated by exceptionally short homework completion time coupled with high homework scores. AI users who maintain similar homework completion time as non-AI users experience small learning losses.
Edit: moving my comment up here
Just in case as it’s formatted a bit weirdly
X-Axis: Homework scores
Y-Axis: Exam scores
Dafuq is this graph? No labeling and arbitrary numbers. Was this graphic made by ai?
I scrub the linked paper and pretty sure this is the graph they are using.
The axis still makes little sense. Why would they plot scores relative to scores? Thanks for checking out the article, my frustration isn't with you it's with their visualization choices.
You can use AI when doing homework (score is high), but you can't during an exam (score plummets).
What it shows is: the higher your homework score, the higher your exam score in non-AI using students, but with those using AI, that relation inverts - meaning that those who use AI to get stupid high scores in homework, just stop using their own brains and learning things.
Right, but the scores don't exactly "plummet" I would have expected to see below 50% for non-proficiency. I would want to see a graph of a control group of student scores pre-AI as a 3rd line here.
Edit: Clearly the "smart" students are not using AI, but I can't help but wonder if the "dumb" students would have either gotten those same scores anyway, or somehow did marginally better.
Thank you for explaining it to me. Makes much more sense now
It's comparing homework scores against exam scores
The kids who scored highly on homework because of AI did poorly in exams. This shows that AI use is just doing the homework for the kids and the kids aren't learning.