The Generative AI Learning Penalty: Evidence from Chinese Secondary Education
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning. The data combine monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. We exploit staggered AI adoption in a difference-in-differences design. AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years. The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. The learning losses are concentrated among roughly 80% of AI users whose behavior is consistent with homework outsourcing, as indicated by exceptionally short homework completion time coupled with high homework scores. AI users who maintain similar homework completion time as non-AI users experience small learning losses.
Edit: moving my comment up here
Just in case as it’s formatted a bit weirdly
X-Axis: Homework scores
Y-Axis: Exam scores
Wouldn't a bar graph be better for this sort of data? The line keeps suggesting change over time to me and it's making the whole thing hard to read, even with OP's note.
Disagree-- line graphs are good for most continuous x axes, not just time. There's no reason to group the homework scores but not the exam scores, as most usages of a bar graph here would, or to use a bar graoh with 1 bar per possible hw score (which is basically just a line chart anyways).
The issue with the chart is just that the axes aren't labelled well at all. X is hw, Y is exam, and it looks like rather than raw score it's using... score divided by the average score, as a %? So how much better or worse the score is than the average, as a percentage?
It got me confused too at first but once you think about it it makes a lot of sense : the gray line is mostly straight, efforts get rewarded, but the more they use AI in homework, the worst they do on exams, even worst than the worst of the no AI crowd
Funny is that when I saw the graph I immediately thought that most people would be confused.