The Generative AI Learning Penalty: Evidence from Chinese Secondary Education
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning. The data combine monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. We exploit staggered AI adoption in a difference-in-differences design. AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years. The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. The learning losses are concentrated among roughly 80% of AI users whose behavior is consistent with homework outsourcing, as indicated by exceptionally short homework completion time coupled with high homework scores. AI users who maintain similar homework completion time as non-AI users experience small learning losses.
Edit: moving my comment up here
Just in case as it’s formatted a bit weirdly
X-Axis: Homework scores
Y-Axis: Exam scores
I disagree. Bad students could fear being punished for using AI as it's clearly cheating.
True. It could factor in the response if the students don't believe the survey is anonymous. It would be dependent on how the survey was given out. Electronic surveys are less likely to have students believe that. They have no way of knowing if the survey is truly anonymous. It's full trust based.
Paper surveys with no names and just simple bubble filling that they see immediately go into a stack of papers usually removes this worry in collecting data. Though this is much more expensive to collect survey data this way. Thankfully, for students this can be done for free in class.
But, yes, this fear of "being honest" could hurt the dataset if the students don't feel it is truly anonymous.
Good survey collectors will collect data both ways in order to compare the two methods to see if their is any bias in one data collection than another.
It's a problem. But it's one likely the paper accounted for already by comparing datasets. The problem I mentioned (positive categorization bias) is a potential flaw to the survey that they did not account for.
Though, I just realized you might be talking about a positive categorization bias for students that performed badly. A student not wanting to admit they used AI so they select "no ai" even though they did use it and scored poorly on exams. This is likely negligible though. A student that performs badly knows that they performed badly. This actually makes the positive selection bias favor them to be honest. People will select "used AI" as it lets their bad performance be attributed to something other than themselves. It's basically the same reasoning for the other bias I mentioned.
A bad exam score student that used AI has a positive personal bias to be honest and attribute their failure to AI use. They are likely to blame AI use for their failure. Answering honestly if they feel anonymous.
A good exam score student that used AI has a positive personal bias to attribute their success to their hard work and less about their use of AI in homework. The survey answer offers them no option to categorize themselves in this way without being categorized in the same way as bad students. So they "lie" and select the "no ai use" option as it better categorizes what they attribute their success to. They attribute it to their hard work. And there is no option that allows them to express this with these binary options.