Programming

26127 readers

945 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 2 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

UlrikHD@programming.dev

bugsmith@programming.dev

Spyro@programming.dev

How AI assistance impacts the formation of coding skills - Anthropic study (www.anthropic.com)

submitted 1 month ago by idriss@lemmy.ml to c/programming@programming.dev

15 comments fedilink hide all child comments

A company not making self-serving predictions & studies.

you are viewing a single comment's thread
view the rest of the comments

[–] entwine@programming.dev 3 points 1 month ago (1 children)

In a randomized controlled trial, we examined 1) how quickly software developers picked up a new skill (in this case, a Python library) with and without AI assistance; and 2) whether using AI made them less likely to understand the code they’d just written.

We found that using AI assistance led to a statistically significant decrease in mastery. On a quiz that covered concepts they’d used just a few minutes before, participants in the AI group scored 17% lower than those who coded by hand, or the equivalent of nearly two letter grades. Using AI sped up the task slightly, but this didn’t reach the threshold of statistical significance.

Who designed this study? I assume it wasn't a software engineer, because this doesn't reflect real world "coding skills". This is just a programming-flavored memory test. Obviously, the people who coded by hand remembered more about the library in the same way students who take notes by hand as opposed to typing tend to remember more.

A proper study would need to evaluate critical thinking and problem solving skills using real world software engineering tasks. Maybe find some already-solved, but obscure bug in an open source project and have them try to solve it in a controlled environment (so they don't just find the existing solution already).

[–] Miaou@jlai.lu 1 points 1 month ago (1 children)

The study is about the impact AI use has on learning. Their experiment seems to test just that, unlike what you're describing.

Besides, remembering what you did an hour ago seems like a real world problem to me. Unless one manages to switch project before the bug reports come in

[–] entwine@programming.dev 1 points 1 month ago

The study is about the impact AI use has on learning. Their experiment seems to test just that, unlike what you’re describing.

The title is literally "How AI assistance impacts the formation of coding skills". Memorizing APIs isn't what most people would consiser a "coding skill".

Debugging, systems design, optimization, research and evaluation, etc are what actually make someone a useful engineer, and are the skills a person develops as they go from junior to senior. Even domain knowledge (like knowing a lot about farming if you're working on farming software) is more useful than memorizing the API of any framework. The only thing memorization does is it saves you a few minutes from having to read some docs, but that's minimal impact, and it's something you pick up normally throughout the course of working on a project anyways. When you finish that project, you might never use that API again, or if you do it might have changed completely when a new version is released.

remembering what you did an hour ago seems like a real world problem to me.

Sure, humans have shitty memory, but that has nothing to do with AI code assistance. There are plenty of non-AI coding assistants that help people with this (like Intellisense/LSP auto complete, which has been around for decades)