The verses have a progression of A, G, F, E, and the choruses go A, G, E.

I think that the verses are subtly clever because A leads to E in a I V I motion, but in A Mixolydian, G is part of the scale, and I think that descending to F#m would make it sound too "major." So they opt for F instead, which is a half step above E, the target. You still have the line cliche, but with that Talking Heads wit thrown in. I also find it fascinating that, in root position, we have the backwards alphabet in the fifths (E D C B A) with F and G occurring in the root notes during the sequence. The song is about the absurdity of being an infant, so it makes sense.

A C# E / G B D / F A C / E G# B