I can only speak from a British perspective, but it's not even the size of the cars that bothers me, it's the seemingly endless abdication of responsibilities or driving skills that come with it.
"Oh this car has a 4x4 capability" - then drives through the snow like it's a hot summer's day.
"Oh it has adaptive cruise control" - then stares blankly into space rather than anticipated hazards the cameras won't have seen.
It's really frustrating, like "nah fuck everyone outside the bit of metal around me". The size seems somewhat immaterial (with the exception of children and persons of smaller stature) - a chunk of metal around an SUV coming at you at 30 or 70 mph is going to ruin your day as much as a Smart car is, sadly.

