One thing that definitely contributed to this: when these games were coming out, those of us who were reading the gaming magazines at the time were aware that Sony had taken the IP away from the original devs, Singletrac, between 2 and 3. So we went in skeptical, and then… the controls were squishy, the power ups all looked the same (replaced the 2D icons with 3D pickups in an age where that just did not work), the weapons didn’t pack as much visceral ppunch… it just didn’t feel right. And knowing that this was an entirely different dev team, “not feeling right” felt like a betrayal. So while 3 probably isn’t a massive downgrade from 2 in an objective sense, that feeling of betrayal turned mild disappointment into HAAAAATE.
And then Rogue Trip came out, which was the new car combat game from Singletrac, and there was a collective “oh, this is what TM3 was supposed to feel like.” And that didn’t help matters.