History disagrees with your opinion. A large part of the population does not vote in any election, and far right citizens are committed to making their voice heard in every one. When left-wing voters withhold their votes, the Overton window naturally shifts right.

Electors are generally bound to the local popular vote. Candidates can absolutely win the election without winning the popular vote overall though.