I know of thee reasons:

1. Affordable housing needs to be near services and work, otherwise transit costs make the housing unaffordable. Some affordable housing in Matagimi isn't helping, even if it's free. The attractiveness of those services and work make the locations valuable.

2. It's not just luxury and affordable a city is trying to achieve, they ideally want all ranges of housing affordability mixed together everywhere. This mixture reduces segregation and promotes positive socioeconomic outcomes.

3. The bottom up push for affordable housing (at least in Montréal) is coming from areas undergoing gentrification. So the citizen push isn't to stick affordable housing in very expensive areas, it's to not have affordable housing removed when the very expensive housing comes to them (Montréal examples of Verdun, Griffintown, and PSC). So your example of scraping one affordable unit to build two elsewhere still displaces an existing family/residents.