NYC itself doesn't have much (although it still has some! see image below) low-density zoning, but their suburbs sure do. The city itself also has a lot of other bureaucratic barriers to development that result in it having abysmal housing construction rates.

As for vacancy, yes, the threat of not being able to sell is what stops builders from building too much. For example, it's the reason no one's even trying to build the Burj Khalifa in Bakersfield. But if you make it legal and reasonably easy to build, yes, people will build.

Perhaps Tokyo is the best example. Biggest city in the world, and yet it's actually relatively affordable, thanks largely to good land use policy:

In the past half century, by investing in transit and allowing development, the city has added more housing units than the total number of units in New York City. It has remained affordable by becoming the world’s largest city. It has become the world’s largest city by remaining affordable.

Two full-time workers earning Tokyo’s minimum wage can comfortably afford the average rent for a two-bedroom apartment in six of the city’s 23 wards. By contrast, two people working minimum-wage jobs cannot afford the average rent for a two-bedroom apartment in any of the 23 counties in the New York metropolitan area.

...

In Tokyo, by contrast, there is little public or subsidized housing. Instead, the government has focused on making it easy for developers to build. A national zoning law, for example, sharply limits the ability of local governments to impede development. Instead of allowing the people who live in a neighborhood to prevent others from living there, Japan has shifted decision-making to the representatives of the entire population, allowing a better balance between the interests of current residents and of everyone who might live in that place. Small apartment buildings can be built almost anywhere, and larger structures are allowed on a vast majority of urban land. Even in areas designated for offices, homes are permitted. After Tokyo’s office market crashed in the 1990s, developers started building apartments on land they had purchased for office buildings.

I think the key idea is to not have government bureaucrats or existing homeowners or landlords decide whether there's "enough" housing, but rather let builders determine if there's unmet demand. If there is unmet demand, they will build if you let them. If there truly is enough housing in a certain city, then you don't need to tell builders not to build -- they'll simply stop building if they sense there's not enough demand for it.