You're really going to have to define your metrics for "tend to do really poorly over time".

Countries which prevent markets from operating efficiently tend to do really poorly over time. The more authoritarian, the worse they perform.

In general it means less government control over the markets. And less means libertarian concept (see article again). If you mean something in between , there is need in very detailed scale to find difference between current regulated markets, non regulated markets (libertarian nonsense) and balance that you want.

libertarianism... does not work

I get the point about anarchy (power vacuum arguments apply across implementations), but libertarianism is such a huge category that I'm not sure what you're getting at. Libertarianism isn't an economic system, there are socialist and capitalist extremes. It's also not a government structure, it houses both anarchists and bigger government ideas.

It's a philosophy that values the principles of individual liberty and non-aggression first and foremost, and everything else is discussed on those terms (I.e. how can we solve the problem with more liberty). There are different views about property rights, validity of certain types of taxes, etc, so you usually can't generalize unless you believe we need authoritarianism or something.

If you could be more specific, we could probably have a constructive conversation.