Yes, performance fees were, and still are a thing. That's why there are separate streaming services for playing background music in e.g. restaurants and malls, which have licences that cover that use case. You can't just use e.g. Spotify even if you pay the appropriate fees separately, at least where I'm from (Finland). Same goes for playing TV and radio in the background – if it's specifically as background noise or e.g. TV at a bar you typically need separate licencing on top of the one used by the channel itself for distribution. Might be different in the US for example, but copyright laws tend to be quite similar across countries (unfortunately).

The smooth jazz in malls part is likely a lot due to the (historically) overwhelming market share of the company Muzak. They were the pioneer of background music in commercial environments and got a strong enough presence in the market that their name ended up a generic one. AFAIK they're still a large player in the business.

Mall music basically ended up as its own genre due to Muzak and how big it became, and started influencing other media as well. As an example, the style was referred to in the soundtrack of the first Sims game in the buy mode, where it brought with it the feeling of rampant consumerism and buzz of purchasing new stuff. At least according to this video essay on the subject, which in my mind makes its point quite convincingly. (the relevant part from 11:46 onwards, but the video is altogether quite interesting)