I wouldn't say it's a pejorative term, it just sounds weird to use it for talking about the whole language (language spectrum?) when a) the name just means "Fujian (province)" but Fujian has like six different non mutually intelligible languages but the name is only used for one of them and b) this name for the language is pretty much only used in singapore, malaysia and philippines

so it's weird to talk about Southern Fujianese people or Taiwanese people speaking "Hokkien", they would say Min-nan/Bannam(?) or Taiyu/Daigi respectively. wikipedia also refers to the language as "southern min" in english which sounds better.

i guess it's kinda similar to saying "español" instead of "castellano", but imagine that castilian, galician and catalan were all equally national languages, and only overseas castilian speakers called castilian "spanish", and overseas castilian speakers made up less than 50% of speakers worldwide