Absolutely correct. A true average can be either good or bad on the whole depending on context. For example, an American's average standard of living is pretty good compared to the North Korean's average standard of living. So, when most people say "average" are they, on average, talking about things that are good on the whole or things that are bad? My guess would be a bias towards things that are good on the whole.