"Man" is a gender neutral term, as it was the original use (from proto-germanic "mann"). Previously, wer and wif were used to differentiate between sex. Using "man" to refer to male gendered individuals is based on context.
Inclusive language is important, which means not creating new exclusions for terms that are already neutral.