If your code isn't up to par, or your feature isn't relevant enough and doesn't fit "the vision", it's correct to deny it. On top of diluting the project contributed code add a maintainership cost that the random contributor will probably not be footing.

Accept everything in your cake and tomorrow it'll be an inedible mess that nobody wants. It's ok for software to be aimed at different people.