I mean, what’s another way of easily conveying the same idea. “Those people who you might not typically consider to be X”?
I feel like I’ve seen and heard normies used in a variety of contexts, to refer to people outside of a particular group. Not to say they can’t be in the group, or that there’s anything wrong with them.