Intent doesn’t really matter if the entire butt of the joke is that this person is asking like a “lesser” group.
Even if intent did matter… How would a passive observer be able to tell if the person meant it or not? You really can’t without knowing both parties involved, their history together, and if that person regularly does similar styles of joke.
It’s easier to just pick a different word to insult someone’s intelligence instead of one that’s designed to hurt a specific group of people.