The main issue that seems to be the disconnect is simply that you are using the word as an insult.
Calling someone unintelligent is fine (I feel at least) but specifically using words that are associated with disabled people as an insult towards abled people is blatantly putting one group on top of the other.
The insult in that case cuts both ways. The person you are directly talking to is insulted because they are acting like a “lower group” and you’re also insulting the entire group by saying they are inherently lesser.
You can not use a group of people as an insult without inherently viewing that group as insult worthy.