The issue is, though you may make a distinction between “I’m using this slur as an insult and not against its targeted oppressed minority”, bigots make no such distinction. Hearing others use the slur and normalize it emboldens these bigots to use it against vulnerable minorities, backing up to “I didn’t mean it that way” when they get called out. The word’s legacy also tangles with a fair bit of racism, as children of minority races were often labeled “mentally retarded” for poor English skills or just so they could be shuffled out of class after school segregation was ended. It’s just a word, yes, but one with a lot of ugly history in the US at the very least.
Plus, the dislike of the word really isn’t new, it just has more support these days. We have lots of other words to choose from, what’s the harm in avoiding this one?