They are entirely different words. No one is calling for an outright ban on those letters; that’s a sentiment you made up.
Don’t use it as a slur. If you are using the word in another, legitimate context where it’s not a slur, I don’t give a fuck. But stop arguing that those two uses are somehow indistinguishable because that’s just not true.
Edit:
Unlike the F-slur, N-word, and all the other colorful terms…
This is false. Examples: “cracker,” the b-word, the f-slur (in UK contexts), “queer,” “gay.”
All of these have other legitimate meanings. So, please reconsider your defense of this specific term, because you’re not even arguing it based on facts.