What you described is a prejudice which is not great. This prejudice, like all prejudices, can be mitigated through exposure. Because of the stigmitization which you described this exposure has additional barriers than just population size. You’re right that this is enough of a problem.
What I’m talking about is weaponizing this prejudice through politics. There has been a major push to pin “the fall of western society” on “trans ideology.” Idiots may repeat these moronic talking points to one another, but they are not the ones who wrote those talking points for them to parrot mindlessly. In fact, these talking points are derived from a movement which has existed for just over a century.
One of the reasons fascism is so attractive to many people is that it unites the “proper” people against the enemy which is simulaneously corrupt and foolish but also masterminds who will undermine and destroy the culture unless they are stopped. Because these foes exist, the state must exercise greater social control and violence to manage the “menace.” Because this population is small and stigmatized (by coincidence, relatively about the size of the Jewish population in 30’s germany), the state can establish the apparatuses for social control and violence. Because these mechanisms are ostensibly to use against just trans people, idiotic bigots support them and a far greater number of people simply don’t care. The thing is though, that from a fascist politician’s point of view the excuse for creating these authoritarian devices is not important really. The point is to establish an authoritarian fascist government. Any defenseless group can be demonized and persecucted, they just have to pick a group that the population will let them persecute. As long as the greater population doesn’t mind what they do to the scapegoat they can build up a system which can eventually be deployed against any threat to fascist power.