Forcing people to host speech they don't want to is far more draconian than not doing so.
You'd probably be more than a little annoyed if I put a swastika sign on your front yard and then told you that you were infringing on my right to free speech when you went to go remove it.