If you want to get philosophical, society always has to turm authoritarian when it needs to deal with people who reject society. You either forcefully reject the ones who refuse to partake or you let them warp society in their image.
Being anti-vax is the same as being against society. Now, completely trying to get rid of them, like nazis tried with jews, is a bit extreme but penalizing antivaxxers IMO is perfectly fine. You can’t simply endanger others just because you refuse to partake in society.