Honestly (and this is the point I thought you were going to make) there are more Nikki Haley voters than leftists in this country. I think nazis/stormfront/proud boys are too many, but not enough to influence an election.
The issue I’m seeing with Biden not giving concessions to leftists and progressives is that they’re mainly the ones who have boots on the ground helping Biden get elected. Without politically active people who believe they can influence Biden helping him win, I’m not seeing any hope for his chances. The secondary effect is that once you lose leftists and progressives, they’re VERY vocal online about why they aren’t happy, which also hurts Biden’s chances. So leftists and progressives might not be as numerous as Nikki Haley voters, but they still have influence over how people perceive this administration. I’m not sure if that effect is even quantifiable, or what matters more between the Nikki haley voters who choose to vote for him or the progressives/leftists who inadvertently cause apathy. Who knows. But if he cares about winning (which is still debatable as far as I’m concerned), he’d start giving concessions fast.