Agreed. You could make that argument perhaps for German civilians. And maybe even some of the Wehrmacht at a stretch. But the SS answered directly to the Nazi party and not the Army. They were the enforcers of Nazi ideology across Europe, and most were committed to Nazism. They knew what they were doing and carried out war crimes with enthusiasm.
Anyone who defends the SS in any shape or form is a very bad person.