I don’t think it destroys art as a form of communication any more than the possibility of being misunderstood over texting or even in person destroys those media.
The chance for miscommunication exists in every form of communication, it’s the consequence of letting an idea out of your own head and into the world. And art is inherently less clear a method for communication than something more straightforward would be.
If you create a work that nazis can see a bit of their worldview in, congratulations! They see their worldview in the world, so you’ve created a decently accurate facsimile of reality. Shitty people seeing their own shitty ideas in your art doesn’t say anything about you, y says something about them. The same “death of the author” that lets them have that take insulated the author from that take.
But the reason I like it is that it also allows decent people to come to decent conclusions about art made by shitty people. Even if I didn’t like it, I know it exists. Art can speak to someone about experiences the author didn’t imagine, and that can be powerful and significant and beautiful, even if it can also be shitty.