Well, it’s not all your Matrix data, but if you don’t trust Discord with writing an app that runs client-side, I’m not sure why it’s helpful to trust them with holding onto your conversions with other Discord users either…
I’ve also run a Matrix server and I can tell you from experience… You shouldn’t trust me with your conversations. Even if I was a good friend, I’m definitely not a security professional!