More likely the downvotes come from users you’ve already blocked or an instance with which yours is not federated. Supposedly, there are also instances that only show local votes as well.
Regardless, you are the only one who has said they didn’t see any downvotes, so far.