Lemmy is small enough that “brigading” doesn’t feel entirely appropriate. Maybe “platooning?”
In any case, we know from other sites that downvotes increase the probability of getting more downvotes, and nasty comments increase the probability of getting more nasty comments. The same goes for upvotes and positive comments. It’s just social dynamics. Some subs on reddit existed almost exclusively to call out other subs, but I think that Lemmy’s user base is small and spread out enough that it’s not a major contributing factor in voting.
I think it’s mostly just people scrolling along, running across a hot take, and interpreting it according to the voting.
In any case, I would suspect that people would be more impacted by hurtful comments than downvotes.