I think you have the legal right to prohibit flying above your property. Do it, make a sign and report everyone who violates it to the police. Flying on someone’s property with a camera drone is illegal anyways.
EDIT: Also check if you legally can physically eliminate the drone in that case. It’s the most effective strategy.