I'd be scared too.
Any possibility of you guys installing an alarm system? They're not too expensive, sometimes they are free is you purchase a monitoring package.
I'd call everyone you know and ask them to drive by the house whenever they can, pull into the driveway maybe and walk around your property. This would make that neighbor think twice about setting foot on your property.