Geez, what a mess! I went through something similar with my grandparents years ago. They had owned their property for YEARS, then a new neighbor moved in and parked their trailer right on my grandparents' land. It was a nightmare to get it all straightened out, and their neighbors were much like yours. In the end, my grandparents won and YOU will win too because you're in the right.
I don't know how your deed is written, but I was amazed at the one my grandparents had. It was OLD, they had lived there forever. So, it was like X many feet to the oak tree....well, the oak tree wasn't there anymore. But, there was a pin in place.
Make sure they actually had a real surveyor there because my grandparents' neighbors tried to fake one lol.
Good luck, I hope you get it all straightened out! Try not to let them get to you. That's what they want, so don't give in.