I share your frustration. A couple of my neighbors are very intrusive. I am very quiet, low key, private, and hard working, especially when it comes to the house and yard. Apparently this is very aggravating to some people.
Be polite, but be assertive in establishing boundaries. Like you, I dislike confrontation and awkward situations. I wish people would simply behave appropriately. The worst is when these people ignore subtle signals to back off. I'm still working on being blunt.
Hope you can work things out for the better. Maybe team up with the other neighbor for moral support.