Sometimes it's scary having a pool. You can further protect yourself by putting up "NO TRESPASSING" signs on the fence, where they are most visible to those idiots that would climb over to get to your pool.
You know, when I was a teenager, we used to "pool hop" and never thought a thing about it
Of course, people weren't sue-happy back then, either
And we were respectful, we didn't leave any trash or make a mess, we just went for a quick, quiet dip and left.
Hopefully they won't be back!