This has happened to me twice. The first time I panicked, gasped, and said "Omg, you're holding my dog! I don't let anyone hold my dog", and I grabbed her back. An extreme reaction, I know...but I kinda freaked out.
Perfect Response lol!!! Let them think you're the one who can't handle them holding your dog, so they won't get offended...And who cares anyway, it's our babies and we make the rules for them; we don't make the rules for the rude people, they make them for themselves....