I think he'll be fine, it's more stress if you leave him with the little brat. You'll have a crap time on vacation worrying about him and who knows what could happen to him with that child.
If he's at home, you know he'll be well rested when you get back and have double the Yorkie kisses.