I actually think he would be more comfortable at his own house than at some strange house around rowdy kids! Whenever my parents go out of town, they leave their yorkie at home and a dog sitter comes over 3-4 times a day to check on her and let her out and feed her and such. Obviously you have the dog sitter part taken care of, so I think he should be fine! Leave some dirty clothes of yours out, like worn tshirts, socks, shorts, and stuff, so he can lay on them and get the sense that maybe you're there. Make sure when the people come over to check on him that they maybe take him for a walk once a day and stuff. At least it is only for a weekend. I think he'll be fine