If I read correctly, the reason for the blow up sounds really petty. He probably can't control how horribly he behaves, or realize what he was actually saying. I would at least approach them about visitation rights for Toby.
Side note: when I first read the title it sounded like you were 11 and were grounded and can't see your dog
