Aww well I don't blame you for being upset. I would have a hissy fit if my b/f did that I was taking care of HIS daughter and household while he goes out to have a ball.
I wouldn't back down if I was you. Relationships are about compromise and by the sound of this particular incident, you have more than compromised by letting him go without you, and merely asking him to come back and spend time with you and his daughter.