It's your baby , take responsibility and have it home with you. I don't care how defensive the mom is, I would be more defensive. your baby does NOT belong at your grandparents house, it belongs with you and take responsibility .
if something happens you will regret it so do the right thing now.
Good Luck
