I think your baby will be just fine!! It will be very good for him
Our yorkie, Miko, is my baby and my husband's baby, but he looooves to see his grandmothers and his aunts and uncles!! He used to spend a lot of time with them, not more than with us, of course, but he just gets so excited when he gets to see them. It's so cute!