Oh, I agree....talk with him about your concerns and see where that takes you.
Let him know you can't really relax as much there because of worrying about what your babies will do. I'm sure he has an idea of how much more is involved for you to come to his place and maybe he doesn't mind coming to your place most of the time anyway.
Just be open and honest....and talk.
