I say go for it hun! You're really not moving too far away, only 3 hours away from where you are now. It wouldn't be that far away to go visit your family for the weekend. Plus if he will be making that much more money and have top notch benefits, you can totally afford to drive down to visit your family!
I think it would be great for you, your hubby, and your baby. Research the towns around his work and find a great safe city (with a great school district) to move to. I mean what's the worst that could happen? It doesn't work out and you move back home where you live now, right? I think you will regret it if you don't take the plunge. Good luck sweetie!