A great big huge thanks to all of you that responded! I feel so encouraged and much less guilty about the way I was feeling, actually I think I just need to close up my office and go home right now and play with him! LOL But I have another 4 hours to go....
Just reading all the comments and stories helps confirm that I was a bit panicked, like someone said on the way home I was thinking "what have I done???!!!" But looking back it's easy to see it was a crazy day; he was picked up early morning, went on errands, had a 3 hour drive and was late arriving so we had waited in the parking lot almost 3 hours for them and then he looked so pitiful...I think all of it combined just overwhelmed me a bit. Not to mention how he must have felt.
He did wonderfully through the night. It will take us a few days to get a new routine going in the mornings - getting them to pee and poop at the same time is a must, so I'm not making 4-6 trips outside before work! That was a little hectic, and then 2 poop accidents to clean up because I missed their signals when they finally DID decide they needed to go! LOL And there is a winter storm coming - I am sure THAT will be fun!
We're having our 2 daughters, SIL's and 3 grandkids over for my grandson's birthday tonight, so everyone will get to meet him. My daughter is very compassionate with animals and LOVES to rescue, so I am sure I will be convinced that he's staying by the time she leaves!