I would bring him home even if he is a little groggy, which is normal anyway, because you can watch him and he will be much more comfortable. I've had many neutered while fostering, and only a couple were sent home with meds and that was because of a few teeth being extracted. He will do just fine, so don't worry so much. I know, easier said than done.
