I always bring my pups and foster pups home after neuter/spay surgery. I will not leave them in an unattended office. I do not feel like I am stepping on my vet's toes...nor should you! If you feel comfortable bringing him home...and will rest better...I say bring him home!
I am sure he will be fine....there really is not much to a neuter on a male and they recover quickly!
