Did the little guy have a good vet experience when he had his shots? He could be blaming you for taking him, they do have some sulks about things like that.
The vet said he was ok so my guess is give him time to come round they can't resist human contact for long.
