I had hubby drop Hot Rod off the day of his neuter but I felt just as bad when he went out the door at home. At least I knew that the staff at the vets office were totally in love with Hot Rod and that he would be in really good hands!
I know what you mean about cats being independent, I don't remember going through this when they each got spayed (Maybe I was just a whole lot younger and not as sentimental-mental being the operative word)

.
I just want to hold onto Hot Rod because he seems so small and fragile and more dependent than the cats ever were. So I can totally related to what you are going through. Don't worry, he'll be fine.
I did, however drive like a nut when I was able to pick him up in the afternoon and all was well! He'll be fine!