Have you tried just holding him when you go outside and stroking him to keep him calm? Then if he stays quiet for about 5 minutes go inside give him cuddles and come back out again and try for a longer period of time. I know its probably not going to be the best answer but i think it might help him
