Welcome to YT! That poor baby has been through it.
. Four different families has to be rough. I'm sure he does have trusting issues. Once he knows that you are going to love him and not mistreat him I'm sure it will get better. My Mikki hated men because of her past. Now she is my husbands dog! Be patient with him. You may even have to look at some training. Right now I would just love him and let him know he has his new furever home! I am so glad you have him!