tdjmom has some great advice.
I have seen this similar situation with children. If they are taken from an abusive home at a young age, there is a lot of work to be done so that they can trust again. I'm glad he has you!
I would try to correct this as soon as possible. His distrust with males could become an aggression issue later.