Was he a rescue? This just sounds very abnormal for a Yorkie. Every Yorkie that I've met is very outgoing and friendly. Are you sure nothing had happened to him in his past? Maybe from a man in the house? One of your gf's ex's? That's the only thing I can think of which would make him dislike men. Oh, and don't take it personally. I'm sure he wants to like you, he's just had something happen to him and he's afraid. If you're more caring to him and prove your love to him and truly like him (I know you're not crazy about him now) then I'd hope he'd warm up to you. It's not the dog's fault he's like this. He has a disorder. I'm sure you wouldn't "not like" a friend or family member if they had a disorder. Right? I mean, coming onto this forum is proof enough to me that you really care, and I commend you for that. I hope you learn to like the dog even if he never 100% opens up to you. Give it time, keep trying, and don't give up.
