I'm not trying to be rude at all, so please don't take it that way. If this situation is getting to the point where you are starting to resent your Yorkie...why don't you give him to your girlfriend? It sounds like they get along great!
Honestly, if you're the second man to be spanking or swatting at this poor little guy, I think that it will be hard for you to win him over. If you do decide to keep him, I hope that you find alot more patience. Yorkies can be the BEST little dogs ever, if you treat them right.
P.S. It's great that you care enough about this to come here and ask for advice.