Not to discount the experiences of others in any way--I have heard many people say that their Yorkies have a stronger attachment to one person than another--but your pup is so young, I'd hesitate to think that any particular preferences he shows now are necessarily permanent. Two of my three have been with me since they were infants and they have gone through so many phases of likes and dislikes--even food tastes can change suddenly. Just like there are times in a Yorkie's development when fears may be more likely to manifest, there are bound to be changes in other feelings as well. One of my Yorkies has "issues" but the other two have grown to be just as eager to be with either me or my spouse--15 months old and a 5 yr old recent rescue.
Don't give up! Even 
human babies can be temporarily a one-parent child. 
