A lot of people don't realize that Yorkies can differ greatly in looks/size/colouring. They have a picture in their head of what a Yorkie "should" look like and they think that all Yorkies look like that. I met a lady who was talking about her neighbor who had 3 Yorkies that all looked completely different. She seemed really suprised/shocked and didn't understand why the Yorkies didn't all look the same. If you look at dogs like German Sheppards or Labs/Retreivers, they generally all have the same look and don't vary very much. A lot of people think that Yorkies are like that, too.
People who don't think your baby is a Yorkie, are people who clearly don't know anything about them. Yet, ironically enough, they probably act like they know more than you do