LOL, I hear it all the time and to be honest, it doesn't bother me. He IS big for a yorkie. Very big. Almost 18lbs and shaped more like a Border Terrier or a JRT. I really love his size, personally, so if people don't like it well... whatevz.
To be honest I usually get compliments though. Like "Oh I know so-and-so has one and I feel like I can break it! I like his size, more sturdy".
I usually get the "Is he a ..... Yorkie?" question, and I just reply yes. Some think he has to be mixed with something, but he doesn't look like anything else (well, besides a Silky, which is so closely related, so yeah). I will sometimes say "Well he might be a Silky Terrier, but he was sold to me as a Yorkie". It's just hard to explain to people. But he's not mixed with anything else. He's just big.