Far from an expert (I'm just a first time yorkie owner!), but my little Zoey is similar. She doesn't back away when we try to touch her, but sometimes, she will back up when others try to touch her...even other people that she knows well! She is also very very stubborn. She will usually come if I use my stern, I'm-not-playing-and-may-get-mad-soon voice. But, my goodness, if she sees a squirrel or a rabbit, she doesn't hear anything!
Now, Zoey is a girl, but somehow, she has the same "delicate" problem...she has one particular toy dog...it has slowed down over the years (she's 3years), but when she's bored, she'll still go at it!