Aww... Poor little guy
You gotta remember ... yorkies are small
Everybody wants to come over and pet him and pick him up
and say "How cute he is"
Thos big hands coming at him from all directions and lifting him up can get to be scary
For now tell everyone who comes to him they need to come into a room where he is ... act like he's not there and take a seat
Let HIM go to them and when he does YOU praise him and tell him what a good boy he is and THEN your inlistee can touch him and play with him
once he gets the idea that people are not scary and good things happen when new people are around he'll be fine