It sounds like your baby is just too young to leave his Mother, IMO the Breeder should keep him until he is at least 12 or even 15 weeks!
I bought Artie off of the Internet and he was 20 weeks old by the time I received him b/c he was sold/transfered 2 times before I had purchased him, for whatever reason or maybe he was returned? But I am Thankful that he was that much older.....he was much easier to Train and seems more Stable physically & mentally! Abigail on the other hand I purchased @ 7 weeks old and I too had a $350.00 Vet./Hospital bill for a "Hypoglycemia" siezure, but I never thought to give her back to the Breeder b/c I was so in Love with her that I felt obligated and determined to make it work! I had a Litter of 3 puppies 3 weeks ago w/c is a product of my committment to her! Thank goodness for Nutrical its a cure all! I learned an expensive Lesson and as a result of that I am keeping her puppies for a minimum of 12 weeks and the smallest Baby/Runt for 15 weeks just in case? But I would search your Heart and follow your gut feeling? Sometimes when people decide to get a Yorkie, they want it so bad that they don't do enough Research, they are so impatient that they take the first available one they come across? Sometimes it could take up to a year to find the right one. My Sister met Artie 2 yrs. ago and had been waiting for me to have a Litter, last May her Husband bought her a Yorkie for Mother's Day as a surprise! Of course she was thrilled but to this day she regrets not waiting another 4 months for mine, afterall whats another 4 months compared to 2 yrs.? And guess what she asked me for her Birthday present (on 9/28)? Let us know what you decide? Sorry I'm no help I'm a Sucker, I'll always cheer for the Underdog it's just more gratifying in the end, Aloha & Good Luck!