If you really love him I would give it another try but in my opinion is have the breeder have a bile acid test done to this pup to rule out livershunt at 12 weeks old. Its a fairly inexpensive test to have done
http://glassyorkies.homestead.com/livershunt.html
Good luck with your new pup, if you decide to give him another try. Maybe because hes so little he should stay with mom a bit longer