Welcome to YT!
I can relate with your worry and concern. I'm a full-time working Yorkie mommy.
I would have to agree with your breeder. I brought home my babies at 5 months old. Personally, I totally would not have wanted them any sooner. At 5 months, I still had to coordinate someone to come over for an afternoon potty break so that they were only crated at a maximum of 4 hours at a time for their potty training period. It's crucial to have them relieved; otherwise, if they have accidents, it's nearly impossible to get them potty trained. Even at 5 months, it was not easy! Accidents are not only frustrating, but it risks letting them know accidents are okay, and making it harder for them to learn.
Adopting retired show dogs from reputable breeders are an
excellent idea. Note that you'll still have to go through a potty training period regardless what the age, because when a dog goes to a new home, they will have to be retrained. Plus side is, being an adult dog, you know they have the ability to hold in their potty with proper training measures.
Like you, I had initially looked into rescue organizations, but not only were there no hyper-allergenic small dogs available at all, I also found that many have health issues.
I would keep looking for reputable responsible breeders and let them know of your interest in getting a retired show dog. It does take time and patience to get that perfect baby, so don't rush it. If you make a good impression, you never know. A good breeder may keep you in mind and retire one earlier, knowing that their baby will go to someone caring and responsible. Sometimes it's worth it even if you have to drive a little further to another state.
As mine are both adults now at 1.5 years and 3 years, they are able to stay at home during my 8-hour work days. Prior to letting them have free-access everywhere, I spent a lot of time training them to prevent separation anxiety so they won't bark or terrorize my home while I'm away. I also monitored them when I first let them roam around freely. Hehe... Everything was a very slow transition. It's all part of being the worrying and super cautious mommy!
But I do believe that the more cautious you are and the more preventative measures you take, the safer and happier your babies will be.