I don't have much advice as I am in the same "boat" as you. You are not being silly, I am feeling the same way about my vet. We have been seeing her for 15+ years. She was great with my Dalmatian and she is good with our cats and my Lab/ Retriever, Mazey but I feel I need to find someone who deals with Yorkies more regularly...especially one as small as my Lulu Belle (she is 3.5 #).
Good luck in your search...hope you find someone you and your baby are happy with.