I would definitely do some allergy testing before bringing a pup home. Each Yorkie is different, just like people, you may be allergic to one Yorkie and not another. I would strongly suggest you do more than just visit. Take the pup to your cleaned out car and sit and hold him for an hour or so, take a hair sample home for a couple days etc....I'd hate to see you take a pup you might not be able to keep.
My YorkieDaddy is highly allergic and yes he is allergic to our Yorkie, but she gets baths weekly, and he is medicated daily