Not to question how you handle him but maybe something inadvertantly happened when you first blow-dryed him or brushed him that scared him and now he can't forget it. I don't bother blowdrying my yorkie at all anymore and she is about 3.5 lbs. She is very good with brushing and when I brush her out after a bath, she mostly does well but it was all a matter of working up to it. She was exactly how it sounds like your little one is acting when I first started bathing, brushing, and blowdrying.
When I first started bathing her, I started out with a little tiny tub and filled it with water and let her stand in the water so that the sound of the running water wasn't overwhelming for her and would just bathe her like that for her first several baths. I recently purchased a neat little thing called Rapidbath and you can bath dogs this size in under 3 minutes and graduated her up to that. She stands there the whole time I spray her down. Its soap and water all at once.
When i blow-dryed her, I would wrap her in a towel and have her on the counter and hold her against my body and blowdry her and she would mostly stay against me but blow-drying was scary for her and so I don't bother with it nemore since brushing her seems to help her dry off quickly. I can imagine being that tiny and having this loud hot sound blowing all over your delicate skin and hair. What I noticed when blow-drying her was that she would burn really fast since her skin is so thin and so is her hair. I think brushing them out and letting them run around for a few hours is sufficient. I think the heat drys their hair out too much neways.
Brushing her took me a long time to work up to it but I would let her feel out the brush when I first started and gave her treats as I worked. I would make the sessions short since I know she disliked it but now she has fully accepted it 3 months later.
As you said, time is really of the essence to train them to accept grooming. If you start out really young they will learn to accept it if not enjoy it. Make it fun and rewarding and I'm sure he'll grow to like it and its really good bonding time too. It takes patience and gentle hands! Good luck!
