My husband is about 6'4''. Our Zoey girl is about 4 1/2 lbs. It is quite funny to watch him walk her. He doesn't care. He knows who he is and he loves Zoey. He always meets other guys out walking their "small" breed doggies and they are all either married or have gf's. They have a quick giggle, the dogs play, and that's it. They go on with their walk.
Besides, your pup has an aggressive, rough and tumble typical boy personality it seems. He sounds entertaining and fun! I would keep him.