I had Biscuit neutered at around the same age. He was a bit over six pounds. We didn't do it earlier because we were struggling with the idea and were a bit worried, but Biscuit was completely fixated on mounting everything that came in his path. He also wanted to go after every dog in the neighborhood even if he was inside the house. Months of trying to get to him to relax were not working out.
He is much more at ease now and hasn't gained any significant weight or had any problems with his health. I struggled with putting him through surgery and I definitely think your worries are justified. I don't think you should feel forced to neuter your dog if you feel that it would do more harm than good. Each situation is different and you should weigh your pros and cons.
Good luck with everything
