Wow, there are lots of stories in here!
Sugar, you have to do what is best for you. If you can't have peace of mind with this little furboy around now, then find him a good home and let him go.
It's your life - no one else can live it for you even though we're all good at giving advice.

I wish you well whatever your decision.