I don't know if you've ever considered home schooling but that's what we do and it's worked out famously for all of us! Just graduated my oldest with a 4.9 as a matter of fact. #2 has a full scholarship to Duke and #3 is 6 yrs old, working at about a 10 yr old level. All are very well rounded, smart, kind people and yes, they have tons of friends and have more of a social life than I ever had lol.
Homeschooling is not what it used to be, it's SO much better. You might wanna have a look. Those feelings you have are natural and worth listening to.
Good luck with whatever your choice is. I wish you and your lil one all the best.