No, I don't think you should and I certainly don't think you are selfish. A million people get dogs for company. Just because he likes to see/play/spend time with other dogs/people, doesn't mean he doesn't love spending time with you.
Is there a dog park in your area? or a meet up group?
Don't feel bad. And I really don't think you should feel like you have to rehome him. Hugs!
EDIT:
That's a great suggestion from Ashley!