I think you are going to get a wide range of answers on this.
My first thought is, do you have a dog already? If so, what is it's personality? If it is outgoing and active, this may not be a good fit for your pack. When I picked out Gracie Ruth, I initially was drawn to her sister. But Bella was more reserved and quiet and I had Boo to think about. He's active and loves to play. I really thought that Bella would get too frustrated with Boo's attention. Gracie is very active and was just going to be a better fit for our pack. She's turning into a really special girl.
Similar thing with Yogi. His brother Pablo is timid and Yogi is more outgoing, yet the last time we all got together for a playdate with some other friends, it was Pablo being the outgoing one and playing with the new visitors. Yogi just wanted to hide behind his Momma.
And there's no telling how much of this pups personality right now, is due to him always being picked on. He might blossom into an amazing fellow once he gets away and you help him get some confidence.
I'd try to go see him again, maybe even several times. Try interacting with him in another room, were it's just the 2 of you. Good luck. Let us know what you decide.