Are the people approaching him correctly? Have them kneel down, and scratch his chest, rather than patting him on the head. They should also avoid eye contact. I would suggest letting him get used to that before progressing to him being picked up.
Also, if he likes the squeaky toy, definitely bring it with you and give it to people who want to befriend him.
Rather than letting a stranger reach out for him, let the person squeak the toy and wait for your baby to approach.