I've gotten into the habit of picking up Kaji to allow people to pet him when asked. That way, he's securely in my arms and people can pet him all they way. It also helps because Kaji hates people (he's terrified) and I can easily flip him around so he's facing me and the stranger can pet his back and not his face.
I think that's less scary for him when they're not touching his face.