I think keeping the harness and leash on is a great idea. I have done this because when chewy was a baby and my daughter Aubrey had him in her arms we had a few times where people would come close to pet him...(We thought) but they grabbed him right out of her arms instead!
We had to figure out what we could do about this!!!! So the leash works, we have a front pack to hold our babies in and when people come rushing in at us we step back and it sends a message to not come to close! People can pet our dogs when they ask.