It's always better to be cautious...I'm glad it wasn't some scary person in a van. I'm always a little lerie of vans (no offense to anyone who owns a van) but they are always the cars that the crazy serial killers drive in the movies. So I don't blame you one bit for turning and running home. I would have done the same thing.
I'm always super cautious leaving my baby in the car, most of the time he comes with me but there is the occasion when it isn't appropriate for him to come with me.