I've been on both sides of the fence too - I've delt with rude cart riders and now my husband has to ride a cart in large stores like Walmart. He is not allowed to do that much walking due to a severe heart and lung problem. But trust me, he looks perfectly fine - no outward signs of impairment at all. He's only 60 so he doesn't even look old and decrepit!
When he needs something off the shelf - he just gets off the cart and gets it. That is not a problem for him and he is allowed to do that much physical exertion. Believe me though, he gets so many disgusted looks from people who probably think he is perfectly fine. Often I walk beside him in the cart and we talk so that people know we are together. I think they are more tolerant because they realize he would be walking with me, if he could.
If he does not use the cart, he has to use oxygen, and that really bothers him because people do stare a lot if you have a hose up your nose! LOL!
When he first started using the cart, he was so tired of people acting like he shouldn't be using it, that he wore his oxygen while in the cart. Then people just looked so pitifully at him - and he did look pretty pitiful - that he doesn't do that anymore!
Anyway, I agree that there are rude people on both sides of the fence and it is difficult to know what to do or how to react.