I think you've gotten some really good advice here. It is a more common issue than I would have thought. For me one of the Perks of living in another country is this. If anyone says something rude to me...I don't understand it! haha

My German is very limited so unless I understand a word or two, I generally just smile and say (in german) " I'm sorry i only speak a little german i didnt understand" I have yet to have an English rude comment though so we'll see...