I wouldn't take offense to it after all your brother doesn't love Oliver like you do but I would try to work on Oliver's issues. I really don't personally believe in spraying them in the face. Personally, I think that will make things worse. It sort of confirms what he already feels, when men are around, something unpleasant is going to happen to him so even though I know you don't mean to, you are reinforcing his bad behavior. I have been studying SATZ training for over a year now and you would be amazed at home much your dog can understand and learn with this method. I would highly recommend that you check it out as Oliver sounds like a prime candidate for this! Here is a link that has more info, I hope I see you there!
SATZ_Main : Safe_Alternative_Teaching_Zone