Oh my goodness...yes of course it will REALLY harm your little one - my heart bleeds for him.
Can you imagine being a teensy little baby, and a great big 21 year old man comes looming over you, shouting and shoving you into a bucket of dirt? Absolutely DISGUSTING.
Your brother is a bully
Are your parents around? Is there a responsible adult about at all times, when you're not there?