I am so sorry.
You are not over-reacting. And I feel the same way you do... I don't like to block Jackson off from the family, he's PART of the family, and if he's in HIS home, I will not lock him up somewhere for the sake of someone else.
I live with my much-younger-than-me sister. She is 3 1/2 now but her terrible 2's were difficult. Jackson and her really love each other and have a great bond but I still didn't trust her. If she got angry at something else, and Jackson happened to be sitting next to her, she'd swing at him or something and I'd have to be right there to tell her 'no!' or get Jackson out of the way. He's always been very understanding of her meltdowns and would put up from a lot with her but I still don't want him to have to, it's my job to protect him first and foremost. Whenever she wants to play with Jackson, I still supervise even though she's older now.