I totally understand! My ex & I used to go our seperate ways on holidays

kind of stunk bc then we were not together... Im with a new guy now and we are trying to make it work so we can go to both our families houses. His for brunch and mine for lunch! You should offer to host it, that way you can invite BOTH families to your house and then you dont have to choose. When I lived with my ex we had a huge house, so we hosted Easter one year with like 20 people from his family and mine. It was fun and like the first holiday we really spent together! (10 years together crazy huh?)
Yeah if I were you Id suck it up and go! Sounds like hes done it for you a few times! It will all work out nicely Im sure, you'll be happy to make him happy too!