I think DH is being rude to you, to expect this is OK with you.
I think he needs a reality check. If it were me and my DH expected to have his EX around me no matter what the reason,he would soon have another EX to deal with, I have 2 EXs and both times I was the one who kicked them to the curb.

I am very happy today
He should be the one to say NO! This situation is not being respectful to you
Some men are slow, I would tell him one time exactly how you feel. Pretending to be OK with you will only escalate the problem
Deana
Prestigeous