I think you should just give Rusty all the love and attention you can whenever he is at your house. That may be all the positive attention he gets. I agree with your husband, I definitely wouldn't take him home.
I think by them saying they were just going to leave him out, they sort abandoned responsibility for him. The next time they came over, I would just ask if I could have him.
I hope that they will see that it is better for Rusty that he live with your family.
