Three weeks isn't very long. I would have your husband take him outside and spend some time with him. Perhaps he could even take him for a walk or a car ride with just the two of them. That is, if your husband is willing.
He will learn that good things happen when he is with dad.
As for the children, I wouldn't rush this. He will warm up to them if they aren't too rambunctious around him.