The next time you go there, don't take them. He'll probably ask why you didn't bring the dogs. Tell him the truth (nicely) and he'll most likely either (A) say "ok" which will let you know that he really didn't like having them over OR (B) he'll say he really doesn't mind that you bring them and tell you to bring them next time and maybe he'll stop complaining all the time (maybe he really likes them but doesn't want to admit it

) See what happens.