If his previous owner gave him milk daily with no problems etc,I think stopping now would be sad for him.Having said that,him pooping 3 times a day may be due to the milk,so maybe you can just give him half a bowl a day instead of what he is drinking now.
I know that my two only poop once a day and they are 2years old so 3 times a day some seem like alot for an adult.As long as its not diarrea though,I guess its ok
