Of course you are NOT doing a bad thing. You are doing one of the tough things in parenthood (whether for a pet or a people child) -- letting them grow some independence. It is good for your Oliver to socialize and be away from you a little while. It's all good!
I know you will worry a little the first few times, even when you know your mom will take very good care. But think of how loving you both will be when you pick him up! A little absence makes the heart grow fonder is an old and true saying. He will be fine and so will you. Enjoy you night out and give lots of kisses when you go get him.