Well, your asking for opinions so I will give mine
First off, I think when both of you have cooled off you should sit down together and in calm manner discuss the problem, both yours with him, and him with you. Calmly...with true listening on each side, not just waiting for your turn to tell your part, but to listen and HEAR, both of you. Without this, I think you will get nowhere fast. This may be an easy fix or it may take lots of work on both of your parts...the question is...how hard do you want to work and how important is your marriage to you? Only you each know the answer to that question.
For my part, I think people say "I'm leaving" or "I'm done" too fast. That is why the high divorce rate these days in my opinion. I'm not knocking you at all, I'm speaking in a general sense. People think "I'm so mad, he's such a so and so and I am leaving" and they don't try to work on it. When my husband and I don't agree or just flat out don't like each other that day, we never say, "I want a divorce, or I am leaving" and we don't think it either. Always there is love even when there is no "like" that day or days
If you love someone you owe it to them and yourself to try and work things out...it might not always be enough and sometimes you SHould get out while the gettins good, but don't let leaving always be your first thought. Okay, that is my soapbox for the day. I hope if You want to work it out you do, or if leaving is the best thing for you, I wish you all the best!
BTW, I will have been married 23 years in just a few days (Sunday) is our anniversary, and I may not always "like" him, but I will LOVE him forever!! (Sorry for the sap
)