I don't think I remember reading in your post (and if you did post this I'm sorry for asking it) but how long have you had the dog for? It sounds to me like he is WAY better off with you. If you've already had him for a while, I would say that your sorry but that you two made an agreement and you've already bonded with the dog and he has already bonded with you. I mean I understand that it was her dog and all, but it doesn't sound like he was being very well taken care of and I would be extremely concerned about that.
I would do everything in your power to keep that sweet baby