He is really soooo cute! When you get a dog that has had another home, I think sometimes you really have to retrain some things they do. The previous owner's situation may have been different and they may not have put a leash on them and maybe let them out into a back yard to go potty. I personally don't take ours out without a leash because they are so small.  I don't want a big bird to pick them up!! Does he know how to sit on command? If so, maybe he'll calm down after you work with him a while and know what you want.  If not, its a good place to start training.
I noticed you are in Locust Grove, Ga.  I am close by in Hampton, Ga. 
Good Luck!!