I personally wouldn't risk it. I don't know where you are located, but here in South, TX it is WAY too hot to leave my baby outside. And even though we have a fenced backyard, you never know what is going to get back there and possibly get your baby
Last night there was a huge POSSUM!!!! Thankfully, my other dog was out there and killed it (she is a bigger dog), but that thing was WAY bigger than Stedman and it could have gotten him if my other dog hadn't of gotten IT FIRST
Stedman has an x pen that we put him in any time we have to leave him. If I were you, I would do the same...however, I can understand you not wanting to leave him enclosed in a little pen for hours during the day. Sooo, if I were you, I'd get a baby gate and either put your put in the bathroom with the baby gate blocking it off so that he can't get out, or maybe in the hall or kitchen closed off so that he can't roam the rest of the house. Good luck!