I felt that way when he was younger. but I have to work a full time job as well. But that doesn't mean I don't spoil my baby. They need some time on their own as well. They need to be a little independent. I also have a cat and they keep each other company.
Maybe if your baby had a friend you wouldn't feel so bad..and I take my baby on every trip I go on. Everynow and then I take him to the kennel. He really loves it there. He comes home and can't even talk.. He just quacks like a duck..hehe.
Good luch