Aw! How sweet! I had similar problems with my Peek A Boo, he wasn't from a puppy mill, but he had too many homes and if he bonded with anyone during that time, he would be moved and any bonds were broken. It took him 8 months to begin to trust me, and now at 15 years old, he's still not really into toys, but will bite at my hands under the blanket and get all worked up... Sometimes it just takes alot of time and patience, but even the smallest things can be major breakthroughs...and so heartwarming.
Best of Luck with your baby...
