I can't say for sure about your situation but my dog gets cranky if I disturb or try to carry him when he's sleeping / lying around on the carpet. On the other hand, he's always happy to be carried when in his playpen because he knows he's getting out for some play time
Perhaps what you could try is if he's food/toy motivated, you could bring a small treat/toy every time you approach him. If you do it consistently, hopefully he would not see you approaching as a sign of disturbance but as something rewarding to him.