Keeley does that too!
When he falls asleep on me and I want him to stay and snuggle, I feel like I have to be super careful not to "bug" him, because he will totally ditch me.
How funny, I always thought that it was just my stinker, I am glad it isn't anything personal.