They're definitely fragile, but you'll feel better about it once you establish rules and learn your puppies personality. Some of my rules are that you can't hold Billi unless you're sitting down and she's in your lap (she wiggles furiously when she's excited), kids aren't allowed to hold her unless they're sitting AND their parent or I have to be right next to them.
She's not allowed in the backyard without the gate being closed and me present, she's not allowed out front without being on her harness/leash.
Thankfully Billi knows her limitations and doesn't try to jump off things. She uses her ramps that we made for her and she doesn't push herself too much. She fell off the couch once a couple days after we brought her home (about 9 weeks old) and it scared the daylights out of her. She wasn't hurt, but that was all it took and we can trust her not to jump now.
Yorkies are a lot like children and you have to take extra precautions, because even though they're dogs, they're small dogs and there's always a risk of injury and theft. The smaller your yorkie is, the more cautious you'll find yourself being. Not to make you totally paranoid. The older they get, the wiser they are, but just keep an eye out for them and make it easy for them to choose to be safe.