Stress
Considering that everyone in your family probably had to sacrifice their own social lives and any scrap of luxury they ever wanted to pay for your training, if you fail, or get a crippling injury that takes you out of the sport before achieving fame, you'll never be able to pay them back and it'll all be for nothing.
That's kind of stressful.
If you're going to manage the stress in figure skating, you have to be okay with failing. A lot. Each time you fall, your nerves may become a little more fragile, and your resolve a little less strong. The pressure to make the next jump will increase. It'll be a daily struggle, and you'll have to face constant setbacks and still keep up your confidence that you can be the best. No biggie.
Even Olympians have off days. Except when they screw up, everyone's watching. Everyone. And big bucks are on the line. And the pride of their country may be at stake.
That's enough to make anyone forget how to land a double axel mid-air.