Physical Danger
Diving can be an incredibly arduous and painful sport, even if you don't do a belly flop. Constant bounces on the board take a toll on your knees, back, neck, and joints, which can lead to chronic pain and injuries (source).
There's also the incredibly dangerous risk of hitting your head on the board, like famous Olympian Greg Louganis. Ouch. Luckily, that's rare.
Most likely, you'll get a painful strain or injury that requires some rest and relaxation, but most diving injuries tend to be fairly minor (source).
Of course, there's always the looming possibility of hurting your pride. Be careful out there.