Stress
The hours are long, the deadlines are tight, the work is demanding, and some of the egos involved can be unbearable. For the most part, though, you're doing what you love to do and you're making magic happen, which makes up for a lot of the icky parts.
Some of the most stressful moments can arise out of differences of opinion with the director. The director has the final say on just about all the major decisions, but remember: you're an artist, not a laborer.
You can argue your case, but you're probably going to have to give in when you meet resistance. If the move works and the film is a huge success despite the difference of opinions, both of you will be able to ride off together into the sunset.
But it'll more likely be fairly stressful and not at all like sailing off to a happy ending.