Absolutely. And I say this as someone who's turned down many lucrative opportunities to "follow my heart". If you're going to do it, don't do it for the money. Turning down a mid-six-figure offer to start a billion dollar company is most certainly an outlier, on both sides.
If you're going to turn down money in hand for something speculative in your own life, do it knowing how much time, money, and opportunity you can afford to lose, and do it because it's something you really believe in. Even if you fail, you can land on your feet with an experience you'll never forget.
Just don't do it out of some crazy notion that destiny rewards the reckless, which is where the survivorship bias comes into play.
http://en.wikipedia.org/wiki/Survivorship_bias