Well, when I left my first job is was when I woke up and realized I was making minimum wage writing legacy applications when I could learn more, put my skills to better use outside the company and make more money doing it. We didn't have source control, and our development systems were consistently down or our favourite editor wasn't able to run on them.
When I left my first, real, salaried position it was more of a location question. Where I was, there were no other technology companies. Zilch, nada, none. I knew that to stay competitive I had to move, and it helped that my wife wanted to move. ;) To add to that though, I was out growing the team. I needed to learn more, and that just wasn't in the companies' budget or on its radar. With production servers grinding to a halt and version control getting messy it was time to move on.
Both of those companies I highly respect, and strongly believe they will do very well in the years to come. Growing pains come with the small company lifestyle, but so do good relationships.
Disclaimer: I no longer work for a small company, and the last time I wrote my own freelance work was at least a year ago. I'm quite content at the moment learning everything I can from .NET to Oracle and it looks like I'll be sticking around for quite awhile, despite that urge to create my own start up and live like Kevin Rose.
As far as making the right decision goes, I would have to say that I definitely missed the fast paced 'do or die' environment for awhile. I also quit just before the first company I mentioned sold their start up to Yahoo which hurt a little. Still, in the end it was the best thing I could do for my schooling and career at the time.
The second time around, well, I wouldn't be where I am without that company but I also wouldn't be here if I hadn't left. I learned a ton working with them, and that really helped when I moved to where I am now. Was it the best decision? When I find out, I'll let you know. :)