There are a few options:
- Throw out the outliers
- Filter
- Use a better GPS
- Use an external data source (snap to road)
- Combination of the above
I like to use filters - A Kalman filter is the typical (and often best) solution - it uses an amount of predictive averaging which is better than a cheap IIR (Infinite Impulse Response) filter:
FilteredValue = FilteredValue * 0.75 + NewValue * 0.25
You can get GPS modules which give you 4-5 fixes per second, which will allow you to use the above 'cheap' filter with reasonable response times.
You can also simply get a better GPS (SiRF III or better) that isn't as noisy and has better indoor reception (where possible).
Consumer GPS units "snap to road" where possible, so errors off the road are not seen by the consumer, as well as a few of the other techniques.
A Kalman isn't easy to implement, but without an external dataset or sensor (such as road speed), it's the best option. Check out http://www.google.com/search?q=open%20source%20kalman%20filter for code and tutorials on it.