Without knowing additional context about your application, there could be a couple of things going on. The first that comes to mind is that you have a very "heavy" visual tree. Each item in your ListBox is going to create some graph of visual objects. The more you have, the longer it takes to render.
When those items are not visible on the screen, Silverlight is still doing the work to render them. Basically this means that the Layout() and Measure() pass are being performed on all of your data-bound ListBox items, even if they aren't visible.
The trick is to use virtualization to force Silverlight into not performing the Layout/Measure passes on controls that are not visible within the control's visible region at the time.
Here is a blog post that describes how to do this with a StackPanel. It shouldn't take much to abstract this to a ListBox or better yet, you could simply make the control container for your ListBox a virtualizing stack panel and then you'd be done :)
http://blogs.msdn.com/uberdemo/archive/2009/02/18/a-virtualizedstackpanel-control-in-silverlight-2-0.aspx