Why on earth would you want to write your own java GUI toolkit? You already have the choice of Swing and SWT, can you really do better than these two teams?
If you've already written the rest of the toolkit, I don't understand why the scrollbar would stump you. Without knowing anything about your event system, or how your custom components are structured, it's impossible to give much advise. I don't see this being particularly maths intensive - just maintain the height of the scrollable component, and the view it's in, and the scrollbar size should match the proportion of the component that is visible. The position of the scrollbar should match which part of the component is visible (this will have to be scaled). Specifically, what do you want to know?