How are the contours currently stored?
You might try first getting the contours as series of straight-line segments then attempt to fit smooth curves to portions of the straight line segment series. For example the 'one' would probably turn into an edge ring of 4 straight lines and 3 circular arcs.
You'll have to develop some heuristics as to what constitutes a 'good' fit.
But yes this is a pretty difficult problem. You should look up some research papers on character recognition. That's probably the best place to get ideas.
EDIT
I'll assume that you're only interested in closed contours. To extract a straight-line representation you can construct a geometric graph from the bitmap representation. Each pixel in the contour is a node and contour pixels from the bitmap that are touching (or within a specified tolerance) are connected by an edge. Each node retains the pixel's position. Extracting the maximal edge ring from each connected graph will give you the contours as straight lines.
Extracting the edge ring is fairly straightforward. Start with the farthest left node. Sort the connected edges clockwise around the node (where 12-o'clock is minimum). Take the last edge (e) from the sorted list. Add e to the ring you are building. Traverse the edge e to its connected node n and remove e from the graph. Repeat the sorting process at n where e is now the minimum for sorting the edges clock-wise. Select the last edge in the sorted list, add it to your ring, traverse it and remove it from the graph as you did for the last node. Repeat this process until you wrap back around to the starting node.
The edges in your ring are your closed contour. Because the pixels are so close together you'll have to post-process the ring to remove collinear edges and probably snap them to some larger grid to collapse points. But the general approach is sound.
From a geometric standpoint this is like walking around the outside of a large building keeping your left-hand against the wall. You don't know what the exact boundary of the building is but if you keep your left-hand to the wall you're guaranteed to traverse the entire (and only) outer boundary.
It has the benefit of being reasonably fast (basically linear with the number of pixels in the contour).
Just let me know if anything is unclear or you have any questions and I'll try to clear it up.