Just implement whatever you find interesting. Over at ompf.org you can find links to data sets to render however you like. The site is dedicated to ray tracing, but we aim to do that in real time too. Or define some simple cubes or spheres and render those. Use OpenGL, build your own rasterizer, do ray tracing, radiosity, photon mapping, whatever you find most interesting. If you're stuck at square one because your imagination is way out there and you're at the bottom of the learning curve, then pick something simple you think you've got your head around and do that first just to verify you understand it. Parts will be reusable as you build ever more complex software.
And if that's not specific enough, here is a series of tasks:
1) write code to open a window and draw a square in it.
2) modify #1 to draw a 3d box using OpenGL or whatever method interests you.
3) introduce a loop so you can update the image at some fixed frame rate.
4) rotate the box at that fixed frame rate.
5) get into shading and loading some interesting scenes.
Then just go where you feel the need.
EDIT Since you've asked for math problems to solve.
This reflects my background, but try deriving the equations for ray/object intersection distance for a number of primitives. The results can be compared to solutions on the net, or possibly in the book (I haven't read it).
1) Ray/Plane intersection
2) Ray/Sphere intersection
3) Ray/Triangle intersection
4) Ray/Tetrahedron intersection
Also, create a library with vectors (3 and 4) and matrix operations (4x4 and 3x3). While this is a programming problem, you can use it to numerically verify anything you derive mathematically.
Aside from that, if you want actual problem sets with answers, pick up a college level introductory text on linear algebra. That is the basis for everything in 3d and physics.