There are several variations, but when building an octree, one approach is this:
- Start with one node (ie. a cube) encompassing your entire scene or object being partitioned.
- For each element in your scene/object (eg. a mesh, or a poly, or whatever granularity you're working to):
- Check if that element fits completely inside the node.
- If yes, subdivide the node into eight children, then recursively do Step 2 for each child.
- If no, then continue to the next node until there are no nodes left.
- Add the element to the smallest node that can contain it.
It's also common to stop recursion based on some heuristic, like if the size of the nodes or the number of elements within the node is smaller than a certain threshold.
See this Flipcode tutorial for some more details about building the octree.
Once you have the octree, there are several approaches you can take to render it. The basic idea is that if you can't "see" a node, then you can't see its children either, so everything inside that node (and its children) don't need to be rendered.
Frustum culling is easy to implement, and does the "can you see it?" test using your view projection's frustum. Gamedev.net has an article discussing frustum culling and some other approaches.
You can also go further and implement occlusion culling after frustum culling, which will let you skip rendering any nodes which are covered up by nodes in front of them, using the z-buffer to determine if a node is hidden. This involves being able to traverse your octree nodes from closest to furthest. This technique is discussed in this Gamasutra article.