Because a heading is a block element and therefore gets pushed to the next line (since a block-level element displaying as block (or without an otherwise specified display-type [inline-block, inline, etc...] gets) can't share a line; I don't know if a list-element should (or shouldn't), according to spec, contain block level elements inside itself, or if it should be presumed to be 'sharing' the line with the contained element.
I could be wrong, about all of this, though; it's just the only explanation I could think of.
You may, also, find that there are default margins, padding or positioning being applied to the heading elements in Firefox. You could test with Firebug (or alternatives) to see where the positioning is coming from.
edit
After copy-pasting your code into my template file, and viewing in FF3.0.10 I don't see the problem you report. The resulting code I used is pasted below, if you haven't already resolved this, try the code, below, and see if the problem persists:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title></title><link rel="stylesheet" type="text/css" href="css/stylesheet.css" />
<style type="text/css">
/* css reset */
h1, h2, h3, h4, h5, h6, a { display: block; }
/* list styling */
ul { width: 14em; margin: 1em auto;}
</style>
</head>
<body>
<ul>
<li>
<h3>Primary</h3>
<ul>
<li>
<h4>Secondary</h4>
<ul>
<li>
<h5>Tertiary</h5>
<ul>
<li><a href="#">Tertiary Link</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<ul>
</body>
</html>