In terms of performance... it's not important. Readability of the code is king, a tiny fraction of a percent of performance difference isn't going to change anything.
The raw HTML version is usually the easiest to read (and probably has the best performance too for what it's worth - but what it's worth is: nothing). This is no surprise: PHP is an HTML templating language, the whole point is interleaving HTML at a language syntax level.
Look at nickf's code to see how to keep it readable. Indenting is important! Put a level of indenting inside each PHP control structure too, so you can keep track of them. eg.:
<?php if ($error) { ?>
<p> Oh no, error! </p>
<?php } ?>
Finally, when outputting content, such as $container in your example, you must always htmlspecialchars() it, or you'll have an application full of HTML-injection security holes, like every other PHP newbie (and even many professional developers, sadly). This matters whichever method you use to output content.
Because htmlspecialchars is quite an annoyingly long function name, you could try defining your own shortcut function:
<?php
function h($s) {
echo(htmlspecialchars($s, ENT_QUOTES));
}
?>
<ul>
<?php foreach ($things as $thing) { ?>
<li> <?php h($thing['name']) ?> </li>
<?php } ?>
</ul>