How to dispose of resources in iTextSharp append method on exception

Question

I have the following method using the iTextSharp library which should process and move the source document to a target document (which may or may not already exist). There is an append flag which will append the contents if the document already exists.

I'm throwing an exception if append isn't set, but I'm having some problems working out how to clean up all the resources.

public void CopyToDocument(string destinationFile, string sourceFile, bool append)
{
    string uniqueName = Path.ChangeExtension(Guid.NewGuid().ToString(), ".pdf");
    string tempFile = Path.Combine(Path.GetDirectoryName(destinationFile), uniqueName);

    Document document = new Document();

    var fs = new FileStream(tempFile, FileMode.Create);
    PdfWriter writer = PdfWriter.GetInstance(document, fs);

    document.Open();

    PdfContentByte content = writer.DirectContent;

    if (File.Exists(destinationFile))
    {
        if (!append)
        {
            fs.Dispose();
            File.Delete(tempFile);
            throw new InvalidOperationException(" /* whatever... */");
        }

        //... etc

The line File.Delete(tempFile); throws an exception here saying the file is in use by another process.

Might this be because the file isn't released immediately by fs.Dispose()?

The only other thing I can see is that it may be in use by the PdfWriter object, but that doesn't implement Dispose() and trying to call Close() on it throws a document has no pages error.

Any clues how to resolve this?