What you're trying to do is "chain" the alerts.
To do this you need to call orderOut:
on the alert window.
Here's the documentation:
If you want to dismiss the sheet from
within the alertDidEndSelector method
before the modal delegate carries out
an action in response to the return
value, send orderOut: (NSWindow) to
the window object obtained by sending
window to the alert argument. This
allows you to chain sheets, for
example, by dismissing one sheet
before showing the next from within
the alertDidEndSelector method. Note
that you should be careful not to call
orderOut: on the sheet from elsewhere
in your program before the
alertDidEndSelector method is invoked.