What happens if I disable an interrupt inside of its ISR?

Question

What happens if you disable an interrupt inside that interrupt's ISR?
For example, if I am transmitting data over USART from a buffer and that buffer runs out of data then I want to stop transmitting temporarily, so after sending the last byte in the buffer, I disable the interrupt.

(This is on a PIC18F4580)

The datasheet for the PIC18F4580 states that "erratic behaviour may occur" if an interrupt is disabled inside of an ISR.

Answer 1

The interrupt stops happening. But why do this? And do you have a plan for when you're going to re-enable the interrupt?

Just have your ISR do nothing if there is nothing to do.

Answer 2

I don't know specifically about the PIC18F4580, but as a generaly rule,
there is nothing wrong with disabling an interrupt while in its ISR

Essentially this will prevent the ISR to be called again (i.e. until the interrupt is somehow re-enabled), but it shouldn't affect the logic of the ISR, cause it to terminate or other odd thing one may thing about.
Indeed, may ISRs will typically start by disabling interrupts lest they would get interrupted while servicing a first interrupt (and hence avoiding various re-entrancy issues, at the cost of possibly missing events).

Of course you need to plan on how/where the interrupt would be re-enabled, if that is important to the rest of your program/logic.

Answer 3

Disclaimer: Speaking from PIC32 and dsPIC experience here combined with what I read from the PIC18F4580 datasheet and errata (PIC18F4580 References).

According to the PIC18F4580 datasheet's Interrupt section:

Do not use the MOVFF instruction to modify any of the Interrupt Control registers while any interrupt is enabled. Doing so may cause erratic microcontroller behavior.

So to get that unknown behaviour you need to modify an INTCONx register. As you specified that you wanted to toggle the interrupt enable for the USART module inside the handler, you shouldn't have to worry about that warning as you will be changing PIE and not INTCONx. Note that the PIE register holds enables for specific peripheral interrupts. There is also a general interrupt enable (GIE) that masks ALL maskable interrupts in one shot. You don't want to be messing with this one inside the interrupt as it should be handled for you automatically by the handler's entry/exit procedure (also its in register INTCON).

So like mjv said, masking the USART peripheral interrupt inside its handler is a perfectly reasonable thing to do.