What's the address that the prefetch abort is triggering on. It can occur because the program counter (PC or R15) is being set to an address that isn't valid on your microcontroller (this can happen even if you're not using an MMU - the microcontroller's address space likely has 'holes' in it that will trigger the prefetch abort). It could also occur if you try to prefetch an address that would be improperly aligned, but I think this dpends on the microcontroller implementation (the ARM ARM lists the behavior as 'UPREDICTABLE').
Is the CPU actually in Abort mode? If it's executing the Prefetch handler but isn't in abort mode that would mean that some code is branching through the prefetch abort vector, generally through address 0x0000000c but controllers often allow the vector addresses to be remapped.