Some old x86 instructions are undefined in 64-bit mode. For instance LDS, LES and LSS, or short opcodes of the INC r16 (40 + rw) and INC r32 (40 + rd) instructions.
Are there any instructions that are defined only in 64-bit mode, and not in 32-bit protected mode?
Edit: The context is development of an x86 processor. I want to make sure I'm compatible to the spec.
There is an addressing mode, which has no counterpart in 32 bit: Instruction pointer relative addressing. In 32 bit mode, this is only available with branches (aka branch offset), in 64 bit mode you can form effective addresses for any memory operand relative to the (r)ip.
So, the instruction
mov rbx,$100[rip]
does exist in x86-64, while
mov ebx,$100[eip]
does not in x86.
You can always look into the different manuals at http://www.intel.com/products/processor/index.htm?iid=dev%5Fcenter+tech%5Fhw%5Fdesign+prods+processors to find such instructions. Will be a lot of work tough ;)
Setting aside old instructions that now operate on 64 bit registers (they are really new instructions as they use a distinct binary representation), I can think of at least two instructions that are 64-bit only: syscall and sysret instructions.
the opposite is much easier to come up with;) like cmpxchg8b .... you'd never use that one in 64-bit mode.