The main advantage of a 64-bit CPU is the ability to have 64-bit pointer types that allow virtual address ranges greater than 4GB in size. On a 32-bit CPU, the pointer size is (typically) 32 bits wide, allowing a pointer to refer to one of 2^32 (4,294,967,296) discrete addresses. This allows a program to make a data structure in memory up to 4GB in size and resolve any data item in it by simply de-referencing a pointer. Reality is slightly more complex than this, but for the purposes of this discussion it's a good enough view.
A 64-bit CPU has 64-bit pointer types that can refer to any address with a space with 2^64 (18,446,744,073,709,551,616) discrete addresses, or 16 Exabytes. A process on a CPU like this can (theoretically) construct and logically address any part of a data structure up to 16 Exabytes in size by simply de-referencing a pointer (looking up data at an address held in the pointer).
This allows a process on a 64-bit CPU to work with a larger set of data (constrained by physical memory) than a process on a 32 bit CPU could. From the point of view of most users of 64-bit systems, the principal advantage is the ability for applications to work with larger data sets in memory.
Aside from that, you may get a native 64-bit integer type. A 64 bit integer makes arithmetic or logical operations using 64 bit types such as C's long long
faster than one implemented as two 32-bit operations. Floating point arithmetic is unlikely to be significantly affected, as FPU's on most modern 32-bit CPU's natively support 64-bit double
floating point types.
Any other performance advantages or enhanced feature sets are a function of specific chip implementations, rather than something inherent to a system having a 64 bit ALU.