It's said to illustrate that pointers merely store addresses, and that addresses may be thought as numbers, much like integers. But usually addresses have a structure (like, page number, offset within page, etc).
You should not take that by word. An integer literally stores a number, which you can add, subtract etc. But which you cannot use as a pointer. An integer is an integer, and a pointer is a pointer. They serve different purposes.
Sometimes, a cast from a pointer to an integer may be necessary (for whatever purposes - maybe in a OS kernel to do some address arithmetic). Then you may cast the pointer to such an integer type, previously figuring out whether your compiler guarantees correct sizes and preserves values. But if you want to dereference, you have to cast back to a pointer type.