The pointer returned by malloc() points directly to the memory on the heap that will be used by your program.
However, this isn't the only memory that's allocated. A few bytes are allocated in the memory locations immediately preceding the pointer returned that indicate the size of the chunk on the heap. This isn't used by your program, but it will definitely be needed by free.
When free(p) is called, the information about its chunk on the heap is contained in, say, the locations from p-4 through p-1. This depends on implementation of course, but the details need not concern the programmer. The only thing that the programmer needs to know is that free uses that area of memory to free the chunk of memory from the heap, and that area is derived from the original pointer p.
In other words, if you call free on p, it will only make sense if malloc once returned exactly p.
If you pass in a pointer that wasn't created with malloc, who knows what will lie at p-1, p-2, etc.? It will probably result in a catastrophic failure.