This statement has several problems:
p=(int (*(*p)[b])[a])malloc(b*sizeof(int (*p)[a]));
First, malloc returns a void*. You are casting that pointer using (int (*(*p)[b])[a]) which yields a value, not a data type. That isn't a valid cast, so that's one reason that the compiler is yelling at you. At this point, p hasn't been initialized so the de-referencing taking place here can crash your program if this statement was executed.
Inside your malloc call, you are using sizeof(int (*p)[a]). The statement int (*p)[a] isn't a valid C statement.
It seems that you are making this a bit more complex that it needs to be. There are two ways of building a 2D array. You can build an array using malloc(a * b * sizeof(int)) as Reinderien explains. You can also build a 1D array of pointers, each pointing to an array of type int. From your code, it seems you are trying to do the latter.
The easier way to do this would be something like this:
int **p;
... get input from user ...
// Declare an array of int pointers of length b
p = malloc(b * sizeof(int*));
// For each int* in 'p' ...
for (i = 0; i < b; ++i) {
// ... allocate an int array of length 'a' and store a pointer in 'p[i]' ..
p[i] = malloc(a * sizeof(int));
// ... and fill in that array using data from the user
printf("\t\bEnter Column %d\t\n");
for(j = 0; j < a; j++)
scanf("%d", &p[i][j]);
}
Using this method of building a 2D array allows you to use the syntax p[x][y]. Since p is a pointer-to-pointer, p[x] is a pointer to an array and p[x][y] is an item in the pointed-to array.