You have a few options:
1) Magic error value. Not always good enough, for the reason you describe. I guess in theory for this case you could return a NaN, but I don't recommend it.
2) Define that it is not valid to pop when the stack is empty. Then your code either just assumes it's non-empty (and goes undefined if it is), or asserts.
3) Change the signature of the function so that you can indicate success or failure:
int pop(double *dptr)
{
if(sp > 0) {
*dptr = val[--sp];
return 0;
} else {
return 1;
}
}
Document it as "If successful, returns 0 and writes the value to the location pointed to by dptr. On failure, returns a non-zero value."
Optionally, you could use the return value or errno
to indicate the reason for failure, although for this particular example there is only one reason.
4) Pass an "exception" object into every function by pointer, and write a value to it on failure. Caller then checks it or not according to how they use the return value. This is a lot like using "errno", but without it being a thread-wide value.
5) As others have said, implement exceptions with setjmp/longjmp. It's doable, but requires either passing an extra parameter everywhere (the target of the longjmp to perform on failure), or else hiding it in globals. It also makes typical C-style resource handling a nightmare, because you can't call anything that might jump out past your stack level if you're holding a resource which you're responsible for freeing.