I'm a bit "damaged goods" when it comes to this subject. I used to design and maintain fairly large APIs for embedded telecom. A context where you cannot take anything for granted. Not even things like global variables or TLS. Sometimes even heap buffers show up that actually are addressed ROM memory.
Hence, if you're looking for a "lowest common denominator", you might also want to think about what language constructs are available in your target environment (the compiler is likely to accept anything within standard C, but if something is unsupported the linker will say no).
Having said that, I would always go for alternative 1. Partly because (as others have pointed out), you should never allocate memory for the user directly (an indirect approach is explained further down). Even if the user is guaranteed to work with pure and plain C, they still might for instance use their own customized memory management API for tracking leaks, diagnostic logging etc. Support for strategies like that is commonly appreciated.
Error communication is one of the most important things when dealing with an API. Since the user probably have distinct ways to handle errors in his code, you should be as consistent as possible about this communication throughout the API. The user should be able to wrap error handling towards your API in a consistent way and with minimum code. I would generally always recommend using clear enum codes or defines/typedefs. I personally prefer typedef:ed enums:
typedef enum {
RESULT_ONE,
RESULT_TWO
} RESULT;
..because it provides type/assignment safety.
Having a get-last-error function is also nice (requires central storage however), I personally use it solely for providing extra information about an already recognized error.
The verbosity of alternative 1 can be limited by making simple compounds like this:
struct Buffer
{
unsigned long size;
char* data;
};
Then your api might look better:
ERROR_CODE func( params... , Buffer* outBuffer );
This strategy also opens up for more elaborate mechanisms. Say for instance you MUST be able to allocate memory for the user (e.g. if you need to resize the buffer), then you can provide an indirect approach to this:
struct Buffer
{
unsigned long size;
char* data;
void* (*allocator_callback)( unsigned long size );
void (*free_callback)( void* p );
};
Ofcourse, the style of such constructs is always open for serious debate.
Good Luck!