views:

1657

answers:

8

I have a C application with many worker threads. It is essential that these do not block so where the worker threads need to write to a file on disk, I have them write to a circular buffer in memory, and then have a dedicated thread for writing that buffer to disk.

The worker threads do not block any more. The dedicated thread can safely block while writing to disk without affecting the worker threads (it does not hold a lock while writing to disk). My memory buffer is tuned to be sufficiently large that the writer thread can keep up.

This all works great. My question is, how do I implement something similar for stdout?

I could macro printf() to write into a memory buffer, but I don't have control over all the code that might write to stdout (some of it is in third-party libraries).

Thoughts? NickB

+1  A: 

You can "redirect" stdout into file using freopen().

man freopen says:

The freopen() function opens the file whose name is the string pointed to by path and associates the stream pointed to by stream with it. The original stream (if it exists) is closed. The mode argument is used just as in the fopen() function. The primary use of the freopen() function is to change the file associated with a standard text stream (stderr, stdin, or stdout).

This file well could be a pipe - worker threads will write to that pipe and writer thread will listen.

qrdl
This does not solve my problem. I am trying to move the disk writing off of the thread making the printf() call. Using freopen() would still have my printf() calls writing to a file, albeit a different file to stdout. Is it possible for me to specify a "file" to freopen() which is not a disk file?
NickB
Sure. Use pipe instead of file.
qrdl
A: 

You can change how buffering works with setvbuf() or setbuf(). There's a description here: http://publications.gbdirect.co.uk/c_book/chapter9/input_and_output.html.

[Edit]

stdout really is a FILE*. If the existing code works with FILE*s, I don't see what prevents it from working with stdout.

Bastien Léonard
A: 

Why don't you wrap your entire application in another? Basically, what you want is a smart cat that copies stdin to stdout, buffering as necessary. Then use standard stdin/stdout redirection. This can be done without modifying your current application at all.

~MSalters/# YourCurrentApp | bufcat
MSalters
+1  A: 

If you're working with the GNU libc, you might use memory streams.

You can call me Chuck
A: 

One solution ( for both things your doing ) would be to use a gathering write via writev.

Each thread could for example sprintf into a iovec buffer and then pass the iovec pointers to the writer thread and have it simply call writev with stdout.

Here is an example of using writev from Advanced Unix Programming

Under Windows you would use WSAsend for similar functionality.

Robert S. Barnes
+2  A: 

I like the idea of using freopen. You might also be able to redirect stdout to a pipe using dup and dup2, and then use read to grab data from the pipe.

Something like so:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#define MAX_LEN 40

int main( int argc, char *argv[] ) {
  char buffer[MAX_LEN+1] = {0};
  int out_pipe[2];
  int saved_stdout;

  saved_stdout = dup(STDOUT_FILENO);  /* save stdout for display later */

  if( pipe(out_pipe) != 0 ) {          /* make a pipe */
    exit(1);
  }

  dup2(out_pipe[1], STDOUT_FILENO);   /* redirect stdout to the pipe */
  close(out_pipe[1]);

  /* anything sent to printf should now go down the pipe */
  printf("ceci n'est pas une pipe");
  fflush(stdout);

  read(out_pipe[0], buffer, MAX_LEN); /* read from pipe into buffer */

  dup2(saved_stdout, STDOUT_FILENO);  /* reconnect stdout for testing */
  printf("read: %s\n", buffer);

  return 0;
}
Nate Kohl
This sounds like it solves my problem. I'll try it out. Thank you!
NickB
A: 

w/r/t Bastien Léonard:

is this approach correct:

size_t bufSize = 4096; /* big enough? */

char bigbuf[bufSize];

char cpybuf[bufsize];

sprintf(cpybuf,"dummy blank data to be overwritten");

if( !setvbuf(stdout, bigbuf, _IOBFB, bufSize) ){

printf("test line sent to standard out\n");

strncyp(cpybuf, bugbuf, bufsize);

} else {

fprintf(stderr,"Something awful must have happened!);

}

fprintf(stderr."Here's what was sent to stdout: %s",cpybuf);

Do I understand correctly that we can thus capture stdout into a buffer, our only concern being that we copy the data before bufSize characters have been written to stdout, and the buffer flushed?

Thanks!

A: 

The method using the 4096 bigbuf will only sort of work. I've tried this code, and while it does successfully capture stdout into the buffer, it's unusable in a real world case. You have no way of knowing how long the captured output is, so no way of knowing when to terminate the string '\0'. If you try to use the buffer you get 4000 characters of garbage spit out if you had successfully captured 96 characters of stdout output.

In my application, I'm using a perl interpreter in the C program. I have no idea how much output is going to be spit out of what ever document is thrown at the C program, and hence the code above would never allow me to cleanly print that output out anywhere.

Bob