Seg fault with open command when trying to open very large file

Question

I'm taking a networking class at school and am using C/GDB for the first time. Our assignment is to make a webserver that communicates with a client browser. I am well underway and can open files and send them to the client. Everything goes great till I open a very large file and then I seg fault. I'm not a pro at C/GDB so I'm sorry if that is causing me to ask silly questions and not be able to see the solution myself but when I looked at the dumped core I see my seg fault comes here:

if (-1 == (openfd = open(path, O_RDONLY)))

Specifically we are tasked with opening the file and the sending it to the client browser. My Algorithm goes:

1. Open/Error catch
2. Read the file into a buffer/Error catch
3. Send the file

We were also tasked with making sure that the server doesn't crash when SENDING very large files. But my problem seems to be with opening them. I can send all my smaller files just fine. The file in question is 29.5MB.

The whole algorithm is:

ssize_t send_file(int conn, char *path, int len, int blksize, char *mime) {
  int openfd; // File descriptor for file we open at path
  int temp; // Counter for the size of the file that we send
  char buffer[len]; // Buffer to read the file we are opening that is len big

  // Open the file
  if (-1 == (openfd = open(path, O_RDONLY))) {
    send_head(conn, "", 400, strlen(ERROR_400));
    (void) send(conn, ERROR_400, strlen(ERROR_400), 0);
    logwrite(stdout, CANT_OPEN);
    return -1;
  }

  // Read from file
  if (-1 == read(openfd, buffer, len)) {
    send_head(conn, "", 400, strlen(ERROR_400));
    (void) send(conn, ERROR_400, strlen(ERROR_400), 0);
    logwrite(stdout, CANT_OPEN);
    return -1;
  }
  (void) close(openfd);

  // Send the buffer now
  logwrite(stdout, SUC_REQ);
  send_head(conn, mime, 200, len);      
  send(conn, &buffer[0], len, 0);
  return len;
}

I dunno if it is just a fact that a I am Unix/C novice. Sorry if it is. =( But you're help is much appreciated.

Answer 1

It's possible I'm just misunderstanding what you meant in your question, but I feel I should point out that in general, it's a bad idea to try to read the entire file at once, in case you deal with something that's just too big for your memory to handle.

It's smarter to allocate a buffer of a specific size, say 8192 bytes (well, that's what I tend to do a lot, anyway), and just always read and send that much, as much as necessary, until your read() operation returns 0 (and no errno set) for end of stream.

Answer 2

Rather than using a variable length array, perhaps try allocated the memory using malloc.

char *buffer = malloc (len);

...

free (buffer);

I just did some simple tests on my system, and when I use variable length arrays of a big size (like the size you're having trouble with), I also get a SEGFAULT.

Answer 3

I suspect you have a stackoverflow (I should get bonus points for using that term on this site).

The problem is you are allocating the buffer for the entire file on the stack all at once. For larger files, this buffer is larger than the stack, and the next time you try to call a function (and thus put some parameters for it on the stack) the program crashes.

The crash appears at the open line because allocating the buffer on the stack doesn't actually write any memory, it just changes the stack pointer. When your call to open tries tow rite the parameters to the stack, the top of the stack is now overflown and this causes a crash.

The solution is as Platinum Azure or dreamlax suggest, read in the file little bits at a time or allocate your buffer on the heap will malloc or new.

Answer 4

You're allocating the buffer on the stack, and it's way too big.

When you allocate storage on the stack, all the compiler does is decrease the stack pointer enough to make that much room (this keeps stack variable allocation to constant time). It does not try to touch any of this stacked memory. Then, when you call open(), it tries to put the parameters on the stack and discovers it has overflowed the stack and dies.

You need to either operate on the file in chunks, memory-map it (mmap()), or malloc() storage. Also, path should be declared const char*.