tags:

views:

108

answers:

3

I want to read 8byte chunks from a binary file at a time till I reach end of the file. Why doesn't this code work? What are the alternatives?

// read a file into memory
#include <iostream>
#include <fstream>
using namespace std;

int main () {

  long long int * buffer;

  ifstream is;
  is.open ("test.txt", ios::binary );

  // allocate memory:
  buffer = new long long int;

  // read data:
  while(!is.eof())
      is.read (buffer,sizeof(long long int));
  is.close();


  delete[] buffer;
  return 0;
}

If I replace all long long int with char, the code works perfectly.

Ref: code adapted from www.cplusplus.com

+3  A: 

The problem is !eof: it tells you whether the last operation hit eof, not whether the next will, and not whether the last failed!

Test the stream's truthiness itself (or use the fail method which is the same thing, negated), after doing the IO:

while (is.read(buffer, sizeof(long long int))) {
  // use buffer
}
assert(is.fail());  // must be true

Additionally, you don't have to use new at all:

long long buffer;
// ...
is.read(reinterpret_cast<char*>(&buffer), sizeof buffer)
  // the reinterpret_cast is needed in your original code too

Your source, cplusplus.com, also fails to check whether the read succeeded before using the data. In general, I find that site great for listing parameters, methods, etc. and horrible for most else.


Putting it all together, with some example output:

#include <climits>
#include <fstream>
#include <iomanip>
#include <iostream>

int main () {
  using namespace std;

  ifstream in ("test.txt", in.binary);
  cout << hex << setfill('0');
  for (long long buffer;
       in.read(reinterpret_cast<char*>(&buffer), sizeof buffer);)
  {
    cout << setw(sizeof buffer * CHAR_BIT / 4) << buffer << '\n';
  }

  return 0;
}
Roger Pate
Even if I comment out the While statement it gives me an error. Actually, it should have read atleast one 8byte chunk, doesn't it?
Sunil
@Sunil: How do you expect anyone to help you with errors if you never say what they are? "Doctor, somethings not right with my arm." "Okay, what?" "Somethings wrong with it."
GMan
Sunil
@Sunil: Yes, it is possible to use any size that fits within memory and type limits. The cast is because istreams work with chars, not long longs. Because long long is a POD type, we can do the cast and write directly into the int.
Roger Pate
So the datatype of the buffer would actually be a long long int and not a char[8]. Am I right?
Sunil
@Sunil: In my code, the type of buffer is long long, yes.
Roger Pate
@Roger: I tried your code using a 5GB binary file, i replaced the cout statement with i++ and initialized i = 0 at the beginning to count the number of iterations. It turned out to be exactly 28000000. so the total bytes read from the file is 28000000*8 = 224000000 bytes. Roughly 224MB.I don't get. why is this?
Sunil
@Sunil: I just tried it on a 1411776512 byte file, and it worked. (Boy, I'd forgotten how slow small reads can be.) You will miss bytes at the end of a file, if the file size is not a multiple of the size of a long long, but I can't see anything to explain the behavior you report.
Roger Pate
@Roger: There is nothing wrong with the file too. I created a purely binary file of 5GB using dd command in unix. I'm not sure why I'm not getting it.
Sunil
@Roger: I found the error. I removed the `cout << hex` statement and the output is fine. While printing out the `i` atlast the `hex` statement does something (I say something because I don't see how the value of `i` which originally is 671088640 gets conerted to 28000000). :)
Sunil
@Sunil: 0x28000000 == 671088640; std::hex modifies an ostream to output in hexadecimal (base 16). I used it to modify how buffer is output in my sample.
Roger Pate
@Roger: I've one more doubt if you could help me out. The line `for (long long buffer; in.read(reinterpret_cast<char*>()` reads 8byte chunks till it reaches the end of the file. How can I modify this so that I can read only a part of the file (say 500MB from 2GB file)?
Sunil
@Sunil: istreams have a seekg method to seek for "getting" (the 'g').
Roger Pate
@Roger: Thanks. I will look into it.
Sunil
@Roger: For now I implemented like this which is much simpler. Let me know what you think. `int i=0; for(long long int buffer;i<=sizeiwant;i=i+sizeof buffer) in.read(reinterpret_cast<char*>(`. Again thanks for your time.
Sunil
A: 
char buffer[8];
while (in.read(buffer,8)) {
    long long data = *(reinterpret_cast<long long*>(buffer));
    // process data
}

This is probably what you are looking for.

Alexander Rafferty
Local char arrays are not guaranteed to be correctly aligned.
Roger Pate
@Alexander: As Roger and you say using reinterpret_cast works well but what would be the datatype of the buffer after this operation?
Sunil
A union would be better than `reinterpret_cast` since it also guarantees alignment.
Ben Voigt
@Sunil: The buffer variable doesn't change at all, but with the cast we are "viewing" it differently. (@Alexander: I notice now you should change 8 to sizeof(long long).)
Roger Pate
@Ben: It's UB to read from a union member that wasn't the last one set, but yes, it works in practice.
Roger Pate
+1  A: 

Another way to do the buffer allocation is with a union. This removes the need for casting.

#include <iostream>
#include <fstream>
using namespace std;

int main ()
{
    union
    {
        long long int n;
        char ch[sizeof(long long int)];
    } buffer;

    ifstream is("test.txt", ios::binary);

    while (is.read(buffer.ch,sizeof(buffer)))
        cout << buffer.n << '\n';

    return 0;
}
Reinderien
I'm not familiar with union so correct me if I'm wrong. Doesn't the size of buffer here be 16? size of the variable n + size of ch[8].
Sunil
@Sunil: Read up on unions, they're class-types but different from other classes.
Roger Pate
It's UB to read from a union member that wasn't the last one set, but yes, it works in practice.
Roger Pate