Using the endianness of the CPU (no matter little or big) gives you the speed benefit on arithmetics: you can add, subtract etc. multibyte integers directly in memory.
Using a predefined, prescribed endianness (no matter little or big) in a file format gives you the benefit of being able to read the file on any system, no matter the endianness of the CPU of the other system. Systems with the right endianness can read the file faster (if the read routine is written and optimized properly), but even systems with the wrong endianness can read it. Usually, the speed difference is negligable (except for very large files with lots of integers), so it is a good idea to first measure the maximum possible speed gain of optimizing the read routine.
Some file formats (for example TIFF) support both endianness. In this case it is a good idea to generate the file with the CPU's endianness, assuming the file would be post-processed on the same machine, or a similar machine.