Splitting long input into multiple text files

Question

I have some code which will generate an infinite number of lines in output. So, I can't store those values in a single output file.

Instead, I split the output file into more files. I am splitting the file according to the index numbers. Now my doubt is I don't know how many numbers my file will be having. So is it possible to split the file into different output without giving index? For example:

• first 100,000 lines in m.txt
• from 100,001 to next 200,000 in n.txt

Answer 1

If you don't need to be able to find a particular line based on the file name, you can split the output based on the file size. Write lines to m1.txt until the next line will make it >1MB; then move to the next file - m2.txt.

Answer 2

Generate files with a running index. Start with opening e.g. m_000001.txt. Write a fixed nuber of lines to that file. Close file. Open next file, e.g. m_000002.txt, and continue.

Making sure that you don't overflow the disk is an housekeepting task to be done separately. Here one can think of backups, compression, file rotation and so on.

Answer 3

You may want to use logrotate for this purpose. It has a lot of options: check out the man page.

Here's the introduction of the man page:

"logrotate  is  designed to ease administration of systems that generate
large numbers of log files.  It allows automatic rotation, compression,
removal, and mailing of log files.  Each log file may be handled daily,
weekly, monthly, or when it grows too large."

Answer 4

split(1) appears to be exactly the tool for your job.

Answer 5

4 ways to split while writing:

A) Fixed no of characters (Size)

B) Fixed no of lines

C) Fixed Interval of time before writing

D) Fixed Counter of a function before calling a write

Based on those splitings, You can name the output file.