Select columns of data from .txt to .csv

Question

hi there! I am quite new to python (well more like I've only been using it for the past week). My task seems fairly simple, yet I am struggling. I have several large text files each with many columns of data in them from different regions. I would like to take the data from one text file and extract only the columns of data that I need and write it into a new .csv file. Currently they are tab delimitated but I would like the output to be comma delimitated.

I have:

#YY  MM DD hh mm WVHT  SwH  SwP  WWH  WWP SwD WWD   MWD
#yr  mo dy hr mn    m    m  sec    m  sec  -  degT  degT
2010 07 16 17 00  0.5  0.5  5.0  0.3  4.0 SSE SSE   163
2010 07 16 16 00  0.6  0.5  5.9  0.3  3.8 SSE SSE   165
2010 07 16 15 00  0.5  0.5  6.7  0.3  3.6 SSE  SW   151
2010 07 16 14 00  0.6  0.5  5.6  0.3  3.8 SSE SSE   153

I only want to keep: DD, WVHT, and MWD

Thanks in advance, Harper

Answer 1

You need to format this question a little more legibly. :)

Take a look at the python csv module for writing your csv files from your now-stored data: http://docs.python.org/library/csv.html

EDIT: Here's some better, more concise code, based on comments + csv module:

import csv

csv_out = csv.writer(open('out.csv', 'w'), delimiter=',')

f = open('myfile.txt')
for line in f:
  vals = line.split('\t')
  # DD, WVHT, MWD
  csv_out.writerow(vals[2], vals[5], vals[12])
f.close()

Answer 2

One easy way to achieve this is by using the csv module in the standard library.

First, create a CSVReader and a CSVWriter object:

>>> import csv
>>> csv_in = csv.reader(open('eggs.txt', 'rb'), delimiter='\t')
>>> csv_out = csv.writer(open('spam.csv', 'w'), delimiter=',')

Then just put the information you want into the new csv file.

>>> for line in csv_in:
...    csv_out.writerow(line[2], line[5], line[-1])

Answer 3

One of the problems appears to be that all of your data is on a single line:

2010 07 16 17 00 0.5 0.5 5.0 0.3 4.0 SSE SSE 163 2010 07 16 16 00 0.6 0.5 5.9 0.3 3.8 SSE SSE 165 2010 07 16 15 00 0.5 0.5 6.7 0.3 3.6 SSE SW 151 2010 07 16 14 00 0.6 0.5 5.6 0.3 3.8 SSE SSE 153

If this is the case, you will need to split the input line up. If you know your data are regular, then you could be sneaky and split on the 2010:

f = open('data.txt')
for line in f:
    for portion in line.split(' 2010') #space is significant
    # write to csv

If your data span multiple years, then Python itertools module can be very handy. I often find myself using the grouper recipe.

import csv
from itertools import izip_longest

csv_writer = csv.writer(open('eggs.csv', 'wb'), delimiter=',')

def grouper(n, iterable, fillvalue=None):
  """
  >>> grouper(3, 'ABCDEFG', 'x')
  ['ABC', 'DEF', 'Gxx']
  """
  args = [iter(iterable)] * n
  return izip_longest(fillvalue=fillvalue, *args)

f = open('spam.txt')
for line in grouper(22, f.split('\t')): 
    csv_writer.writerow(line[2], line[12])

Answer 4

Here is a basic thing since it is a basic need and since there is no extensive use of csv, here's a snippet without the csv module.

DD = 2
WVHT = 5
MWD = 12
INPUT = "input.txt"
OUTPUT = "output.csv"

from os import linesep

def main():
    t = []
    fi = open(INPUT)
    fo = open(OUTPUT, "w")
    try:
        for line in fi.xreadlines():
            line = line.split()
            t.append("%s,%s,%s" %(line[DD], line[WVHT], line[MWD]))
        fo.writelines(linesep.join(t))
    finally:
        fi.close()
        fo.close()

if __name__ == "__main__":
    main()