CSV File Imports in .Net

Question

I realize this is a newbie question, but I'm looking for a simple solution - it seems like there should be one.

What's the best way to import a CSV file into a strongly-typed data structure? Again simple = better.

Thanks.

Answer 1

Check out FileHelpers.

Answer 2

A good simple way to do it is to open the file, and read each line into an array, linked list, data-structure-of-your-choice. Be careful about handling the first line though.

This may be over your head, but there seems to be a direct way to access them as well using a connection string.

Why not try using python instead of c# or vb? It has a nice CSV module to import that does all the heavy lifting for you.

EDIT: @NotMyself - just because we commented poorly on your trolling question, please don't vote down our answers. sasb and i appreciate it.

Answer 3

If the file is small:

Read each line, tokenize what you've read and assign each value to its respective place in the data structure

Answer 4

If you can guarantee that there are no commas in the data, then the simplest way would probably be to use String.split.

For example:

String[] values = myString.Split(',');
myObject.StringField = values[0];
myObject.IntField = Int32.Parse(values[1]);

There may be libraries you could use to help, but that's probably as simple as you can get. Just make sure you can't have commas in the data, otherwise you will need to parse it better.

Answer 5

Two articles on CodeProject that provide code for a solution, one that uses SteamReader and one that imports CSV data using the Microsoft Text Driver.

Answer 6

I was bored so i modified some stuff i wrote. It try's to encapsulate the parsing in an OO manner whle cutting down on the amount of iterations through the file, it only iterates once at the top foreach.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.IO;

namespace ConsoleApplication1 {

class Program
{

    static void Main(string[] args)
    {

        // usage:

        // note this wont run as getting streams is not Implemented

        // but will get you started

        CSVFileParser fileParser = new CSVFileParser();

        // TO Do:  configure fileparser

        PersonParser personParser = new PersonParser(fileParser);

        List<Person> persons = new List<Person>();
        // if the file is large and there is a good way to limit
        // without having to reparse the whole file you can use a 
        // linq query if you desire
        foreach (Person person in personParser.GetPersons())
        {
            persons.Add(person);
        }

        // now we have a list of Person objects
    }
}

public abstract  class CSVParser 
{

    protected String[] deliniators = { "," };

    protected internal IEnumerable<String[]> GetRecords()
    {

        Stream stream = GetStream();
        StreamReader reader = new StreamReader(stream);

        String[] aRecord;
        while (!reader.EndOfStream)
        {
              aRecord = reader.ReadLine().Split(deliniators,
               StringSplitOptions.None);

            yield return aRecord;
        }

    }

    protected abstract Stream GetStream(); 

}

public class CSVFileParser : CSVParser
{
    // to do: add logic to get a stream from a file

    protected override Stream GetStream()
    {
        throw new NotImplementedException();
    } 
}

public class CSVWebParser : CSVParser
{
    // to do: add logic to get a stream from a web request

    protected override Stream GetStream()
    {
        throw new NotImplementedException();
    }
}

public class Person
{
    public String Name { get; set; }
    public String Address { get; set; }
    public DateTime DOB { get; set; }
}

public class PersonParser 
{

    public PersonParser(CSVParser parser)
    {
        this.Parser = parser;
    }

    public CSVParser Parser { get; set; }

    public  IEnumerable<Person> GetPersons()
    {
        foreach (String[] record in this.Parser.GetRecords())
        {
            yield return new Person()
            {
                Name = record[0],
                Address = record[1],
                DOB = DateTime.Parse(record[2]),
            };
        }
    }
}

}

Answer 7

Brian gives a nice solution for converting it to a strongly typed collection.

Most of the CSV parsing methods given don't take into account escaping fields or some of the other subtleties of CSV files (like trimming fields). Here is the code I personally use. It's a bit rough around the edges and has pretty much no error reporting.

public static IList<IList<string>> Parse(string content)
{
    IList<IList<string>> records = new List<IList<string>>();

    StringReader stringReader = new StringReader(content);

    bool inQoutedString = false;
    IList<string> record = new List<string>();
    StringBuilder fieldBuilder = new StringBuilder();
    while (stringReader.Peek() != -1)
    {
        char readChar = (char)stringReader.Read();

        if (readChar == '\n' || (readChar == '\r' && stringReader.Peek() == '\n'))
        {
            // If it's a \r\n combo consume the \n part and throw it away.
            if (readChar == '\r')
            {
                stringReader.Read();
            }

            if (inQoutedString)
            {
                if (readChar == '\r')
                {
                    fieldBuilder.Append('\r');
                }
                fieldBuilder.Append('\n');
            }
            else
            {
                record.Add(fieldBuilder.ToString().TrimEnd());
                fieldBuilder = new StringBuilder();

                records.Add(record);
                record = new List<string>();

                inQoutedString = false;
            }
        }
        else if (fieldBuilder.Length == 0 && !inQoutedString)
        {
            if (char.IsWhiteSpace(readChar))
            {
                // Ignore leading whitespace
            }
            else if (readChar == '"')
            {
                inQoutedString = true;
            }
            else if (readChar == ',')
            {
                record.Add(fieldBuilder.ToString().TrimEnd());
                fieldBuilder = new StringBuilder();
            }
            else
            {
                fieldBuilder.Append(readChar);
            }
        }
        else if (readChar == ',')
        {
            if (inQoutedString)
            {
                fieldBuilder.Append(',');
            }
            else
            {
                record.Add(fieldBuilder.ToString().TrimEnd());
                fieldBuilder = new StringBuilder();
            }
        }
        else if (readChar == '"')
        {
            if (inQoutedString)
            {
                if (stringReader.Peek() == '"')
                {
                    stringReader.Read();
                    fieldBuilder.Append('"');
                }
                else
                {
                    inQoutedString = false;
                }
            }
            else
            {
                fieldBuilder.Append(readChar);
            }
        }
        else
        {
            fieldBuilder.Append(readChar);
        }
    }
    record.Add(fieldBuilder.ToString().TrimEnd());
    records.Add(record);

    return records;
}

Note that this doesn't handle the edge case of fields not being deliminated by double quotes, but meerley having a quoted string inside of it. See this post for a bit of a better expanation as well as some links to some proper libraries.

Answer 8

I had to use a CSV parser in .NET for a project this summer and settled on the Microsoft Jet Text Driver. You specify a folder using a connection string, then query a file using a SQL Select statement. You can specify strong types using a schema.ini file. I didn't do this at first, but then I was getting bad results where the type of the data wasn't immediately apparent, such as IP numbers or an entry like "XYQ 3.9 SP1".

One limitation I ran into is that it cannot handle column names above 64 characters; it truncates. This shouldn't be a problem, except I was dealing with very poorly designed input data. It returns an ADO.NET DataSet.

This was the best solution I found. I would be wary of rolling my own CSV parser, since I would probably miss some of the end cases, and I didn't find any other free CSV parsing packages for .NET out there.

EDIT: Also, there can only be one schema.ini file per directory, so I dynamically appended to it to strongly type the needed columns. It will only strongly-type the columns specified, and infer for any unspecified field. I really appreciated this, as I was dealing with importing a fluid 70+ column CSV and didn't want to specify each column, only the misbehaving ones.

Answer 9

If you're expecting fairly complex scenarios for CSV parsing, don't even think up of rolling our own parser. There are a lot of excellent tools out there, like FileHelpers, or even ones from CodeProject.

The point is this is a fairly common problem and you could bet that a lot of software developers have already thought about and solved this problem.

Answer 10

I agree with @NotMyself. FileHelpers is well tested and handles all kinds of edge cases that you'll eventually have to deal with if you do it yourself. Take a look at what FileHelpers does and only write your own if you're absolutely sure that either (1) you will never need to handle the edge cases FileHelpers does, or (2) you love writing this kind of stuff and are going to be overjoyed when you have to parse stuff like this:

1,"Bill","Smith","Supervisor", "No Comment"

2 , 'Drake,' , 'O'Malley',"Janitor,

Oops, I'm not quoted and I'm on a new line!

Answer 11

Use an OleDB connection.

String sConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\\InputDirectory\\;Extended Properties='text;HDR=Yes;FMT=Delimited'";
OleDbConnection objConn = new OleDbConnection(sConnectionString);
objConn.Open();
DataTable dt = new DataTable();
OleDbCommand objCmdSelect = new OleDbCommand("SELECT * FROM file.csv", objConn);
OleDbDataAdapter objAdapter1 = new OleDbDataAdapter();
objAdapter1.SelectCommand = objCmdSelect;
objAdapter1.Fill(dt);
objConn.Close();

Answer 12

You could use the TextFieldParser Microsoft supply in the .NET framework?

If you're using VB.NET it seems like an obvious choice (the question is tagged both VB.NET and C#). You could use it from C#, even though it's part of the VB.NET namespace it's still installed with the .NET framework.