What language should I use to write a text parser and display the results in a user friendly manner?

Question

My company's proprietary software generates a log file that is much easier to use if it is parsed. The log parser we all use was written by another employee as a side project, and it has horrible performance.

These log files can grow to 10s of megabytes very quickly, and the parser we currently use has issues if a log file is bigger than 1 megabyte.

So, I want to write a program that can parse this massive amount of text in the shortest amount of time possible. We use Windows exclusively, so running on Windows is a must. Our current implementation runs on a local web server, and I'm convinced that running it as an application would have to be faster.

All suggestions will be helpful. Thanks.

EDIT: My ultimate goal is to parse the text and display it in a much more user friendly manner with colors and such. Can you do this with Perl and Python? I know you can do this with Java and C++. So, it will function like Notepad where you open a log file, but on the screen you display the user-friendly format instead of the raw file.

EDIT: So, I cant choose the best answer, and that was to choose a language that can best display what I'm going for, and then write the parser in that. Also, using ANTLR will probably make this process much easier. I changed the original question, since I guess I didn't ask what I was really looking for. Thanks everyone!

Answer 1

I would suggest using Python or Perl. Parsing large text files with regular expressions is really fast.

Answer 2

I believe perl is considered a good choice to parse text.

Answer 3

Perl is good for text processing.

A number of very good text processing programs have been written in Perl. Ack (a grep replacement) is one.

Answer 4

Sounds like a job for Perl, much as I don't particularly care for it as a language myself. ActivePerl is a reasonable distribution of Perl for Windows.

Answer 5

I've used both Python and Perl. Perl is a more natural fit for this but can be hard to maintain. Python will do it just as well and is easier to read. Go for Python.

Answer 6

Maybe a finished product such as the MS LogParser (usage podcast here) may do what you need and it's free.

Answer 7

c/c++ or java... for c/c++ i have snippet that might help you:

FILE *f = fopen(file, "rb");
if(f == NULL) {
    return DBDEMON_OPEN_ERROR; // open fail
}

for(int i = 0; feof(f) == 0; i++)

{

fscanf(f,"%d %s %s %c\n",  &db[i].id, &db[i].name[0], &db[i].uid[0], &db[i].priviledge);

db_size++;

}

fclose(f);

this is reading a file with the following format:

int string string char

1 SOMETHING ANYTHING Z

to a struct define as follows:

typedef struct {

    unsigned int    id;
    char        name[DBDEMON_NAME_MAXSIZE];
    char        uid[DBDEMON_UID_MAXSIZE];
    char        priviledge;
} DATABASE;

Use fscanf with care, since no types are checked, etc, it can result in errors. But I think this is pretty efficient.

Answer 8

You should use the language that YOU know... Unless you have so much time available to complete the project that you can also spend the time learning a new language.

Answer 9

Whatever language your coworker used.

(I could tell you that any macro assembler will let you write code that would rip through your data, but seriously, are you going to spend months writing assembly just to save a few seconds of CPU time? Rewriting a program is fun but it's not practical.)

Whip out your profiler, point it at your horribly performing log parser, and fix the performance problems. If it's a common language, there will be people here who can help.

Answer 10

Hmmm, "go with what you know" was a good answer. Perl was designed for this sort of thing (but imo is well suited for simple parsing, but I'd personally avoid it for complex projects).

If it gets even a little complex, why not use a proper syntax and grammar set-up?

Lex & Yacc (or Flex & Bison) spring to mind, but personally I would always reach for Antlr http://www.antlr.org/

Define various "words" in terms of patterns (syntax), and rules to combine those words (grammar) and Antlr will spit out a program to parse your input (you can have the program in Java, C, C++ and more (you are worried about parse time, so choose a compiled language, of course)).

I personally find it tedious to hand-craft parsers, and even more tedious to debug them, but AntlrWorks is a lovely IDE which really makes it a piece of cake ...

That bit at the bottom is defining a grammar rule.

If you mess up your grammar rules, you will be informed. This is not the case with hand-crafted parsers, where you just scratch your body part and wonder about the "strange results"...

Check it out. Even if you think your project is trivial now, it may well grow. And if you have any interest in parsing you do owe it to yourself to at least be familiar with lex/yacc, but especially Antlr(Works)

Answer 11

I'd suggest Perl. It was practically built for parsing log files. As for output I agree with ghostdog74, HTML is the way to go. Perl has dozens of modules that allow you to build and/or template HTML.

I'd parse out the data using regular expressions, then use Template::Toolkit (on CPAN) to create nice pages using HTML and CSS templates.

Answer 12

Parse this massive amount of text in the shortest time possible.

Consider the PADS Project from AT&T. It's a special-purpose language, compatible with C, that's designed exactly for high-speed parsing of log files and other ad hoc data formats. There's even a feature where it can try to learn your log format from examples, although I don't know if that has hit production yet. The people behind the project are really smart, and it's had a big impact within the phone company. PADS gives very high performance on data streams that produce gigabytes. Joe Bob says check it out.

If "massive text in the shortest time possible", Perl and Python are not the answer. But if you need to whip up something not too slow, and it's OK to take longer, Perl and Python could be OK. Tems of megabytes is not actually that big.