Does anyone know of any tools to provide simple, fast queries of flat files using a SQL-like declarative query language? I'd rather not pay the overhead of loading the file into a DB since the input data is typically thrown out almost immediately after the query is run. Consider the data file, "animals.txt": dog 15 cat 20 dog 10 cat 3...
Hi guys, I'm working on a site which requires a very simple CMS - basically there's a block of text on the homepage that needs to be editable by the client. Their current hosting plan doesn't allow for a database, and including one will cost an extra $X a month which I think is unnecessary for such a basic system. The site is currently...
I am having a problem uploading data from tab-delimited flat files (TSV files) into SQL Server 2005 using the SSIS Data Import wizard. I did not experience this problem using the equivalent procedure in SQL Server 2000, and I have checked that the internal structure of the files I am trying to import is unchanged since well before the SQ...
I'm looking for any helpful links or advice on translating an incoming EDI 940 (X12) to a (|) Pipe delimited flat file in c#.net ...
(Flat File Location) Samba Share | Windows Share (SSIS) _______________________________ | | XP 32bit | Works | Works | | 2003 Serv 32bit | Works | Works | | Vista 64bit ...
Just looking for a component that can be programmatically called in a fairly simple way to import a flat file of data. The data is typically 100,000-500,000 rows, each row contains about 200 fields of text anywhere from about 5 to 250 characters long. Data could be CSV, tab-delimited, etc. There is some budget for this, but would like...
I have a process that currently runs in a Delphi application that I wrote and I need to convert it to a Java process that will run on our web application. Basically our State Financial (legacy) system requires this file in a specific output. In Delphi it is like this: procedure CreateSHAREJournalFile(AppDate : string; ClassCode : stri...
We receive lots of data as flat files: delimitted or just fixed length records. It's sometimes hard to find out what the files actually contain. Are there any well established practices for embedding the schema of the file to the beginning or the end of a file to make the file self-explanatory? Just to get an idea, imagine something l...
Hello, I currently work in the position of Data Warehouse programmer and as such have to put numerous flat files through ETL process. Of course prior to loading the file I have to be aware of its content, the problem is that majority of the files are > 1 GB large and I can not open them using my dear old friend "notepad". Kidding. I usu...
I am working on upgrading our site to work in different languages. My plan is to store paragraphs of text in mulitple languages and give each paragraph an identifier. Example id => '1' brief => 'welcome_paragraph' en => 'Welcome to our website!' de => 'Willkommen auf unserer Website!' Requirements quickly retreived edited via CMS a...
I have a Biztalk project that imports an incoming CSV file and dumps it to a database table. The import works fine, but I only need to keep about 200-300 records from a file with upwards of a million rows. My orchestration discards these rows, but the problem is that the flat file I'm importing is still 250MB, and when converted to XML u...
A friend and I were debating about whether he should use MySQL or a flatfile database for his website's backend. I told him to go with MySQL because it was structured, held records well, and was consistent. He on the other hand said that he would rather go for speed. Reading files is a lot quicker than connecting to MySQL and it made me ...
Hi, Our client sends us a flat file as input, which we then take and convert to an XML file before sending to the destination system. The flat file consists of multiple lines, each line is delimited by LF or CRLF. How do I create a Flat File Schema so that Biztalk can interpret each line of data regardless of whether the line was deli...
I was wondering if anyone knows. Is 'flat' the opposite of hierarchal? Does it come from a phrase like, "this is a flat-out regular file"? ...
hi. im creating a game right now and im a bit stuck on how to implement storage of levels. i need to be able to download level files from the internet ota. im not so familiar with transferring files ota, but i have some experience with databases (mysql). what would be a better way of storing the game's level data? ...
I have a flat file that looks something like this: junk I don't care about \n \n columns names\n val1 val2 val3\n val1 val2 val3\n columns names \n val1 val2 val3\n I only care the lines with values. These value lines are all fixed width format and have the same line length. The other junk lines and column names c...
I need to save a few informations about some files. Nothing too fancy so I thought I would go with a simple one line per item text file. Something like this : # write io.print "%i %s %s\n" % [File.mtime(fname), fname, Digest::SHA1.file(fname).hexdigest] # read io.each do |line| mtime, name, hash = line.scanf "%i %s %s" end Of course...
I have an SSIS package with a Data Flow that takes an ADO.NET data source (just a small table), executes a select * query, and outputs the query results to a flat file (I've also tried just pulling the whole table and not using a SQL select). The problem is that the data source pulls a column that is a Money datatype, and if the value i...
Hi, which structure returns faster result and/or less taxing on the host server, flat file or database (mysql)? Assume many users (100 users) are simultaneously query the file/db. Searches involve pattern matching against a static file/db. File has 50,000 unique lines (same data type). There could be many matches. There is no writing to...
We receive a nightly data export from a vendor in the form of about 10 tab-delimited flat file without column headers. In addition, the vendor provides us with the SQL scripts for the database tables so that we can import the files into our system. Unfortunately, the vendor recently changed the schema for the flat files. Each file has ...