data

Data verifications in Getter/Setter or elsewhere ?

I'm wondering if it's a good idea to make verifications in getters and setters or elsewhere in the code. This might surprise you be when it comes to optimizations and speed-ing up the code, I think you should not made verifications in getters and setters but in the code where your're updating your files or database. Am I wrong ?...

Peak detection of measured signal

We use a data acquisition card to take readings from a device that increases its signal to a peak and then falls back to near the original value. To find the peak value we currently search the array for the highest reading and use the index to determine the timing of the peak value which is used in our calculations. This works well if t...

Are there any decent free JAVA data plotting libraries out there?

On a recent JAVA project, we needed a free JAVA based real-time data plotting utility. After much searching, we found this tool called the Scientific Graphics Toolkit or SGT from NOAA. It seemed pretty robust, but we found out that it wasn't terribly configurable. Or at least not configurable enough to meet our needs. We ended up dig...

Configurable Table Prefixes with a .Net OR/M?

In a web application like wiki or forums or blogging software, it is often useful to store your data in a relational database. Since many hosting companies offer a single database with their hosting plans (with additional databases costing extra) it is very useful for your users when your database objects (tables, views, constraints, and...

Large Data Sets

I'm always looking for large data sets to test various types of programs on. Does anyone have any suggestions? ...

Data Auditing in NHibernate and SqlServer

I'm using NHibernate on a project and I need to do data auditing. I found this article on codeproject which discusses the IInterceptor interface. What is your preferred way of auditing data? Do you use database triggers? Do you use something similar to what's dicussed in the article? ...

What is the best strategy for retainment of large data sets?

I'm leading a project where we'll be recording metrics data. I'd like to retain the data for years. However, I'd also like to keep the primary table from becoming bloated with data that, while necessary for long term trending, isn't required for short term reporting. What is the best strategy for handling this situation? Simply archive ...

Program for working with large CSV Files

Are there any good programs for dealing with reading large CSV files? Some of the datafiles I deal with are in the 1 GB range. They have too many lines for Excel to even deal with. Using Access can be a little slow, as you have to actually import them into a database to work with them directly. Is there a program that can open large CS...

C# NetCDF Library

I am currently working on (or at least planning) a couple of projects that work with large amounts of repetitive data. The kind of data that works well in a spreadsheet or database, but is nasty in XML. :) NetCDF seems like a great option for a file format. However, my work is in C# and there is no "official" NetCDF implementation avail...

What is the best way to implement soft deletion?

Working on a project at the moment and we have to implement soft deletion for the majority of users (user roles). We decided to add an "is_deleted='0'" field on each table in the database and set it to '1' if particular user roles hit a delete button on a specific record. For future maintenance now, each SELECT query will need to ensure...

Astoria vs. SQL Server Data Services

What are in your opinion big differences between areas of usage for "Astoria" (ADO.NET data services) and SQL Server Data Services? ...

Embed data in a C++ program

I've got a C++ program that uses SQLite. I want to store the SQL queries in a separate file -- a plain-text file, not a source code file -- but embed that file in the executable file like a resource. (This has to run on Linux, so I can't store it as an actual resource as far as I know, though that would be perfect if it were for Windows...

Mending bad BAD database design once data is in the system.

I know that that is not a question... erm anyway HERE is the question. I have inherited a database that has 1(one) table in that looks much like this. Its aim is to record what species are found in the various (200 odd) countries. ID Species Afghanistan Albania Algeria American Samoa Andorra Angola .... Western Sahara Yemen Zambia Zim...

Infopath 2007 - How do I perform data validation on the current view ONLY?

I have an infopath 2007 form that I am developing which uses 3 different views. The 3 different views are basically the same form, but have different text boxes shown, depending upon what button the user selects. I run into a problem where 'view 1' has some form validation, but the user has selected 'view 2' and submits it. The form val...

Sql Server compatibility mode

We're currently running a server on Compatability mode 8 and I want to update it. What are the implications of just going in and changing it? What is likely to break? Is there anything that checks the data will survive before I perform it? Can I rollback to mode 8 without performing a restore and without loss of data? ...

How do I protect my file data from disk corruption?

Recently, I read an article entitled "SATA vs. SCSI reliability". It mostly discusses the very high rate bit flipping in consumer SATA drives and concludes "A 56% chance that you can't read all the data from a particular disk now". Even Raid-5 can't save us as it must be constantly scanned for problems and if a disk does die you are pret...

What's a good way to store raster data?

I have a variety of time-series data stored on a more-or-less georeferenced grid, e.g. one value per 0.2 degrees of latitude and longitude. Currently the data are stored in text files, so at day-of-year 251 you might see: 251 12.76 12.55 12.55 12.34 [etc., 200 more values...] 13.02 12.95 12.70 12.40 [etc., 200 more values...] [etc., ...

Advice on handling large data volumes.

So I have a "large" number of "very large" ASCII files of numerical data (gigabytes altogether), and my program will need to process the entirety of it sequentially at least once. Any advice on storing/loading the data? I've thought of converting the files to binary to make them smaller and for faster loading. Should I load everything...

Suggestions for human editable data file format/parsing library

for example, right now I have a roll-my-own solution that uses data files that include blocks like: PlayerCharacter Fighter Hitpoints 25 Strength 10 StartPosition (0, 0, 0) Art Model BigBuffGuy Footprint LargeFootprint end InventoryItem Sword InventoryItem Shield InventoryItem HealthPotion end human editable (...

Looking for tool to generate random CSV data

Hello All, I've been Googling fpr a product, but have yet to find something that I like for generating random/semi-random Data. Ideally, I'd be able to use some of my own input lists, combined with email generation, phone number, etc... to be able to generate test CSV files with multiple records. Also, open source is best. What tools ...