tags:

views:

50

answers:

1
Q: 

DataDeduplication idea

Good morning

Just built a small Data De-duplication chunk of code in C# and want to check if anyone has done something simular before, and if so, how? is there publicly available code for this?

The code i wrote is on GitHub at http://gist.github.com/273880.

At the moment, there is no physical backing store, and no way of storing what blocks belong to what files, but was thinking of chucking these into a DB. Also, using SHA512 for the check-summing, but that might be overkill...

+1  A: 

I think this is basically a limited dictionary coder.

Kaleb Brasee  2010-01-11 00:35:03
looking at the entry, correct, it would be. but it seems to be the way all these de-dupe systems are built... the one in ZFS, from what i can gather, is built based on the same principle: each block, before written to the disk, is checked for duplication and if there is a dupe, its just a link... i am thinking of building this into a DB or file backed store and sharing over SMB... cant think of another way of doing this...
TiernanO  2010-01-11 00:55:05

related questions

Displaying Flash content in a C# WinForms application
How to get the value of built, encoded ViewState?
Unhandled Exception Handler in .NET 1.1
How do I connect to a database and loop over a recordset in C#?
How do I most elegantly express left join with aggregate SQL as LINQ query
Get a new object instance from a Type in C#
.NET Testing Framework Advice
Automatically update version number
What is the difference between an int and an Integer in Java/C#?
How to write to Web.Config in Medium Trust ?
WinForms ComboBox data binding gotcha
How do you sort a C# dictionary by value?
Adding Scripting functionality to .NET applications
Floating Point Number parsing: Is there a Catch All algorithm?
How do I print an HTML document from a web service?
Decoding T-SQL CAST in C#/VB.net
Anatomy of a "Memory Leak"
How do I get a distinct, ordered list of names from a DataTable using Linq
Reliable Timer in a Console Application
How do I fill a DataSet or a DataTable from a LINQ query resultset ?
What's the difference between Math.Floor() and Math.Truncate() in .NET?
How do I calculate relative time?
How do I calculate someone's age in C#?
Are there any conversion tools for porting Visual J# code to C#?
When setting a form's opacity should I use a decimal or double?