Does Microsoft SkyDrive have an API?
So with the recent news that Microsoft Skydrive is going to get bumped to 25GB of storage per account, does anyone know if SkyDrive has an API? (And if so, where are the docs?) ...
storage
Game Terrain Database Model
I am developing a game for the web. The map of this game will be a minimum of 2000km by 2000km. I want to be able to encode elevation and terrain type at some level of granularity - 100m X 100m for example. For a 2000km by 2000km map storing this information in 100m2 buckets would mean 20000 by 20000 elements or a total of 400,000,0...
software that store data in flat file, what are common encoding/file format techniques
Hi, I have been spoiled by either using sql server to store data, or using xml files. What are common techniques for storing data in flat files other than xml and CSV. I know many times when I open files that data is all jumbled up, which means it is encoded right? Are there any common techniques that I could read about somewhere? ...
Lightweight Store Mechanisms
I'm about to write a small utility to organze and tag my mp3s. What is the best way to store small amounts of data. More importantly, are there databases which exist where I don't need to install a client/server environment, I just include the library and I'm good? I could use XML, but I'm afraid that the file size would become large ...
How to make Linux GUI "usable" when lots of disk activity is happening
If I start copying a huge file tree from one position to another or if some other process starts doing lots of disk activity, the foreground app (GUI) slows way down. For example, take a 2gb file tree with 100k files in it. Open a console and do cp -r bigtree bigtree2. Then go to firefox and start browsing. Firefox is almost unusable. Ev...
Is there a market amongst programmers for an OID-keyed blob store?
A friend has developed a very amazing blob store and I think it needs to be used, but I'm wondering whether people think such a thing has a market, and if programmers ever get to make these kinds of decisions. It has support for online backups using deltas and is much faster than anything I know of, it's undergone rigorous testing and b...
What file format do you use for your application and why?
I'm most interested in in-process (single user) solutions for large amounts of mutating object-oriented data, where any part of the data may change. Such systems generally suffer from these problems: Writing large files out from scratch is inefficient xml is too verbose SQL blobs aren't a good match So how do you do it? ...
Highest Performance Database Storage Mechanism
I need ideas to implement a (really) high performance in-memory Database/Storage Mechanism. In the range of storing 20,000+ objects, with each object updated every 5 or so seconds. I would like a FOSS solution. What is my best option? What are your experiences? I am working primarily in Java, but I need the datastore to have good perf...
Best way to store data for Greasemonkey based crawler?
I want to crawl a site with Greasemonkey and wonder if there is a better way to temporarily store values than with GM_setValue. What I want to do is crawl my contacts in a social network and extract the Twitter URLs from their profile pages. My current plan is to open each profile in it's own tab, so that it looks more like a normal br...
What's the best practice for storing huge amounts of text (into a DB or as a file?), and what about compressing it?
I'm building a web-app that handles internal emails and other frequent small-to-medium sized chunks of text between users and clients. What's the best method for storing this data? In a database (MySQL) or as thousands of individual files? What about compressing it (PHP's gzcompress() or MySQL's compression features)? This will not be a...
Recommended location for document storage - in database or elsewhere?
Background: We have an in house document storage system that was implemented long ago. For whatever reason, using the database as the storage mechanism for the documents was chosen. My question is this: What is the best practice for storing documents? What are the alternatives? What are the pros and cons? Answers do not have to...
The best way to store and access 120,000 words in java
Hello, I'm programming a java application that reads strictly text files (.txt). These files can contain upwards of 120,000 words. The application needs to store all +120,000 words. It needs to name them word_1, word_2, etc. And it also needs to access these words to perform various methods on them. The methods all have to do with S...
Images management
Im building a site were users can upload images and then "use" them. What I would like some thoughts and ideas about is how to manage temporary uploads. For example, a user uploads an image but decides not to do anything with it but just leaves the site. I have then either uploaded the file to the server or loaded it to the server memo...
Best way to collect and store data daily?
I have a bunch of statistics: # of users, # of families, ratio user/family, etc. I'd like to store these daily so I can view this data historically. However, I'm looking for the most effective way to store this data. Should I run a cron job that writes to the database DATE: today USERS: 123 FAMILIES: 456 RATIO: 7.89 or whatever? (or sh...
Anybody tried mosso CloudFiles with Google AppEngine?
I'm wondering if anybody tried to integrate mosso CloudFiles with an application running on Google AppEngine (mosso does not provide testing sandbox so I cann't check for myself without registering)? Looking at the code it seems that this will not work due to httplib and urllib limitations in AppEngine environment, but maybe somebody has...
Storing Images for Multiple Users
The platform isn't really as important as the theory. For the record, it is ASP.NET (C# on 3.5 SP1), SQL Server 2005. For the sake of argument, I have unlimited space (filesystem and database) and unlimited bandwidth. I am working on a project that would allow multiple users to upload their own images which could be managed by that us...
Why would you ever use asp.net's ViewState storage object over the Session storage object?
Other than because session storage is session-global to more than one page, why would you ever want to use the viewstate to hold values? It seems kind of ridiculous to send any kind of information other than a few small query string like values, back and forth from the client to server. I mean what a waste of bandwidth(!), simply for s...
Do you recomend Sql Server for storing and indexing files (pdf, office, etc) ?
I need to storage and index files, like PDF and office files. Currently I'm using Sql Server 2k8 to perform this task using the Full text search with IFilters. My question is: Is this the "best" way? Should I switch, for instance, to Lucene for indexing? Thanks ...
User images - database vs. filesystem storage
I’m writing an upload function in .NET for presentation images on user profiles. I’m choosing between two different ways of doing it, and would like to ask on advice and opinions as to the pros and cons. Store the images directly on the file system Store the images in the database So far I’ve been mostly pro the first option – naming...
What is the best way to deal with collections (lists or sets) in key-value storage?
I wonder what can be an effective way to add/remove items from a really large list when your storage is memcached-like? Maybe there is some distributed storage with Java interface that deals with this problem well? Someone may recommend Terracotta. I know about it, but that's not exactly what I need. ;) ...