Since the Google App Engine Datastore is based on Bigtable and we know that's not a relational database, how do you design a database schema/data model for applications that use this type of database system? ...
When would you use a bigtabe/simpledb database vs a Relational database? ...
New school datastore paradigms like Google BigTable and Amazon SimpleDB are specifically designed for scalability, among other things. Basically, disallowing joins and denormalization are the ways this is being accomplished. In this topic, however, the consensus seems to be that joins on large tables don't necessarilly have to be too ex...
I need a way to do key-value lookups across (potentially) hundreds of GB of data. Ideally something based on a distributed hashtable, that works nicely with Java. It should be fault-tolerant, and open source. The store should be persistent, but would ideally cache data in memory to speed things up. It should be able to support concur...
Am wondering if anyone might provide some conceptual advice on an efficient way to build a data model to accomplish the simple system described below. Am somewhat new to thinking in a non-relational manner and want to try avoiding any obvious pitfalls. It's my understanding that a basic principal is that "storage is cheap, don't worry ...
I'm importing Brazilian stock market data to a SQL Server database. Right now I have a table with price information from three kind of assets: stocks, options and forwards. I'm still in 2006 data and the table has over half million records. I have more 12 years of data to import so the table will exceed a million records for sure. Now, ...
Is it Oracle or MySQL or something they have built themselves? ...
Hello, I have the 3 following tables in a MySQL 4.x DB : hosts: (300.000 records) id (UNSIGNED INT) PRIMARY KEY name (VARCHAR 100) paths: (6.000.000 records) id (UNSIGNED INT) PRIMARY KEY name (VARCHAR 100) urls: (7.000.000 records) host (UNSIGNED INT) PRIMARY KEY <--- links to hosts.id path (UNSIGNED INT) PRIMARY KEY <--- links ...
I've been looking into writing a web app that will run on Google App Engine, but before I commit myself to the platform I'd like to know what, if any, limitations there are. I'm aware of the basic CPU/bandwidth restrictions that Google places on the free service, but I'm wondering more about development restrictions like how BigTable com...
Hi, I'm pretty new to app engine, and I'm trying to set a bit of text into the app engine database for the first time. Here's my code: def setVenueIntroText(text): venue_obj = db.GqlQuery("SELECT * FROM Venue").get() venue_obj.intro_text = text # Works if I comment out db.put(venue_obj) # These two lines This ...
I want to build something to store and serve up time series data, which is coming in from a variety of sources at different time intervals. this includes both raw data and computed data. for example, let's say I want to log an every-30-seconds temperature reading, and a temperature forecast I'm calculating separately every 5 minutes. I ...
Let's say I have a booking database consisting of users: user_id fname lname and their tickets ticket_id user_id flight_no and associated flights flight_no airline departure_time arrival_time What would I need to change to move this Google AppEngine? I understand AppEngine doesn't allow joins. Does that mean my table should be...
Duplicates Why should I use document based database instead of relational database? Pros/Cons of document based database vs relational database I don't know much about Google's BigTable but am wondering what the difference between Google's BigTable and relational databases like MySQL is. What are the limitations of both...
Can anyone point me to a reference or provide a high level overview of how companies like Facebook, Yahoo, Google, etc al perform the large scale (e.g. multi-TB range) log analysis that they do for operations and especially web analytics? Focusing on web analytics in particular, I'm interested in two closely-related aspects: query perf...
In GAE, I have a model called Foo, with existing entities, and attempt to add a new property called memcached to Foo that takes datetime values for the last time this value was set to memcache. If I try to query and sort on this property, or even filter for entities that do not have a value for memcached, entities that haven't had a val...
Hello all, In a classic relational database, I have the following table: CREATE TABLE Person( Id int IDENTITY(1,1) NOT NULL PRIMARY KEY, MotherId int NOT NULL REFERENCES Person(Id), FatherId int NOT NULL REFERENCES Person(Id), FirstName nvarchar(255)) I am trying to convert this table into a Google App Engine table. ...
I basically have the classic many to many model. A user, an award, and a "many-to-many" table mapping between users and awards. Each user has on the order of 400 awards and each award is given to about 1/2 the users. I want to iterate over all of the user's awards and sum up their points. In SQL it would be a table join between the ma...
I need some properties to be unique. How can I achieve this? Is there something like unique=True? I'm using Google App Engine for Python. ...
Is there a way to keep a timestamped record of every change to every column of every row in a MySQL table? This way I would never lose any data and keep a history of the transitions. Row deletion could be just setting a "deleted" column to true, but would be recoverable. I was looking at HyperTable, an open source implementation of Goog...
We're thinking of building some of our infrastructure on to Google AppEngine. But we're worried that if it does not scale, we'll need to export the data and run it on our own servers in future. Is there a way to export Bigtable to MySQL? ...