scalability

Caching a resultset

My app will retrieve a countylist from MySql using a datasource bean. Since all pages will potentially use the same recordset every time I could store the countrylist as a List in some global bean, safe in this case. I could manage to refresh the list any time I want... but when things become more complex what is the best strategy for it...

How digg (or other high load category websites) are storing user sessions?

Hi. How does digg or any other high-traffic website store user sessions? What do they use for storing the user sessions? File system, DB (which one?), memcache or both? Let's imagine a simple situation. Logged user has set the flag "Remember me" during login. We've set a session cookie with expiration date 1 year. For example, we are ke...

dynamically horizontal scalable key value store

Hi, Is there a key value store that will give me the following: Allow me to simply add and remove nodes and will redstribute the data automatically Allow me to remove nodes and still have 2 extra data nodes to provide redundancy Allow me to store text or images up to 1GB in size Can store small size data up to 100TB of data Fast (so w...

Can you really scale up with Django...given that you can only use one database? (In the models.py and settings.py)

Django only allows you to use one database in settings.py. Does that prevent you from scaling up? (millions of users) ...

What makes Oracle more scalable?

Oracle seems to have a reputation for being more scalable than other RDBMSes. After working with it a bit, I can say that it's more complex than other RDBMSes, but I haven't really seen anything that makes it more scalable than other RDBMSes. But then again, I haven't really worked on it in a whole lot of depth. What features does Ora...

What is the best architecture for tracking field changes on objects?

We have a web application that is built on top of a SQL database. Several different types of objects can have comments added to them, and some of these objects need field-level tracking, similar to how field changes are tracked on most issue-tracking systems (such as status, assignment, priority). We'd like to show who the change is by,...

open source alternatives to oracle coherence?

Are there any open source alternatives to oracle coherence? (btw, how much does coherence cost anyways?) ...

What aspect of relational databases makes it difficult for them to scale sufficiently on services like Google App Engine?

Apparently the reason for the BigTable architecture has to do with the difficulty scaling relational databases when you're dealing with the massive number of servers that Google has to deal with. But technically speaking what exactly makes it difficult for relational databases to scale? In the enterprise data centers of large corporat...

Does the Thundering Herd Problem exist on Linux anymore?

Many linux/unix programming books and tutorials speak about the "Thundering Herd Problem" which happens when multiple threads or forks are blocked on a select() call waiting for readability of a listening socket. When the connection comes in, all threads and forks are woken up but only one "wins" with a successful call to "accept()". In ...

Massive table in SQL 2005 database needs better performance!

Hi Guys, I am working on a data driven web application that uses a SQL 2005 (standard edition) database. One of the tables is rather large (8 million+ rows large with about 30 columns). The size of the table obviously effects the performance of the website which is selecting items from the table through stored procs. The table is index...

Scalable MySQL databese for mail-like messaging

Assume we have a popular site. We need to implement mail-like messaging between users. Typical solution is to use 2 tables: Users (user_id) Messages (message_id, sender_id (references user_id), receiver_id (references user_id), subject, body ). This method has 2 significant limitations All messages of all users are stored in one tab...

How to prevent your code from becoming obsolete?

Some programmers here have been developing a project in VB6, and they say they now need to upgrade to vb.net if they want their apps to run on newer/future systems as vb6 is going to be history soon. So there is this huge application they have been working on that they are going to have to rebuild from scratch (they tried to use the upg...

JMS for email scalability

Does it make sense to use JMS and JavaMail together to build a scalable email solution? Presently these are the 2 options I am considering: -- Build a RESTful email center API and call it from all over the web app. -- Use JMS in between the web app and the email center. The 'email center' as I call it will probably be implemented us...

(When) are parallel sorts practical and how do you write an efficient one?

I'm working on a parallelization library for the D programming language. Now that I'm pretty happy with the basic primitives (parallel foreach, map, reduce and tasks/futures), I'm starting to think about some higher level parallel algorithms. Among the more obvious candidates for parallelization is sorting. My first question is, are p...

Random MySQL row while avoiding holes

I have a big MySQL users table and need to get six random rows from it (I'm using PHP). The table has an index column that is auto incremented. The only problem is that some rows are marked as inactive, because some users have disabled their accounts or whatever. That means I can't just count the rows and then grab a random number from t...

process of commenting and improving already written program?

Please allow my intro to properly define the scope of my question: I'm still very new to the programming world. This all started for me when I had an idea for a software program, but no programming experience. I ended up going the outsourcing route to get the program, and after almost a year, we do have it live and functioning. This ...

What are the Best Practices For SQL Inserts on Large Scale in reference to ad impressions?

I am working on a site where I will need to be able to track ad impressions. My environment is ASP.Net with IIS using a SQL Server DMBS and potentially Memcached so that there are not as many trips to the database. I must also think about scalability as I am hoping that this application becoming a global phenom (keeping my fingers cros...

Config IIS for serving Static files / CDN?

Is there anyway to config IIS to serve static file like CDN? Like Gzip, Cache, E-tag, Mod-date? And how should we config to make it very robust on massive requests? I know it's a short question but that's all I want to ask. To the question on the IIS version, i prefer 6 and 7 IIS. You can give your answer on either 6 or 7 :) Thanks...

IIS Authentication and scaling

Hi me and my team are about to build a Webb service application with Silverlight with a potential to be used by hundred of thousands users. Each user has roles and privileges and belongs to a company and each company has their own database and so on. When a user logs in I thinking of storing the user-id and the connection-string to the ...

Is it expensive to do a WHOIS lookup using TcpClient.Connect() in C#?

A customer has asked that we do a dynamic whois lookup on the homepage of their ASP.NET site, based on the IP of the user accessing the site. The implementation would be something like what's described here: http://www.aspheute.com/english/20000825.asp However, I noticed that this code connects to whois.networksolutions.com. I am thin...