tags:

views:

71

answers:

4

I need to run calculations on a very large database. The application slows down significantly when I run the calculations. A possible work around would to copy this into a second database and run the calculations offline. When it's done switch it with the live database. Can this process be automated?

is this what Google does with their search index?

+1  A: 

Sure, and it can be automated. But data cannot be altered if you are going to swap back over it.

I would only transfer the data that needs to be transferred, and aggregate what data can be aggregated during the transfer - and index the destination tables to optimize the calculation. However, both these options are possible without taking the data out of the database and just making other tables in the database.

If you're doing the operation on the same machine, it's possible you will still experience the same slowness since the server will be carrying out the same operations...

If the "slowness" is actually caused by locking/blocking behvaior and the data is not going to be changed, you can use the WITH(NOLOCK) option on your tables to avoid blocking other operations during your long operations.

Cade Roux
so it will be on a second machine. do you know where i can find some sample automation code? right now i have to transfer the files manually over ftp...
SSIS has an FTP component and comes with SQL Server 2005 and above and you can automate it with the SQL Server Agent. Without further details on the type of calculation and what you mean by large (either in number of rows or number of columns and nature of table joins), it's hard to recommend a suitable architectural decision.
Cade Roux
+1  A: 

Yes, this can be automated with replication, agent tasks (assuming SQL Server), or even a custom application.

John Gietzen
A: 

I would say the best idea would be to extract the calculations out into their own module, then run that separately in another process or on another computer using the same database. It's hard to tell exactly if this would be the right thing to do without more information about your specific setup.

McWafflestix
A: 

There are several ways to do this. You can automate using sql server agent.

without knowing more details, and what the calculations are for you might implement a data warehouse, use aggregation if possible

Stuart