What is the most common performance bottleneck that is not caused by the database structure?
The Applying scalar function to each row in the result set
SELECT fn_EveluateSomeValue([ColumnName]) FROM LargeTable
Triggers can be a HUGE bottleneck (and headache) when you are processing large batches of commands.
- Server - Like memory and storage types.
- Network - Latency and bandwidth issues.
- Indexing - not sure if you consider this database structure
- Queries - Bad queries with join issues that may cause full table scans.
table scan because:
- index does not exist
- stats out of date
- functions in where clause prevent usage
Let's see (in no particular order)
Cursors
non-sargable where clauses
failure to index foreign key fields
failure to index fields commonly used in the where clause
correlated subqueries
accidental cross joins causing the need to distinct the result set
Badly performing code coming from ORMs
Code that causes too many reads or is called more than once when it didn't need to be (I've seen applications that send the same code many times when they didn't need to or every time a page is opened)
network pipe overloaded
User defined functions causing row-by-row processing
Parameter sniffing
out of date statistics
Union instead of union all
If the I/O system in the server isn't up to the job, you can get latch contention on tempdb and this in turn can cause serious performance problems.
I tend to run into the following bottlenecks (in order of frequency):
- Missing or incorrect indices (resulting in table scans)
- Poorly written queries
- I/O contention
Having a poor my.cnf configuration can kill a server even with proper database design and indexing. Read:
http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/ http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/ http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/
http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/ http://www.mysqlperformanceblog.com/2007/09/17/mysql-what-read_buffer_size-value-is-optimal/
http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html
Helpful my.cnf analytical tool: http://www.day32.com/MySQL/tuning-primer.sh
Using table data one row at a time instead of one table at a time (i.e. cursors).
Unnecessary (or badly designed) locks.
Logs logging things that don't need to be logged (Delete from table instead of Truncate table, etc.)
Having constraints on when bulk loading data. Seriously slows down the insert. Turn them off, load the data, then turn them back on.
Ditto ref cursors, also badly written client apps that get a huge recordset back then process it themselves because the developer didn't understand that sql was anything but a huge heap to hold data in.
Trying to do everything in one huge query instead of breaking it down into sensible steps. Sometimes this will not give SQL a chance to correctly reduce the amount of index (hopefully !) pages that it has to look at. Related to this, not understanding how to use temp tables to hand off manipulations prior to updating large indexed tables.
Not having indexes on joins or commonly queried fields.
Not including fields in indexes when you are always seeking and return the same value (for instance formatted name for an employee when you look up using EmployeeID) causing two actions instead of one.
I've never managed to find much useful information on breaking up large queries, in Oracle it seems your advised more to keep everything together in one query rather than use temp tables. You can also get redo log issues if your temp table is holding a lot of data. I'd like to know more/get some links?
Clauses to avoid (if possible) :
- DISTINCT / UNIQUE
- UNION
- GROUP BY
- ORDER BY