views:

58

answers:

1

Right now we are dealing with a bit of a conundrum in my corporate environment where we are being blamed for a server crash, but I'm not 100% sure we are the culprit. Here's the server environment: We have a primary Coldfusion and its MSSQL database. We then also have a secondary database (MySQL) hosted on a different cloud which is used for miscellaneous tasks. The main reason the system is structured this way is because the primary server is operated by our Content Management System thus we are not allowed to modify it, add tables, or any operations like that, so we use the alternate database for that. By design there are no mission critical items on it, and pages are built in a way such that if the alternate DB returns no rows, the pages will continue to render properly.

Basically, I am being told that when the alternate MySQL server goes down, or stops accepting connections, that it is taking the entire primary cloud with it, including 5 other sites hosted on it. I do not have access to the primary Coldfusion or database logs, because the CMS provider will not give them to me. Thus I can only judge based on the validity of the explanation they are giving me.

The explanation for this behavior coming from our CMS provider is that when Coldfusion queries a database it creates a thread, and that if the DB doesn't respond the threads continue to stack. Eventually the processor is capped, and the server goes down. Is that an accurate explanation of how Coldfusion operates? If so, is there anyway to prevent it, possibly with shorter DB timeouts and the like? Or is the entire explanation posed by our CMS a red herring and something else is really causing the crashes.

Any guidance would be greatly appreciated.

Question answered - Documents found http://kb2.adobe.com/cps/180/tn_18061.html http://www.adobe.com/devnet/server_archive/articles/cf_timeouts_and_unresponsive_requests.html

Setting timeout requests globally does not timeout internal processes waiting on external resources (cfquery/cfhttp etc). The only way to time those out is by manually setting the timeout attribute. Not setting this could result in thread overload and a crashed server as was occurring with us.

+5  A: 

http://kb2.adobe.com/cps/180/tn_18061.html

From reading bullet point 3 and depending on your traffic, your CMS guy might be right.

Also from the link above:

If the database is down and unresponsive, how many times will ColdFusion Server try to reconnect to the database? Will it eventually restart the ColdFusion Server?

If the database is down or the network link to the database goes down when a query request occurs, the connection will timeout (you can customize the timeout period with the timeout attribute in the cfquery tag) and return an error to the user. Please note that the ability to set the timeout for the connection depends on which driver you are using. You can trap this error and handle it programmatically with thecftry/cfcatch tags.

The catch here is that the timeout variable on the cfquery tags are not compatable with the MySQL ODBC driver. Could not find what the default timeout is. Let's say 5 minutes. If you get more than one request in those 5 minutes, it does appear that the connections will start to 'pile up'.

Tommy
Hmmm, good find. What do you mean by "not compatible" with the MySQL driver? As in the timeout value is ignored in the cfquery tag? What about the application wide timeout value, or limit simultaneous requests. Does it operate like a stack or a queue? IE once full accepts no more, or once full it drops the oldest? With all of this in mind, is there anyway to prevent these crashes?
Owen Allen
Not compatable - according to their KB article, it means not supported. I would think that a combination of these two variables on the server settings would help solve the problem. Timeout (min):The number of minutes that ColdFusion MX maintains an unused connection before destroying it.Interval (min):The time (in minutes) that the server waits between cycles to check for expired data source connections to close.
Tommy
Also, you could make use of the try/catch, but since you don't call a connection then run the SQL like .NET/Java; I think your try/catch would wait for the server set ODBC timeout to occur before throwing an exception. But, you should be able to call some type of close call for that database connection. Sorry, didn't really get this deep when I was doing CF :/
Tommy
Just found this document http://www.adobe.com/devnet/server_archive/articles/cf_timeouts_and_unresponsive_requests.html it states "However, because ColdFusion can only time out processing occurring internal to the ColdFusion server, the Timeout requests setting does not time out actively running database queries or any other external calls (such as cfhttp, cfftp, cfobject, cfx, etc.)"That pretty much says the app-wide Timeout requests won't work, so will the cfquery timeout function? In addition where do you see that MySQL cfquery timeout isn't supported, I can't find that verbiage.
Owen Allen
Bullet point 2 on the link above: The timeout attribute is currently supported by the SQL Server ODBC driver, and the DB2 and Informix native drivers. It is not supported by the Oracle native driver, Sybase native driver, the Oracle ODBC driver, and many other ODBC drivers. Since they did not mention MySQL, I am going with it is one of the many other ODBC drivers.
Tommy
I just ran a test, and I believe mysql timeout works. I used a complex query which takes 15s to run. No timeout, it works fine. I set the timeout to 5s, it returned a mysql error. I set timeout to 30s, it works fine. With that, I believe we can close the topic and solution is to set a timeout on all secondary queries. Appreciate your help Tommy, thanks for taking the time!
Owen Allen
No problem! I would also run some test with your server offline when you can and try to monitor thread pool usage and see if this works for that scenario as well. Good Luck!
Tommy