views:

239

answers:

0

I have a "Cron Service" that i've had running on our production server for over a year and a half.

It's a regular windows service which every minute connects to the DB, checks on one table whether there's something to do, and if so, does it. This is a server for a website that's not used a lot, so most of the time the service is completely idle.

A couple months ago, this service started throwing the following exception and dying:

MESSAGE: Object reference not set to an instance of an object.
SOURCE: MySql.Data
TARGETSITE: MySql.Data.MySqlClient.Driver CheckoutConnection()
STACKTRACE:    at MySql.Data.MySqlClient.MySqlPool.CheckoutConnection()
   at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
   at MySql.Data.MySqlClient.MySqlPool.GetConnection()
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at DBA.Connect() in D:\xxx\DBA.vb:line 21
   at CronService.TaskMailings.OnExecute() in D:\xxx\TaskMailings.vb:line 54
   at CronService.CronTask.ThreadExecute() in D:\xxx\CronTask.vb:line 99
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

The exception is clearly coming from inside the MySQL Library. Also, my function that calls it is pretty simple:

 Public Shared Function Connect() As MySqlConnection
  Dim strConnString As String = WebConfigurationManager.AppSettings("ConnectionString")
  Dim DBConn As New MySqlConnection(strConnString)
  DBConn.Open()
  Return DBConn
 End Function

And the function where I'm calling DBA.Connect() (CronService.TaskMailings.OnExecute) is doing so with a "Using" statement, so the connection is getting closed, this shouldn't be a problem of connections being left open and getting exhausted (although it might, if there's a bug inside the MySQL library).

This is the connection string I'm using for the Service:

 server=localhost;port=13306;uid=xxx;pwd=xxx;database=xxx;Pooling=True;charset=utf8;

Any ideas what might be going on here? The strangest part is that this has worked for well over a year before it started happening out of the blue, and load to the server hasn't changed enough to justify it being a load problem, I believe.


EDIT: Some of the times it dies, it gives this other error message:

MESSAGE: error connecting: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
SOURCE: MySql.Data
TARGETSITE: MySql.Data.MySqlClient.Driver GetConnection()
STACKTRACE:    at MySql.Data.MySqlClient.MySqlPool.GetConnection()
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   atDBA.Connect() in D:\xxx\DBA.vb:line 21

Although the error message is pretty clear, like I mentioned before, we are releasing all connections, so this shouldn't happen. Any ideas?