Connection Pool returns Same Exception Instance to Two Threads Using the Same Bad Connection String?

Question

Ok this looks like a major fundamental bug in .NET:

Consider the following simple program, which purposely tries to connect to a non-existent database:

class Program
{
    static void Main(string[] args)
    {            

        Thread threadOne = new Thread(GetConnectionOne);
        Thread threadTwo = new Thread(GetConnectionTwo);            
        threadOne.Start();
        threadTwo.Start();

    }



    static void GetConnectionOne()
    {
        try
        {
            using (SqlConnection conn = new SqlConnection("Data Source=.\\wfea;Initial Catalog=zc;Persist Security Info=True;Trusted_Connection=yes;"))
            {
                conn.Open();
            }    
        } catch (Exception e)
        {
            File.AppendAllText("ConnectionOneError.txt", e.Message + "\n" + e.StackTrace + "\n");
        }

    }


    static void GetConnectionTwo()
    {
        try
        {
            using (SqlConnection conn = new SqlConnection("Data Source=.\\wfea;Initial Catalog=zc;Persist Security Info=True;Trusted_Connection=yes;"))
            {
                conn.Open();
            }
        }
        catch (Exception e)
        {
            File.AppendAllText("ConnectionTwoError.txt", e.Message + "\n" + e.StackTrace + "\n");
        }

    }
}

Run this program and set breakpoints on the catch blocks. The DBConnection object will attempt to connect for 15 seconds (on both threads), then it will throw an error. Inspect the exception's stack trace, and the stack trace will have TWO call stacks intermingled, as follows:

at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.SqlClient.SqlConnection.Open()
at ZoCom2Test.Program.GetConnectionOne() in C:\src\trunk\ZTest\Program.cs:line 38
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at ZoCom2Test.Program.GetConnectionTwo() in C:\src\trunk\ZTest\Program.cs:line 54

You may have to try it several times to get this to happen, but I'm getting this to happen right now on my machine. How is this possible? This should be totally impossible at the VM level. It looks like the DBConnection.Open() function is simultaneously throwing the same exception on two threads at once, or something bizarre like that.

Answer 1

You are getting the same exception thrown. I don't understand why however. Have a look at the output window, notably that exception1 == exception2.

class ThreadingBug
{
    private const string CONNECTION_STRING =
        "Data Source=.\\wfea;Initial Catalog=catalog;Persist Security Info=True;Trusted_Connection=yes;";

    static void Main(string[] args)
    {
        try
        {
            Thread threadOne = new Thread(GetConnectionOne);
            Thread threadTwo = new Thread(GetConnectionTwo);
            threadOne.Start();
            threadTwo.Start();

            threadOne.Join(20000);
            threadTwo.Join(20000);

            Debug.WriteLine("Same?" + (exception1 == exception2));
        }
        catch (Exception e)
        {
            Debug.WriteLine("error main" + e);
        }
    }

    static Exception exception1;

    static void GetConnectionOne()
    {
        try
        {
            using (SqlConnection conn = new SqlConnection(CONNECTION_STRING))
            {
                conn.Open();
            }
        }
        catch (Exception e)
        {
            Debug.WriteLine("Error Con one" + e);
            exception1 = e;
        }
    }
    static Exception exception2;

    static void GetConnectionTwo()
    {
        try
        {
            using (SqlConnection conn = new SqlConnection(CONNECTION_STRING))
            {
                conn.Open();
            }
        }
        catch (Exception e)
        {
            Debug.WriteLine("Error Con two" + e);
            exception2 = e;
        }
    }
}

Edit: The below was my original response.

It's very likely your "random" filenames are similar, if not the same, as they will sometimes be called within very close timeframes. Often, when you have a problem that randomly appears, and you have a Random.Next call, it should be the first place you look.

Answer 2

Try this instead, and see what happens:

class ThreadingBug
{
    private const string CONNECTION_STRING =
        "Data Source=.\\wfea;Initial Catalog=catalog;Persist Security Info=True;Trusted_Connection=yes;";

    static void Main(string[] args)
    {
        try
        {
            Thread threadOne = new Thread(GetConnectionOne);
            Thread threadTwo = new Thread(GetConnectionTwo);
            threadOne.Start();
            threadTwo.Start();

            threadOne.Join(2000);
            threadTwo.Join(2000);
        }
        catch (Exception e)
        {
            File.AppendAllText("Main.txt", e.ToString());
        }
    }

    static void GetConnectionOne()
    {
        try
        {
            using (SqlConnection conn = new SqlConnection(CONNECTION_STRING))
            {
                conn.Open();
            }
        }
        catch (Exception e)
        {
            File.AppendAllText("GetConnectionOne.txt", e.ToString());
        }
    }

    static void GetConnectionTwo()
    {
        try
        {
            using (SqlConnection conn = new SqlConnection(CONNECTION_STRING))
            {
                conn.Open();
            }
        }
        catch (Exception e)
        {
            File.AppendAllText("GetConnectionTwo.txt", e.ToString());
        }
    }
}

---

I believe there is a bug here, though it's neither major, nor fundamental. After working to narrow this down (and to do things like removing one thread), it looks like the same instance of the Exception class is thrown by the Connection Pool implementation on both threads (kudos to Gregory for discovering this). This sometimes shows up as a corrupt ("intermingled") stack trace, and sometimes simply as the same stack trace on both threads, even when the code is quite different between the two threads.

Commenting out one of the Thread.Start calls shows an entirely different stack trace, demonstrating that the odd part is in the connection pool implementation - the odd stack traces are being handed out by the connection pool, since both threads use the same connection string and credentials.

I've submitted a Connect issue on this at https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=522506. Everyone should feel free to vote on how important (or unimportant) you feel it is, whether you can reproduce it, or whether you have a workaround. This will help Microsoft prioritize a fix.

---

Update: The Connect issue has been updated. Microsoft acknowledges it as a bug, and plans to fix it in a future release.

Thanks to nganju, Gregory, and everyone else who participated in solving this problem. It was indeed a bug, and it will be fixed, and it's because of us.

Answer 3

OK, I managed to reproduce this (VS2008, FX3.5SP1, dual core) both inside and outside(*) the debugger. And after altering your catch logic a little it even is reliably reproducable. And, like Gregory mentioned, it is the same exception instance thrown in both threads.

This should be totally impossible at the VM level.

Where did you get that idea?

Both threads are trying to connect through the connection pool. I don't know anything about how the Pool works, but I'll take a guess: It is serializing the 2 simultaneous requests. That sounds like being nice to the Server. And then when the attempt fails it has 1 exception and 2 waiting threads.

I too would have expected the CLR or the ConnectionPool to duplicate the exception and prepend 2 separate stacktraces but instead it merges the 2 calling traces.

So I think your bug could very well be feature, status: by design.
Because it is not really an 'intermingled' stacktrace but more of a deliberately Y-shaped one. It does not look accidental.

It would be nice if somebody found a reference for this behaviour though. Right now I'm not sure if this is a CLR or a ConnectionPool 'feature'.

(*) Edit: I think I saw it outside the debugger once, but now I'm unable to reproduce that. So it could be a debugger or a timing issue.