views:

864

answers:

10

Hi All -

Final Edit: I found a solution to the problem (at the bottom of the question).

I've got an Nunit problem that's causing me grief. Edit: actually it looks more like a SQLite problem, but I'm not 100% certain yet.

My TestFixture has a setup that generates a random filename that's used as a SQLite database in each of my tests.

[Setup]
public void Setup()
{
    // "filename" is a private field in my TestFixture class
    filename = ...; // generate random filename
}

Each of my tests use this construct in each method that accesses the database:

[Test]
public void TestMethod()
{
    using (var connection = Connect())
    {
        // do database activity using connection

        // I've tried including this line but it doesn't help
        // and is strictly unnecessary:
        connection.Close();
    }
}

private DbConnection Connect()
{
    var connection = DbProviderFactories.GetFactory("System.Data.SQLite").CreateConnection();
    connection.ConnectionString = "Data Source=" + filename;
    connection.Open();
    return connection;
}

So that one helper method Connect() is used by all the methods. I'm assuming that the using() { } construct is calling Dispose() on the connection at the end of TestMethod() and freeing up the connection to the SQLite database file.

The problem I have is in my [TearDown] method:

    [TearDown]
    public void Cleanup()
    {
        File.Delete(filename); // throws an IOException!
    }

With every test I get an exception:

System.IO.IOException: The process cannot access the file 'testdatabase2008-12-17_1030-04.614065.sqlite' because it is being used by another process.

All of the tests fail when they get to the [TearDown], so I end up with a directory full of temporary databse files (one per test, each with a different name) and a whole bunch of failed tests.

What process is accessing the file? I don't get how a second process could be accessing the file. The connection has completely gone out of scope and been Dispose()d by the time I'm trying to delete the file, so it can't be something SQLite related. Can it?

Note that I get the same result if I run all the tests or just a single test.

Update: So I tried Dispose()ing of my DbCommand objects as well, since I wasn't doing that (I assumed that every other ADO.NET provider that Dispose()ing the DbConnection also Dispose()s any commands on that connection.) So now they look like:

[Test]
public void TestMethod()
{
    using (var connection = Connect())
    {
        using (var command = connection.CreateCommand())
        {
        // do database activity using connection

        }
    }
}

It didn't make any difference -- the File.Delete() line still throws an IOException. :-(

If I remove that one line in [TearDown] then all my tests pass, but I'm left with a whole bunch of temporary database files.

Another Update: This works just fine:

var filename = "testfile.sqlite";
using (var connection = BbProviderFactories.GetFactory("System.Data.SQLite").CreateConnection())
{
    connection.ConnectionString = "Data Source=" + filename;
    connection.Open();
    var createCommand = connection.CreateCommand();
    createCommand.CommandText =
        "CREATE TABLE foo (id integer not null primary key autoincrement, bar text not null);";
    createCommand.ExecuteNonQuery();
    var insertCommand = connection.CreateCommand();
    insertCommand.CommandText = "INSERT INTO foo (bar) VALUES (@bar)";
    insertCommand.Parameters.Add(insertCommand.CreateParameter());
    insertCommand.Parameters[0].ParameterName = "@bar";
    insertCommand.Parameters[0].Value = "quux";
    insertCommand.ExecuteNonQuery();
}
File.Delete(filename);

I don't understand!

Update: Solution found:

    [TearDown]
    public void Cleanup()
    {
        GC.Collect();
        File.Delete(filename);
    }

I ran the unit tests through the debugger, and when the [TearDown] method starts there are definitely no references to the SQLite DbConnection around any more. Forcing a GC must clean them up though. There must be a bug in SQLite.

+1  A: 

try calling Close on the dbconnection

make sure the sqllite process is terminated

you can see what process has your file locked with the Unlocker (free) utility

this may be a 'known' issue with SqlLite; the forum banter suggests closing the connection and disposing the command, and suggests that this will be fixed in a future version (since this behavior is not consistent with other ADO providers)

Steven A. Lowe
His using clause should call Dispose which calls Close. But worth a try nonetheless!
TheSoftwareJedi
Calling Close() doesn't help -- I get exactly the same result.
Stewart Johnson
@[Stewart Johnson]: see edits; Unlocker will tell you who has your file open, it's probably SQLLite...
Steven A. Lowe
Is there some way I can force SQLite to release the file? I've tried Close() and the DbConnection goes out of scope, I'm not sure what else to do.
Stewart Johnson
@[Stewart Johnson]: have you verified with Unlocker that it is SqlLite still holding the file open?
Steven A. Lowe
@[Stewart Johnson]: this may be a known issue w/SqlLite, see edits
Steven A. Lowe
@Steven: the people in that thread seem to be not calling Dispose() and getting file lock issues. I *am* calling Dispose() -- at the closing brace of my using(). Unless I'm reading that thread wrong?
Stewart Johnson
@[Stewart Johnson]: the way i read the thread, you also need to call Dispose on the Command object using the connection, due to a bug in the SqlLite provider
Steven A. Lowe
Ah okay, I'll give that a shot (when I'm at home -- don't have the code at work). Thanks.
Stewart Johnson
So I wrapped all my DbCommands in using() blocks as well, so they're being Dispose()d immediately before the connection is Dispose()d. It made no difference -- same problem. :-(
Stewart Johnson
@[Stewart Johnson]: I'm calling "bug" on SqLite then. You might try the developer mailing list http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-dev or pay for support or if it's open source debug the @#$% thing yourself. Sorry I can't help more, but my google-fu is exhausted!
Steven A. Lowe
Thanks for your help anyway!
Stewart Johnson
I found a solution! Adding GC.Collect() immediately before the File.Delete line in my [TearDown] method gets rid of the exceptions. When I'm debugging there are no longer any references to the connection so I'm not sure how the GC.Collect works, but it does.
Stewart Johnson
@[Stewart Johnson]: I'm glad that works for you, but I think that pretty much proves that this is a bug in SQLite - holding a lock on a file until an unreferenced object is finalized is baaaaad!
Steven A. Lowe
A: 

Tear down is executed after each test.

This attribute is used inside a TestFixture to provide a common set of functions that are performed after each test method is run.

You should try to delete them all with TestFixtureTearDown:

    [TestFixtureTearDown]
    public void finish()
    {
        //Delete all file
    }

Maybe one test is using the file that you try to delete in an other test. <-- [Stewart] As I stated in the question, it happens when I run only one test, so this isn't possible.

Update You do not give enough information about what you are doing. Have you try to clean up all the Test file with only 1 test in your test file and try it? [Stewart] Yes. If it works [Stewart] (it doesn't) then it's that you have multiple test problem (they access each other). You need to cut down the problem to find the source. Then, come back here we will help you. for the moment it's only guessing that we can give you. [Stewart] I have already done these cut-downs of the problem that you suggest, you'll find that they're in my original question!

Daok
I understand that [TearDown] is executed after each test -- that's exactly what I want. I don't keep a huge list of all the filenames, since each one is used for exactly one test, and I want to delete it at the end of the test. How will using [TestFixtureTearDown] make any difference?
Stewart Johnson
"Maybe one test is using the file that you try to delete in an other test." Not possible -- see my last comment in my question: I get the same result even when I run each test by itself in isolation.
Stewart Johnson
You do not give enough information about what you are doing. At least try what we suggest you. Have you try to clean up all the Test file with only 1 test and try it? If it works then it's that you have multiple test problem (they access each other). Bad question detail, bad answer sorry.
Daok
Yes, I've tried running a single test and having [TearDown] and [TestFixtureTearDown] delete the file, and I get the exact same problem. I think there is plenty of detail in my question. What more do you need to know?
Stewart Johnson
First, calm down. If you had so much good detail the problem would have been already solve. Second, down voting me will not solve your problem. You just lost someone who was ready to help you.
Daok
Second, detail about does it works in your principal program? What is the query? You say it fail then go to teardown, what happen when success query? etc.
Daok
@Daok - I'm not sure what you're talking about. I am calm. You're telling me there's not enough detail in the question, but you won't tell me what else you need. The Delete() fails no matter what query is performed. Why do you think it was me who downvoted you?
Stewart Johnson
+1 for hanging in there
Steven A. Lowe
@Daok: "You say it fail then go to teardown" -- I didn't say that. Everything works *except* that single line in [TearDown].
Stewart Johnson
A: 

I'd need to see your filename creation logic, but it is possible that you are opening the file to create it but not closing it once it is created. I think if you use System.IO.Path.GetTempFileName() it will just create the file and return you the name of the file with the file closed. If you are doing your own random name generation and using File.Open, you'll need to make sure it's closed afterwards.

On another note, I heartily recommend pursuing a mocking strategy to abstract out the database rather than reading/writing from an actual database in your unit tests. The point of unit tests is that they should be very fast and file I/O is going to really slow these down.

tvanfosson
The file creation logic is already there -- in the call to connection.Open(). SQLite creates the file for me.
Stewart Johnson
A: 

Do you have Antivirus running? Some AV products scan the file when they see them being closed. If the AV software still has the file open when you come to delete it, you'll see this problem. You could retry the delete after a small delay.

I've also seen this happen with search indexers.

Roger Lipscombe
Nope. No AV nor search indexers.
Stewart Johnson
A: 

First step would be to find who is holding a handle to the file in question.

Get a copy of Process Explorer Run it. Press Ctrl+F and enter the name of the file. It would show you a list of processes accessing it.

Gishu
+1  A: 

Is it possible that the file is in the process of being closed (I.e. SQLLite doesn't release it immediately)?

You could try putting the db close in a delay loop, (maybe one second between each attempt), and only throwing the exception after a few (5 or so) iterations through the loop.

Andrew Rollings
That sounds pretty hacky. I would've thought the explicit call to connection.Close() would cause SQLite to release the file (if indeed it is SQLite holding the file).
Stewart Johnson
you cheap hacker! Is that how you get things done so quickly?
Steven A. Lowe
Hehe... I didn't say it would be the final solution... It would help you diagnose if that's the problem though :)You have no direct control (other than calling Close()) over what SQLLite does with the file.
Andrew Rollings
A: 

What do you use to open your database ? Do you use the ADO 2.0 connector from here. If have an application that use it and I can do multiple connection with it (close/open). If you do not use this connector, you might give a try. What does your method Connect() return?

Daok
Connect() is right there in my question -- it returns a DbConnection, which is actually a SQLite DbConnection.
Stewart Johnson
(And yes, I am using System.Data.SQLite).
Stewart Johnson
I know Connect is in the question, does it return a valid connection? You can do your queries? You say it all fail then go to teardown, what happen if it doesn't fail?
Daok
Yes it returns a valid connection, since the database access works just fine.
Stewart Johnson
"You say it all fail then go to teardown" -- I didn't say that. The tests work perfectly, the only thing that fails is the single line in the [TearDown] method. If I comment out that line all my tests pass!
Stewart Johnson
A: 

Thanks for pointing this out!

The problem is not related to SQLite, but rather to memory management in general. After your test has run, the objects pointing to files have no scope anymore, but they still exist in memory.

Hence, there's still references to the files and you can't delete / move them.

A: 

I know this answer is over a year late, but for anyone reading in the future...

I had a similar problem to yours - attempting to delete test databases in between tests failed because of the SQLite database file remaining open. I traced the problem in my code to a SQLiteDataReader object not being explicitly closed.

SQLiteDataReader dr = cmd_.ExecuteReader();
while (dr.Read())
{
    // use the DataReader results
}
dr.Close();  //  <-- necessary for database resources to be released
jumpalongjim
A: 

Call static method

SqliteConnection.ClearAllPools()

After this call the database file is unlocked and you can delete the file in the [TearDown].

TTT