Troubleshooting failover cluster problem in W2K8 / SQL05

Question

I have an active/passive W2K8 (64) cluster pair, running SQL05 Standard. Shared storage is on a HP EVA SAN (FC).

I recently expanded the filesystem on the active node for a database, adding a drive designation. The shared storage drives are designated as F:, I:, J:, L: and X:, with SQL filesystems on the first 4 and X: used for a backup destination.

Last night, as part of a validation process (the passive node had been offline for maintenance), I moved the SQL instance to the other cluster node. The database in question immediately moved to Suspect status.

Review of the system logs showed that the database would not load because the file "K:\SQLDATA\whatever.ndf" could not be found. (Note that we do not have a K: drive designation.)

A review of the J: storage drive showed zero contents -- nothing -- this is where "whatever.ndf" should have been.

Hmm, I thought. Problem with the server. I'll just move SQL back to the other server and figure out what's wrong..

Still no database. Suspect. Uh-oh. "Whatever.ndf" had gone into the bit bucket.

I finally decided to just restore from the backup (which had been taken immediately before the validation test), so nothing was lost but a few hours of sleep.

The question: (1) Why did the passive node think the whatever.ndf files were supposed to go to drive "K:", when this drive didn't exist as a resource on the active node?

(2) How can I get the cluster nodes "re-syncd" so that failover can be accomplished?

I don't know that there wasn't a "K:" drive as a cluster resource at some time in the past, but I do know that this drive did not exist on the original cluster at the time of resource move.

Answer 1

Random thought based on what happened to me a few months ago... sound quite similar

Do you have NFTS mount points? I forget what it was exactly (me code monkey and relied on DBAs), but the mount points were either "double booked" or not part of the cluster resource or the SAN volumes were not configured correctly.

We had "zero size" drives (I used xp_fixeddrives) for our log files but we could still write to them.

Assorted reboots and failovers were unsuccessful. Basically, it was a thorough review of all settings in the SAN management tool.

A possibility for your K: drive...

The other thing I've seen is the mounted drives have letters as well as being mounted in folders. I used to use mounted folders for SQL Server but the backup system used a direct drive letter.