Error activating more than one site's Publishing (Site) Feature concurrently

Question

We have a custom self-service SharePoint solution that allows people to come to a provisioning SharePoint app, follow a 3-page wizard, answer a bunch of questions, and based on those questions, calls one of several web services around the world which in turn in the local farm creates a new site collection based on one of 3 site types (Team, Project, or Community) - all sites are created as an out-of-box Team Site that have the Publishing Features stapled to it, which we then (in code) modify in a number of ways (create libraries, drop content types, etc).

The problem arises when two users try to create the same type of site (i.e. 2 Project sites, or 2 Community sites) in the same farm. Activating the site collection publishing feature seems to fail, and I can't figure out why. Naturally, when we attach a debugger to walk through, it all works fine. We've reviewed all SPSite/SPWeb disposal patterns and everything else we could think of - problem being, with the crash happening within Microsoft assemblies, it's a hard fix.

This is a sample crash, taken from the SharePoint ULS log. Two site creations were launched within seconds of each other; the first succeeded, the second failed (the second always loses). I've manually added line breaks to make reading it easier. Any ideas?

Failed to activate feature 'PublishingSite' (Id f6924d36-2fa8-4f0b-b16d-06b7250180fa) associated with site template 'STS#0' at scope "http://projects/sites/heisenbug2".

Exception: Microsoft.SharePoint.SPException: Provisioning did not succeed.
Details: Failed to create the 'Reusable Content' library.
OriginalException: Cannot remove file "Item". Error Code: 16388. ---> 

Microsoft.SharePoint.SPException: Cannot remove file "Item". Error Code: 16388. ---> 

System.Runtime.InteropServices.COMException (0x81070207): Cannot remove file "Item". Error Code: 16388.
     at Microsoft.SharePoint.Library.SPRequestInternalClass.AddOrDeleteUrl(String bstrUrl, String bstrDirName, Boolean bAdd, UInt32 dwDeleteOp, Int32 iUserId, Guid& pgDeleteTransactionId)
     at Microsoft.SharePoint.Library.SPRequest.AddOrDeleteUrl(String bstrUrl, String bstrDirName, Boolean bAdd, UInt32 dwDeleteOp, Int32 iUserId, Guid& pgDeleteTransactionId)

     --- End of inner exception stack trace ---

     at Microsoft.SharePoint.Library.SPRequest.AddOrDeleteUrl(String bstrUrl, String bstrDirName, Boolean bAdd, UInt32 dwDeleteOp, Int32 iUserId, Guid& pgDeleteTransactionId)
     at Microsoft.SharePoint.SPFolder.DeleteCore(DeleteOp deleteOp)
     at Microsoft.SharePoint.SPFolder.Delete()
     at Microsoft.SharePoint.SPContentType.DeleteFolder()
     at Microsoft.SharePoint.SPContentTypeCollection.Delete(SPContentTypeId id)
     at Microsoft.SharePoint.Publishing.Internal.ProvisioningHelper.ReplaceAllContentTypesOnList(SPList list, SPContentType contentType)
     at Microsoft.SharePoint.Publishing.Internal.RootProvisioner.CreateReusableTextList()

     --- End of inner exception stack trace ---

     at Microsoft.SharePoint.Publishing.Internal.RootProvisioner.CreateReusableTextList()
     at Microsoft.SharePoint.Publishing.Internal.RootProvisioner.<Provision>b__0()
     at Microsoft.SharePoint.Publishing.CmsSecurityUtilities.RunWithAllowUnsafeUpdates(SPWeb web, CodeToRun secureCode)
     at Microsoft.SharePoint.Publishing.Internal.RootProvisioner.Provision()
     at Microsoft.SharePoint.Publishing.PublishingResourcesFeatureHandler.<>c__DisplayClass2.<FeatureActivated>b__0()
     at Microsoft.SharePoint.Publishing.CmsSecurityUtilities.RunWithWebCulture(SPWeb web, CodeToRun webCultureDependentCode)
     at Microsoft.SharePoint.Publishing.PublishingResourcesFeatureHandler.FeatureActivated(SPFeatureReceiverProperties receiverProperties)
     at Microsoft.SharePoint.SPFeature.DoActivationCallout(Boolean fActivate, Boolean fForce)
     at Microsoft.SharePoint.SPFeature.Activate(SPSite siteParent, SPWeb webParent, SPFeaturePropertyCollection props, Boolean fForce)
     at Microsoft.SharePoint.SPFeatureCollection.AddInternal(Guid featureId, SPFeaturePropertyCollection properties, Boolean force, Boolean fMarkOnly)
     at Microsoft.SharePoint.SPFeatureCollection.Add(Guid featureId, Boolean force)
     at Microsoft.SharePoint.SPFeatureCollection.CheckSameScopeDependency(SPFeatureDefinition featdefDependant, SPFeatureDefinition featdefDependency, Boolean fActivateHidden, Boolean fForce)
     at Microsoft.SharePoint.SPFeatureCollection.CheckFeatureDependency(SPFeatureDefinition featdefDependant, SPFeatureDependency featdep, Boolean fActivateHidden, Boolean fForce)
     at Microsoft.SharePoint.SPFeatureCollection.CheckFeatureDependencies(Guid featidDependee, Boolean fActivateHidden, Boolean fForce)
     at Microsoft.SharePoint.SPFeatureCollection.AddInternal(Guid featureId, SPFeaturePropertyCollection properties, Boolean force, Boolean fMarkOnly)
     at Microsoft.SharePoint.SPFeatureCollection.Add(Guid featureId, SPFeaturePropertyCollection properties, Boolean fForce)
     at Microsoft.SharePoint.SPTemplateAssociationElement.EnsureTemplateAssociatedSiteFeaturesActivated(SPSite site, String sTemplateName)

Answer 1

It's hard to know where the error lies, but if it's a shared-memory concurrency error, Jinx might help you track it down. Jinx controls thread interleaving to force heisenbugs to appear more quickly and predictably. http://petravm.com/jinx

Answer 2

I don't know sharepoint, I've never had this problem, I'm just playing captain obvious here.

From what I understand, the bug doesn't occur when to people create sites of a different type. That and the error messages "Failed to create the 'Reusable Content' library" and "Cannot remove file "Item"" indicates to me, that somewhere they may both be trying access the same directory structure while creating a new site.

Possibly both sites use a common index as the basis for the directory structure of the site and thus run into a collision when trying to "CreateReusableTextList()".

This would also explain why Team, Project, or Community sites don't collide, since they probably use different root directories.