I got the answer to the problem (with some help from one of my colleagues)
Here is the solution:
1. The program needs to add (pseudo?)entries to the IIS metadata before it access the file/folder under the virtual directory, before we access the entry:
try
{
// make pseudo entries:
DirectoryEntry folder = rootDir.Children.Add("Folder", "IISWebDirectory");
folder.CommitChanges();
file = folder.Children.Add("File.aspx", "IISWebFile");
file.CommitChanges();
}
Then voila it works
PS:
DirectoryEntry dir = new DirectoryEntry("IIS://" + serverName + "/W3SVC/1/ROOT" + vDir, @"adminusername", @"password");
dir.AuthenticationType = AuthenticationTypes.Secure;
dir.RefreshCache();
Directory.Refresh does not help