I am assuming you are trying to access a file in the local filesystem.
If so, you cannot access files like that. Silverlight does not have the access priveleges u expect. If you want to add a file to your Silverlight Application at runtime. You will need to have Silverlight 4, running Out of the Browser with Elevated priveleges. There are certain limitations to this too. You can only access files in Special Folders like My Documents, Pictures, Music etc. For more info about access files this way. You can look at John's tutorials on Silverlight 4 elevated priveleges in Channel 9 MSDN.
I would doubt your FileInfo usage too. Here is a sample code to get file data using a simple drag and drop feature.
private void list_Drop(object sender, DragEventArgs e)
{
FileInfo[] files = (FileInfo[])e.Data.GetData(DataFormats.FileDrop);
for(int i=0;i<files.Length;i++)
textblock.Text += files[i].Name;
}
You can get the properties of the file such as "Name". You wil not hit any access denied errors. You cannot access properties like "DirectoryName", "FullName" etc. The reason being they are declared as SecurityCritical properties for Security reasons. The advantage of elevated permissions is that you can get to local file system (special folders) to access the FullName and DirectoryName properties without any exceptions.
Hope this helps