I don't think that System.Numerics.dll is part of the Silverlight 4.0 distribution. But that's not the real point. What you are looking at is a special version of the reference assembly. You'll find one in c:\program files\reference assemblies\microsoft\framework\.netframework\v4.0
for example.
The assemblies there have all their IL stripped from them. Their metadata is otherwise identical to the "real" reference assemblies in c:\windows\microsoft.net\framework\v4.0.30319
I have no clue what the function of these stripped assemblies is supposed to be. I can only imagine they are meant to speed-up compilation, since the compiler only needs the metadata. But that's a bit of a long shot. I can also imagine it has something to do with the mysterious new [TargetedPatchingOptOut] attribute, a very long shot as well since the mechanism behind it isn't explained anywhere that I could find. I had a conversation with JaredPar about this, he was going to ask inside MSFT about it. Haven't heard back.
Well, no real answer, but it does explain what you saw.
Following up on this, I do have one more theory, inspired when I noted that the folder is named "v4.0". Note that the build number is not part of it, as it is in c:\windows\microsoft.net. That ought to have interesting effects when they release new builds, similar to the updates to the base assemblies of .NET 2.0 when the service packs were released.
Infamously, one thing that went badly wrong is that these updates had changes in the core classes, without a change to the [AssemblyVersion]. The most visible one was the WaitHandle class, it acquired the WaitOne(int) overload. Very useful, because nobody could ever figure out what to pass for the exitContext argument. Using this new overload was easy to do, targeting .NET 2.0 doesn't prevent it, but hell breaks loose if the target machine has the original .NET 2.0 RTM release installed without getting the service packs.
My guess: these reference assemblies are the core assemblies for any current and future version of .NET 4.0. Their public interface is frozen. And prevents you from accidentally using a public method that gets added in a later build. It follows that IL isn't useful because that's going to change.