Interesting question.
First, consider the difference between anonymous methods and lambdas. From the compiler writer's perspective, the most important difference is that lambdas can require the compiler to infer the type of the parameters from the target to which the lambda is being assigned; C# 2 anonymous methods do not have this feature. This feature seems like a small difference but in fact it has major ramifications on the implementation of the compiler. See my blog series on this topic for some thoughts on why that is:
http://blogs.msdn.com/ericlippert/archive/2007/01/10/lambda-expressions-vs-anonymous-methods-part-one.aspx
So now let's come to your actual question: why can we not infer outness/refness from the target type to the parameters of the lambda. That is, if we have delegate void D(out int x) then surely D d = x=> { x = 10; } could infer that x is "out int".
There's no technical reason I'm aware of why we could not do that. Internally in the compiler the out/ref types are represented as types like any other.
However, features do not get done just because they can be done; they get done because there's a compelling reason to do so. For lambdas, the compelling reason to do type inference in the first place is LINQ; we want to be able to do a simple syntactic transformation on a query comprehension into a method call with lambdas, and let the method type inference engine work out the types of all the lambda parameters. None of the LINQ methods generated have delegates with out or ref parameters.
So, we have no compelling reason to do the feature. Delegates which have out/ref parameters are relatively rare. And assignment of lambdas to those delegates is rarer still. So this is a feature that we don't need, and that benefits almost no one.
C# 3 was the "long pole" on the Visual Studio schedule; we had the most number of days of work scheduled of any team that ships a component in VS. That meant that every day we slipped the schedule, the entire division slipped. That was a powerful disincentive to spending time on unnecessary features that benefitted no one. So the work was never done.
I agree that it would be nice to be more consistent here, but it's unlikely to happen. We have many higher priorities.