Puzzling Enumerable.Cast InvalidCastException

Question

The following throws an InvalidCastException.

IEnumerable<int> list = new List<int>() { 1 };
IEnumerable<long> castedList = list.Cast<long>();
Console.WriteLine(castedList.First());

Why?

I'm using Visual Studio 2008 SP1.

Answer 1

That's very odd! There's a blog post here that describes how the behaviour of Cast<T>() was changed between .NET 3.5 and .NET 3.5 SP1, but it still doesn't explain the InvalidCastException, which you even get if you rewrite your code thus:

var list = new[] { 1 };
var castedList = from long l in list select l;
Console.WriteLine(castedList.First());

Obviously you can work around it by doing the cast yourself

var castedList = list.Select(i => (long)i);

This works, but it doesn't explain the error in the first place. I tried casting the list to short and float and those threw the same exception.

Edit

That blog post does explain why it doesn't work!

Cast<T>() is an extension method on IEnumerable rather than IEnumerable<T>. That means that by the time each value gets to the point where it's being cast, it has already been boxed back into a System.Object. In essence it's trying to do this:

int i = 1;
object o = i;
long l = (long)o;

This code throws the InvalidCastException you're getting. If you try to cast an int directly to a long you're fine, but casting a boxed int back to a long doesn't work.

Certainly an oddity!

Answer 2

Hmm... interesting puzzle. All the more interesting given that I just ran it in Visual Studio 2008 and it didn't throw at all.

I'm not using Service Pack 1, and you might be, so that could be the issue. I know there were some "performance enhancements" in .Cast() in the SP1 release that could be causing the issue. Some reading:

Blog Entry 1

Blog Entry 2

Answer 3

Enumerable.Cast method is defined as following:

public static IEnumerable<TResult> Cast<TResult>(
    this IEnumerable source
)

And there is no information about initial type of IEnumerable's items, so I think each of your ints is initially converted to System.Object via boxing and then it's tried to be unboxed into long variable and this is incorrect.

Similar code to reproduce this:

int i = 1;
object o = i; // boxing
long l = (long)o; // unboxing, incorrect
// long l = (int)o; // this will work

So solution for your problem will be:

ints.Select(i => (long)i)

Answer 4

I wish they could have done something intelligent, like using any implicit or explicit cast operators defined for the type. The current behavior and inconsistency is unacceptable. Absolutely useless in its current condition.

After realizing that Cast<Type> was throwing an exception instead of using the cast operators that I defined for the type, I became irritated and found this thread. If it's defined for IEnumerable, why wouldn't they just implement it to use reflection to get the type of object, get the target type, discover any available static conversion operators, and find an appropriate one to do the cast. It could cast a heterogeneous IEnumerable into an IEnumerable<T>.

The following implementation is a working idea...

public static class EnumerableMinusWTF
{
 public static IEnumerable<TResult> Cast<TResult,TSource>(this IEnumerable<TSource> source)
 {
  Type source_type = typeof(TSource);
  Type target_type = typeof(TResult);

  List<MethodInfo> methods = new List<MethodInfo>();
  methods.AddRange( target_type.GetMethods( BindingFlags.Static | BindingFlags.Public ) ); //target methods will be favored in the search
  methods.AddRange( source_type.GetMethods( BindingFlags.Static | BindingFlags.Public ) );
  MethodInfo op_Explicit = FindExplicitConverstion(source_type, target_type, methods );

  List<TResult> results = new List<TResult>();
  foreach (TSource source_item in source)
   results.Add((TResult)op_Explicit.Invoke(null, new object[] { source_item }));
  return results;
 }

 public static MethodInfo FindExplicitConverstion(Type source_type, Type target_type, List<MethodInfo> methods)
 {
  foreach (MethodInfo mi in methods)
  {
   if (mi.Name == "op_Explicit") //will return target and take one parameter
    if (mi.ReturnType == target_type)
     if (mi.GetParameters()[0].ParameterType == source_type)
      return mi;
  }
  throw new InvalidCastException( "Could not find conversion operator to convert " + source_type.FullName + " to " + target_type.FullName + "." );
 }
}

I can then call this code successfully:

    //LessonID inherits RegexConstrainedString, and has explicit conversion operator defined to convert string to LessonID
List<string> lessons = new List<String>(new string[] {"l001,l002"});
IEnumerable<LessonID> constrained_lessons = lessons.Cast<LessonID, string>();

Answer 5

Here are some things to think about...

1. Do you want to cast or convert?
   
2. Do you want the result as a List<T> or an IEnumerable<T>.
   
3. If the result is an IEnumerable<T>, do you want the cast/convert to be applied lazily (i.e. the cast/convert will not actually occur until the iterator reaches each element)?

Useful distinction between cast/convert, since a casting operator often involves constructing a new object, and could be considered a conversion:
"Cast" implementations should automatically apply conversion operators defined for the types involved; a new object may or may not be constructed.
"Convert" implementations should allow one to specify a System.Converter<TInput,TOutput> delegate.

Potential method headers:

List<TOutput> Cast<TInput,TOutput>(IEnumerable<TInput> input);
List<TOutput> Convert<TInput,TOutput>(IEnumerable<TInput> input, Converter<TInput,TOutput> converter);
IEnumerable<TOutput> Cast<TInput,TOutput>(IEnumerable<TInput> input);
IEnumerable<TOutput> Convert<TInput,TOutput>(IEnumerable<TInput> input, Converter<TInput,TOutput> converter);

Problematic "Cast" implementations using existing framework; suppose you pass as input a List<string>, that you want to convert with any of the previous methods.

//Select can return only a lazy read-only iterator; also fails to use existing explicit cast operator, because such a cast isn't possible in c# for a generic type parameter (so says VS2008)
list.Select<TInput,TOutput>( (TInput x) => (TOutput)x );
//Cast fails, unless TOutput has an explicit conversion operator defined for 'object' to 'TOutput'; this confusion is what lead to this topic in the first place
list.Cast<TOuput>();

Problematic "Convert" implementations

//Again, the cast to a generic type parameter not possible in c#; also, this requires a List<T> as input instead of just an IEnumerable<T>.
list.ConvertAll<TOutput>( new Converter<TInput,TOuput>( (TInput x) => (TOutput)x ) );
//This would be nice, except reflection is used, and must be used since c# hides the method name for explicit operators "op_Explicit", making it difficult to obtain a delegate any other way.
list.ConvertAll<TOutput>(
 (Converter<TInput,TOutput>)Delegate.CreateDelegate(
  typeof(Converter<TInput,TOutput>),
  typeof(TOutput).GetMethod( "op_Explicit", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.Public )
 )
);

Summary:
Cast/Convert methods should involve defined explicit conversion operators, or allow one to specify a conversion delegate. C#'s language specification for conversion operators -- specifically their lack of a method name -- makes it difficult to obtain a delegate except through reflection. The alternative is to encapsulate or replicate the conversion code, increasing the (maintenance) complexity of your code unnecessarily, since really, possible/allowed conversions are implicit in the presence or absence of conversion operators, and should be handled by the compiler. We should not have to manually search for cryptically-named definitions (e.g. "op_Explicit") of appropriate conversion operators with reflection at RUN TIME on the types involved. Furthermore, Cast/Convert methods for bulk/list conversions using explicit conversion operators should really be a framework feature, and with List.ConvertAll<T>, they are... except the language spec makes it hard to get a delegate for the conversion operators efficiently!!!

Answer 6

I'm at it again!
Here's the solution to all your List<T> and Enumerable<T> conversion problems. ~150 lines of code
Just be sure to define at least one explicit or implicit conversion operator for the input/output types involved (if one does not exist), as you should be doing anyway!

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Reflection;

namespace System.Collections.Generic //purposely in same namespace as List<T>,IEnumerable<T>, so extension methods are available with them
{
    public static class Enumerable
    {
     public static List<TOutput> ConvertAll<TInput,TOutput>( this IEnumerable<TInput> input ) {
      return BuildConvertedList<TInput,TOutput>( input, GetConverterDelegate<TInput,TOutput>() );
     }

     public static IEnumerable<TOutput> ConvertAll<TInput,TOutput>( this IEnumerable<TInput> input, bool lazy ) {
      if (lazy) return new LazyConverter<TInput,TOutput>( input, GetConverterDelegate<TInput,TOutput>() );
      return BuildConvertedList<TInput,TOutput>( input, GetConverterDelegate<TInput,TOutput>() );
     }

     public static List<TOutput> ConvertAll<TInput,TOutput>( this IEnumerable<TInput> input, Converter<TInput, TOutput> converter ) {
      return BuildConvertedList<TInput,TOutput>( input, converter );
     }

     public static List<TOutput> ConvertAll<TInput, TOutput>( this List<TInput> input ) {
      Converter<TInput, TOutput> converter = GetConverterDelegate<TInput,TOutput>();
      return input.ConvertAll<TOutput>( converter );
     }

     public static IEnumerable<TOutput> ConvertAll<TInput, TOutput>( this List<TInput> input, Converter<TInput, TOutput> converter, bool lazy ) {
      if (lazy) return new LazyConverter<TInput, TOutput>( input, converter );
      return input.ConvertAll<TOutput>( converter );
     }

     public static List<TOutput> ConvertAll<TInput, TOutput>( this List<TInput> input, Converter<TInput, TOutput> converter ) {
      return input.ConvertAll<TOutput>( converter );
     }

     //Used to manually build converted list when input is IEnumerable, since it doesn't have the ConvertAll method like the List does
     private static List<TOutput> BuildConvertedList<TInput,TOutput>( IEnumerable<TInput> input, Converter<TInput, TOutput> converter ){
      List<TOutput> output = new List<TOutput>();
      foreach (TInput input_item in input)
       output.Add( converter( input_item ) );
      return output;
     }

     private sealed class LazyConverter<TInput, TOutput>: IEnumerable<TOutput>, IEnumerator<TOutput>
     {
      private readonly IEnumerable<TInput> input;
      private readonly Converter<TInput, TOutput> converter;
      private readonly IEnumerator<TInput> input_enumerator;

      public LazyConverter( IEnumerable<TInput> input, Converter<TInput, TOutput> converter )
      {
       this.input = input;
       this.converter = converter;
       this.input_enumerator = input.GetEnumerator();
      }

      public IEnumerator<TOutput> GetEnumerator() {return this;} //IEnumerable<TOutput> Member
      IEnumerator IEnumerable.GetEnumerator() {return this;} //IEnumerable Member
      public void Dispose() {input_enumerator.Dispose();} //IDisposable Member
      public TOutput Current {get {return converter.Invoke( input_enumerator.Current );}} //IEnumerator<TOutput> Member
      object IEnumerator.Current {get {return Current;}} //IEnumerator Member
      public bool MoveNext() {return input_enumerator.MoveNext();} //IEnumerator Member
      public void Reset() {input_enumerator.Reset();} //IEnumerator Member
     }

     private sealed class TypeConversionPair: IEquatable<TypeConversionPair>
     {
      public readonly Type source_type;
      public readonly Type target_type;
      private readonly int hashcode;

      public TypeConversionPair( Type source_type, Type target_type ) {
       this.source_type = source_type;
       this.target_type = target_type;
       //precalc/store hash, since object is immutable; add one to source hash so reversing the source and target still produces unique hash
       hashcode = (source_type.GetHashCode() + 1) ^ target_type.GetHashCode();
      }

      public static bool operator ==( TypeConversionPair x, TypeConversionPair y ) {
       if ((object)x != null) return x.Equals( y );
       if ((object)y != null) return y.Equals( x );
       return true; //x and y are both null, cast to object above ensures reference equality comparison
      }

      public static bool operator !=( TypeConversionPair x, TypeConversionPair y ) {
       if ((object)x != null) return !x.Equals( y );
       if ((object)y != null) return !y.Equals( x );
       return false; //x and y are both null, cast to object above ensures reference equality comparison
      }

      //TypeConversionPairs are equal when their source and target types are equal
      public bool Equals( TypeConversionPair other ) {
       if ((object)other == null) return false; //cast to object ensures reference equality comparison
       return source_type == other.source_type && target_type == other.target_type;
      }

      public override bool Equals( object obj ) {
       TypeConversionPair other = obj as TypeConversionPair;
       if ((object)other != null) return Equals( other ); //call IEqualityComparer<TypeConversionPair> implementation if obj type is TypeConversionPair
       return false; //obj is null or is not of type TypeConversionPair; Equals shall not throw errors!
      }

      public override int GetHashCode() {return hashcode;} //assigned in constructor; object is immutable
     }

     private static readonly Dictionary<TypeConversionPair,Delegate> conversion_op_cache = new Dictionary<TypeConversionPair,Delegate>();

     //Uses reflection to find and create a Converter<TInput, TOutput> delegate for the given types.
     //Once a delegate is obtained, it is cached, so further requests for the delegate do not use reflection*
     //(*the typeof operator is used twice to look up the type pairs in the cache)
     public static Converter<TInput, TOutput> GetConverterDelegate<TInput, TOutput>()
     {
      Delegate converter;
      TypeConversionPair type_pair = new TypeConversionPair( typeof(TInput), typeof(TOutput) );

      //Attempt to quickly find a cached conversion delegate.
      lock (conversion_op_cache) //synchronize with concurrent calls to Add
       if (conversion_op_cache.TryGetValue( type_pair, out converter ))
        return (Converter<TInput, TOutput>)converter;

      //Get potential conversion operators (target-type methods are ordered first)
      MethodInfo[][] conversion_op_sets = new MethodInfo[2][] {
       type_pair.target_type.GetMethods( BindingFlags.Static | BindingFlags.Public | BindingFlags.FlattenHierarchy ),
       type_pair.source_type.GetMethods( BindingFlags.Static | BindingFlags.Public | BindingFlags.FlattenHierarchy )
      };

      //Find appropriate conversion operator,
      //favoring operators on target type in case functionally equivalent operators exist,
      //since the target type's conversion operator may have access to an appropriate constructor
      //or a common instance cache (i.e. immutable objects may be cached and reused).
      for (int s = 0; s < conversion_op_sets.Length; s++) {
       MethodInfo[] conversion_ops = conversion_op_sets[s];
       for (int m = 0; m < conversion_ops.Length; m++)
       {
        MethodInfo mi = conversion_ops[m];
        if ((mi.Name == "op_Explicit" || mi.Name == "op_Implicit") && 
         mi.ReturnType == type_pair.target_type &&
         mi.GetParameters()[0].ParameterType.IsAssignableFrom( type_pair.source_type )) //Assuming op_Explicit and op_Implicit always have exactly one parameter.
        {
         converter = Delegate.CreateDelegate( typeof(Converter<TInput, TOutput>), mi );
         lock (conversion_op_cache) //synchronize with concurrent calls to TryGetValue
          conversion_op_cache.Add( type_pair, converter ); //Cache the conversion operator reference for future use.
         return (Converter<TInput, TOutput>)converter;
        }
       }
      }
      return (TInput x) => ((TOutput)Convert.ChangeType( x, typeof(TOutput) )); //this works well in the absence of conversion operators for types that implement IConvertible
      //throw new InvalidCastException( "Could not find conversion operator to convert " + type_pair.source_type.FullName + " to " + type_pair.target_type.FullName + "." );
     }
    }
}

Sample use:

using System;
using System.Collections.Generic;

namespace ConsoleApplication1
{
    class Program
    {
     static void Main(string[] args)
     {
      List<string> list = new List<string>(new string[] { "abcde", "abcd", "abc"/*will break length constraint*/, "ab", "a" });
      //Uncomment line below to see non-lazy behavior.  All items converted before method returns, and will fail on third item, which breaks the length constraint.
      //List<ConstrainedString> constrained_list = list.ConvertAll<string,ConstrainedString>();
      IEnumerable<ConstrainedString> constrained_list = list.ConvertAll<string,ConstrainedString>( true ); //lazy conversion; conversion is not attempted until that item is read
      foreach (ConstrainedString constrained_string in constrained_list) //will not fail until the third list item is read/converted
       System.Console.WriteLine( constrained_string.ToString() );
     } 

     public class ConstrainedString
     {
      private readonly string value;
      public ConstrainedString( string value ){this.value = Constrain(value);}
      public string Constrain( string value ) {
       if (value.Length > 3) return value;
       throw new ArgumentException("String length must be > 3!");
      }
      public static explicit operator ConstrainedString( string value ){return new ConstrainedString( value );}
      public override string ToString() {return value;}
     }
    }
}