How to solve Assembly.Load performance hit?

Question

I'm refactoring some code and I've been hit with a dilemma.

Let's say we have the following scenario:

1. A Core Assembly that contains many common interfaces and classes
2. A Library Assembly that contains more specialized classes.

The Library Assembly references the Core Assembly. So far so good.

Due the fact that I'm refactoring this, there's a need for the Core Assembly to create objects that are declared in the Library Assembly.

OK, in order to avoid a circular reference problem, I decided to load the Library Assembly when needed (and it's needed only in a very specific point at the type initialization).

However, the loading performance of the whole thing plummeted to a dark abyss...

Does anyone know how to solve this?

---

Edited to add

Some people have requested the code I use to load... It's quite trivial, really.

/*
 * Load the Library Assembly
 */
Assembly asm = Assembly.Load("Library, PublicKeyToken=...");

/*
 * Get desired type
 */
Type   t = asm.GetType("Library.DesiredType")

/*
 * Get the default constructor
 */
var ctor = type.GetConstructor(new Type[] {})

Answer 1

The assembly should only be loaded once into the AppDomain. Repeat calls to load assembly X should return the already-loaded assembly. Can you post the code for how you're attempting this? How are you measuring "performance"? Have you profiled your application to verify the performance hit is indeed coming from loading the assembly?