There are two ways to go about this. The first involves using the Plug In / Module pattern. Start here for that.
A second way is to simply expose your API logic through well documented assemblies.
The way you pick really boils down to how you want it implemented. For example, Adobe Photoshop uses the plug-in pattern to great extent. MS Office uses both plug-ins as well as provides an object interface for working with the application.
UPDATE:
Objects in .Net assemblies can be exposed to be consumed by external applications simply by setting the objects scope to Public. This includes windows forms, business objects, enum's, etc.
Just think about it from the external developer's perspective. What information will they need to have in order to be successful at extending / controlling your product? Whatever that is, put it in the help file.
Also, make use of the auto-documentation features of .Net. This includes adding xml documentation the the IDE already knows how to read and display via intellisense. For example:
/// <summary>
/// Gets all active documents for the specified customer.
/// </summary>
/// <param name="customerId">The Id of the customer.</param>
/// <returns>A list of documents for this customer.</returns>
public static List<Document> GetDocuments(int customerId)
{
//... do something ...
}