Plugin architecture for file type recognition

Question

I'm working on an app that is needs to be able to process "file paths" to import resources of different kinds. The issue is that in different situations (OS's for example), some of the file kinds are not applicable. What I want to do is define a very basic set of types that everything supports and then have a plug in architecture that supports adding new types with minimal code modification, ideally (for now) adding a single line to main() per type and linking in the needed files. Later I would like to be able to do this loading at run time via DLL/SO loading.

For now the file types I need to support include

• binary files
• text files
• directories
• zip files
• any of the above in zip files

in the future I'd like to support things like:

• HTML/XML files
• docx files
• image files
• MP3 files
• SQL sever quieries
• HTTP URL's
• e-mail via POP/IMAP
• etc.

A solution to this problem would have to support (at least) two levels of processing.

1. "path" based identification to determine where to look (local files vs. MySQL vs. HTTP va. MSSQL...)
2. resource type content probing to determine the exact type of the resource once it is found. (it's file, what type of file?)

Another part of the same app solves a simalar problem for output by building a hash table of output types and output handlers where the plugins register themselves with the main app.

The big question end up being: Does anyone have any advice as to how to proceeds? Has anyone build a simalar system and have any advice based on that experiences?

Answer 1

This reminds me of the XMMS2 plugin chain. XMMS2 is a music player which is able to play various formats from various sources (files, http, shoutcast, daap, ...).

The basic idea of the XMMS2 plugin chain is that every plugin registers what kind of input it may process and what its output will be (ranging from unknown/encoded binary to 44khz 16bit integer audio).

Transport plugins like file or daap handle strings with a specific prefix (file:// daap://). Then, they output binary, which will then be chained to the next suitable plugin. So the mp3 plugin uses a such called "magic" test to recognize if what's coming in is an mp3 stream. If it is not, the next plugin (like ogg vorbis, or modplug) will try.

You can do the same, for examply magically recognizing ZIP files (they have a standard prefix), so you have a plugin unpacking them in your chain. Then, several ressource plugins like PNG, JPG could further recognize the output.

An interesting feature of this design is that there is no architectural difference between a plugin talking to a webserver, a plugin unpacking zip files or a plugin reading in a piece of music / a picture. You could for example even read from a zip file within a zip file. Plugins don't have to know each other or talk to each other, they only pass the data further down the chain.

If you are interested in the implementation there - which is obviously best at dealing with audio - you can find the project here: http://wiki.xmms2.xmms.se/ Did I mention that the code is x-platform and plugins are loadable libraries?