Design questions about a plugginable Java framework

Question

Hello,

During the last 4 years, I was working on a project in my spare time having two goals : to improve myself as a software engineer and to build something which(I hoped) it will help me to build faster and better Java applications. I've always liked modular applications, so I though a plugginable framework wouldn't be bad to have.

Since I'm always looking first before building something(no need to reinvent the wheel, right?) I was looking at Eclipse and NetBeans but there was something missing, it didn't have what I was looking for :

• Eclipse and NetBeans are great platforms for client side applications, nice plugin systems but how about writing server side applications based on plugins? How about client/server plugginable applications?
• a little to heavyweight if you just want to write a small application; also something like Swing Application Framework http://appframework.dev.java.net/ looks too simple to me (I'm not criticizing, it is a good starting point to write Swing applications - it has a small and simple API).
• unsuitable to build client side, server side or client/server applications with the same API(well not exactly the same for both sides but a more unified API).

Of course, during these years the framework went through a few major milestones, it was used in several projects, and it served its purpose(more or less). The API changed to meet some needs, but I wasn't satisfied by some design decisions which at that time seemed good. Furthermore, there are still some things which bother me and this is the source for my questions. Why am I asking these questions after so many years? Good question :) , maybe because now I'm thinking of make it public, and before I do it, I want to clarify some things(or maybe there wasn't a website like StackOverflow before ? :) )

First of all, let me explain very succinct what it provides:

• design based on plugins for modularization.
• application functionality is provided by services, different plugins create and register services, which provide a certain functionality : templates, persistence, scheduling, jobs/tasks execution, security, automatic update, etc.
• the application is described in an XML file, every application is structured in multiple modules, every module contains 1..N plugins;plugins are shared between modules(basically a small set of plugins creates the application core and the rest can differentiate the application in certain ways, like a different application UI. Of course this makes sense only for the client side, on the server side all the plugins are instantiated to provide support for any client type).
• every plugin is loaded in its own classloader and there is an application life cycle : load plugins, init plugins(at this time all services are registered), init services, post init services, post init plugins, init UI plugins, post init UI plugins, start plugin.
• every plugin has its own XML descriptor file to add/register/configure the application, there is an API to allow easily to configure the application through XML, similar with Apache Commons Digester.
• plugins were loaded from several application repositories, in the same was as Maven resolves dependencies; also Maven 1&2 repositories are supported.
• automatic application deployment (Ant task) within 5-10 lines(basically in 2-3 lines - task definition and invocation - but there are some settings involved about how to change the application layout - where to put the plugins, libraries, boot libraries, application startup scripts, application data, etc.)

Now here are the questions:

1 a) The previous version was built on the idea that the application should only a small code base : boot library - for basic application support - and the application APIs(the framework API + the application API/APIs).There was no visibility between plugins and every plugin has to import only a few jar files. This seemed as a nice thing to have, releasing the developer from the jar's/classes' hell, sort of code isolation, you see only what you need to see. This had also a few disadvantages, testing was not really unit testing, more like an integration testing since you only see the "public" API, and you start the whole thing and just query if the module works well as a whole. There were also some problems with classloading in certain cases(solved issue but ...).

b) Since the unit testing was the main issue, the current version changed the way how everything is loaded : exactly, as any regular application, you have to import every jar you need - of course those jars are also loaded dynamically at runtime. Classloading was changed so there would be no difference in how plugins/libraries are loaded at test time compared with runtime, you had access to every public implementation available in all imported plugins/libraries. In the end, even if the code is more testable I'm still ending in writing more like integration tests instead of unit tests because (I'm lazy) the framework it's still more handy when it comes to give me already instantiated services instead of providing lots of stubs/mocks. And the thing is I lost something that I had before, the simplicity of having just a small visible code base during development.

Question : Does the need of have better unit testing is worth more than a certain simplicity(in this context, of course)? It was really "better" or I just get used with the idea that was better and having more testable code(or at least the chance to write more testable code) means droping everything else?

2 One thing that I found useful was to keep the application instance available anytime attached to the current thread. Since the framework controls all the threads(there is no need to create threads, just create a Worker and schedule it) and all the threads are under a controlled environment(before and after the thread was executed the application context was attached to / removed from the thread) it was easier to just call ApplicationContext.getApplication() instead of passing the application around.

Question : Is this approach good practice or acceptable?

3 I've always thought to split the framework in several projects instead of a "monolitic" project and have a minimal core of plugins/services.

Question : What would you expect to see in a plugginable framework in terms of "services" / functionality (other than loading the application code and application life-cycle)?

I would appreciate any response and advice, and I hope this will not start a discussion like "oh no, not another framework, there are already too many...". My intention is to build something that will help me learn, and maybe to become useful to other developers.

Thanks.

Edit : Let's take a real life example : a newsletter manager(see http://www.phplist.com). So you're creating a new plugin called plugin-newsletter which will need persistence support(store things in DB), messaging support(send emails) and template support (to generate nice emails) The API provides a PersistanceService, MessagingService and a TemplateService and their implementations resides in plugin-persistance, plugin-messagin and plugin-template.

• PersistanceServices provides a similar functionality like Apache iBatis and uses SpringFramework JDBC module + an EvaluationService(expression language like JEXL, MVEL) in order to generate SQLs based on parameters. In conclusion, PersistanceService will bring a new dependency, plugin-el.
• NotificationService uses Java Mail API to send and receive notifications
• TemplateService provides an unified API for a few available template engines available(Velocity, FrameMaker, etc)

With for implementation (a) when the only thing visible to you was the API(mostly interfaces) and you had to load the application to have everything you need to test plugin-newsletter. Of course you could just implement on the spot everything you need to test plugin-newsletter but then you will have to do something similar also for other plugins, which mean duplicate code, which is not very elegant.

With the second implementation (b) you will see the implementation code for all plugins, including their libraries. You may instantiate only what you need to test a specific class. Of course, I end up with instantiating the application(but a test aware application instance, which doesn't resolve/load code since it is already in the class path) because I don't want to test a specific class with stubs(mainly because I'm lazy?) because I can write a test faster using the real application.

I know that in the spirit of TDD you should test your code in isolation with mocks/stubs and then write integration tests to see how different modules work together.This works great for libraries but when you're writing applications you want to use the application context and lots of tests looks like integration tests : a bunch of real modules are initialized and passed to my classed(or queried directly with ApplicationContext.getWorkerService() for example).

I was switching from (a) to (b) thinking it will be better, but I end up the opposite : I think the first version was better, the isolation offered by first version looks better than what I have now. Any opinions?

Answer 1

Q1: I guess you don't expect a plug-in programmer to rely on other code, but rather just assume that there will be a component that does X (separation of concerns), i.e. implements a certain interface Y? If so, I think extending the test scope should be fine; you will detect much more problems. Just be careful that you test the framework in different situations, e.g. with many/little plug-ins installed, etc.

Q2: I would think so... probably depends on the actual application (and how likely it is that the application instance won't be static at some point).

Q3: I would expect that a plug-in can clearly define its dependencies (on other plug-ins, libraries, etc.), and that they are able to find other plug-ins for specific tasks at runtime (e.g. if I want to store an internal data structure to a file - which plug-ins for exporting the data are available for me, to let the user choose?). A plug-in management that detects configuration problems and deactivates malfunctioning components is also nice to have (just as in Eclipse).

Answer 2

I have a similar interest and was considering OSGI because it allows independent components to co exist with their own dependencies without worrying about version clashes. The only problem that i need to solve is the ability to query a bundle for what implementators of an interface it wishes to expose in a standards fashion.

Answer 3

From what I heard, IBM's Jazz is very similar in purpose - so it might be interesting to see where it's similar to your framework and where it differs.