Designing an AJAX driven Quicksilver-style search with multiple search plugins for a website

Question

I'm trying to build a Quicksilver style search system for the internal web app that we develop at work. There are plenty of examples of really cool front ends for this using JQuery or MooTools or whatever. None of these examples really talk about the back-end. As far as I can tell, these examples assuming the back-end is searching a single table or at least, performing a single query. What I want to do is design a system where you can, literally, type anything at all at it and find what you were looking for. Idealy, I want to be able to just write plugins for this system, drop them in, and start searching.

I have a solution where the back-end uses the observer pattern to send the query to different plugins for each type of search. However, this will return the results from all the plug-ins as one chunk. This could get noticeably slow if there are many kinds of searches. I'd like it to be quick and return the results in a more asynchronous fashion where results are displayed as they come in, a la OS X's Spotlight or Quicksilver.

Another solution is to write, on the fly, a javascript array with the names of the plug-ins to be used. I could then fire off separate calls to the server with the query, one for each plug-in. Something about this solution seems... off to me. I can't exactly put my finger on it though.

So, my question is: does anyone have any better solutions for building a plug-in based search system where the individual search types are not known before the page is loaded and the results are returned ASAP?

Answer 1

You could do a Comet style solution that used long polling in Ajax to get results for the search. Make one place for the script to call that will give back the results of all the plugins as they come in. This method allows you to get the quick results displayed sooner.

Having an array of plugins is an option but some browsers are limited to 2 requests at a time so that would limit the amount of request just being kicked off and could cause a fast process to have to wait for the slow processes.

It sounds like you are getting close with with back end you have just make it provide up the data as it comes in. Also this will allow you to add and remove plugins on the fly without effecting the JS so no worries about cached array lists.

A few thoughts on the back end from comment. Build a work queue so search requests can be farmed out to many workers. It would be possible to implement the work queue in a DB or through a web service so you could use other languages or even computers to do the work for each search. The work call would need some id to pass back to direct the data at the correct client. Also you would want a way to remove jobs from the queue or at least mark all work for a client as void if that client goes away. (You should be able to detect this if you are using long polling.)

Connection limits

IE7 for HTTP1 4

IE7 for HTTP1.1 2

IE8 for HTTP1 6

IE8 for HTTP1.1 6

Answer 2

Another solution is to write, on the fly, a javascript array with the names of the plug-ins to be used. I could then fire off separate calls to the server with the query, one for each plug-in. Something about this solution seems... off to me. I can't exactly put my finger on it though.

This does not seem like that bad of an option. It gives you everything you need.

1. You need search results to come back as soon as they can.
2. It allows you to use your existing plugin architecture, I believe.
3. It follows the KISS principle.

It is not a new solution, but I think that it is the easiest.

Regards.

Answer 3

From all the comments and talk it seems like you want to build this on the front end.

Don't build an array of plugins to call it forces you to worry about caching when changing out plugins you should do instead is build a bootstrap system. It would be a simple ajax call that got a list of plugins with there URL to call. This will allow you to turn on and off plugins from a central location and it will work better.

You will have to make each of your plugins into a web service instead of a plugin so each can be called independently. Make sure to use mediasalve's link about the number of connections because it will be limited by browsers if you don't get around it.