Best Practices for Building a Search App?

Question

I will be starting a simple datastore-and-search project soon. Basically, one of those "put my huge Excel spreadsheet into a database, build a web GUI for it, and make it searchable" type-things.

One thing that's been bugging me is the actual search logic that will be used when the user enters some criteria. I am imagining a search interface with a text field, and a few other filtering tools - drop down combo boxes and check boxes and such.

While that gives me very strong, granular control over the filtering I can perform, I am wondering what SO's thoughts are on actually performing the search. I'll be using ASP.NET, MS SQL Server, and Linq-To-SQL here, so keep those technologies in mind.

Off the top of my head, I think I'd do something like:

var results = from s in db.Stuff
              where (s.Prop1.Contains(textFilter) ||
                     s.Prop2.Contains(textFilter) ||
                     s.Prop3.Contains(textFilter)) &&
                     checkbox1.IsChecked ?
                          s.Prop4.ToLower().Equals(combobox1.Text) : true
              select s;

Here's what I know:

• How to do grouping and joins if necessary
• I can use the Contains() method on individual properties to generate SQL LIKE queries
• I can filter things property-by-property, building my search logic as above.

Here's what I'm asking:

• Is there a way to search all properties (without pulling all objects into memory - which I assume means building a list of each object's properties with reflection, stringifying them, and then checking is out)? If not, this seems incredibly cumbersome as I'd have to build new logic for every new property I might add. Something like s.Contains(textFilter) in the above would be ideal.
• How does a SQL LIKE query actually work? Is that something I want to do?
• Is there a standard way of implementing search rules such as quoted strings for full-matching and logical operators such as AND and OR? I would be surprised if every application that implemented them did so with custom parsing logic.
• Am I barking up the wrong tree? Did I miss something?

Answer 1

I had to create a similar search for a comments system recently. What I did was I created some extension methods off of the comments which allowed me to pass in a generic filtering object.

Here is the sample code that I used:

This is just a partial method and does not have the return but it will give you a picture of what I am doing:

public List<oComment> GetComments(oCommentSearch filters)
{

    using (CommentDataContext db = CommentContextFactory.CreateContext())
    {
        var query = from comment in db.COMMENTs.FilterComments(filters)
                    select comment;
    }
}

As you can see off the COMMENTs i have FilterComments. This is an extension method. This method looks like this (this is the entire class I have):

public static class CommentExtensions
    {
        public static IQueryable<COMMENT> FilterComments(this IQueryable<COMMENT> Comments, oCommentSearch Filters)
        {
            Filters = CheckFilter(Filters);

            IQueryable<COMMENT> tempResult = Comments;

            if(Filters.Classes.Count() > 0)
            {
                tempResult = from t in tempResult
                             where
                                 Filters.Classes.Contains(t.CLASS_ID)
                             select t;
            }

            if (Filters.Flags.Count() > 0)
            {
                tempResult = from t in tempResult
                             where
                                 Filters.Flags.Contains((int) t.FLAG_ID)
                             select t;
            }

            if (Filters.Types.Count() > 0)
            {
                tempResult = from t in tempResult
                             where
                                 Filters.Types.Contains(t.CommentTypeId)
                             select t;
            }
            return tempResult;
        }

        private static oCommentSearch CheckFilter(oCommentSearch Filters)
        {
            Filters.Classes  = CheckIntArray(Filters.Classes);
            Filters.Flags =  CheckIntArray(Filters.Flags) ;
            Filters.Types =  CheckIntArray(Filters.Types) ;
            return Filters;
        }

        private static int[] CheckIntArray(int[] ArrayToCheck)
        {
            return ArrayToCheck == null || ArrayToCheck.Count() == 0 ? new int[] {} : ArrayToCheck;
        }
    }

This should get you started in the right direction for what you are trying to do.

Hope this helps!

Answer 2

You didn't mention it in your list of technologies that you're using, but don't overlook using Lucene.NET. It does searching very well and is fairly easy to setup. Basically, you add documents to an index, and Lucene efficiently manages the index. That way, you can search the index instead of loading documents one by one and looking at their properties.

Answer 3

Is there a way to search all properties (without pulling all objects into memory - which I assume means building a list of each object's properties with reflection, stringifying them, and then checking is out)? If not, this seems incredibly cumbersome as I'd have to build new logic for every new property I might add. Something like s.Contains(textFilter) in the above would be ideal.

We are using MsSql full text search functionality for this.