Jim Holden Blog

Change for the better

November 23rd, 2005

Work has decided to allow Fridays (assuming other things are ok) to be a day where we do new innovative research projects.

We’re currently looking closely at building a comprehensive ajax based search system for the site and it’s going to be something remarkable. Stepping back from our current search problem, the solution seems simple but it’s taken a while to get there and some time to see it as such.

Part of the problem is trying to search data and producing a decent ranking algorithm. There are always search systems out there but it’s often the case that the kind of tweaks we need are such that developing a bespoke system is a better solution. Caching aside, the raw data comes from sql and i’ve come up with a decent ranking system for the search which produces pretty accurate results at fast speeds.

if ( fieldName like 'keyword', 20,0) + if ( fieldName like '% keyword %',15,0) + if ( fieldName like '% keyword', 10,0) ) as ranking

Now, whilst that is a reduced version it somewhat highlights my ideas about it. Matching on words in different ways produces differing ranking values, which when summed up produce and overall ranking value. Other conditions such as the popularity of the item, “custom tags”, section etc all add to the overall ranking. Most of the ranking values can be added to a small table or file and adjusted outside the code which enables fine tweaking to the search system without redeployment which is always a good thing.

The other guys are seperated out to work on the Ajax implementation, the automated (cron) system which will produce the core indexed data table and the sql database tests (we are going to trial mysql 5 and postgres). I’ve been into Ajax and the whole web2 thing for a while now and its excellent. I think this will put a new lease of life into our development and hopefully let us think of more innovative things in every day projects and maintenance to make our life and that of our users better.

Entry Filed under: Technology