This week I gave a talk at Ignite Melbourne (Slides are below), the topic of which is how to beat google at relevancy. My opinion is that with a small team of dedicated and passionate people it is possible to build a search capability that can rival Google.
Let me first say I’m not talking about web search, at the moment Google has that market pretty much sewn up and it will take a new level of technology (possible the semantic web) to better them, however I firmly believe that if you focus on a market segment or consumer niche you can build a more relevant search experience (Step 1).
Once you have defined your niche, understand why users are searching your site, what is their goal. Users don’t search because they have a spare 10 minutes over lunch, they have a problem they need to find a solution to, search is a mechanism by which they can select a supplier to solve their problem (Step 2).
Next consider the context of the searchers query. What do you know about the searcher, the intent of the search, the device they are searching from, their location, the language they use, even the time of day they are searching, all of this information can be used to deliver an individualised search as opposed to a generic search for the masses (Step 3).
Once you understand the context of your user, think about their end goal and identify what information would allow them to compare and contrast between the results you return (Step 4).
Finally build you search algorithm so that it can dynamically take the context of the user, their intent and the content that can support the compare and contrats process and deliver results based on all of this information (Step 5).
So my 5 step process to beat google is:
1) Pick your niche
2) Understand your users goal
3) Consider all of the context you know about your user query
4) Identify the information that allows a user to compare results
5) Build an algorithm that considers all of the points noted above.
Even if you don’t want to beat Google, using the process I’ve noted you can improve on the user experience that you deliver to your users and this can never be a bad thing.
This is the video of my talk at the Lucene revolution conference in San Francisco. If you don’t want to listen to my voice the slides are availble on slideshare..
Search, APIs, capability management and the Sensis journey Presented by Craig Rees, Sensis from Lucene Revolution on Vimeo.
I’ve always thought that managing an internal capability or technology platform requires the same processes, artifacts and resources as you use in managing an external technology product such as Websphere or Vignette.
6 months ago, a couple of colleagues (Craig Lonsdale and Pete Crawford) and I were discussing our plan for improving search within our company, we wanted to document our progress as well as define where we were on our journery and where our target for the future was. I had used the CMMI framework before to quantify how well organisations delivered projects and it felt perfect to try and use the same framework in defining on well an organisation managed their search capabilities.
The Search Capability Maturity Model that we came up with looks to quantify the stages an organization goes throught as they become more mature in managing search as a true capability as opposed to just installing it and hoping it works.
Search Capability Maturity Model
Organisations move through the different levels as they grow and become more adept and knowledgable, fundamental to the model is that it’s very difficult to jump levels, I know this from experience, especially when you look at the upper levels of the model such as utilizing external collaborators.
SCMM detail
I have worked in a number of organisations and they rarely get past the adhoc level unless you are talking about the Google’s of the world and because of this they don’t get to see the huge benefits a good search capability can deliver (to be honest this model can be taken beyond search and can be used to track the management maturity of any type of capability).
Search is a capability that changes over time, search patterns change as user experience changes, language moves forward or content is updated all of this means you need to dedicate resources to managing search.
Start off easy, set up a reporting program in order to quantify how well your search is meeting user expectations. Simple things like zero results can point you in the direction of big problems. Once you have identified the obvious issues you can start to use path analysis to pick up commons search types. Focus on these and you will see your user satisfaction soar. Then in the words of agile development, lather, rinse and repeat.
There is a concept in sales call benefits based selling, this is where you sell your product or service based on the benefit it will give your customer not on the features of the product being sold. For example if I was selling you home cleaning services I would talk about the fact you could get 2 hours back a week to go to the cinema not on services I could provide such as dusting or cleaning the oven.
This same approach can be used in a search context either using a simple keyword interface or with content filters / navigators or wizards.
The key advantage of using a benefits based searching technique is that you can reduce or if your lucky remove the complexity of the search scenario. As the searcher you no longer need to be an expert in the topic you are searching, you only need to consider the reason behind your search.
For example lets look at the scenario of a consumer searching for a new camera, you could allow them to search on size of memory card, number of megapixels, size of battery or even length of lens. All of these attributes would require me to be knowledgeable about photography. How about you switch this around and allow the consumer to think about how they will use the camera. Do they want to take pictures for 3 months without downloading them, do they want to print pictures off the camera and create large posters, or do they want to take pictures of distant objects. All of these questions can be used to identify the attributes that are important to the consumer, more importantly these are all questions that can be answered without having a deep knowledge of the topic being searched. This is the advantage of benefits based searching.
Retail is a great example of how benefits based searching can deliver you consumer advantage, I think local search is the next sector in which this paradigm will take off. When you consider the disparte topics that local search sites cover, you can’t expect consumers to be experts in all of the these fields. Educating consumers with content is one mechanism to improve the search experience, benefits based searching is another. Just make sure you have the ability to link the benefits users are searching on to the attributes you have stored in your database.
Search hasn’t really changed since the early years of the internet, you type in some words into a box press the search button and away you go. If you are on a Yellow Pages style site then you might have a two input boxes, one for what you are searching for and one for where you are searching for it. This means a 45 year old man searching at home on a laptop will get exactly the same results as a 17 year old school child out in the city searching on his mobile phone. Both of these individuals will have completely different needs and will find different results relevant to them.
It’s obvious that we should be giving different users different results based on the context of their search. Context can include the device being used to search with, the time of the day, month of the year, the location of the user, past search history and personal information such as a users social graph. All of these different contexts can deliver wildly different results and increase the quality or results returned.
Even context can change depending on the search being executed. For example take location, if I am looking for a pizza takeaway close by on a Friday night, location is very important to me, the difference between 5 minutes away and 25 minutes away is huge. However if I am looking for the nearest DIY shop to drive too because I want timber on a Sunday morning because I am building a deck the difference between 30 minutes and 45 minutes may not bother me. This information can help the search engine determine what is the most relevant result.
Over the next few years we will see more complex search engines being developed, they will gather as much information that they can about the searcher and deliver tailored results to a query based on the context of both the query and the searcher.
What with Google place search, Groupon types sites, Paypal bricks and mortar payments and Facebook launching deals and venue search, the local search ecosystem is coellecing together, but search is just a means to an end.
A consumer has a need that can be serviced by a local business, they don’t care how they find the business or if it was recommed by a close friend or if 99 other people use that service they get 10% off the price, what they really care about is fulfilling their need whether it be to take their partner out for an aniversairy meal or gettng an extension built to house their growing family. The ecosytem is not about local search but about local commerce and this is what Yellow Pages companies were all about, to bring buyers and sellers together.
The local commerce ecosystem is made up of a number of key components:
- Search – Google places
- Coupons – Groupon
- Payments – Paypal
- Social graph – Facebook
- Check ins – Foursquare
- Reviews – Yelp
It seems that they is an opportunity to deliver an end to end capability across the local commerce ecosystem, but not as a stand alone platform. Web 2.0 / 3.0 has show us that the web is no longer about monolithic stand alone sites, its about integration and interpolation across different destinations, platforms and devices, its about the glue that brings it all together and maybe this is where Yellow Pages has an opportunity, it can be the glue that simplifies how a consumer interacts with local businesses, a friendly trusted guide across this new environment.
More thoughts to follow on this topic.
Google has just launched their new search UI after months of A/B testing, for me the biggest change from a functionality perspective is the prominence given to the search refinements options (now occupying the new left hand column). You can refine your results by a particular time period, the type of media you are looking for or even particular forms of information such as reviews.
Google serp
Search refinement or feature filtering as it’s also called is not a new concept, online retailers have been using this feature for a while, think about buying a TV, you might filter on the type of panel, the size of the screen, the types of input or the resolution. You might have also seen this type of functionality being utilized on recruitment, car sales and holiday web sites as seen below.
Search refinement is a mechanism that allow users to create complex boolean type search queries using a graphical interface. It allows a user to filter down a large set of data based on a number of different attributes, great for when a user has an idea about what they are looking for but needs a guiding hand on getting to their correct result. It a bit like a crossing a browse feature withe a search feature and getting the best of both worlds.
So far, local search players have not jumped on this band wagon, the major players still use a single search box a la Google old school, is this because they don’t feel that it has value or perhaps that they don’t have the type of structured data required to drive a refinement style capability or the search platform to deliver this capability in a cost effective manner? Without structured data such as product catalogues or defined service style options it can be difficult to deliver a successful refine by interface, especially when you want to guide a user to a successful result and not to lead them down a blind alley into a no results page.
As we have seen in the past Google drives users expectations, as users become comfortable using this type of feature when doing a general web search they will start to expect it when they are doing any other type of onsite search, so my view it that it is more of a when rather than if this type of capability will be delivered, so start to get your data structured, it can be a slow and painful process but your user base will thank you for you hard work.
I’m in the process of writing a content strategy for my employer and I’ve been looking at the digital landscape to understand what key trends are going to affect our industry, this has allowed me to delve deeper into some of my pet subject areas.
The semantic web, metadata and creating context around content keeps being mentioned in both blog posts and industry articles. This along with an article I read on FUSMI ( by Fran Alexander) last week about how user generated folksonomies and author led controlled vocabularies can improve different parts of the interaction model reminded me of a diagram I created for a conference I was talking at last year on Information Architecture.
I was explaining that in addition to the benefits seen in using controlled vocabularies and folksonomies in different parts of your interaction model (searching vs content aggregation), you can also create a feedback loop between your user led tags and author led controlled vocabularies that can reduce management costs and increase effectiveness.
The theory is that you can use crowd sourcing principles to give you additional intelligence on modern use of terms as well as expertise in areas that are highly specialised, this works well for deeply vertical content sites.
Firstly user search can be improved as soon as user generated metadata is assigned, but secondly by looking at the terms attached to content along with the particular terms that are searched upon (I call this direct and indirect metadata creation) the managed controlled vocabularies can be kept up to date and improved without the need for large backend teams of information architects constantly analysing the information landscape, which in our costs saving times is a significant benefit.
The real barrier to implementing a feedback loop like this is trust, is your organisation open enough to trust the expertise and knowledge of its user base, I would argue that its too costly not too.
Since starting to play around with Twitter, I’ve been amazed at how quickly the twitter ecosystem has changed from random tweets about people waiting for a train or listening to the radio to crowd sourcing for answers to difficult questions.
Last week I noticed someone asking the crowd for a restaurant to go to while they were in San Francisco. There are many stats floating around at the moment pertaining to the level of influence that user reviews can have on product selection but this seems to be going to a new level (A form of digital word of mouth that crosses geographical and social boundaries and delves into the remit of local search providers).
I think this is still an early adopter usage trend but it could have huge implications on how people search for products and services, it may not be the Google game breaker but it will definitely have a disruptive influence on search as we know it today.
In the old days we tried to put all of our content into one big (normally expensive) system so that it could be reused, once you went down this route a couple of times you realized it was like trying to boil the ocean with the benefits rarely ever seen.
A few years later it was about enterprise content management systems that could federate using standards such as JSR170, again high costs and big implementation timescales.
Perhaps the answer to the problem was working in a separate team in your organization.
Search is all about finding relationship between pieces of content without knowing the underlying structure of that content, once you add in rich metadata and semantic mark up a search engine could be your new content federation and publication platform, plus it tends to be cheaper to implement.
The BBC and the guardian newspaper are creating rich content propositions using their search tools, I think the rest of us should start looking in this direction as well.