Vertical search within results

Just spotted this up on Techmeme from SearchEngineLand – Google Tests Additional Search Box Within Search Results. Check the full post for images to explain. Here’s the quick summary.

Your search results include sites like Amazon, Wikipedia and New York Times. For those results, you can perform a follow up search that covers just that site. It’s an interesting way to keep people using Google as the start location for search.

Sitting in on a session with FAST at the SharePoint Conference today, FAST had a great slide up showing online search as a long tail. Only 30% of searches actually start from the main gateways – Google, Yahoo, Microsoft. So that means Google actually only has a 60% market share of 30% of the market… in terms of direct searches. I daresay a fair few of the locations that account for the other 70% of searches use Google for their search… by the by, the idea of ‘search within search’ is another way of protecting and expanding that starting point for online search.

To click or not to click

One of the topics I spend some time on when delivering enterprise search workshops is helping organisations to identify and understand when you want people to click on a search result versus not.

When someone is seeking interactive help – for example: I want to submit a question to an internal discussion forum – then you want the search results to display that forum as the first link on the page, with enough information for the user to be confident that it is the right forum. I click on the link, enter the forum, post my question and start a conversation. Similarly, when someone is seeking detailed information – for example: I need last year’s budget – you want enough information displayed in the results page for me to be confident that the 10Mb file I am about to download over a flaky Internet connection is the right one.

When someone is seeking a snippet of information – for example: I want to know the telephone number for a customer – then you can improve productivity by displaying enough information in the search results for people to not have to click anything. I don’t need to view a page or open a document about the customer, I just want the telephone number because I need to call them.

This can be done within SharePoint Server 2007 (and it’s sibling Search Server 2008) using managed properties. You can modify the results pages to display additional information by tweaking the XML that determines what and how information is displayed.

Interestingly, Yahoo has made an announcement to enable similar behaviour on their Internet search engine – An open approach to Search. Web site owners can submit data and Yahoo will display their results in a more informative format. Here is the example (‘before’ on the left/’after’ on the right) given on their blog post:

From a user perspective, the ‘after’ is a big improvement. If that’s the restaurant I was looking for, I can see it has some good reviews and I’ve got the telephone number to make a reservation. But I’m not sure the web site owner will agree. That result isn’t from the restaurant itself, it is from a review/directory site. The argument behind search engine optimisation (SEO) is that many people start their Internet journey at a search engine. If you want your web site to be the next destination, then you need to be at the top of the search results. In this example, improving the visual display of search results means that there is no next destination. The search engine becomes the start and end. If you have a web site with a business model dependent on online advertising revenue (dependent on people visiting your site), the search engine just ate your lunch.

Naturally, there is a solution. If you are an intermediary web site, you need the search result to display information that will still bring a visitor to your site or keep you in the loop of the transaction. In this example, perhaps being able to offer a 10% discount on the meal if booked through the review site…

Technorati tags: Search ; Enterprise Search

All web sites great and small

Spotted a depressing article on Techmeme on Friday – Hackers turn Google into a vulnerability scanner (Infoworld). I suppose it was inevitable that this would happen.

Hacking group Cult of the Dead Cow (CDC) have kindly released a tool that uses Google to automatically scour web sites for sensitive information. Because it is automatic, it means that new and novice web sites are no longer protected by relative anonymity. If you are storing information anywhere in ‘the cloud’ and are worried about it being kept private and secure, the best approach is to run the tool for yourself and find out if your site needs fixing.

Whether Google likes it or not, they are as good as a monopoly on the Internet. There isn’t the proprietary lock-in achieved by a certain other technology company. But Google is the one location that most* people go to in search of stuff and therefore the one location most web sites aim to be discovered by. The trouble with technology monopolies is the lack of diversity. It’s what makes Microsoft software so vulnerable. Give a cold to one computer and you can pass it on to them all. Now the Internet is the focus and Google is the target to exploit. The CDC tool doesn’t care if your web site is on page 1 or page 1,000,001 of Google’s search results. It can and will find you (cue Terminator music).

The ultimate irony – the tool takes advantage of Google’s index, has been written using Microsoft .NET and is licensed as free open source… it’s not often you see those three areas come together as a single solution. Pity it had to be this one.

*According to comScore World Metrix, Google hosted 62.4% of web searches in December 2007. Next nearest rival was Yahoo with 12.8%, trailed by Microsoft with 2.9%

How relevant is your content?

Last year I wrote a blog post that generated a bit of offline feedback – Search Lessons. Or rather, it was explaining The Google™ Effect that sparked a debate.

¨Why can’t our enterprise search just be like Google?¨

People are used to Google when searching the Internet – simple (and single) user interface, fast results. Relevance varies depending on the search but is usually good enough for most queries. When it comes to searching for information within the boundaries of your organisation, the answers are not so easy to find…

There’s one simple answer: Intranet ≠ Internet.

They might look similar, thanks to web standards, but looks can deceive. A mushroom looks like a toadstool but one is edible and the other isn’t.

On the Internet, pages want to be found. The same is rarely true within the enterprise.

On the Internet, if your site doesn’t appear in search results, it doesn’t exist. That’s a huge incentive to ensure anyone publishing content uses tags and any other tricks to attract the attention of a search engine. Enough of an incentive to create a whole new industry – Search Engine Optimisation (SEO). The same incentives rarely apply within the enterprise and few organisations budget for in-house SEO. Added to that, the search vendors have to continuously optimise the index to balance any negative effects caused by SEO. In-house? You need to do both! Enterprise search is a continuous improvement process, not an isolated project.

On the Internet, there is no relationship between the search engine and the end-user. If you don’t find what you are looking for, you try a different query or look somewhere else. There is no contract stating that the Internet must provide you with answers. There are no guarantees that what you find will be accurate or useful. When an enterprise search engine fails to perform, the IT department usually hears about it.

On the Internet, all users are equal (and anonymous). You can’t login to Google with a different ID and suddenly access previously hidden search results. In the enterprise, security is rarely considered optional.

The Internet doesn’t care what you do with what you find. An enterprise search engine should.

When an enterprise search engine fails to deliver, the technology usually gets the blame:

¨Our search engine is rubbish, the results it finds are never relevant.¨

But is it really the search engine that is to blame? When it comes to calculating relevance, it all depends on the content. Jenny Everett, over on the SharePoint Blogs, published the results of a survey that clearly demonstrate the real challenge facing enterprise search solutions.

Source: SharePoint Search is not Psychic by Jenny Everett

85% of search issues had nothing to do with relevance ranking. The numbers may vary by organisation, but I would wager that the top two issues are consistent. Imagine if you performed a search on the Internet and the first page of results consisted of items with the title ‘Blank’.

Microsoft Enterprise Search Product Comparison

I’m using the following table in the Enterprise Search workshops I am currently delivering on behalf of Microsoft. People seem to be finding it useful so I thought I’d post it here. It is an expanded version of the table you can find on Microsoft’s web site (see end of post for link). The table is not a comprehensive list of all features and aims to highlight the differences between each product from a search perspective. (SharePoint Server 2007 has additional features not listed here.)

(Click to view)

Windows SharePoint Services and Search Server 2008 Express have identical licensing requirements. They can only be installed on Windows Server 2003 SP1+ or later. That means you will need a Windows Server license but nothing more. If you perform a basic installation of either, you get the database included with the Windows license – MSDE for Windows SharePoint Services, SQL Server 2005 Express Edition for Search Server 2008 Express. If you perform an advanced installation of either, you can opt to use the full SQL Server product instead.

(Tip: You can also use SQL Server 2005 Express Edition with Windows SharePoint Services, but you have to download and install it first, and then use the Advanced Installation to tell Windows SharePoint Services to use it – SQL Server 2005 Express Edition was released after Windows SharePoint Services and has replaced MSDE.)

Search Server 2008 licensing has not yet been announced by Microsoft. Given it is effectively a subset of SharePoint Server 2007, I’d guess it will be a per-server license but we’ll have to wait and see.

SharePoint Server 2007 requires a per-server license and per-client access licences (CALs) for internal use. There are two versions – Standard Edition and Enterprise Edition.

Search Server 2008 is due to be released during the first half of 2008. The new features being introduced (you can download the release candidate if you want to play with them) will also be made available to both editions of SharePoint Server 2007.

If the FAST acquisition does go ahead, expect to see Microsoft’s search strategy expand deeper into enterprise information assets.

Related blog posts:


Microsoft Enterprise Search Workshops

[Update: 26 Jan] February workshops are now full and registration has been closed

[Update: 15 Jan] January workshops are now full and registration has been closed

To kick off 2008, amongst other things, Microsoft UK is running a series of Enterprise Search workshops. The focus of the day is to explore how Microsoft products and services can be configured and used to improve information findability*. And the workshop content is being designed and delivered by yours truly 🙂

The workshops are being held at Microsoft’s UK training facility in Chertsey. If you are interested in attending, select one of the following dates to view the agenda and register online (no fee):

The workshop will be a mix of presentation, demonstration and discussion, with slides kept to the minimum (handouts will be provided) and the emphasis on interactive demos (no scripts) supported by case studies. The final session of the day will be a walkthrough installing and configuring Microsoft’s free** search product – Microsoft Search Server 2008 Express Edition. Anyone who brings along a laptop can be assisted in setting up their own installation. You will need to be running Windows Server 2003 or later, either on the laptop or inside a virtual machine and it will help if you have some content ready to be indexed.

If you are planning to attend and there is anything specific you would like to see or ask about, either drop me an email or leave a comment here. Each workshop will be individually tailored to ensure attendees get as much out of the day as possible.

*Findability is still not in the Oxford English Dictionary, but give it time…

** Free as in can only be installed on a Windows Server and you do need a license for that

Enterprise Search Trends for 2008

CMS Watch has posted a summary of Enterprise Search Trends for 2008. Given it’s already a summary, I can’t really do any more summarising here 🙂

The benefits of a good search solution are obvious – helps you find what you are looking for (and even shows you stuff you didn’t know you needed to look for until it found you). The challenge for organisations is that ever-improving search solutions – particularly on the Internet – make it easier for customers and competitors to find out information about you that you may not realise exists. Handling the fall-out from search needs to be an integral part of an information strategy. Customer complaints are getting organised (link courtesy of Cognitive Edge)

Microsoft Federated Search – Image Example

Following Microsoft’s Enteprise Search Announcement, a partner – IDV Solutions – has created a great demonstration of federated search, combining standard web search results with Flickr photos and Virtual Earth. Below is a screenshot, showing a search for ‘Rome’. As you select each image, it’s location is displayed alongside using Virtual Earth:

They have made their search connector available for download. You can find out more at their web site – Everybody’s Searchin’ for Somethin’. You can view a live demonstration (requires ActiveX control) at

Filed under: Microsoft Search

Technorati tag: SharePoint 2007, Search

Microsoft Enterprise Search Announcements

Yesterday, Microsoft announced some changes to their enterprise search strategy. To read all about it, go to Here’s the summary:

New Products

Server-based enterprise search solutions are now available in three flavours:

  • Microsoft Office SharePoint Server 2007
  • Microsoft Search Server 2008
  • Microsoft Search Server 2008 Express Edition

Microsoft Office SharePoint Server 2007 (MOSS) is the full blown platform for information and knowledge solutions that includes features for information management, collaboration, portal and indexing/search

Microsoft Search Server 2008 (MSS) is a dedicated search server. It’s the same indexing and search technologies provided within MOSS 2007 but without all the other features. In other words, if you’re using MOSS, you do not also need to buy MSS. MSS used to be known as the mouthful that was Microsoft Office SharePoint Server 2007 for Search. MSS does not have the full set of search and indexing features provided within MOSS. Specifically, you do not get People search (dependent on MOSS for providing user profiles and MySites) and you do not get the Business Data Catalog (indexing structured applications – dependent on MOSS for managing application integration and authentication)

Microsoft Search Server 2008 Express Edition (MSSE) is a limited version of MSS that is being made available as a free download. Yup, Free. No licenses required. If you choose to use SQL Server Express as the back-end database (also free), then the only license you will require is for the operating system – Windows Server 2003. MSSE has all the features of MSS. The only limit is that it is a single server solution. If you wanted to scale it to multiple servers (e.g. dedicated server for indexing, dedicated and load-balanced servers for processing search queries), then you would need to upgrade to MSS.

New Features

Also in the announcement, but less easy to spot, is the introduction of some new features within enterprise search:

  • Federated Search Connectors
  • Streamlined Installation
  • Unified Administration Dashboard

Federated Search connectors will enable you to submit a search query to multiple different indexes – both local and online – and bring back results grouped by the different index. (It is not possible to bring them all back together as a single results set because different indexes apply different ranking algorithms). You can build your own or download add-ons from the new Search Connectors Gallery. They are not yet available, but partners listed include: Business Objects, EMC Documentum, Endeca, FAST ESP, Handshake Software, OpenText, SAS, and Symantec

The Streamlined Installation process (think wizards) should make it a lot easier to set-up and configure an enterprise search solution using Microsoft technologies.

The Unified Administration Dashboard will provide a better user interface (UI) for administration and maintenance. Currently, within MOSS 2007, the search settings are somewhat buried in the Central Administration console. The new dashboard should make it much easier and quicker to keep track of your search infrastructure

What’s the deal?

There is a fourth product in this Microsoft-specific playground that also warrants a mention, the underlying platform that serves MOSS 2007 – Windows SharePoint Services (WSS). With this announcement, organisations planning information management and collaborative solutions have some new choices:

  • If you want to build an internal enterprise platform for information management and collaboration, MOSS 2007 remains the product of choice if you go down the Microsoft route (naturally there are other vendors also in this space)
  • If you are less concerned about enterprise stuff and are more interested in building agile collaborative workspaces that operate independently of each other, chances are you would be looking at just using Windows SharePoint Services (WSS). By adding on Search Server Express Edition, you would now be able to search across all of those different workspaces using a single query. (WSS alone means you can only search within one site at a time.) This provides a rich solution with no extra licence fees required other than Windows Server 2003 (WSS is provided as part of Windows Server 2003)
  • If you are not, for whatever reason, using or planning to use MOSS 2007, you can whack Search Server Express into your environment for minimal cost. No limits on what you index, as long as one server can cope with the load
  • And what about Search Server 2008? Well, if the Express edition is not enough and you need to scale your search infrastructure, you will need to upgrade. But if you are that serious about your search solution, chances are you’ll be interested in social networking (people search) and collective intelligence (combining structured and unstructured sources of information). I’d wager that route will lead to MOSS rather than MSS although it will depend on the different licensing costs involved versus the calculated benefits

Introducing Federated Search Connectors is a big deal, at least it will be for the customers I have been working with. At last, Microsoft is providing a solution that acknowledges there is life outside The One Index. (It’s similar to the early days of Active Directory – The One Directory – later being extended, thanks to Identity Management solutions, to connect and synchronise with other directories.) As the volume of information being indexed continues to grow – internally as well as out on the Internet – it becomes much harder to maintain high relevance within results from a single index. The most effective solution is to send a search query across different indexes that can be individually tailored to serve up the best possible results.

The only thing you can actually get your hands on today is the release candidate for Search Server 2008 Express edition. The new features are not yet available for MOSS 2007. When will they be? Well that’s for Microsoft to announce but I think we can expect to see updates available in early 2008. No vendor wants its free products to have more features than the licensed versions…

I should include a disclaimer: I’m an independent consultant running Enterprise Search workshops funded by, and on behalf of, Microsoft. I’ll be delving into some of the finer details within this announcement within future events.

[Update: 09/11/07] The announcement has also been covered over on the product group’s blog:

Filed under: Microsoft Search

Technorati tags: SharePoint 2007; MOSS 2007; Enterprise Search