Views 3 + Apache Solr + Acquia Drupal = The Future of Search

For the last six months, Scott Reynolds has been keeping a big juicy secret. As the maintainer of the Apache Solr Views module, he knows just how cool the future of Drupal Search is going to be. His module, based on an idea and code from Thomas Seidl, lets you make custom searches against the Solr index the same way you currently make views against the MySQL database. Want to build a search that just includes videos and MP3s, and renders the results as a playlist? Or how about a search that is limited to the current user's images, displayed in a slideshow? How about a block that shows the latest results that contain the phrase "badgers are the new pony"? Well, even if you didn't want a block like that, with Views 3 and Apache Solr Views, you can have it.

Thomas Siedl's brilliant idea was that Views should be able to build "queries" against any data source, not just databases. Earl Miles agreed, and inagurated the Views 3 branch by commiting the patch by Thomas (with great help from Jeff Miccolis and others). With Views 3 I predict you'll be able to build Views using data from Flickr, or from RDF databases using SPARQL, or from the local file system, or from any other data source that has an API.

To test it all out I used the Acquia Drupal Stack to create a new site (I just love the stack's multisite functionality!). I then signed up for a trial Acquia Network subscription because I wanted to get my hands on 30 days of free Acquia Search (it's easier than setting up Solr myself). I then downloaded Views 3 and Apache Solr (DRUPAL-6--2, just for fun. DRUPAL-6--1 works, too). I had to get the Apache Solr Views module from CVS (Scott, make a devel release!). I put these in sites/all/modules so that they'd override the versions in the Acquia Drupal Stack.

The CVS command for getting Apache Solr Views

$ cvs -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal-contrib \
co -d apachesolr_views contributions/modules/apachesolr_views

I installed Apache Solr manually which means I also needed to get the SolrPhpClient library. Since I have Drush, and since Apache Solr DRUPAL-6--2 has Drush integration, I did it like this:

$ drush solr phpclient

I <3 Drush!

I then used FeedAPI to grab all sorts of content from Planet Drupal. I could have just as well used Drush and the Devel module to generate some content, but lorem ipsum gets mighty boring. Finally I used Drush to run cron and even did a search (from the command line!) to check that the content was in the index.

$ drush cron
$ # wait a few minutes for the search index to commit the changes...
$ drush solr search drupal
node/175 by admin (user/1)
title: Agile and Scrum Videos
 This is likely to become a pretty big collection of videos about Scrum and
other other Agile based managements processes. (Drupal 5, Drupal 6, Drupal 7,
Drupal Planet, Drupal Video) ...

node/1 by admin (user/1)
title: Welcome to your new Acquia Drupal website!
 If you are new to Drupal, follow these steps to set up your web site in
minutes: Step 1 ... , forums, polls, tags, comments, ratings, and more. Acquia
Drupal comes with many modules to power social publishing capabilities on your
site. Hundreds of additional Drupal 6.x compatible modules ...    

The good stuff

Now for the good stuff. When you make a new view in Views 3 you get asked what data source to use. Here you can see that I use the Apache Solr search index as a data source.

Views 3 asks for data source

Then I added some fields. These are not the same fields that are available to node based views. They are specific to the underlying data source.

Apache Solr fields in Views 3

I also added a sort so that the results would be displayed according to the search score (keyword relevance).

Adding a sort on score

In order to make this view seem like a "search" screen, it needs a search box, right? You get that by adding a search filter and exposing it. I could add more filters, too, like a filter to limit it to just one content type.

Exposing the search filter

This shouldn't just be a copy of the normal search screen. The results should look different. To that end I told Views to render the results in a table.

Tell views to theme the results as a table

Since we want this to be a page view it needs a path, and I went ahead and stuck it in the menu as well.

Finally, I want to be able to use Apache Solr's facet blocks along with the view. This is a three-step process.

  1. First I have to enable the facet filters (I enabled the block for Tags - a taxonomy vocabulary I have) at q=admin/settings/apachesolr/enabled-filters
  2. Then I have to actually place the blocks at q=admin/build/block
  3. Finall, and this is important, I have to pass in an argument to the view to filter by the same "things" that I'm filtering for with the facets. In my example I have the taxonomy facet block enabled, and therefore need to pass in an argument for taxonomy term.

Add an argument for every facet filter you use

How does it taste?

It tastes great! Feast your eyes on this marvelous search screen.

Search results with Views 3 and Apache Solr

The keyword search and the facet block interact seamlessly.

Search term modules and facet filter Drupal 6

An interesting point to note is that there are no database queries used in retrieving the data or displaying it. No complex views query with lots of joins, and no node_load() calls for displaying the results. This method of querying Solr is just as efficient as using the normal Apache Solr search module.

To my mind, Views 3 and Apache Solr Views are the future of Solr search for Drupal. Even though they are both in heavy development, you can try them out and enjoy the great control you have over your search experience. There are many more handlers that need writing, too, so jump into the Apache Solr Views issue queue and help out. Since it all works with Acquia Drupal and Acquia Search, you can easily get up and running using an Acquia subscription. Enjoy!

Comments

Its more than just search

This sort of functionality really goes way beyond search. We've been implementing these sorts of rich Solr driven features for a client in far less elegant ways for the last year. Using ApacheSolr, FacetedSearch and heaps of bespoke tomfoolery, we've been able to significantly increase the quality of suggested content in blocks and custom nodes.

Thanks to this integration, we should get an explosion of Views widgets and interfaces to really enhance the user experience with Solr. This is a milestone indeed.

did you really say that?

Did you really just say "bespoke tomfoolery"? That's awesome, I wish I had said it =)

Yeah, jump in. The sky's the limit and there is SO much that can be done.

-Rob

To see the crazy-ness that

To see the crazy-ness that we are doing with Apache Solr Views check out MothersClick.com

All the landing pages are powered by Solr Views.

great, just great!

great, just great!

TOO MUCH!

That is too much awesomeness for me to handle. TOO MUCH!

Pushing Search as far as it can go

I have been reviewing websites and older articles about search during the last few days.
http://pajamadesign.com/2009/01/14/acquia-search-installation/
http://developmentseed.org/blog/2009/feb/05/extendr-flickr-and-views
http://www.smashingmagazine.com/2009/09/28/search-results-design-best-practices-and-design-patterns/

example sites with different search configurations
http://www.wired.com/
http://lifehacker.com/
http://www.target.com
http://www.ebay.com/
http://www.smashingmagazine.com/

Are there any plans to make it easier to extend, override and use the "Search box form" in a more user friendly way for non-coders?
It would be nice if thru integration with views, context and features that the user could specify simple widgets that could be used with the search box form, and get specific search results.

Here are the some use cases.

1 ) Toggle search between the site, web-services, and 3rd party search sites.
User has an photo site. So they grab a feature that setups the photo gallery on the site.
Then downloads something like Extendr to access all the older photos on their Flickr account.
The user also wants to be able to redirect into or use results from http://images.google.com ; so they use something like external-search.
They would then need a coder to modify the search box form to put in the three radio button options.

http://developmentseed.org/blog/2009/feb/05/extendr-flickr-and-views
http://drupalmodules.com/module/external-search

2 ) Select list with search box form to quickly jump into search results for a context.

Basically an integration example for a site using context and features.
Website has the following contexts ( videos, podcasts, teams, store ) where a different filter box is in each context.
The search results for each context are visually and functionally different views.
The search box form has a select list drops down to the four context items.

3 ) Another select list with search box form to quickly jump into search results for multiple drupal sites.

Intranet with multiple site instances of drupal. Site_IT, Site_HR, and Site_some_other_dept, yes this happens.
So from drupal Site_HR the search box form has a select list for all the sites.
After a search on Site_HR can't find "travel reimbursement policy" so the user unsets a filter item and expands the search across the all the intranet sites that are connected.

4 ) Views Display filters

Search view with multiple displays ( Grid, Table, Teaser/condensed, "Full Node"/Expanded).
In the current search results have a tab, menu or widget that lets the user see the results in the display of there choice.

I have seen parts of the above can be done in “template files”/theming, views, and custom modules but nothing that integrates seamlessly for non-coders. With Views 3 + Apache Solr + Acquia Drupal will any of the above be easier or are we looking farther into Drupal 8 roadmaps?
http://acquia.com/community/projects/acquia-search-roadmap

Nice to see

It's nice to see that you expect a lot out of your search =) I do too. A lot of the things you mention will likely require some integrations glue code for some while. We're looking first at a phase where people get familiar with the new opportunities made available with Views 3, and lots of feature build-out has to happen. All the views handlers for sorting, filtering, adding new fields, and so forth - still has to be built. What is there is a great start, but there are dozens of features yet to add.

Apache Solr (the Drupal module) also has some growing to do. A big refactoring isin order to make it easier to develop with, and finding a way to modularize the facet widgets is of critical importance.

The views display filters request you have should be possible. I could have shown the above results in a grid view as well.

-Rob

follow up

Kewl. A Views display filters, would be one of those enhancements that would be nice if it was integrate with search and views.
I basically use the views tabs for different displays.
http://drupal.org/node/272818

I have read articles and seen different modules with better tabs; its on my todo list.
http://www.maybeor.com/blog/drupals-views-display-tabs-module-made-pretty-blog-build-learning-1
http://drupalmodules.com/module/views-display-tabs

There are number of posts about custom search box forms with drop-downs or other widgets.
Has anyone posted an abstract or how-to for that? I have used the filters-views instead of the custom module route.

post about the form widgets
http://acquia.com/blog/drupals-search-framework-the-execution-a-search#comment-12133
http://groups.drupal.org/node/15675
http://drupal.org/node/345942
http://acquia.com/node/370000
http://acquia.com/node/429908

Some other posts similar to the other user case examples
replacing drupal 6 core search with views search
http://groups.drupal.org/node/18582

Multi-site search with seperate contents?
http://acquia.com/node/907175

Is there a primary landing page that organizes the different search patterns into how-to articles or comparisons of the search modules that complement ApacheSolr modules and "Acquia Search"?
I have been referring people to a scatter shot of articles and blogs (the best have been on acquia)

http://acquia.com/blog/s-files-adding-search-fields-and-filters-views
http://acquia.com/blog/understanding-apachesolr-cck-api
http://acquia.com/blog/making-module-installation-easy-acquia-search
http://acquia.com/node/53332
http://groups.drupal.org/search-group

There's no centralized source

The list of links you've put together is very very useful. Blog about it! =)

-Rob

One alternative to the

One alternative to the external search module that works great is the Adjusti-Search module. It gives you a search block where you can customize the search engines as well as what is searched for. Example.. You can create a search block that only searches a particular node type.

The only downside to the module is that you are limited to one block per site as of right now.

Incredible!

I feel like I've just found God!

Awesome stuff.

I just wet my pants...

Guys, this is so hot!
In an couple of days I'll try to reproduce what you have built so far.
But having views deliver the search results & structure on top of any data collection is... well.. indescribable!

Keep up the excellent stuff!

Wim

Checkboxes in Blocks

Great tutorial - Thank you very much!

One question: How did you got the checkboxes in the filter blocks?

Thanks and greetings,
Filburt

DRUPAL-6--2

It's a feature in the DRUPAL-6--2 branch of the module.

-Rob

cck - field - imagecache thumbs

Hi

Is it possible to add cck fields to the search results? Like imagefield. I have created a site where people can create car-ads etc. When using faceted search It would be a great feature to show nodetitle and the main picture from the ad instead of just the nodetitle and published date.

:-)

Morten

Not yet,

at least without overriding theme_search_item.

-Rob

facets

Great article. Everything works fine except I cannot seem to link the facets to the corresponding argument in my apachesolr view.

Should the uri change when I click on an item in the faceted block? At the moment my URI doesn't change when I hover over any of the faceted items, it is always http://mywebsite/search_solr?text=london.

Thanks,

Colin

holy crap you guys just made

holy crap you guys just made my year

Acquia/Drupal Install

Love the article.

I downloaded the latest Acquia Drupal:

acquia-drupal-1.2.21

How much of the customization you describe in this article is relevant to this version (relating to how many modules do I need to install)?

I installed this and Solr from LucidImaginations and can query content from the Solr example within the Solr Admin Page. And my Acquia Drupal can send searches to the Solr instance, but return no results.

So I am just trying to figure if I have to install the modules you suggest, and do I need to run the cron jobs for the index if I have gone through the Solr and updated docs within that index?

Thanks for your posts!

jay

Yes, run Drupal cron

And you need the Views 3 module, and the apachesolr_views module. Do you have both of those? Some things may have changed, too, since I wrote this, since Views 3 is in heavy development.

-Rob

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote>
  • You may post code using <code>...</code> (generic) or <?php ... ?> (highlighted PHP) tags.
  • Lines and paragraphs break automatically.

More information about formatting options