State Of Flux

I’ve been at Sensis for a year now and my latest project is Yellow Lab . The Yellow Pages is a significant part of the Sensis and Telstra portfolio earning 1,273 million dollars in revenue (p34) during 2007/8 which includes both print and online revenue. Yellow Pages is a product that you only change when you are absolutely sure you’re going to make it better. Labs was created to trail ideas for Yellow Online and mitigate risk in delivering new features. The ideas we trail are product, technical and user experience related.

It’s built on Rails

This is the first public site that Sensis has built in Rails. The momentum has been building within Sensis for a while but nothing of this scale has made it out to the public. Rails was chosen because it allowed Sensis to get to market quickly. From development to deployment, Rails has been a win. We’ve had many developers rotate through the project already, all of them have picked up Ruby and Rails very quickly with more experienced devs passing on their knowledge. That’s a win for Ruby and Rails .

Microformats everywhere

The Yellow Pages has a lot of data. Every single search result, and listing, is microformatted. I’ll be writing a separate post about this very soon, but it is really exciting stuff.

Building a search engine

We put a lot of time and effort into search. It’s hard and it’s fun. It also crosses all elements of the business from product management to user experience to technology. It is certainly more complicated than performing LIKE queries on a database or installing Sphinx or Solr . Business rules, physical locality, voting and tags all need to contribute towards getting a good result.

Aggregated Listings

Yellow Pages is a directory at it’s heart which lends itself to browsing rather than searching. Let’s say you’re looking for hardware in Port Melbourne. In a print world you’d flick open the book, look for the hardware category and then thumb your way through until you find a business that works for you. Big businesses like Bunnings would list themselves under hardware, but they may also list themselves under Nurseries as they also sell plants. That’s an awesome solution to the book because Bunnings can now be found in several places throughout the book. That’s a win for the business advertising and its a win for the user as they can find what you are after more easily.

When directories put their info online you usually have a modified browse experience where you have to select a category and then you can search within a category. For example, you say your looking for a Hardware store in Port Melbourne and find that Bunnings turns up. This is the way the core site works.

We wanted change the experience of finding businesses from a directory browsing experience to an internet style search. One hitch, if you put these listings online and search for Bunnings in Port Melbourne you’ll now find two entries (actually 5 on the site), one for Hardware and one for Nurseries – and that sucks if you are looking online. So we’ve aggregated listings together that are from the same businesses using some jiggery pokery to provide a single view of a business. It’s not perfect but it makes a big difference – it definitely solves 80% of the problem.

Sweet, sweet maps

Whereis has freaking good maps, despite Google having customer mind-share on maps. You can zoom in and see buildings with names and they have map data that gets updated regularly. You can send a location right to you GPS device, but in general, it is really well tailored to the Aussie market. Besides they look much nicer than the current Yellow Pages maps and so we’ve put them on Yellow Labs .

User generated content

Dude… We are so Web 2.0 – finally. We’ve got tagging (which we index immediately in our search engine so you can find em quickly) and user ratings in the form of “WOMming”. The tech is straight forward, and the features may seem obvious but how does this fit in to being an innovation platform? Finding out how things work in the real world is hard and the questions we are asking are more social. So we’ll build it and test if and how people use it? Do we moderate tagging contect? How? Does a ‘positive’ only recommendation work? Let build something and find out.

The future starts now

We have just dropped the very first iteration of Yellow Lab. It is rough and ready. This is the beggining of the adventure, not the end. We are going to put new features in and rip out features once we’ve finished learning from a feature. Trying out a new idea, measuring its success and deciding to go forward – or to throw it away – helps the business make better decision. There is some good stuff coming down the pike. Keep coming back to the Lab on a regular basis to check out what is new as we update the site several times a month – and please send us feedback.

I wanted a nice looking 404, 500 and maintenance pages for my Rails app and I couldn’t serve them from Rails.

My requirements were:

• I didn’t want to hand code the pages – I’m using a web framework for a reason!
• I wanted to use the application’s layout
• I needed that pages to be static so I could serve them when the Rails app was either down or when it had ‘issues’

My solution was to:

1. create a controller for the purpose of rendering the static pages
2. tailor your views so you have nice 404, 500 and maintenance pages
3. modify the layout so that signin and register were not longer present
4. create a rake task to render the pages and write them out to a file

Items 1 & 2 are just standard Rails stuff – so go nutz young coders.

Item 3 was pretty straight forward, in the layout I put:

  <% unless controller.controller_name == "errors" %>
     put your sign-in code here
  <% end %>

Item 4 was a bit trickier, but this rake task should get you up and running.

  namespace :generate do
    task :pages => :environment do
      require 'action_controller/integration'
      app = ActionController::Integration::Session.new
      app.host! "stateofflux.com"
      [['/errors/error_404', 'public/40 4.html'],
       ['/errors/error_500', 'public/500.html']].each do |url, file|
        begin
          app.get url
          File.open(file, "w") { |f| f.write app.response.body }
        rescue Exception => e
          puts "Could not write file #{e}"
        end
      end
    end
  end

We run the rake task in the development environment then check it in, but you could run it in production if there was production data that you needed to create the page.

If you get the following message

Please install the postgresql adapter: `gem install activerecord-postgresql-adapter`

It means you don’t have the new ‘pg’ postgresql library installed. This is easily fixed with a bit of

sudo gem install pg

I often want to grep for a word recursively within a directory. That is easy! But quite often I know there are a bunch of directories that I can ignore. For instance, in my rails apps I don’t want to hear about subversion directories and I don’t want to search the vendor tree.

I also don’t want to look at files that are really big – they are usually log files or binaries of some sort that I just don’t care about. So I put together the monster find and grep string that works for me.

find . -path '*/.svn' -prune \
    -o -path '*/vendor' -prune \
    -o -path '*/log' -o -path '*~' -prune \
    -o \( -size -100000c -type f \) \
    -print0 | xargs -0 grep -ne "SEARCH"

Let’s break it down.

find .

start searching for files recurively, starting from the current directory

-path '*/.svn' -prune

ignore .svn directories

-o -path '*/vendor'

OR ignore the vendor directory too.

-o -path '*~'

OR ignore all the emacs backup files

-o -path '*/log' -prune

ignore the log directory

-o \( -size -100000c -type f \)

OR make sure the size is less than 100,000 characters and the type of thing is a file (not a pipe or a director or socket) – this usually gets rid of all non source files

-print0

tells find to terminate the line with a 0. Which sounds really dull, but you can’t pipe the data (get the output from ‘find’ to ‘grep’) if there is a space in the filename.

|

push the data from the command on the left to the command on the right

xargs -0

take the input from the fine command, terminated with a 0 (the print0 from find) and send it to grep.

grep -ne

search each file with a regular expression and when a match is found print the line number with the filename.

The easiest way to use this is to put it into a shell file – I’ve put it into s.sh.

My s.sh file looks like this

#!/bin/sh

if  [ ${#1} -gt 0  ]
then
    find . -path '*/.svn' -prune -o -path '*/vendor' -prune -o -path '*/log' -o -path '*~' -prune -o \( -size -100000c -type f \) -print0 | xargs -0 grep -ne  $1
else
    echo "Not enough parameters.  Usage: ./$0 searchstring"
fi

Then I make it executable by

chmod 755 s.sh

If you want to find all of the before filters you would do this

s.sh “before_filter”

and enjoy you super fast find command.