Google hires thousands of humans to rank search results manually

Anyone who thinks Google's results are solely the product of fancy-pants algorithms and clever engineering -- think again. The Register has taken a look into Google's 160-page guidebook which is essentially a reference manual for human "raters" -- yep, that's...

Read more

Just another reason why Google's search results are better than Bing's.

Google results in last three years 2010-2012 went to totally irrelevant.

Source?

Have a few buddies that work for leapforce and lionbridge. They make killer money around the holidays as bonuses to contracts are really high.

Sadly, I'd prefer algorithms. I don't trust human beings.

How about rating new or outdated info? it is a lot of old / junk on the web.

I actually was a quality evaluator/rater for Google through Workforce Logic, and did this a couple years back when I was in College. So it's nothing new

Nothing new. They need human judges to produce relevance assessments to evaluate their retrieval models. this is a customary practise done since the early 1960s. whenever a change to an algorithm is recommended you need to assess it in terms of effectiveness performance. relevance judgements are what facilitate these evaluations. being a large company such as google, you can afford to refresh your training collections by outsourcing relevance assessments on an occasional basis. humans are not actually sitting behind the scenes rating your results