Explanatory algorithms

There is a trend in recommender systems that I think is extremely interesting: systems are starting to explain themselves. The first place I noticed this was at Amazon in their personal recommendations section, at the bottom of a given suggestion:

Amazon recommendation

In this case, Amazon recommended Moon Palace because I had rated another book by Paul Auster. This makes perfect sense, namely I rated something by an author, so the system recommended other books by the same author. The second place this popped up was at the new social music service iLike. Every time a user views another user’s profile, the system calculates a compatibility score based on how similar your favorite artists are, as shown here:

iLike recommendation

In this case, I share interest in the bands ESG, TV on the radio, et al. with this user, so our compatibility is high. When I share more popular artists like Miles Davis or Bob Dylan, my compatibility score is lower. This makes sense since rarer bands suggest a closer connection. Last.fm has added a similar feature called Taste-o-meter.

What’s interesting about these examples is not the algorithm, some augmented form of collaborative filtering, but rather in the way that the algorithm explains itself to the user. Many years ago, with the likes of Firefly and CDNow showing off the power of recommender systems, this sort of behavior would have been considered crazy. Showing to users elements of how your algorithm works? What if they reverse engineer it and copy your methods and copy your system and steal all your users?!

Not likely. For most intents and purposes, recommender systems are within wiggling distance of each other. Netflix is holding a contest to see if theirs can be improved, offering a cool $1M to anyone who can show a 10% gain over their current algorithm. While the current leaderboard shows the best contenders at a 4% gain over the original algorithm, Netflix does not expect people to make the 10% gain necessary anytime soon, suggesting the contest could run until 2011. But companies like Amazon and iLike are making improvements through the way that these algorithms are explained.

Explanation creates understanding, and understanding leads to trust.
What if all systems started to take this approach? We mostly assume that search providers keep their ranking algorithms in a 6-foot safe behind a wall of lasers, but at the same time Google is starting to release more information about PageRank through various systems. Someday we might have search results that explain themselves, while keeping the special sauce away from SEO geeks and spammers. Imagine if a top search result said “This result is first because: your search term was in the title, the author is a well known writer, and the host is a reputable newspaper.” I would probably say “that makes sense,” and in turn I would trust that system even more.

Amazon launches answers site

Askville LogoToday I received an invite to join a new community at Amazon called Askville:

You’re Invited!

As a valued Amazon customer, you’ve been specially picked to get an early look at a new website called Askville where you can ask any question on any topic and get real answers from real people. It’s new, and best of all, it’s free!

This site will compete with Yahoo! Answers and Microsoft Q&A in the free question-answering space except that it might be able to leverage the Amazon community of experts. For those that have not been following this area, these systems enable knowledge creation by allowing users ask questions that are then answered by other users in exchange for reputation within the system. The first success in this space was a startup in Korea named Naver that took control of the search market share in a very short period of time.

Amazon’s system is similar to all of its American counterparts, with its large fonts and friendly messaging (“ask.. answer.. meet.. play”), except for a few subtle distinctions:

  • Users are rewarded for asking questions as well as answering them
  • Questions are limited to 5 answers total
  • Best answers are chosen by the group of question asker and answerers, where the asker gets one more vote than the answerers

Probably the most significant change is the flow of the question/answering exchange. In Yahoo! Answers, and elsewhere, answers are shown publicly as they are received; in Askville, answers are hidden to the public until 5 answers have been received. Any discussion or clarification can happen in a public message board attached to the question. After 5 answers have been collected, the group of asker and answerers vote and the whole thing is made public.

Askville rewards users with “coins,” a virtual currency that will be redeemable in another community named Questville slated for release in early 2007.

The system has given me 25 invitations for other accounts. If you’re interested in trying out the system, shoot me an email.

Update: I apologize, but all of my invitations have been distributed! It seems like the invitations are spreading though, so look for one on a weblog near you…