The 2 Graphs That Will Define The Future of Search

Posted: September 23, 2010 | Author: | Filed under: Predictions, Search, Trends | 1 Comment

The term “graph” is used to mean the underlying data structure of a set of objects.

The major initial innovation of Google was to map the links on the web and use this data to rank search results. This data could be called the “Webpage Graph”. Of course, Google has done a lot to improve search since then, adding over 200 relevant factors. The next major wave of improvement in search results will come from hooking into the social graph and the taste graph.

The Social Graph – how people are connected together on a social network

Facebook got this right by focusing on real identity and making the site ubiquitous so one’s facebook connections could fully replicate their real-world connections. By combining people’s interests (their profiles) with the connections advertisers can now target you by what you and your friends like. This hasn’t led to high CPMs on Facebook (yet) but it still proves to be very valuable information.

Google wants this information. They wish that Facebook would just open up this information but Facebook doesn’t allow users’ data to be used for doing targeted advertising by third-parties. Google says they are going to create a social layer for search. It is still cryptic what they mean by this but basically they think knowing how people are connected and having profile information could lead to higher quality search and advertising quality.

The Taste Graph – Measures what people will like by what other people like

Two systems: popularity based and recommendation engine based.

Amazon was the first to use this concept well. Amazon used the popularity concept by having user-written product reviews. This helps users make informed decisions on what to purchase and made Amazon a trusted source to find what to buy online.

Netflix decided instead just showing the average ratings, Netflix would allow each user to have a customized expected rating for each film using a recommendation engine. This method is definitely superior to a purely popularity based taste graph in that it provides more accurate information for each user.

Hunch is trying to do what Netflix did for movies reviews but for everything–an ambitious goal for sure. They are trying to get there by having users answer multiple-choice questions and say whether they like or dislike certain things. I think eventually more ways of getting feedback will eventually be required but that is for another time. Right now it is just important to see the trend of the rising importance of the “Taste Graph”

Google measures which user clicks on which links and this leads to a lot of interesting data that goes into improving search but it is still not a full “taste graph”. Google could benefit from a taste graph in a similar way they could benefit from a social graph. This information could lead to better algorithms and thus better search results.

Conclusion

Eventually all of these graphs are going to converge and it is going to drastically improve personalized search.

Most likely, Google will find a way to get a detailed profile of each user and a detailed social graph of whom they are connected to. On top of that they will know what each user likes based on having information on what decisions they have made and how they have rated different things. All this information will go into improving personalized search results.

It will be a big challenge for Google to obtain this information and it won’t happen overnight but in the very long term, Google will most likely find a way to get this information.

“The perfect search engine,” says co-founder Larry Page, “would understand exactly what you mean and give back exactly what you want.” In order to get to this point, Google has to know who you are and what you like.

Introducing a new Theory: The Recommendularity

Posted: August 20, 2010 | Author: | Filed under: Random Ideas, Trends | 4 Comments

The Singularity is the technological creation of smarter-than–human intelligence. I present something that is akin to this theory though on a much smaller scale: the recommendularity.

The Recommendularity is the technological creation of better-than-human intuition recommendations. By this, I mean that an advanced recommendation algorithm with sufficient data will be able to predict how a person will like something better than the person himself or herself.

In recent years the internet has seen a proliferation of recommendation engine based products:

I greatly enjoy using each of these products along with millions of others. And the best thing is: the more that you use them the better they get.

Recently, something crazy happened. When it comes to movie recommendations, I now trust netflix more than my own intuition. The process has been slow and gradual. Overtime I have fed more film ratings into the system–now over 1,000. Also the netflix algorithm has been constantly improving during the same period of time.

Netflix, realizing the importance of the accuracy of their recommendations hosted a million dollar prize competition for who could improve their recommendation algorithm the most. Beyond this being a damn good publicity stunt and way to get more than their money’s worth of labor, this is just a hint to what may be in store.

As recommendation technologies improve and these technologies begin to have access to more and more data about us, the recommendations that they will be able to make will only improve.

It is too early to predict the nature and shape of this improvement curve.  On the top end it could follow an exponential improvement curve like Moore’s Law. On the opposite end, there could be a natural limitation to how good recommendation engines can get leading improvement to reach a screeching halt.

Even if there are limitations to how good the recommendation engines get, there is still a ton of room for growth in the amount of information that can be fed into a recommendation engine leading to great improvement in the results.

(This is where the SciFi part begins)

Where could this lead given enough time? (Hopefully, in our lifetimes)

Recommendation engines will be better than humans at making nearly any decision. People will rely on the personalized recommendations to make more and more decisions in their lives–anything from where to eat lunch, to where to go to college. Eventually even the the most important decisions in life could be left to an algorithm.

Ultimately, choice will remain with the individual but as the computer recommendations get better and better more and more weight will be placed upon a machines recommendation.

The recommendularity represents the ultimate discovery tool. It will help ensure that everyone gets everything out of the world that is best for them according to processing the preferences of people with similar interests.