Thursday, February 7, 2008

Digging into the workings of social news aggregators

An ex-colleague of mine, Kristina Lerman, wrote a very interesting paper analyzing and modeling the process in which news items are discovered and make their way onto the front page of the social news aggregation site Digg. The paper is called "Social Information Processing in Social News Aggregation".

Interesting points made in the article regarding the social nature of the system, include:
  • People subscribe to feeds of news items that are posted or voted for ("dugg") by their friends, increasing greatly the probability of viewing (and thus voting on) news items that their friends found interesting.
  • This "social filtering" phenomenon is necessary in order to deal with the huge number of stories that are posted to Digg each day.
  • Stories propagate very fast through the system: anecdotal evidence suggests that they are faster than clustering-based approaches such as Google News, (which makes sense, as a story must break on multiple news sources before it will appear on a clustering-based news aggregator).
  • According to mathematical model devised to explain the process by which news items are propagated to the front page, a highly relevant/interesting story (relevance=0.9) may not get onto the front page if the original poster is not well connected - i.e. doesn't have a large number of readers.
  • And inversely, relatively uninteresting articles (relevance=0.1) may make it to the front page if the contributor is particularly well connected (i.e. famous).
Reading the article made me think of another article I read recently on using agent-modeling techniques to analyze human social behavior. This time the author was interested in the divergence of opinions rather than the consensus opinion (i.e. newsworthiness on Digg). In his article "Mobility and Social Network Effects on Extremist Opinions", André Martins created agent-based simulations of public opinion so as to analyze the emergence of extremist positions in society. The finding in that case was that more interaction between diverse groups could lower the overall amount of extremism.

(Thanks to the physics arXiv blog for bringing my attention to the second article. - They point out that in this age of terrorism fears, closing borders and restricting travel may well have a negative effect on curbing extremism.)
Posted by at

3 comments:

Anonymous said...

Something it really resonates with what I think as well. Read the bibberish (I know it is not a word ;-) at the end of
http://www.gnuband.org/papers/trust_metrics_on_controversial_users_balancing_between_tyranny_of_the_majority_and_echo_chambers-2/

Or just go to the really interesting point, made by Cusstein in the book Republic.com ;-)

February 27, 2008 at 4:55 PM
Mark Carman said...

Wow Paolo, just read your excellent paper! I will never consider trust a global metric again. And your discussion on consensus opinion at the end is very interesting.

February 27, 2008 at 6:18 PM
Junkie said...

I had read that paper many months ago and found it very interesting, like most of her work on swarm robots you probably already know. To me, the most exciting idea they have about multi-agent system seems to be the fact that they can take design decisions by looking at the mathematical model! It may seem trivial, but I think this is something software engineers tend to ignore …

March 19, 2008 at 10:03 PM

Post a Comment

Subscribe to: Post Comments (Atom)