there are several objective ways to determine someone’s rank or to fit someone into an artificial hierarchy on a socially driven site, and i will cover some of the existing ones and offer some thoughts of my own. what you will note is that no single objective measure is enough and we need some way to calculate participation across all these metrics.
- promoted stories: this is the absolute number of stories out of all your submissions, that have been promoted to a site’s homepage. the problem here is that people can submit as many stories as they find, in the hopes that some of them will make it to the homepage, and not enforce any quality control (i.e. they will hedge their bets by banking on volume rather than quality).
- promotion ratio: this is the percentage of your submissions that end up on the homepage. the problem here is that though a user may have a solid ratio, in most cases a high ratio is the result of very rigorous quality control, which usually means infrequent submissions. users with high ratios usually have low overall participation.
- average votes received: the average votes a user receives per story is a good indicator of average content quality. someone can have a lot of stories promoted with all of them receiving 400 votes or he can have a few stories promoted with 3,000 votes each. just having a story promoted is not enough because often stories are promoted and then get buried or are promoted but don’t really catch on.
- average comment ratings: how well are a community member’s comments generally received? slashdot has one of the most developed comment rating systems among the top news aggregators right now because it allows you to not only rate a comment but also categorize a comment (for example, as humorous, insightful, etc).
those are some of the mechanisms that exist (though only the first two are used by most people). here are a couple of other metrics that could potentially be used:
- quality of votes given: does a user blindly vote everything or heavily reciprocate? or does the user only vote for the content he thinks is good and should be promoted? one of the ways we could judge the quality of votes given (from the general community’s perspective) is to see what percentage of stories a user votes on, ultimately end up on the homepage of the site.
- diversity of participation: diversity of participation is akin to balanced participation. this category will distinguish those that are just submitting apple rumors and ron paul stories from those who contribute good content in a wide-ranging array of categories.
like i said, none of these measures are telling enough on their own but if we are able to use all of them together, we can come up with a better way of recognizing who the best participators in social news are, above and beyond pure popularity. are there any other metrics that you think we should be looking at as well?
Technorati Tags: social news, digg, propeller, reddit, stumbleupon, slashdot, top user
Hi Muhammad,
Regarding the percentage of items voted on that went hot, that would be easily gamed in networks like Sphinn that have a most votes -but-not-hot-yet page.
Besides that, a top social news user, imho, is someone whose name starts with M and then has an “ee” sound in it somewhere… Like Maki, for instance ;).
I like all your criteria for evaluating a social news user, but can’t help but want a few more. IMHO, the criteria listed seem to be affected a lot by popular opinion. While I don’t mean to devalue the opinion of the mass, I think we could use a few measures of quality not based on its opinion. Maybe something like diversity in sources when contributing content, helpfulness of user when replying to another users content. Maybe a few qualitative measurements.
Great post. Wouldn’t it be cool to create some sort of a toolbar that measured a user’s overall “social news user” ranking. We could coin it SNR (ala Google’s PR).
Some thoughts:
* A one-axis scale doesn’t cover everything. You might have a guy who’s a self-promoter but promotes things the community likes… He’s a good guy. There’s another guy who scans the upcoming directory and makes good choices… He’s valuable too. You can’t judge a pitcher by his batting average.
* A big principle behind slashdot is that you’ll find “normal” behavior near the median and trouble far from the median. Users who play the slashdot game and have middle-of-the-road statistics get selected to be moderators… You might throw out a few of the “best”, but the main concern is to get rid of the “worst”.
* I dropped out of Slashdot soon after the moderation system was installed. It seemed to reward ‘averageness’, and the trait of ‘sounding like you know what you’re talking about’ — I didn’t feel rewarded for writing about topics that I understood but wouldn’t be recognized by the community.
A year ago I did a serious stint of Slashdotting and found that the system works a lot better. I generally felt that the community rewarded my good work and buried my bad work, which is what it should do — I felt the system was legitimate, which is an important trait.
Pingback: links for 2026-01-06
A title and clipping with a lasting impression that leads to a quality post with facts arranged in a playful yet concise manner always has my attention.
It has to be something VERY new, unique, original or rare. I need new, hard-hitting and interesting info more than I need stale news.
Incentives for interaction with the post also yields greater results these days.
Nevertheless, spam accumulates at review systems and that spam ranks a site better than my spam-free reviews eg. stumbleupon
Pingback: 52 Post Ideas to Help You Stay Inspired Throughout The Year at Weekly Articles About Blogging - NxE