more than a few people have pointed me towards a tool created by ajax economy that purports to record all bury data for any submission made to digg, asking for comments. i would love to see a tool like this that works and captures 100% of the data but here’s why you shouldn’t get too excited about this particular tool. on the announcement page for version 0.2 of the tool, the site states that the tool is capturing 100% of the data but if you read into how the tool works you’ll see that a more accurate statement is that ‘it captures 100% of all available data’, which is not much by any means.
this application gets the json feed used by digg spy. it does this using ajax (i.e. the xmlhttprequest object) which requires a server side proxy due to domain security restrictions. due to the way that the json is returned from digg spy, it doesn’t set a variable equal to the returned object, which force us to use the before mentioned server side proxy and an eval statement instead of using dom manipulation. the application simply polls for updated data every 20 seconds which makes sure we don’t miss any data and that it doesn’t put too much strain on the server.
the simple problem here is that the tool relies on the json feed from digg spy and yes, it does capture 100% of the data shown by that feed. the feed from digg spy, however, only shows approximately less than 10% of the activity on digg (just do some basic math in your head and you’ll see that there is no way that it could show more data than that and be even remotely readable). so what you essentially have is 100% of 10% of all data, which for all purposes is highly inaccurate because even the 10% is not spread over all stories.
that said, if someone has come up with a better, more accurate way to record information, or thinks the above mentioned is incorrect, please have your say in the comments. for what i know digg shows partial data and there is no way to get the data for the all the activity on the site.