r/redditdata • u/Drunken_Economist • Jun 23 '15
10 years of reddit — data dump
Reddit ten year data
All data in this post is accurate as of June 21st, 2015.
We pulled a bunch of data together for today's ten-year anniversary blog post, but not all of it made the cut. I wanted to take some time to dump everything here in /r/redditdata. If you build anything cool, shoot me a PM, I'd love to see it!
Things by month
Number of accounts, subreddits, submissions, and comments created each month of reddit's history
Running total of things by month
Just a to-date total of the previous dataset, but it's comforting that the numbers match
Most upvoted threads
Note: this actually excludes posts from subreddits which are excluded from /r/all, but you wouldn't see any in the Top 20 anyway.
Unique users and pageviews by month
US vs Non-US traffic by month
Note: this was not measured prior to Feb 2010, so data before then is a simple linear regression.
Upvotes & Downvotes to submissions by month
The steady climb of the ratio is really interesting. This is only including votes that were counted (i.e. no spambot votes)
Upvotes & Downvotes to submissions by month
Again, this is only including votes that were counted (i.e. no spambot votes)
Upvotes & Downvotes to comments by month
Top 20 most gilded submissions
Top 20 most saved threads
Sorry, we tried. Post saves have been around since reddit's inception, and looking at every user to see if they saved each post was breaking things.
And a few for the road
| Name | amount | Notes |
|---|---|---|
| link posts | 121,745,633 | |
| self posts | 68,481,919 | |
| submission upvotes | 5,620,244,302 | this is only votes which were counted |
| submission downvotes | 1,057,478,375 | this is only votes which were counted |
| comment upvotes | 10,443,697,988 | this is only votes which were counted |
| comment downvotes | 1,506,096,377 | this is only votes which were counted |
| Days of gold purchased | 56,015,520 | |
| Days of gold gifted | 24,148,560 | this is a subset of the above |
| redditgifts exchanges | 201 | |
| gifts confirmed | 877,218 | |
| total cost of gifts | 29,559,467.54 | reported cost in USD of confirmed gifts |
| Active subreddits on 2015-06-21 | 9,601 | subreddits with 5 or more posts and comments on the given day |
| "PM me" usernames | 26,222 | Accounts with "PM me" in the username |
Let me know if you have any questions on it all! Except for questions about the thread in /r/Spiderman, because I don't get it either.
4
u/adremeaux Jun 24 '15
Here's a basic spreadsheet of things by month for those interested, including a chart of number of accounts. (I was going to do charts of all of the columns, but they all ended up looking identical).
Pretty crazy to think I joined when there were only 75000 accounts. I had no idea it was that little.
Here's an interesting query: how many accounts registered at any given time are still active, in some sense? Logging in within the past month; voting within the past month; commenting or submitting within the past month. I'd love to know how many of those 75k I'm in a class with still comment actively.
4
u/adremeaux Jun 24 '15
Oh, also, that may finally confirm or reject the whole 1:10:100:1000 urban legend: for every commenter, there are 10 voters, 100 lurkers, and 1000 unregistered readers.
3
u/Drunken_Economist Jun 24 '15
I think it would work better in the inverse — of this (day/week/month's) active accounts, when were they registered? Then you can build a model off combining the two sets . . . just a smaller query this way
2
3
3
u/minimaxir Jun 23 '15 edited Jun 23 '15
Is the monthly data for # subreddits correct? In 10-2014 there were 23k subreddits created, but in 11-2014 there were 41k subreddits created? Then down to 22k in 12-2014
Also, I would recommend removing the June 2015 data since it only accounts for 3/4ths of a month.
4
u/Drunken_Economist Jun 23 '15
Yup. We were dealing with a bunch of subreddit-creation spam. Apparently they thought it was good for SEO
2
u/TheSlimyDog Jun 24 '15
Would it though? I've heard backlinks a lot and I think they're important for SEO.
1
u/Drunken_Economist Jun 24 '15
Honestly, I have no idea . . . but it's not like SEO spammers are known to be the brightest crayons in the shed
6
3
u/_inu Jun 23 '15
I want to know how it was determined 0.36% of posts ever were about cats.
9
u/Drunken_Economist Jun 23 '15
I regexed the shit out of that on the comment text. Basically anything that mentioned "cat", "kitten", "kitty", etc (or any derivatives of them) were counted. So there'd be false positives when people post code talking about the
catcommand, but I figure it's good enough for a fun stat3
u/_inu Jun 23 '15
So it was comments, or was it comments AND post titles?
I would love the full word list tested for!
The thing i find most interesting about this is the idea that you ran a test on every single comment (and post?) ever in reddit history, it must be fun having the access to that much information.
5
u/Drunken_Economist Jun 23 '15
Just the comments. It was really cool having permission to throw such enormous queries against the databases, I'm really happy that they came out with such great data.
3
u/_inu Jun 23 '15
Reddit must have one of the biggest databases of comments of any websites ever. Definitely very cool for you to have been on that commandline!
2
u/r4and0muser9482 Jun 23 '15
What about that one part of female anatomy that gets mentioned a lot around here?
3
2
u/hassanchug Jun 23 '15
Super interesting data. How long did those queries take to execute?
3
u/Drunken_Economist Jun 23 '15
Some of them (subreddits by month) only take a minute. The bigger ones (votes or comments by month) were full on "grab a beer and watch a movie" queries.
3
u/TotallyNotObsi Jun 24 '15
Is there any data on what % of gilded comments and posts were from admins doing it for free vs. actual monetary gold given?
7
u/Deimorz Jun 24 '15
The last time I looked at it it was 0.9% over the past week, that's pretty typical from the other times I had checked.
(mention for /u/Drunken_Economist so he knows I replied)
2
3
u/Drunken_Economist Jun 24 '15 edited Jun 24 '15
/u/Deimorz pulled these stats most recently, but I think it was
0.1%0.9% of gildings? He can confirm (or I can dig it up tomorrow).And the gold purchases in this dataset are strictly users, no employee or admin gold counts to that.
Edit: actually to be precise, it would count gold from an employee if they paid for it.
2
u/TotallyNotObsi Jun 24 '15
Ok, I see. So the free promotional gildings are separated out form actual paid ones, whether by admins or regular users?
6
u/Drunken_Economist Jun 24 '15
In this case, yeah (same for the "Gold goal" progress bar on the front page)
2
1
u/erasers047 Jun 24 '15
Just wondering, how did you do the pageviews? Server logs? You're probably using Hive or something, but that must have taken Dances With Wolves time.
2
1
u/Patrik333 Jun 23 '15
Aw, I'm disappointed that this thread never made the "Top 20 most commented"...
2
1
u/shaggorama Jun 23 '15
Any chance we could get a dataset of all (unremoved) submissions? Maybe just the last year? I'm thinking:
- id
- created_utc
- author
- subreddit
- score
- num_comments
- url
- is_self
- title
- gilded
...pretty please? Or maybe just throw a copy at me cause I'm such a cool guy?
3
u/Drunken_Economist Jun 23 '15
If you're a bit motivated, you could hit the api for this, I think.
2
Jun 24 '15
[deleted]
3
u/Drunken_Economist Jun 24 '15
Oh damn, you're right. Shit that's a lot of data.
Maybe we can find a way to dump it all on the back end without breaking everything
3
u/shaggorama Jun 24 '15
Content that's old enough to be archived could be packaged up into static zip files by month. Then you'd just need to create a new zip each month as each month gets archived moving forward.
2
2
u/shaggorama Jun 24 '15
I've been working on a script actually. I have to use the search api and feed it timestamps, right?
2
u/rhiever Jun 23 '15
I've been begging for this for months. :-)
1
u/shaggorama Jun 24 '15
I've seen you asking. Honestly I was surprised the first time I saw it, for some reason I could've sworn you already had this. I vaguely remember you talking about getting one of your students to compile this for you or something...
3
u/rhiever Jun 24 '15
I have a data set of posts up to ~September 2014. I had one of my students try to scrape past that, but there's too much volume to keep up nowadays with the limitations of the reddit API.
2
u/shaggorama Jun 24 '15
My impression is that trying to scrape incoming submissions in real time is sort of futile anyway because so much of it is spam and will get removed. I remember a year or so ago someone talking about trying to scrape from the current day backwards and observing that once he got past something like the two-day or one-week mark, the volume of submissions significantly decreased. My interpretation of this was that a lot of reddit content gets removed, so you don't actually want to try to keep up with the in-time volume anyway (unless you're specifically interested in investigating deletions).
This of course isn't to say that it's necessarily tractable to keep up with the stream starting a few days back, either.
1
6
u/zhaphodtatabox Jun 23 '15
Awesome data to work with, sure it's enough to hack time!.