Monday, October 24, 2016

Hacker News Dataset October 2016

Our latest project on Sizzle is a visualization of the Top 10k Posts of All Time on Hacker News.

To create the visualization, we first needed to collect the data.

I noticed that there was an old copy of the hacker news dataset available on Big Query. But I needed an up-to-date copy, so I looked into the Hacker News Firebase API.

The API allows you to get each item by Id. You can start by retrieving the current Max ID, then walking backwards from there. (Items my be stories, comments, etc., it's the same API endpoints for all types of items.)

There is no rate limit, so I created the following script that will generate a text file with 10MM lines containing all of the URIs to retrieve. (we will then feed this file into wget using xargs)

Note: 10MM items was ~5 years worth of data.

Script to create the 10MM line file of URIs to retrieve:

That script takes about 10 minutes to produce a file that's around 560MB in size.

After the file is generated, you can feed it to wget using xargs to retrieve all the URIs.

cat hn-uri.txt | xargs -P 100 -n 100 wget --quiet

wget will save the result of each GET request in a separate file with the format {id}.json

Caution: That command took just over 30 hours to complete on my macbook. (it also killed Finder a couple times and I had to disable spotlight on the folder I was saving all the .json files to)

I found that it can be difficult to work with 10MM files in a single directory on my mac, so I will try to save you the trouble.

Here are a couple copies of the dataset I retrieved:

1. Here is a zip of the directory containing 10MM json files. (4GB)

2. Here is a SQL Server backup of a database containing a single table, that contains one record per json file. (2GB)

Hope this helps!

Wednesday, October 19, 2016

Top Cities in the United States

To find a data source for visualizations, I found myself commonly searching for "Top 200 Cities in the US", "largest cities by land area", and "largest meto area by population", etc.

I would then have to combine the list I found with other data sources to get all the info I needed.

I thought I'd try to keep an up-to-date copy of this list, in case others also found this helpful.

You can find that list here:

I will try to keep this list up-to-date with additional information and API keys as I use it over time.

Hope this helps,

Friday, October 14, 2016

Reddit API OAuth C# .NET Example

I had a need to connect to the Reddit API via C# .NET, and thought I'd put this little tutorial together since most I found were in java or python.

Note: General Reddit OAuth API info:

1. Go here and make an app:

2. Use the values from the app you created above, in this code sample:
  more info:

3. Make API requests:

Check out the result here:

interactive data visualization

Hope this helps,