Processing billions of records
Our first step in dealing with large data sets is exploration—working closely with our clients to analyze the structure of their data and helping them glean insights from that data. In this case, we spent quite a lot of time helping ABS make sense of the data they had available and exploring ways to make their huge data set accessible in a performant way.
In order to attain any sort of reasonable site performance, we couldn't directly query the raw data. Instead, we pre-process and aggregate ABS data using AWS Elastic MapReduce with Apache Pig and Hive. We do this hourly and store the aggregated information in MongoDB. This process allows us to distill the raw data into a format that can easily and quickly be queried for display.
Data Dashboard design
With the data now stored in MongoDB, we can easily and quickly extract the data using Ruby on Rails. Backbone.js, D3.js, and Mapbox on the front end allow us to generate data visualizations that can simply and concisely give insights to viewers.
We needed to find a good way to display a lot of these data visualizations in an interesting and compelling way. Among the components we needed to fit into one page were a world map visualizing where bible verses were being read, graphs of views and readers over time, scrolling top ten lists of viewership, and a number of other single statistics. A grid-based layout along with bright splashes of color atop a black and charcoal background allowed us to compactly display all of the information while not overwhelming the user.
Responsive big-screen-first design
This grid-based layout lent itself perfectly to another requirement—a responsive design. The mobile-first approach, in which the mobile interface is designed and developed before the web version, is a common way to handle applications that will be viewed on multiple devices. We flipped this approach on its head for ABS by first designing for very large screens.
An important use case of the site was for it to be displayed on a very large screen in the ABS corporate lobby. Designing with this in mind, we built an attractive and responsive design that works great on a large screen and gracefully scales down to the web, a tablet, or a smartphone.
