DataBeat 2014: Quizlet and New Relic Team Up for Business Success

The following post explores just one of the many talks at DataBeat 2014. To find out more about DataBeat 2014, check out our recap here.

How did you study for your quizzes in high school? Did you meet with friends? Pool your resources? Make flashcards? For Andrew Sutherland, the solution came to him as a sophomore looking for a way to study for his French class. To meet his needs, he created Quizlet , an e-learning platform now used by 25 percent of all U.S. high school students.

At DataBeat 2014, Andrew sat down with Patrick Lightbody, vice president of product at New Relic. Their lightning talk explored how Quizlet uses New Relic’s software analytics system to extract business intelligence from their online learning platform.

The New Relic product in question, New Relic Insights, is still in beta, but customers like Quizlet are already seeing promising results. Lightbody said that when designing this new feature, the team had a specific problem to tackle. Businesses like Quizlet, he said, have all of the most important data flowing through their software applications. This data, including customer data and revenue data, is only valuable insofar as it can be made actionable.

How can it be captured and beamed back to the customer? Out of this dilemma was born New Relic Insights, a real-time analytics platform that promises to “query 200 million events in the time it takes you to blink.” Indeed, the speed of New Relic Insights is one of Andrew Sutherland’s favorite features.

Quizlet started using New Relic about three years ago, at first relying on their performance monitoring features and then adopting the nascent New Relic Insights when it launched this year.

Let’s take a closer look at the New Relic/Quizlet conversation:

What was your approach before to understanding user engagement?

According to Sutherland, Quizlet used to rely on Google Analytics and other similar tools. However, these platforms often require engineers to be proactively thoughtful about which web pages and interactions to measure. Developers must write code, ship it, and then wait for a few days for preliminary stats to be collected. Not only does this mean expanding the team of who must be involved in the process to overworked developers, it also means that there’s a delay before questions are answered, which can be bad news for a business in trouble.

The new system operates with SQL queries, and fundamentally flips the analysis process on its head. Before using New Relic Insights, Quizlet engineers had to come with a question to ask, set up the appropriate parameters to catch that data, wait a few weeks, and THEN analyze. With New Relic Insights, Sutherland says, Quizlet is constantly pulling page and user data, and thus could start querying from day one without waiting for the appropriate data to be generated.

What kinds of questions does Quizlet need to ask that aren’t performance related?

Sutherland shared an example of one specific case where Quizlet reaped the benefits of New Relic Insights. After an idle question from a C-Level executive, he ran a rapid query to identify the ages of the customers looking into purchasing on the Quizlet site, as well as the ages of those who actually follow through and pay for premium accounts. The New Relic Insights data revealed that the people browsing the payment page are largely teens 14-16 years of age, which makes sense when you consider that Quizlet’s key demographic is high schoolers. Those paying, however, are all over the age map, but are decidedly older on average (20-60 years of age).

Observing this trend allowed Quizlet to reallocate valuable marketing dollars, greatly reducing the amount of marketing for paid accounts that the company does to the under-20 segment, and increasing the budget for the older demographic capable of paying. Theirs was a question that no one thought to ask in the beginning, but because New Relic Insights is constantly collecting data, marketers — not developers — were able to go into the system with their question and get immediate results.

Any final thoughts on how this has changed performance of Quizlet/applications? How has it changed how you run the business?

“For a long time,” Sutherland said, “we were not very data oriented. It was difficult to answer questions with the data we had so, to be honest, we just didn’t ask it.” Now, Quizlet is asking more questions and, more importantly, getting more answers all the time. New Relic Insights allows them to proceed with analysis without any lead time and with no need to involve the developers — a positive change by any measure.

Want more from DataBeat 2014? Check out our other posts: