I love data. I would lose my research stripes if I didn’t.
One thing I’ve learned over the years though is that data has its limits.
When I turn on my laptop excited to pour over the results of a new study (researchers are easily entertained), a data point will sometimes jump off the page—that one number that could change everything. But I’ve also learned that when a data point doesn’t look right, 9 out of 10 times it isn’t. Years of experience working with similar data sets from other audio studies brings on a gut reaction towards wonky results that’s right more often than not. So, I dig in and look deeper.
Human intuition adds value to data; they work together to generate actionable results.
Let’s look at a few examples. I’ll start with a personal favorite, baseball.
You’re probably familiar with the Moneyball book by Michael Lewis, or the movie starring Brad Pitt as the Oakland Athletics general manager and Jonah Hill as the geeky stats guy. In the movie, the Hill character brings the advanced data analytics system, ‘sabermetrics’ to the cash-strapped baseball team, casting aside the opinions of the scouts, and propels the lowly As to win their division.
It’s a great story, based on the real-life role that data played to help mold a winning baseball team. But the founder of sabermetrics, Bill James, says it’s not that simple: “[Michael Lewis] presented this conflict between the scouts (and) the analytic community. I never found that to be true. The scouts grew up reading [the data] and know it and understand it.”
In fact, by adding a subjective human factor, baseball scouts help to make better player decisions than analytics on their own. The speed of a 17-year-old pitcher’s fastball speaks to his potential on the mound, but the dinner a scout has with the player and his family can tell you whether the player has the character and commitment to make it to the big leagues. Would you be comfortable basing your next hire exclusively on HR analytics? No interviews, no feedback from past co-workers? Probably not.
Nate Silver, the statistician behind the fivethirtyeight website, has actually tried to quantify the added value of a baseball scout and pegs it at 15 to 20 percent. Maybe it’s even greater.
Artificial intelligence (AI) gets smarter when its models are trained by human experience.
Weather forecasters add similar value to the hyper-sophisticated data models that have vastly improved weather predictions.
Since the 1950s, advances in computer technology and the data collected by weather satellites have made it possible to make predictions with pinpoint precision. Yet, it still needs people to read the maps. Thanks to local knowledge and personal observations, meteorologists still often outperform the algorithms, especially when it comes to the increase of extreme weather events due to climate change.
Likewise, the AI that drives smart speakers continues to depend on human intervention to improve their ‘common sense reasoning.’
Human understanding is even more crucial when it comes to the still large data gaps around audio consumption.
Though the data is getting better, audio—specifically broadcast radio and podcasting—is still well short of having the wealth of data available to weather forecasters or even baseball statisticians.
Personal people meters (PPM) capture exposure to radio signals, but small sample sizes limit their reliability and the ability to deliver data that would help advertisers connect exposure to consumer behaviour. The growth of online streaming of AM/FM radio shows potential by providing census level data on every streaming event rather than a small subsample, but streaming still represents a small slice of overall listening to broadcast radio and can offer relatively little insight on who’s listening.
Podcast data, while improving, still has some gaping holes. Downloads provide census level data but rely on human assumptions within the IAB Podcast Measurement Technical Guidelines on how many people who download an episode actually listen to it. And, like AM/FM or music streaming, download data lack person vs. household demographics (though that is something we are helping Triton Digital develop with their Podcast Metrics Demos+ initiative).
These data shortcomings mean that effective buying of podcast ads still requires heavy lifting from informed and experienced buyers.
The opportunity posed by programmatic ad delivery for podcasts is promising. Its potential to simplify the buying process and move ad dollars further down the long tail of podcasting is exciting. We’re not there yet though. Krystina Rubino of Right Side Up, who witnessed the move towards programmatic buying in other forms of digital media, makes the case in a recent Sounds Profitable article that until podcasting gets better data—and standards around that data—only hands-on ad buys will deliver the kind of ad campaigns that have made such podcasting such an impactful ad medium. Like self-driving cars, programmatic ads for podcasting will be a part of the future, but we have to figure out a few things before we can take our hands off the wheel.
Takeaways for Adding Understanding to Audio Data
- Apply healthy skepticism if the data point feels counter-intuitive to what your experience in audio tells you. Look for context. Ask questions. Just make sure you stay open to the possibility that it might be that new finding that changes everything.
- Consider the source. Does it come from an independent third party? Does the source have a vested interest in the result? It could still be a valid finding, but you may want to look deeper.
- Work with partners who know audio. Numbers are just numbers unless they come with an understanding of what they mean.
- Finally, keep your expectations of AI solutions in check. Though it’s tempting to think that artificial intelligence equals automated intelligence, remember two things: 1) AI depends on human experience to make it better; and 2) AI needs the kind of clean and comprehensive data that we’re still missing in audio.
Signal Hill Insights is an audio research consultancy partnering with publishers, broadcasters and advertisers to tap new opportunities in audio. Our core competency is survey-based research including brand lift studies. Visit us at https://signalhillinsights.com/ or connect with us here.
