(aka “My Epic Journey Back to Doing Science”)
In a little over three weeks, I’m scheduled to present “Exploratory Bandit Experiments with ‘Starter Packs’ in a Free-to-Play Mobile Game” at IEEE-COG. I’m the third author, along with Julian Runge and Anders Drachen.
This is the first time in almost 20 years that I’ve worked on an academic paper. While I’m the author or co-author of over 50 academic papers, I haven’t been an author (or co-author) of one since 2005, when “Supporting Rule System Interoperability on the Semantic Web with SWRL” was published. That’s a long time.
So why am I writing academic papers again? And why am I excited by this?
To understand that, it’s important to understand that I haven’t stopped publishing. I’d simply stopped writing academic papers. In the years since I stopped working on knowledge representation and the semantic web, I’ve published long-form thoughts in many places. Among them are:
- On Game Developer (thoughts on revenue optimization in gaming)
- On Slideshare (Scientific Revenue and Personal Presentations).
- On my mostly aspirational personal blog.
- As collaborations with other people (for example, my article on Generative AI).
If you look at what I’ve produced in those 20 years, about half of it concerns revenue optimization in gaming. And there’s been a steady change in what I’ve published — it switches from pure folklore to empirical science.
The arc from 2010 to 2020 exemplifies my changing perspective.
In 2010, when I was the Head of Product at Live Gamer, I gave a talk at GDC entitled “The New ARPU Playbook: What Safeway Can Teach Us” (which was recorded and put on YouTube: part 1, part 2, and part 3).
This talk repeatedly switches between discussing merchandising tactics in physical retail merchandising and wondering out loud why online gaming doesn’t do similar things. It’s entirely anecdotal, with no data or statistics.
Other talks from Live Gamer (for example, this one from Casual Connect) emphasize the importance of A/B testing but still mainly communicate best practices via folklore and anecdotes. There is absolutely nothing wrong with the following slide except for the complete lack of evidence that any of it really works.
In 2014, when I had just founded Scientific Revenue, I gave a talk at the Wolfram Data Summit entitled “The Strange Case of the N00B Who Didn’t Buy: Big Data and Pricing” (slides here and video here). It’s still all anecdotes (we hadn’t actually done anything at Scientific Revenue yet), but it attempts to make the case that there could be an empirically based scientific theory of digital pricing.
I’m still fond of that presentation and, in particular, of this slide:
By 2017 and 2018, Scientific Revenue was in full swing, and the talks were much more about the empirical, semi-scientific process of pricing optimization.
For example, in What Makes a Price a Good Price or The Unreasonable Effectiveness of Data (from 2017 and 2018 respectively), there were:
- Citations of academic papers: both in the pricing literature and in the predictive analytics literature.
- Examples of specific changes being made in production games.
- Measurements of the value of the changes (including some p-values).
- Citations of underlying economic theory or behavioral principles.
And there’s a growing conviction in the slides that if you haven’t empirically validated the results in a statistically meaningful way, the results are… meaningless. Consider, for example, the following two-slide sequence, which defines a set of payment walls and then talks about revenue optimization. (This is also an early example of thinking about Treatment Effect Heterogeneity in-game data analytics. We’ll talk about TEH in depth in a future blog entry.)
That’s not quite rigorous science. But it’s getting close. There’s a hypothesis (based on the pricing literature), there’s a rigorously defined set of treatments, there’s a carefully designed experiment, and there’s a carefully measured outcome.
All of which is to say, I’ve been traveling along a 20-year arc: from being intrigued by anecdotal folklore to building large-scale systems that enable rigorous statistical testing of scientific hypotheses.
Which led to Game Data Pros and our first paper.
What we’re trying to do at Game Data Pros is simple:
- We’re trying to build robust systems for revenue optimization in digital entertainment.
- We’re moving well past the “point solution” systems of the past (this system does IAP pricing, that utterly disconnected system built by other people does cross-promotion, and this other system that we got from a SaaS vendor does advertising optimization) to building unified optimization across all the revenue touchpoints.
- And we’re doing so in a scientifically valid way that incorporates best practices from across the gaming industry and the statistical inference community (e.g. we’re going way past A/B testing).
And a big part of this is sharing what we learn with the community and building bridges to academic researchers (who often have great ideas but often lack the ability to test them at scale).
In short, we’re doing empirical science as part of a scientific community. And, as part of that, we publish academic papers.
And, while it’s early days, I find that incredibly exciting.
