By Kayla Kassis, Research Manager, California State Lottery
Published August 20, 2024
“But Why?”
“Why did [insert game here] not perform well? This should’ve been a top seller! What did players not like about it? Was it the color, the name, the prize? What??”
I’m sure all of us in the lottery business have heard (or personally had) these questions before, probably multiple times. And for many years, the California Lottery answered them the best we could with hypotheses and educated guesses based on prior player research or broad long-term sales or economic trends. But in 2012, the Business Planning & Research team at the California Lottery decided to take it one step further. It was time for the players to tell us the answers. So, we reached out to our loyal 2nd Chance members and asked them. Through survey research, we started to uncover the player behaviors and perceptions that drove a Scratchers game’s sales. Sales and profits are always key success metrics but diving deeper into the why…the strengths and weaknesses of a game…that’s what gives you the opportunity to diffuse player barriers and grow playership, ensuring long-term product portfolio longevity and growth.
My goal of this article is to share how the California Lottery uses player insights to provide a 360-degree evaluation of the performance of each of its 50+ new Scratchers games every year. And, to hopefully show the value and feasibility of incorporating this type of research and data as part of instant ticket game and strategy development.
ADVERTISEMENT
The Player Study
Currently, the California Lottery’s Scratchers Review Study is a 10–20-minute online survey conducted every other month with the primary objective to gain player insights to evaluate and diagnose the performance of a Scratchers game. We send email invites to a random sample of our 2nd Chance members and (as of 2021) also use a panel sample provider to field the same survey among a broader mix of California adults. The purchased sample allows us to get a better, more representative read on game awareness among non-Scratchers players and trial and attitudes among the less frequent Scratchers players who aren’t in our player program.
We do not currently offer incentives to our 2nd Chance members. When we first started, we would offer an entry into a bonus 2nd Chance drawing, but after comparing response rates with and without incentives, it really didn’t make a difference. Loyal players just want to share their opinions. If response rates start to slip more regularly, we may consider offering incentives again in some form.
We do ensure that our members have not taken a prior survey in the past three months.
Each survey includes the same battery of questions for each of the Scratchers games that were launched over the past 2-3 months (which is about 8-13 games) plus a few more custom questions for specific games of interest. Specifically, we measure a game’s:
Awareness and Consideration. The first stages in the marketing funnel. People can’t buy a product if they aren’t aware of it. And if they have seen it, did they even think about buying it? For all respondents, no matter their prior lottery play, we ask if they have ever seen the specific Scratchers game, while showing them an image of the game. If they have but didn’t buy it, we also ask whether it was something they at least considered buying.
Trial. The metric we were most eager to learn. How many players actually bought the game? Since Scratchers tickets are an anonymous, cash-dominant purchase and submissions into 2nd Chance never proved the most reliable of a proxy for overall playership, a survey with self-reported purchase seemed the best route for measuring it.
ADVERTISEMENT
Repurchase and Attachment. Both of these metrics try to measure the appeal of a game beyond initial trial. “Repurchase” asks those who bought the game if they are still buying it (which is 2-3 months after it was first launched), while “Attachment” measures how the game stacks up against other games on the market. Is it one of their favorite games or do they like other games better? The attachment question is asked on a 6-point scale and considered “positive” if a player thinks the game is at least somewhat better than other games available.
Confusion. People aren’t going to buy a game or make repeat purchases if it seems too complicated. Not every game or playstyle can be pre-tested, so this measure acts as a double check to understand if the perceived confusion of a game was what turned people off from it.
Gifting. It’s not just for the winter/seasonal games. There are many reasons and occasions for people to gift Scratchers, and from our perspective, gifting is a clear indication of a player’s loyalty and advocacy of the Lottery. As such, some Scratchers are launched because they make great “gifts,” so we ask each purchaser if they bought the game only for themselves, only as a gift, or a mix of the two.
Future Play Intent. For players of a game, we ask if this is something they’d buy again in the future, which can help signal strength of the franchise for possible relaunches and also correlates with general appeal of the game. However, we also ask this question of non-Scratchers players; acting like a pre-test of the game’s potential if awareness was higher. If we were able to better boost awareness of a game like this in the future, could we potentially convert them to a player?
The Value
These metrics and their underlying survey questions have been standardized over the years to ensure comparability over time. Asking them in the same way to the same types of people has allowed us to compare results of games to prior launches and games with similar features, and to create averages for price points and demographic groups, which provide rich context and interpretability to the results.
For example, just under half of core/frequent lottery players were aware of the $5 Greetings From California, a third bought it at least once, about half of those who bought it at least once were still buying it after a couple months, 33% had positive attachment, and 21% gifted it. But it is only in comparing that to the average for its price point or to a similar game, that we truly learn why it had below average sales. In this case, it appeared it suffered from lower overall awareness.
Over the years, these insights have been used to make key product development decisions, both at the individual game level and also when considering the full Scratchers portfolio strategy. At the conclusion of each year, an annual Scratchers Game Performance Review is completed, taking a look back and summarizing the performance of every game that was launched in that fiscal year. This review includes an analysis of sales data and these player insights. Combining them provides a 360-degree view on performance and highlights strengths and opportunities for the future. California is focused on reaching more types of players as a critical strategy for long-term sustained growth, which requires additional insights and metrics into game performance and player perspectives.
ADVERTISEMENT
Taking it a step further, with a database of results from over 400 games, some larger trends have emerged. For example, despite the $1 and $2 price points bringing in the lowest sales per game, visibility, trial and gifting rates are higher than the top price points ($20 and $30 in California). But their longevity is much shorter, with softer repurchase and attachment rates, highlighting that these games meet different needs and may be well-suited for diverse and playful themes and shorter time in market. This (along with other price point motivations research) led to our “fun at the $1” strategy: showcasing unique and fun-focused themes rather than just multipliers and money.
Additionally, from a theme perspective, animal- and luck-themed games do well with more casual players and women. Despite average sales performance, Scratchers with animal and luck themes have higher visibility, trial and repurchase than their price point’s average. And extended play games such as Crossword, Bingo and Loteria aren’t just for the core Scratchers players. These themes also do very well among the casual or light players and have high awareness. Games with these themes clearly hold value in the full product portfolio mix even if they don’t have standout sales; they ensure longevity of the category – keeping light players interested or introducing new players to the fun of Scratchers.
The study also revealed that repurchase and attachment are most positively correlated with better sales performance, while awareness can lead to long-term gains through perceptual shifts and stronger player reach. Churn from frequent players is a big driver of a game’s performance in-market. However, long-term strength of a product and brand also requires strong perceptions even among its more casual players. Results from the Scratchers Review Study showed that awareness of certain games was particularly effective at shifting perceptions of the category away from gambling to more excitement and fun, which is a critical strategic direction for the California Lottery. Games like Magic 8 Ball, Uno, and Pet Payout did well at shifting perceptions among casual players; showing better perceptions of Scratchers among those who were aware of the games compared to similar casual players who were not aware of the games.
Best Practices
Developing and conducting an ongoing study like this all in-house had its challenges. We learned and adapted as the years went on. Ultimately, though, despite missteps or some oversights here and there, it’s been a fun challenge regularly asking ourselves how we can do better, how we can optimize and maximize the value of our work. Here are some of the lessons we’ve learned and our advice to those considering a study of this kind:
Start Small and Focused. Don’t try to answer every possible question right at the start. We started with just a few games throughout the year. It took time to figure out the right questions that provided the most useful results. We stayed within our means and resources, so what could we do with what we had? We had a strong player database from our 2nd Chance members, so we started there. It wasn’t until years later that we developed the means to reach out to non-players. Take it one step at a time.
Think Long-Term. This may seem counter to the point above, but there is a balance to be achieved between short- and long-term gains. As seen with many brand trackers, comparability across studies (and in this case, games) is very important, so before making tweaks to wording, think about whether it changes the interpretation of the question and if so, is that OK? When developing questions, think about their longevity. Is it a question you have often or is it unique to that game? Are the answer options so custom and specific that we can’t apply it to other games?
Don’t be Afraid to Test. When you have the advantage of thinking long-term, you don’t always have to get it perfect right away. We’ve worked on the wording of questions over multiple studies, learning from prior results to figure out the best way to measure a metric. How do we tap into the right mindset and what provides us with unique knowledge? When we expanded the study’s objectives to measure perceptual changes, we started with several attributes and questions coming from many, nuanced perspectives. But after a few studies and running correlations analyses we were able to narrow those down to what was most effective, eventually removing and tweaking to improve the insights and survey experience.
Stay Nimble. There is no mistaking that having a standard, repeatable set of questions is important for any long-term study. But if you asked Marketing what they like the most about our study, it would be how we work together to develop adaptations and additional “probe” questions to test theories and hypotheses about various games. It’s important to build in room for custom questions. Why did they buy the prior version but not this one? Did they understand the top prize of this game? Take advantage of your access to player’s input. And it’s not just the questions that need to stay nimble, but also the research objectives and to some extent, the design. As the California Lottery shifted its Scratchers strategic direction to focus on growing the casual player base, we added more questions (and the external sample) to this study to help measure those changes on our games and players.
Use it! Research is only worth the time, effort and cost if it’s used. When we first shifted this study to a more regular basis, there was maybe a year or so when the Product Development team didn’t know how to use the results. Some people who received the data didn’t even know what the metrics meant. We just didn’t do a good job showing its value. We learned that it was imperative to do continuous training and communications with Marketing about the study and its data. Create and provide resources, data dictionaries and training materials. Even small changes to how the data was presented to be more user-friendly to non-researchers made a great difference. Now, the data is used in annual game performance reviews and are integral to key portfolio decisions and marketing strategies. And remember, this is a continuous effort. Communication and regular check-ins on how the research is used provide opportunities for improvements and keeps it relevant and actionable.
ADVERTISEMENT
“Oh, That’s Why!”
On an annual basis, around 9,000 California adults are surveyed about the games on the market; about 1,500 per survey (1,000 2nd Chance members and 500 from the external panel). All for an annual cost of less than $2.25 per person, based only on the cost of obtaining the external panel participants. We learn vital insights into who plays each game and why. Marketing makes data-driven decisions and then measures the impact of those decisions. Not all games are going to be star performers from a sales perspective, but maybe they are adding something else to the portfolio that just can’t be measured through sales and profits. Player insights add that extra value, ensuring you get the most out of every game – whether that means sales or in learnings for the future.
While the California Lottery is fortunate to have the resources to conduct this study on a frequent basis, insights like this, even on a small scale, can prove invaluable. We didn’t start this in 2012 thinking it was going to be an ongoing study or that every game would be tested. But we saw the opportunity and the potential. We started small, a few questions to our loyal players’ club, and inched our way forward to what it is today. If you follow research best practices, even a couple questions can lead to more informed decisions. Just be curious, listen to your players, and keep playing!
Comments