The Truth About Mystery Shopping

Photo Credit: Doublelibra (Flickr.com)

When I was working in retail, one announcement I absolutely dreaded was that sometime in the next few weeks, my store would be mystery shopped.

Don’t get me wrong; I didn’t have anything to hide. But I’d been through enough mystery shops to know that they almost always yielded bad things for me as a front-line manager.

Mystery shoppers are field researchers who are supposed to interact with front-line staff incognito and ensure that they are doing their job properly. Their job is to pose as a customer, pay attention to the details, and then fill out a form describing their experience. They are often asked to answer such questions as, “Was the associate smiling?”, “Was the storefront clean?” and “Was the open sign turned on?”.

The problem is not in the data that’s collected, of course, because these are all important things to know if you are trying to improve the consistency of a store’s service quality. The problem is in how the information is dealt with.

You see, every few months, I’d receive a mystery shopping report that would point out some glaring flaw in my store’s service. Maybe the mystery shopper had to wait in line because an associate was helping another customer. Maybe an associate had forgotten his or her nametag that day and foolishly decided not to wear one. Maybe the mystery shop occurred when I was on a mandatory conference call and trying to man the sales floor at the same time because an associate had called in sick.

The point of course, being that mystery shops didn’t always reflect the reality of what was going on in my store, and were just as prone to outlier behavior as any single sampling technique might be. (To understand what I mean here, imagine that you’re blind and you’re reaching into a fish bowl filled with 500 jellybeans and 10 pieces of gravel. If you grab one of those pieces of gravel, you might make the erroneous assumption that the entire bowl is filled with rocks, even though the odds were high that you’d grab a jellybean.)

What’s more, the way the data was applied was to publicly shame managers during conference calls who did not achieve a 100% rating and to use bad shops as a means of writing up employees (or managers!) for less-than-stellar service.

From a research standpoint, this is probably one of the worst ways you can apply data gleaned from a study. Instead of teaching front-line staff to embrace mystery shops as a valuable tool for improvement, you teach them to dread mystery shops and to resent the results.

There are a few other problems with mystery shopping that need to be addressed:

1) Who are these mystery shoppers? I did some mystery shopping once for a firm I signed up for via the internet. I was paid $10 to go into a bank and pretend to open an account, and then told to back out at the last minute and thank them for their time. All of this was dutifully recorded on a sheet of paper and sent in. It was one of the most awkward experiences I’ve ever had, and it turned me off to mystery shopping. Another time, my mom decided to sign up for restaurant mystery shopping, and she was assigned to a restaurant she didn’t like for a shop. She tried her best to be objective, but she already knew going in that if the service was bad, she was going to hammer them.

Many firms who provide mystery shoppers at the regional or national scale are recruiting folks like me and my mom to do the shops. We both received no training and were given very little direction. Would you want to trust the validity of that data when making decisions about service satisfaction?

2) Mystery shoppers have a built-in bias. Mystery shoppers are not the same as normal customers because they know that they are shopping with a critical mindset. Things they might have ignored in the past are suddenly obvious to them; behaviors they might have excused on their own suddenly become very important. Mystery shoppers are going to be far more critical than the average customer, simply because they’re more involved in the interaction. You have to be very careful in framing your study or you are more likely to see a negative bias in the results.

3) Mystery shoppers can lie. This is absolutely a problem in research, and it’s one of those things that’s very difficult for clients to verify. Mystery shoppers can very easily go into a store, get the bare essential details, and then leave without ever interacting with the staff, but fill out their forms as if they had. I know it happened to my own store on one occasion, and it probably happens more than most researchers would like to think, because mystery shopping requires a lot of nerve. Unless a mystery shopper is doing a quick transaction (like going through a drive-thru at McDonald’s), the whole exchange requires some acting and, sometimes, quick thinking. Mystery shoppers who have a minimal amount of training are going to be more likely to slip in and slip out than those who know how to handle a contingency.

4) Staff are usually warned about shops ahead of time. There is an ethical question as to whether or not mystery shopping should be conducted with the awareness of the staff. Ethically speaking, field research should not be conducted on an unknowing employee unless the employee’s knowledge will invalidate the research process. To get around this, most mystery shopping studies give managers and district managers a window of time in which a shop might occur, such as “the last two weeks of the month” or “sometime this quarter.”

Here’s the problem: any time you put this information out into a hierarchy, you’re going to see downward pressure to master the mystery shops. District managers will announce the mystery shops and then try to hold each manager accountable for getting a perfect score. This may result in an artificial improvement in service until the mystery shopping period is over. This invalidates the data and makes it more likely that the busy people who really are doing their job correctly will get caught in something stupid, while the not-so-busy people who are not doing their job correctly but who are good at faking things to meet the criteria will look like they’re model employees.

5) Information obtained from mystery shops is often sent to managers without qualification. When I would get my mystery shop reports as a manager, I was not told how to deal with them. I was smart enough to know that singling out employees was not a good practice, and I would only address issues with individual employees if they showed up repeatedly. But many managers don’t really know how to manage people very well, and they use the mystery shop reports as leverage. What’s more, when managers are mystery shopped and fail to live up to standards, employees often see these reports as evidence that they should not have faith in their bosses to do their jobs correctly.

It’s an ugly cycle.

So, what can be done?

Tomorrow, I’m going to offer a piece called “How to Conduct a Proper Mystery Shop.” It will explain how you can correct all of these problems and use mystery shopping to its fullest. Don’t miss it!