Stop Asking People How They Feel About Robots

Humans are notoriously bad at self-reporting everything about ourselves, from our nutritional habits to sleep patterns. So, it always strikes me as odd that social roboticists seem to insist on “measuring,” social personality traits with surveys and personal reports.

I’m not here to crap all over qualitative research. Of course, people need to think they feel something in order to buy (or want) it. But, the market tends to show that consistency beats out initial reactions. Some items are instant successes, but others are slow burns, like the iPhone. People thought “why would I need another device that’s like a computer, but small?” Then, upon actually interacting with the device, we all* think “wow it sure is perfect to have another device that’s like a computer, but small!”

So that brings us to social robots — devices which people seem to want to want, but apparently actually don’t. Study after study, researchers ask “how does this make you feel?” They create surveys that ask questions like “how much would you like this robot to be in your home?” or “how much fun was this robot to interact with?” The thing is, and I say this with love in my heart, people are dumb. As much as all the marketing majors are going to fist-bump their bros when I say this, it’s true: People don’t know what they want. And, people will always show you what they want.

The way to see if people think something is creepy isn’t to ask them — though many will volunteer their opinion — but to watch them. Regardless of how they think they feel, what are their interactions like? Here is a short list of questions I think researchers oughta be measuring when it comes to human-robot social interaction:

How physically close do they get to the robot?
How does their face change when it does something surprising, or worrying, or cute?
How much do they try to help it when it’s stuck, or perceived to be broken?
How often do they choose to interact with it unsolicited?
How long do interactions last when the robot solicits them?

So, when we social roboticists design systems and test hypothesis to make robots more fun, engaging, interactive, kind, insert-your-own-subjective-adjective-here, what we must do to evaluate our work is find behavioral proxies to measure our internal inferences. Yes, we are forced to assume “the kid thinks it’s more fun, because the kid played with it for longer, more frequently.” But I’d rather explicitize that assumption than rely on questionable self-reported data.

Ultimately, evaluating social robots is hard. It’s tough to recreate natural social situations in the lab, or even capture them to analyze in the home. But without observing real people really interacting with real robots, we’ll never get to the heart of what makes a robot… human.