Do you agree or disagree with the following statement…?

Data quality is critically compromised when double-barreled questions/statements are used in market research surveys and researchers who write them into their surveys should be shot.

;P

What has Botticelli’s The Birth of Venus got to do with market research? Well, note the crop. But as nice as it is, it’s only the bigger picture that tells us the actual story.

This is at the heart of the next issue for discussion; the ‘new’ (?!) practice of listening online.

Of course ‘listening’ itself, as a method of research inquiry, is pretty obvious and hardly new. What is new* for market research however, is;

• the online location per se,
• some of the online listening technologies, and
• in some cases, the actual content generated online.

But new or not, you still need to know exactly who it is that you’re listening to. And you also need to think about the context.

For example…

If you’re undertaking a market research listening exercise for a client, let’s assume, quite reasonably, that your focus will be on listening to their customers or potential customers.

Is the Internet a good place to listen?

Well, of course it is! But there are some very important questions to ask before you begin;

Of all your client’s customers/potential customers, how many have access to the Internet?

And of all your client’s customers/potential customers who have access to the Internet, how many are confident enough in a) their opinions and b) their writing (or video editing) skills, to express those opinions publicly online?

And of all your client’s customers/potential customers who have access to the Internet, and who are confident enough in a) their opinions and b) their writing (or video editing) skills to express their opinions publicly online, how many bother?

And of all your client’s customers/potential customers who have access to the Internet, who are confident enough about a) their opinions and b) their writing (or video editing) skills to express their opinions publicly online, and who bother, how many express those opinions in an articulate way (ie in a way that a marketer or market researcher might find of use)?

The skews are breaking my brain.

Defining your sample in terms of exactly who it is that you want to get feedback from is absolutely key in terms of determining where and how you should collect your data.

Notably, if your client’s target market comprises a wider group of people than the customers or potential customers who fit the very narrow profile described above, then – and critically – collecting useful data to generate useful output will mean going well beyond the insights you glean from your ‘new’ online listening endeavours.

*relatively speaking and/or as the hype would suggest

There’s been a lot of talk about engaging research participants in this ‘new’ research paradigm.

I’m focusing on qualitative market research here because firstly, that’s my thing…

: )

…and secondly, because I hear that ‘engagement’, within the context of market research online communities, is community-nirvana. The ‘best’ communities are engaged communities.

This strikes me as, paradoxically, both obvious and alarming.

There’s a very fine line between engaging research participants enough to… well…participate in our market research, and over engaging them.

Without due diligence, research effects (pick one of many) are likely to confound the research output in unintended, unexpected and underestimated ways.

I don’t, for one second, suggest that other research methodologies are free of research effects – they aren’t! – but surely this doesn’t automatically generate a license for ‘new’ market research to ignore them.

And while I don’t think you can necessarily control for engagement, some questions;

• To what degree should you try to ‘create’ it?
• How much is too much? When does it start to mess with what you’re looking for from your research?
• How do you disentangle it from your analysis?

I’ve just come back from a fantastic trip to Chicago where I attended the ESOMAR Online Research 2009 conference. You can find Jeffrey Henning’s brilliant recap here (he pretty much live-blogged it – very impressive!).

Anyway, not surprisingly, there was lots of talk of ‘new’.

And it would have been easy to come away with the message that the market research industry really needs to get with the ‘new’ programme or, quite simply, it will wither away and die.

Because the new world of research is here! New methodologies. New technologies. New ways of engaging with respondents (Gasp! Did I say that? I meant ‘participants’, ‘co-creators’ or ‘collaborators’).

All well and good, but what does this actually mean?

New kinds of output?

Let’s all take a moment to think. What, as market researchers, is our purpose?

Market research is about understanding the market. At a very basic level, the end goal is to deliver information that will help our clients make relatively informed/better decisions about how to sell their products or services.

And here’s my point; the output (ie what our clients are paying for) is only ever going to be as ‘new’ as the questions they/we ask*.

Over the next few blog posts I’m going to take a look at some of the elements of ‘new’ I outlined above. Examine them closely. Explore what they mean for researchers at a practical level. Separate, if you like, the hype and theory from the actual task of delivering useful output.

Should be interesting…

: P

*If you’re in the ‘listening’ camp, ie “Oh no, no! We don’t ask questions, we just listen to the conversation!”, I’ll argue that you’re still (implicitly) asking questions when you choose to/not to include any particular content in your analysis.

The term ‘traditional research’ is often used to describe offline methodologies; mostly, I’ve observed, in the context of selling what I assume must be ‘non-traditional’ (??) online methodologies.

In this context, the word ‘traditional’ conjures imagery of antiquated, moth-eaten and fusty research practices.

And clearly, many offline methodologies are anything but. Not to mention that many research solutions require a hybrid of both online and offline approaches.

But if we have to throw labels around, it’d probably be more accurate to substitute the word ‘traditional’ with ‘established’ (as Paul Vittles from TNS usefully suggested during question time at an AMSRS breakfast seminar I attended last week).

So does this make ‘non-traditional’ research ‘unestablished’ research?

: P

Wow. Doesn’t that put a whole new spin on it!?

Great post over at Ray Poynter’s (always) inspiring blog.

The way Ray described a short-term community, I think, delineates a clever, and potentially efficient, research methodology.

But I’m yet to be convinced of the long-term research community concept (although I’ve no doubt some readers are probably pulling their hair out over my apparent inability/reluctance to see the light/‘get it’).

: P

I’m assuming that in a ‘community for market research’ (vs a ‘community for customer relations/brand building’) context, a client will actually have some questions they want to ask and/or issues they’d like to explore.

If they don’t – if it’s a case of just wanting to pick up on issues entirely generated by the research community itself – then I’m guessing that they’d probably get better value by undertaking some basic social media monitoring … (god forbid).

Anyway, assuming we have a community that’s being used for market research, and there are some questions to throw into that research (in whatever shape or form), here are my questions;

1) At what point in time, along the short to long-term spectrum, does the nature of the output you get from a research community change? And probably more importantly, in what way does it change?

(Or is it different from day one because of how the participants are briefed about purpose of the community and their role in it?)

2) Would one analyse the data coming out of long-term vs short-term research communities differently? This, I suppose, brings us to the issue of the research community objectives. From the sales pitch, I sometimes get the impression that the sole objective of the exercise is to get the research community members to bond; never mind the insight, they’re bonding!

: P

Kidding.

But really, what are the issues, the benefits and drawbacks of community members ‘developing a sense of community’ (and from that, I would assume we mean ‘belonging’), particularly over a longer time frame?

Does it make participants more honest, or more willing to share? Maybe. But (it could easily be argued), maybe not. And what impact does, for example, group/clique think etc have on the output? How would you identify/measure the impact in such a wide (uncontrolled? quasi-controlled?) landscape?

(And to take it to the extreme, if one’s aiming for ‘uncontrolled’, then back to the point above; isn’t the ‘community’ just a very limited method for undertaking social media monitoring?)

3) If developing a sense of community/belonging is one of the key operating principles for long term research communities, then how do new recruits (or exiting members for that matter) impact the existing community dynamic and thus affect the output? How about changes in community moderators/managers?

4) Are there any studies looking at the differences in terms of valuable/usable output between short-term vs long-term research communities?

Are there any actual or even theoretical definitions of the ROI (tangible or even intangible) that one might expect from a short-term compared to a long-term research community available?

(I’ll probably come back to this with more questions when I’ve had time to think more).

Coincidentally, after my last post, this came up in my Google Reader today;

More on the Problems with Opt-in Internet Surveys

Here’s the first article;

Study Finds Trouble for Opt-in Internet Surveys

I had the great privilege of attending an AMSRS Professional Development session earlier this year to hear Jon Krosnick speak. He was brilliant.

And so it’s with great interest that I follow this very timely and fascinating debate (hosted on Gary Langer’s excellent blog).

Make sure you follow the links to get the full story/debate. It’s an important one.

P.S And here’s a link to the study itself.

So! Continuing with the #stuffiwonder theme…

The telephone vs online survey debate.

The one that goes;

“Really, given that everyone’s moving from landline to mobile/cell these days, telephone survey sample representivity is seriously compromised”.

More often than not (and, of course, depending on who’s doing the debating), it ends with a nod to online panel surveys. In this context, “…they’re probably just as good as – if not better – than telephone surveys”.

Right?

Well, I don’t know.

Panels are opt in. And yes, the same can (and should) be argued about telephone interviews. You most definitely need research participants to opt in beyond a “Bugger off, I’m eating my dinner” response.

But what differences might we see, in terms of motivation and the research output, between a sample comprising individuals who;

• Have been approached randomly (and I get that it’s not really random; the population will be limited to those with landlines), vs
• Sign up to be part of a/several market research panel/s and/to get paid for their opinions?

How do market research online community providers populate their clients’ communities?

(When I say “market research” communities, that’s exactly what I mean; a community used as a market research tool. I’m not talking about online communities that are used in a marketing/customer relations exercise.

I’m not quite sure that the difference is apparent to all, but they’re not the same; not by a long shot.

In one, you’re giving the community members love because you want to make them happy. In the other, the relationship is somewhat more pragmatic; you want to learn from them. Notably, if you’re giving them love to make them happy, you’re not necessarily going to learn much, because they’ll be all nice and lovely back).

Anyway, focusing specifically on market research communities; what checks are in place to ensure that the people who end up in the community represent the people the client actually wants to hear from (ie the population of interest)?

To borrow from the delightful John Lacey, I’m filing this one under #stuffiwonder.

Here’s another excerpt from my Marketing Magazine series on Qualitative Research…

Within a qualitative research context, there’s no right way to ask a question per se. There are actually many right ways to ask a question. And there are also many wrong ways to ask a question.

The wrong ways

You may have heard about some of the following heinous qualitative research crimes:

• Asking leading questions
• Asking closed ended questions
• Asking vague questions

Why are these ‘wrong’?

Because leading questions ‘lead’ people to a particular answer, closed ended questions can end the discussion prematurely, and vague questions elicit vague answers that have little grounding for interpretation.

Well, theoretically. But all is not what it seems. An experienced moderator might use any of these types of questions purposefully, and with excellent effect:

• A leading question often works well to test a hypothesis, or as stimulus in itself, to get the conversation going
• A closed ended, or vague question can provide a foundation to open the discussion in interesting and new ways

They’re all part of the qualitative researcher’s toolkit and used in a timely and purposeful way, can add tremendous depth to the discussion.

Posted in market research, questions ]]> http://zebrabites.com/2009/08/30/questioning-the-questions/feed/ 4 ZebraBites Questioning the questions