Made for Each Other

[akirlu]

So, our town is sending out a sequence of surveys to citizens gathering opinions on how to improve downtown. Swell. I'm for it. I have opinions. Let me at it.

The most recent survey is here.

Hal & I both got mailed links to the survey. And you know what our totally independent, nearly identical first response was? To e-mail the city back with variants (more politely worded) of "EEEK! ARGH! Horrible, Counterintuitive, Bad, Bad survey design!!!" Well, I mean, come on! First they tell people to rank items in terms of importance, and then they say we should use '10' to indicate most important and '1' to indicate least important? Who works that way? I was halfway through the survey before I re-read it and had to start over. Hal, apparently, had exactly the same experience. We're both betting lots of people will never notice and will use '1' for most important, and so on, and the data will be useless because you can't tell if they read the survey instructions correctly. It's obvious.

But what really amuses me is how much Hal & I think just alike on this. We really were made for each other, the marriage of true minds.

Comments

[billeyler.livejournal.com]

Idjits, obviously. :-)

[cschells.livejournal.com]

I had a similar thought last night when I realized that even though we read completely different news sources online, J and I often manage to read the same articles eventually... I was wondering if you could predict marital compatibility by comparing browsing histories...

[vgqn.livejournal.com]

I agree it's confusing. There are plenty of examples where a higher number is better, but they are usually ratings, not rankings: Olympics and other sports ratings, the movie 10, test scores. Some top 10 lists even -- not everyone does the countdown backwards to 1.

But I do agree that it's not obvious, and whichever way you design a survey, you'll have some respondents do the opposite. What I do is put the words 'Best' and 'Worst' at the ends of the scale rather than trusting people to remember. And come to think of it, I do run mine (mainly tasting surveys for MGs, which are ratings rather than rankings) as 1-worst to 5-best. I also include a line for comments on each item, so I can often tell if someone has reversed the scale, in which case I reverse their ratings (saying 'Yum!" about a 1 and "Hate it" about a 5 is pretty incontrovertible).

[cakmpls.livejournal.com]

Yes--I think adding word cues is a good idea.

[cakmpls.livejournal.com]

I am often taken aback by a survey asking me to rank 1 as best and 10 as worst. That isn't intuitive to me.