|
There are good and bad questions. The qualities of a
good question are as follows:
1. Evokes the truth. Questions must be
non-threatening. When a respondent is concerned about the
consequences of answering a question in a particular
manner, there is a good possibility that the answer will
not be truthful. Anonymous questionnaires that contain no
identifying information are more likely to produce honest
responses than those identifying the respondent. If your
questionnaire does contain sensitive items, be sure to
clearly state your policy on confidentiality.
2. Asks for an answer on only one dimension. The
purpose of a survey is to find out information. A
question that asks for a response on more than one
dimension will not provide the information you are
seeking. For example, a researcher investigating a new
food snack asks "Do you like the texture and flavor
of the snack?" If a respondent answers
"no", then the researcher will not know if the
respondent dislikes the texture or the flavor, or both.
Another questionnaire asks, "Were you satisfied with
the quality of our food and service?" Again, if the
respondent answers "no", there is no way to
know whether the quality of the food, service, or both
were unsatisfactory. A good question asks for only one
"bit" of information.
3. Can accommodate all possible answers. Multiple
choice items are the most popular type of survey
questions because they are generally the easiest for a
respondent to answer and the easiest to analyze. Asking a
question that does not accommodate all possible responses
can confuse and frustrate the respondent. For example,
consider the question:
What brand of computer do
you own? __
A. IBM PC
B. Apple
Clearly, there are many problems with this question.
What if the respondent doesn't own a microcomputer? What
if he owns a different brand of computer? What if he owns
both an IBM PC and an Apple? There are two ways to
correct this kind of problem.
The first way is to make each response a separate
dichotomous item on the questionnaire. For example:
Do you own an IBM PC?
(circle: Yes or No)
Do you own an Apple
computer? (circle: Yes or No)
Another way to correct the problem is to add the
necessary response categories and allow multiple
responses. This is the preferable method because it
provides more information than the previous method.
What brand of computer do
you own?
(Check all that apply)
__ Do not own a
computer
__ IBM PC
__ Apple
__ Other
4. Has mutually exclusive options. A good question
leaves no ambiguity in the mind of the respondent. There
should be only one correct or appropriate choice for the
respondent to make. An obvious example is:
Where did you grow up? __
A. country
B. farm
C. city
A person who grew up on a farm in the country would
not know whether to select choice A or B. This question
would not provide meaningful information. Worse than
that, it could frustrate the respondent and the
questionnaire might find its way to the trash.
5. Produces variability of responses. When a question
produces no variability in responses, we are left with
considerable uncertainty about why we asked the question
and what we learned from the information. If a question
does not produce variability in responses, it will not be
possible to perform any statistical analyses on the item.
For example:
What do you think about
this report? __
A. It's the worst
report I've read
B. It's somewhere between the worst and best
C. It's the best report I've read
Since almost all responses would be choice B, very
little information is learned. Design your questions so
they are sensitive to differences between respondents. As
another example:
Are you against drug
abuse? (circle: Yes or No)
Again, there would be very little variability in
responses and we'd be left wondering why we asked the
question in the first place.
6. Follows comfortably from the previous question.
Writing a questionnaire is similar to writing anything
else. Transitions between questions should be smooth.
Grouping questions that are similar will make the
questionnaire easier to complete, and the respondent will
feel more comfortable. Questionnaires that jump from one
unrelated topic to another feel disjointed and are not
likely to produce high response rates.
7. Does not presuppose a certain state of affairs.
Among the most subtle mistakes in questionnaire design
are questions that make an unwarranted assumption. An
example of this type of mistake is:
Are you satisfied with
your current auto insurance? (Yes or No)
This question will present a problem for someone who
does not currently have auto insurance. Write your
questions so they apply to everyone. This often means
simply adding an additional response category.
Are you satisfied with
your current auto insurance?
___ Yes
___ No
___ Don't have auto insurance
One of the most common mistaken assumptions is that
the respondent knows the correct answer to the question.
Industry surveys often contain very specific questions
that the respondent may not know the answer to. For
example:
What percent of your
budget do you spend on direct mail advertising? ____
Very few people would know the answer to this question
without looking it up, and very few respondents will take
the time and effort to look it up. If you ask a question
similar to this, it is important to understand that the
responses are rough estimates and there is a strong
likelihood of error.
It is important to look at each question and decide if
all respondents will be able to answer it. Be careful not
to assume anything. For example, the following question
assumes the respondent knows what Proposition 13 is
about.
Are you in favor of
Proposition 13 ?
___ Yes
___ No
___ Undecided
If there is any possibility that the respondent may
not know the answer to your question, include a
"don't know" response category.
8. Does not imply a desired answer. The wording of a
question is extremely important. We are striving for
objectivity in our surveys and, therefore, must be
careful not to lead the respondent into giving the answer
we would like to receive. Leading questions are usually
easily spotted because they use negative phraseology. As
examples:
Wouldn't you like to
receive our free brochure?
Don't you think the
Congress is spending too much money?
9. Does not use emotionally loaded or vaguely defined
words. This is one of the areas overlooked by both
beginners and experienced researchers. Quantifying
adjectives (e.g., most, least, majority) are frequently
used in questions. It is important to understand that
these adjectives mean different things to different
people.
10. Does not use unfamiliar words or abbreviations.
Remember who your audience is and write your
questionnaire for them. Do not use uncommon words or
compound sentences. Write short sentences. Abbreviations
are okay if you are absolutely certain that every single
respondent will understand their meanings. If there is
any doubt at all, do not use the abbreviation. The
following question might be okay if all the respondents
are accountants, but it would not be a good question for
the general public.
What was your AGI last
year? ______
11. Is not dependent on responses to previous
questions. Branching in written questionnaires should be
avoided. While branching can be used as an effective
probing technique in telephone and face-to-face
interviews, it should not be used in written
questionnaires because it sometimes confuses respondents.
An example of branching is:
1. Do you currently have
a life insurance policy ? (Yes or No) If no, go to
question 3
2. How much is your
annual life insurance premium ? _________
These questions could easily be rewritten as one
question that applies to everyone:
1. How much did you spend
last year for life insurance ? ______ (write 0 if
none)
12. Does not ask the respondent to order or rank a
series of more than five items. Questions asking
respondents to rank items by importance should be
avoided. This becomes increasingly difficult as the
number of items increases, and the answers become less
reliable. This becomes especially problematic when asking
respondents to assign a percentage to a series of items.
In order to successfully complete this task, the
respondent must mentally continue to re-adjust his
answers until they total one hundred percent. Limiting
the number of items to five will make it easier for the
respondent to answer.
|
