How
to define the goals of the research
Many people begin a survey by
writing the questionnaire items. This is
wrong. Don't do it. If you begin by writing
the questionnaire items you'll lose focus of
what's important. Your survey will become a
fishing expedition with a lot of tributaries
leading nowhere.
Well-designed surveys always
begin by committing your research questions to
writing. Research questions are not the
same as the questionnaire items. Research
questions are global in nature. They are the
goals and objectives of the study. Questionnaire
items, on the other hand, are designed to help
answer the global research questions.
Defining the goals and
objectives of a research project is one of the
most important steps in the research process. Do
not underestimate the importance of this step.
Clearly stated goals keep a research project
focused. The process of goal definition begins by
writing down the broad and general goals
of the study. As the process continues, the goals
become more clearly defined and the research
issues are narrowed.
Here are a few examples of how
you might phrase your goals and objectives:
I want to discover if customers are satisfied
with ...(service, quality, etc.)
I want to learn how employees feel about
...(policy, pay, hours, etc.)
I want to know if there are relationships between
x and y.
I want to understand the differences between x
and y.
How to write research questions
The goals of the study are
easily transformed into research questions. Once
again, research questions are global and broad,
and they are not the same as the questionnaire
items. There are basically two kinds of research
questions: testable and non-testable. Neither is
better than the other, and both have a place in
customer satisfaction and employee surveys.
"Non-testable" means that the
research question cannot be answered by
performing a statistical test. The answers to
these questions might be important to know, but
the decision making criteria does not involve a
statistical test.
Examples of non-testable
research questions are:
What do customers feel is fair price for the new
product?
How do customers feel about our service?
How do customers feel about the quality of our
products?
What are employee's attitudes towards the new
management?
What changes would improve employee productivity?
Respondents' answers to these
questions can be summarized in descriptive tables
and the results might be extremely valuable to
administrators and planners. Business and social
science researchers often ask non-testable
research questions. The shortcoming with these
types of questions is that they do not provide
objective cut-off points for decision-makers.
For example, imagine that we've
done our survey, and now we need to decide what
constitutes satisfactory service? Each of us
might give a different answer. There is no exact
cutoff point where we would say "yes"
our customers are satisfied, or "no"
they are not. When we ask questions like this,
it's important to establish a decision making
guideline before doing the survey.
How
to write testable research questions
It is perhaps more important to
ask questions that involve decision making
criteria. Business research usually seeks to
answer one or more testable research questions.
Nearly all testable research questions begin with
one of the following two phrases:
Is there a significant
difference between ...?
Is there a significant relationship between
...?
Examples of testable research
questions are:
Is there a significant
relationship between a customer's age
and their level of satisfaction
with the service?
Is there a significant
difference between the level of male and
female satisfaction with the
service?
Is there a significant
relationship between managerial level
and support of the new budget?
Is there a significant
difference between the productivity of
workers in plant A and workers
in plant B?
How to convert research
questions into hypotheses and null hypotheses
A research hypothesis is a
testable statement of opinion. It is created from
the research question by replacing the words
"Is there" with the words "There
is", and also replacing the question mark
with a period. The hypotheses for the four sample
research questions would be:
There is a significant
relationship between a customer's age and
their level of satisfaction
with the service?
There is a significant
difference between the level of male and
female satisfaction with the
service?
There is a significant
relationship between managerial level and
support of the new budget?
There is a significant
difference between the productivity of
plant A and plant B?
It is not possible to test a
hypothesis directly. Instead, you must turn the
hypothesis into a null hypothesis. The null
hypothesis is created from the hypothesis by
adding the words "no" or
"not" to the statement. For example,
the null hypotheses for the two examples would
be:
There is no significant
relationship between a customer's age and
their level of satisfaction with the
service?
There is no significant
difference between the level of male and
female satisfaction with the
service?
There is no significant
relationship between managerial level and
support of the new budget?
There is no significant
difference between the productivity of
plant A and plant B?
All statistical testing is done
on the null hypothesis...never the hypothesis.
The result of a statistical test will enable you
to either 1) reject the null hypothesis, or 2)
fail to reject the null hypothesis. Never use the
words "accept the null hypothesis".
When you say that you "reject the null
hypothesis", it means that you are
reasonably certain that the null hypothesis is
wrong. When you say that you "fail to reject
the null hypothesis", it means that you do
not have enough evidence to claim that the null
hypothesis is wrong.
How to make sure your survey
is valid
Pointers on how to write questionnaire items
There are many different ways
to design the questions for a survey. The problem
is how can we check the validity of the survey?
How can we determine if a survey is actually
measuring what it's supposed to measure?
There are no statistical
tests for validity. When a survey is
"validated" it means that the
researcher has come to the opinion that
the survey is measuring what it was designed to
measure, or the researcher has received a
statement from another researcher indicating that
they believe the instrument is measuring what it
was designed to measure. Validity is an opinion;
nothing more.
Here is a quote from our
statistics book
Survival Statistics
"Validity refers to
the accuracy or truthfulness of a
measurement. Are we measuring what we think
we are? This is a simple concept, but in
reality, it is extremely difficult to
determine if a measure is valid. Generally,
validity is based solely on the judgment of
the researcher. When an instrument is
developed, each question is scrutinized and
modified until the researcher is satisfied
that it is an accurate measure of the desired
construct, and that there is adequate
coverage of each area to be
investigated."
Most statistics textbooks will
tell you that the way to "debug" a
questionnaire is to send the questionnaire to a
sample of about 30 potential respondents and
evaluate their responses for potential problems.
We have found that this rarely yields useful information.
Instead, we recommend the
following method to get the kinks out of your
survey. It's fast, costs nothing, and you'll get
immediate and valuable feedback that can be used
to improve your instrument.
- Find someone who will act
as a respondent. They do not need to be
someone from the actual pool of potential
respondents. You can ask a spouse or
friend to "pretend" they are
from the target population. Do not use
someone who helped create the survey.
- Give them a
"final" copy of the survey and
say something like, "Please complete
this survey as if you were a real
respondent. You can just make up the
answers. Feel free to ask me any
questions while you're completing
it". Then give them the survey and
sit there quietly while they take the
survey. The survey you give them should
be a "final" copy... exactly
the way it will appear on the paper when
it is printed. If it's an internet survey, have them take it on the internet. If you use this method
while the survey is in "draft"
form, do it again after the survey is in
final form.
- Any question they ask you
about the survey indicates a defective
item. Real respondents will not have an
opportunity to ask questions, so you must
fix these items now. Modify all items
that were mentioned. Then begin the
process again with a new respondent, and
continue until there are no questions.
Usually, you'll be done after two or
three "pretend respondents".
How to test the reliability
of a survey
Reliability is synonymous with
repeatability. A measurement that yields
consistent results over time is said to be
reliable. When a measurement is prone to random
error, it lacks reliability. The reliability of
an instrument places an upper limit on its
validity. A measurement that lacks reliability
will also lack validity. There are three basic
methods to test reliability: test-retest,
equivalent form, and internal consistency
A test-retest measure of
reliability can be obtained by administering the
same instrument to the same group of people at
two different points in time. The degree to which
both administrations are in agreement is a
measure of the reliability of the instrument.
This technique for assessing reliability suffers
two possible drawbacks. First, a person may have
changed between the first and second measurement.
Second, the initial administration of an
instrument might in itself induce a person to
answer differently on the second administration.
The second method of
determining reliability is called the
equivalent-form technique. The researcher creates
two different instruments designed to measure
identical constructs. The degree of correlation
between the instruments is a measure of
equivalent-form reliability. The difficulty in
using this method is that it may be very
difficult (and/or prohibitively expensive) to
create a totally equivalent instrument.
The most popular methods of
estimating reliability use measures of internal
consistency. When an instrument includes a series
of questions designed to examine the same
construct, the questions can be arbitrarily split
into two groups. The correlation between the two
subsets of questions is called the split-half
reliability. The problem is that this measure of
reliability changes depending on how the
questions are split. A better statistic, known as
Cronbach's alpha, is based on the mean (absolute
value) interitem correlation for all possible
variable pairs. It provides a conservative
estimate of reliability, and generally represents
the lower bound to the reliability of a scale of
items. For dichotomous nominal data, the KR-20
(Kuder-Richardson) is used instead of Cronbach's
alpha.
|
