National Media Forum on Highway Safety
Evaluation: Ho Do You Know You've Been Successful?

INTRODUCTION

“So what?”
The question hangs in the air following implementation of each and every element within a communications plan. The plan only articulates your intent – not your successes and lessons to be learned for the future. The “So what?” question demands a response. Making a difference requires an understanding of what has – and has not – worked. This workshop focuses on:

• Ways and means you can qualitatively and quantitatively measure the success of your media plan by going beyond relying on “reach,” “frequency,” and “impression”;
  
  
• Cost-effective means you can use to evaluate the effectiveness of your communication
  efforts;
  
  
• Challenges of judging success on the basis of behavior change among your target market, and;
  
  
• Reporting your success and publicizing it.
  

KEY QUESTIONS TO CONSIDER

To make this session more relevant, consider the following key questions:

• What resources will you allocate to evaluation in your communications plan to ensure you’re accomplishing what you set out to accomplish, and to revise your plan to build on your successes and lessons learned?
  
  
• Does your communications plan have well-stated, specific and measurable process and
  outcomes goals?
  
  
• What steps, if any, have you taken to build evaluation into the program phase and message development phase of your communication plan?
  
  
• Have you developed and tested the evaluation tool to ensure that it measures what you want it to measure?

back to top

HOW TO MEASURE SUCCESS

INTRODUCTION

The first rule for measuring success is to define (up front) what success is. In other words, have one or more specific campaign goals. Each goal should be measurable and clearly defined.

Too often, an organization sets vague goals. Consequently, they are unable to determine whether the
communications intervention achieved or changed anything.

Different indicators of progress can be assessed to measure changing public or target audience perceptions or behaviors. These indicators include: Reach or Awareness, Knowledge, Understanding, Preference, Attitude, Opinion, and Behavior.

ISSUES

A well-stated, specific goal almost always includes some form of statistical or quantifiable factor on one or more of the indicators listed above. If a benchmark statistic for that factor already exists, then the goal should note what amount of change will occur over a specified time.

At an appropriate point in time during and/or after completion of the communications program, quantitative or qualitative research is conducted to determine what amount of change in the indicator has occurred.

Five of the more commonly used research techniques for these types of measurement are:

• Telephone surveys/interviews
  
  
• Mail surveys
  
  
• E-mail surveys
  
  
• Observation studies
  
  
• Quantifiable numbers or statistics (i.e., fatality rates)

Which research technique you use – and how many people comprise your sample of participants – depends upon a number of factors, including:

• How much money is available for the research?
  
  
• How quickly does the project need to happen (from pre-planning to gathering data to analysis and reporting)?
  
  
• How wide a geographical area do you need to cover?
  
  
• How much data do you want to collect from each person in the sample?
  
  
• Are there limitations that would exclude any particular technique? (For example, unavailability of a good e-mail list of prospective respondents; or measurement of a behavior that can’t be easily observed)

Often it is helpful and important to test the evaluation tool (for example, the questionnaire) to ensure
that respondents understand the questions you’re asking. You may be getting the results you want from your communication intervention, but your evaluation tools may be so poor that you’ll never know what is happening.

RESOURCES

Unless your measurement needs are relatively simple, concise, and straightforward, you’ll want to enlist the assistance of a marketing research specialist (either from within your department, from another department, or from an outside firm). The experience and data tabulation/reporting systems the specialist offers will greatly enhance the usefulness, statistical analyses and overall credibility of the research. Plus it will save you a significant amount of time.

Competent marketing research firms are located in all major and most secondary cities. Also, consider checking with the Marketing Research Association (www.bluebook.org) for sources. Soliciting proposals and cost quotes from two, three, or four firms is worthwhile, for quotes may vary considerably. Good firms often offer useful advice or suggestions that will improve the outcome of your study.

back to top

WHEN CAN YOU ASSESS BEHAVIOR CHANGE OR EFFECTIVENESS?

INTRODUCTION

The appropriate time to measure the effectiveness of a communications program or activity depends on the nature of it. For example: Is the nature of the program/activity one that is on-going, but relatively “quiet” without high-profile, high-visibility elements? Or is it short-term, such as a mobilization or crackdown? Or is it a program to be implemented and assessed within a fixed period of time, such as summertime travel?

IF THE COMMUNICATIONS PROGRAM/ACTIVITY IS OF AN ON-GOING NATURE
These types of programs can be meaningfully assessed during intervals that range from once a quarter to once a year (depending on what the goal is and what time/financial resources are available for assessment). If the program is ongoing, then the assessments take the form of tracking studies – studies that monitor change and progress over time. As such, it’s best to be very consistent in the measurement and research variables and techniques, so valid comparisons can be made from one time period to the next.

IF THE COMMUNICATIONS PROGRAM/ACTIVITY HAS DEFINED TIME FRAMES FOR IMPLEMENTATION
With defined starting and ending times, assessment has two key stages: (1) up-front or advance benchmarking; and (2) measuring the end results, once the program/activity has been implemented. Logically, benchmark measures are established before implementation of the program. If none is readily available, the benchmark may be one based on educated assumptions or a regional or national benchmark reported from other sources (such as NHTSA). However, it is preferable to establish an objective way of determining the benchmark.

Then, during the last day(s) – or immediately after the conclusion – of the program, measurement is done to get data to compare with the benchmark.

In certain cases, subsequent tracking measures may need to be done, especially if you want to learn if behavior has changed even more or has been sustained (even though the program/activity is no longer underway). This kind of tracking might be done one to three months following the conclusion of the campaign.

PUTTING THE EVALUATION FINDINGS TO WORK
Collecting data and writing reports of findings are important, but the real pay-off of the evaluation is using the findings to improve and enhance future activities, strategies and tactics. What those improvements are depends on the type and issues of evaluation. Here are a few examples of how findings can be put to work:

• Refine the media buy in terms of timing, media mix, dayparts, programs, stations, publications, etc.
  
  
• Refine the creative in terms of images shown, words or phrases used, talent, ad sizes/lengths, primary messages, etc.
  
  
• Add, refine, or stop using support materials such as posters, brochures, banners
  
  
• Reallocate expenditures among various elements of the communications program
  

back to top