Conducting an Effective Impact Evaluation Interview Here are some tips for conducting an effective Impact Evaluation interview: Listen effectively. Be a good listener; know how to redirect a conversation to get the critical information as quickly as possible. Listen without judging. Prepare. Review the employee’s Adoption and Impact Survey data prior to the interview. Ease on in. Start with easy to respond to questions—just follow the interview protocol. Set the tone. Let the interviewee know how the data will be used and that everything said will be held in strictest confidence. A standard phrase is this: "Mary, I am an external consultant who has been hired by the company to evaluate the effectiveness of the XYZ Training Course. You have been selected from the people who completed the course survey last week. I am interested in how you have applied what you learned in class and what impact it has had on you, your team, or the company. What you tell me will be held in strictest confidence. I may, however, include your story in a summary report without your name. Are you okay to proceed?" Also, express your appreciation for the time that participants have taken to talk to you. Explain how the interview will proceed and then try to follow that format as closely as possible. Use the script. Follow the interview protocol. Listen to your instincts. As you hear things regarding Adoption and Impact, continue to probe until you feel you have uncovered everything. Know what you want. If you’ve set aside thirty minutes for the interview, do your best to stick with that schedule. But be prepared to cut the interview short and jump to the concluding questions. Don’t waste your time or the interviewee’s if you are not talking about Impact data. The most common things that interviewees bring up that side-track you are discussing how well they liked the class, complaining about the company in general, etc. Write it down. Forget about remembering everything that transpires during an interview. You’ll want to take notes so that you can review at a later time. This is especially important if you’re interviewing many people. Probe. As you uncover areas that you feel are promising, investigate further by asking probing questions. Some common probing questions are as follows: Tell me more about that.  When you did that, what was the result?  How did you do that? What did you do specifically?  What was the result?  You mentioned [insert his/her result]; can you place a metric on that result? What is the metric? Create Impact Profiles Impact profiles show in a short, salient manner the profile of successfully adoption that has significant impact.  To create these profiles: Print the interview summaries for your review. Read each one, and sort them into two piles (the ones that are the best examples of Impact as aligned with the Impact Matrix and those that are not as powerful). Use the following items as criteria for placing an interview summary in the Impact Profile stack: Direct impact on and value to the business can be traced to training via the participant’s Intention Goals and Adoptive Behavior. Adoption of training has made a measurable difference. Results validated in the interview support the data from the Impact Survey.Results are in alignment with the predicted Impact identified in the Impact Matrix. The interviews that pass the preceding test become your Impact Profiles—examples of the best results that the training has produced. These validate the Impact data reported in the Impact Survey and are further proof of Impact created.   The Impact Profile is a one-page synopsis of the participant’s story. Review these profiles, and answer these questions: Do the profiles support the answers to the questions you determined from the Impact Survey data analysis? Those questions are the following: What impact has the company received from training? How does that impact compare to predicted impact? Sample Impact Profile Previous Blogs in the Impact Evaluation Series The Impact Hunt Develop Impact Survey Collect Detailed Impact Data from Successfully Adopted Participants Analyze Impact Survey Data Interview Successful and Unsuccessful Participants (part 1 of 2)

You should now have a clear picture as to the impact that the training is providing and whether or not it is as predicted. An issue that exists with the current impact data is that it is self-reported via the Impact Survey and somewhat validated with the Impact Interviews. This is good and, many times, sufficient for you to produce the Impact Dashboard and draw your conclusions on the value of training. If you choose, you can further build your case by collecting data from existing company records. This type of data is known as extant data. Extant data are existing data, information, and observations that have been collected or that are available for use in the assessment and evaluation processes. Basically, you analyze records and files that the company already has, rather than set out to gather additional information. Where and what files you look at depend on the impact being claimed by participants (e.g., sales figures, attendance figures, call-backs for repair, etc.). Customer e-mails, tech support log files, marketing and business plans, business requirements, documents, and vision statements are just a few of the types of information that help you validate the impact. The quickest way to gather extant data is to ask High-Impact interview participants to provide the documentation that proves their case (assuming they have permission to share it with you). This typically comes in the form of presentations, spreadsheets, sales records, financial statements, manufacturing reports, time and motion studies, etc. When you receive the documentation, review it and ensure that you have sufficient information to validate claimed impact. At times, the participant no longer has the backup documentation or it simply does not exist. This does not mean that you do not include the impact in your Impact Dashboard, but you may need to add a disclaimer to the report stating so. Bringing Meaning to Impact Results It is now time to bring meaning to the Impact results that you have collected.  Do this by: Analyzing results and draw conclusions Is the training program contributing to achievement of the business outcomes? Collect training expense data Compare results & ROI to predictions in the Impact Matrix Previous Blogs in the Impact Evaluation Series The Impact Hunt Develop Impact Survey Collect Detailed Impact Data from Successfully Adopted Participants Analyze Impact Survey Data Interview Successful and Unsuccessful Participants (part 1 of 2) Interview Successful and Unsuccessful Participants (part 2 of 2)

Title: Training Transfer: the Secret Sauce for Making it Stick Date: Thursday, September 12, 2013 Time: 2:00 PM - 3:00 PM EDT I believe that learning is continuous and shouldn’t stop when a training session ends. This approach is a total immersion learning experience that emphasizes training transfer, which drives adoption, which drives impact (company value). We need  to think beyond training sessions to ensure the information "sticks" and will be used effectively in the workplace. This interactive conversation demonstrates how to incorporate training and development lessons into "real world" situations. In this complimentary webinar, I will explain how to integrate training into your teams’ work behavior and implement activities and techniques that will increase your company’s training transfer rate to 60-70% or more. Additionally, I will explain how to use innovative techniques throughout the entire process - from pre-course, training, and post-course - to successfully drive training transfer. In this session, participants will learn: What is "training transfer" and why is it important? More than 20 specific actions to do before, during and after training to improve the "stickiness" of their training. How to partner with management for training’s roll-out, so more skills are transferred to the workplace. The importance of planning for training transfer in the earliest set-up phases of your training - and how this increases training’s ROI. How to cultivate a "training transfer mindset and culture" in your company. Title: Training Transfer: the Secret Sauce for Making it Stick Date: Thursday, September 12, 2013 Time: 2:00 PM - 3:00 PM EDT Reserve your seat at: https://www1.gotomeeting.com/register/264748329

An Impact Dashboard summarizes the Impact found.  A typical Impact Dashboard is made up of four pages/sections: (1) payback, results, and key findings; (2) impact by Adoptive Behavior; (3) impact examples and recommendations; and (4) the Impact Profiles. Sample Impact Dashboard - page 1 Sample Impact Dashboard - page 2 Sample Impact Dashboard - page 3 A Different Impact Dashboard Impact Evaluation Summary An Impact Evaluation addresses the following questions: What business results can be traced to the goal adoption of participants? Are results as predicted to the business? What is the typical profile of a participant who has achieved results? What additional business results (value) could be achieved if graduates who have little/no adoption were to adopt their goals? The reason to train should be to improve the company’s bottom line. By measuring the effect on the company profits from participant intentions, adoption, and impact and comparing that with the predicted value, a company can determine its return on investment and can maximize organizational and business results. Previous Blogs in the Impact Evaluation Series The Impact Hunt Develop Impact Survey Collect Detailed Impact Data from Successfully Adopted Participants Analyze Impact Survey Data Interview Successful and Unsuccessful Participants (part 1 of 2) Interview Successful and Unsuccessful Participants (part 2 of 2) Collect Impact Data from Company Records

A training evaluation is successful when the needs of the stakeholders have been met. A stakeholder is anybody directly or indirectly impacted by the evaluation. As a first step, it is important to identify the stakeholders for your evaluation. It is not always easy to identify the stakeholders of an evaluation, particularly those impacted indirectly. Examples of stakeholders for training evaluation are as follows: The Head of the Training and Development organization Key executives from various business units or staff functions (note: these may be represented by a Steering Committee or other such body) Instructional designers, course developers, and instructors (internal or external) External training suppliers (to a lesser extent) Once you understand who the stakeholders are, the next step is to establish their needs. The best way to do this is by conducting stakeholder interviews. The questions that you ask stakeholders can depend on (1) who you are talking to and (2) how they fit into the overall picture. When formulating stakeholder questions, ask yourself the following: What do you want to find out? How many people can you interview? What will you do with the data to help guide the evaluation? What is the underlying reason for asking a specific question? Some questions that you may wish to use with the stakeholders are: What business issue(s) is this training going to help in solving? Why does this issue exist? Why do you want an evaluation? What kind of information would you like to see from the evaluation? What decisions will you make based on this evaluation information? What is your timetable for getting evaluation information? How would you like to see it and receive it? Who else should I talk to? Take time during the interviews to draw out the true needs that create real benefits. Often stakeholders talk about needs that aren’t relevant and don’t deliver benefits. These can be recorded and set as a low priority.  Once you have conducted all the stakeholder interviews and have a comprehensive list of needs—is to prioritize them. From the prioritized list, create a set of questions the evaluation will answer. Previous posts in this series: Getting Started Coming up in this series of posts: Elements in an Evaluation Plan Evaluation deliverables Evaluation schedule Performing the evaluation Training evaluation and continuous improvement Training evaluation organizational readiness As always, I would enjoy hearing your thoughts on this topic.

In addition to the evaluation questions, other elements of the evaluation plan arise from stakeholder interviews and your research into the course. Each element that you should consider as part of the plan is shown below. Course to be evaluated.  List the title and purpose for the course. Provide a history evaluated (business reason) for the course and what it is intended to achieve. For new courses (ones yet to be developed), these data can be found in the needs analysis and/or instructional design documents. For existing courses,use the current training materials. Purpose of the evaluation.  Documents why the evaluation is being undertaken and how the information will be used (comes from stakeholder requirements). Example: Our approach is based on moving away from the typical measurement of activity or level-based measures in favor of focusing on value-driven continuous improvement efforts.This ensures that the training investment creates value and produces the behaviors and results that lead to the desired business outcomes. Questions the evaluation will answer. These are the basic building blocks for the training evaluation. They are the questions that key stakeholders would like to see answered. Examples: Are participant intentions and beliefs upon program completion aligned with desired values? Are participants using (adopting) the skills on the job? Is the training program contributing to achievement of the business outcomes? If yes, to what degree? If no, why not? Key Stakeholders.  These are the individuals who have a vested interest in the evaluation information. Once this is completed, it’s time to move on and look at the evaluation deliverables (next week’s post). Previous posts in this series: Getting Started Questions Evaluation will Answer Coming up in this series of posts: Evaluation deliverables Evaluation schedule Performing the evaluation Training evaluation and continuous improvement Training evaluation organizational readiness As always, I would enjoy hearing your thoughts on this topic.

Evaluation Deliverables Using the evaluation questions and other planning elements you have defined earlier, create a list of information that the evaluation needs to deliver in order to answer the evaluation questions. Specify when and how each item must be delivered. These are the elements of the evaluation. Add the deliverables to the evaluation plan, with an estimated delivery date. More accurate delivery dates can be established during the scheduling phase, which is next. The Evaluation Schedule Create a list of tasks that need to be carried out for each deliverable. For each task, identify the following: The amount of effort (hours or days) required to complete the task The resource that will carry out the task Once you have established the amount of effort for each task, you can work out the effort required for each deliverable and determine an accurate delivery date. Update your deliverables section with the more accurate delivery dates.  A common problem discovered at this point occurs when an evaluation has an imposed delivery deadline from the sponsor or steering committee that is not realistic based on your estimates. If you discover that this is the case, you must contact the sponsor/committee members immediately. The options you have in this situation are the following: Renegotiate the due dates Employ additional resources Reduce the scope of the evaluation Congratulations! Having followed all the preceding steps, you should have a good evaluation plan. Remember to update your plan as the evaluation progresses and measure progress against the plan. Previous posts in this series Getting Started Questions Evaluation will Answer Elements in an Evaluation Plan Coming up in this series of posts: Performing the evaluation Training evaluation and continuous improvement Training evaluation organizational readiness As always, I would enjoy hearing your thoughts on this topic.

I am honored to have an new article just published in In the article I discuss the elements of my Predictive Evaluation model.  Watch this short video about the PE model. To learn more about the model, take a look at my latest book:

So honored that training magazine just published my article on Getting Managers Engaged in Employee Training.  This is one element of my Learning to Performance (L2P) approach.  L2P is a complete training approach that combines learning, leadership and change management competencies to produce documented, sustainable results and value. I hope you enjoy the article and I would love to hear what you have done in this area.

With the evaluation plan in place and approval to proceed, implement the evaluation by using the following sequence. Choose the course you wish to evaluate; it can be a new course design or a course that already exists. Note: this should have been done when you developed the evaluation plan. Review the course to understand fully what it is, what it is supposed to teach, how participants learn the material, what business issues it addresses, who is to attend, how they attend, what pre-course preparations are in place, what post-course support mechanisms exist, who are the sponsors, what sponsors see as the purpose for the course. Form a committee to sponsor the evaluation and provide oversight while the evaluation is under way. Create evaluation protocols, develop the data collection instruments, and create models to analyze collected data for the evaluation.  Developing evaluation procedures involves taking the approved evaluation plan and drafting the techniques, procedures, and instruments necessary to implement the plan. Collect quantitative & qualitative evaluation data as prescribed in the plan and using the data collection instruments.  This phase of the training evaluation of entails gathering data to address the evaluation questions created in the evaluation plan.  The goal is to obtain the right kind at the lowest-cost to answer the evaluation questions. Analyze the evaluation data and draw truthful conclusions and recommendations.  In the analyze phase the meaning of collected data are interpreted.  This phase uses the collected data to determine the conclusions for which the data supports and the amount of support the data supplies for, or against, those conclusions.  Quantitative & qualitative data analysis procedures are employed. Create an Evaluation Report and Executive Dashboard.  The purpose of this phase is to create the vehicles to communicate training evaluation information to stakeholders so that they can make use of the evaluation information for decision-making or purposes of accountability.  This phase will generate an evaluation report and an executive dashboard detailing the results of the evaluation and the recommendations for to maximize the program’s impact. Previous posts in this series: Getting Started Questions Evaluation will Answer Elements in an Evaluation Plan Evaluation Deliverables & Schedule Coming up in this series of posts: Training evaluation and continuous improvement Training evaluation organizational readiness As always, I would enjoy hearing your thoughts on this topic.