I was watching a show the other day about the making of The Beatles album, Sgt. Pepper’s Lonely Hearts Club Band, and one of the things that struck me about the show was the mention of one of the reasons why The Beatles stopped touring: "We couldn’t hear ourselves onstage."  Ringo mentioned he was just hoping he was playing the right notes as he couldn’t hear the music and was trying to play by reading John and Paul’s lips.  The sound amplification technology of the day just wasn’t good enough to allow The Beatles to keep touring. So they stopped touring and took their creative energies to the studio and revolutionized the concept of a rock album exploiting the existing technology of the day to their advantage.  They took a risk and the result was a masterpiece. The Beatles needed an improvement in technology to keep doing what they were doing - touring and sharing their music to millions of fans - but they were not going to create the new technology.  They were artists, not engineers.  With the technology wall square in front of them, they pivoted in a new direction, and created something the world had never heard. When you hit that wall in your company, whether a startup or corporation, which way do you pivot?  Do you become a technology company and try to create the totally new technology, or maintain your purpose and core skill set to exploit existing technology?  Either answer could be correct (the skill set of your team, available resources, etc. all factor in), but you need to choose wisely. One can only wonder what would have happened if, in late summer 1966, there was a sudden breakthrough in sound amplification technology.  And we can be thankful there wasn’t.    

Thank you Matthew Weiner. Not only for providing 7 seasons of masterful television, but for ending it in a way that leaves the viewer open to creating their own endings, and molding the storylines the way they see fit.  No tidy box with a ribbon on it, with no assembly required; but here are the pieces, there’s the direction we’re heading, put it together so it makes sense for you.  Brilliant! We need more of this in our lives.  Producers crafting stories and letting us think and use our brains to write the story the way we see it, given the parameters set before us.  Too often, in our business world (and when it comes to my passion, Lean), we want it all handed to us in a nice tidy box with a bow.  We want to find out what someone else did, write it down, put it in a box, and then put that box on our shelf.  Guess what.  It doesn’t look right, fit, etc.  So we ignore it, or discard it as something else that "didn’t work here." Admittedly, sometimes it’s nice not to have to think.  To just have someone give us the answer and we don’t have to do anything.  Much of television has been like that for years - a way to escape and not have to think.  The problem is too much of business is becoming that way.  In our quest and zeal for speed and efficiency, we take the easy way out.  We don’t try to fit the methodology and tools to our circumstance, we just want the answer.  No thinking required. Weiner did something that has made many people uncomfortable, and not often done on television - he left it open.  They wanted it "buttoned up" nice and tidy with no loose ends, so they can put it away and not have to think about it anymore.  But that’s not what they got.  They got something that’s a little loose around the edges, needs some pieces put together, and with a little creative thinking they have to, no… get to, wrap it all up We are all writing our story every day.  The story of our personal lives and the story of our work.  Are you just going to take the answer given, or are you going to use what was given and mold it into your own?

The other day, I met a young entrepreneur who is just starting his company with a product that he has a great passion for and thinks will perform very well in the market (due to confidentiality, I won’t mention its name).  He was telling me all about how great it was, how he already had a sales channel set up through a friend, so on and so on…. I listened very enthusiastically, then asked a couple of questions: Have you spoken with any potential customers? Do you understand the issues they have? What problem are you trying to solve? He just looked at me.  Silence.  Then, "No, but that’s a good idea."  I suggested he meet with a few potential customers, get some feedback and see where that takes him. It doesn’t matter if you are a startup, a small company or a Fortune 500 organization - if you are launching a new product or service, you need to make sure you understand what customer problem you are trying to solve. You don’t have to necessarily do an intense amount of customer research or surveys, in fact, doing too much of this could restrict your ability to innovate as customers do not always know what they want; they just know they have a problem. Your objective is not to find out what the customers want, but to understand the customer’s problem.  Your job is then to solve that problem with a product or service that provides a useful value proposition to the customer. It all starts with understanding the problem.  Do you understand your customer’s problem? Until next time…

Kirk Knestis, CEO of Hezel Associates, back again, following up on a previous post about how evaluators’ work in STEM education settings is being influenced by the Common Guidelines for Education Research and Development introduced by the National Science Foundation (NSF) and U.S. Department of Education. Hezel Associates studies education innovations so regularly supports organizations proposing grant-funded R&D projects in science, technology, engineering, and math education (STEM). Sometimes we’re a research partner (typically providing Design and Development Research, Type #3 in the Guidelines); while in other cases we serve as an external evaluator (more accurately, "program evaluator") assessing the implementation and impact of proposed project activities, including the research. Lessons Learned - Work with a wide variety of clients (more than 70 proposals so far in 2014!) has left me convinced that an evaluator—or research partner, if your job is framed that way—can do a few specific things that can add substantial value to development of a client’s proposal. Someone in an external evaluator/researcher role can do more than simply "write the evaluation section," potentially improving the likelihood for proposal success. Hot Tips - 1. Help designers explicate the theory of action of their innovation (intervention, program, technology, etc.) being tested and developed. Any research study aligned with the Guidelines (for example, many if not most NSF projects) will be expected to build on a clearly defined theoretical basis. Evaluators ought to be well equipped to facilitate development of a logic model to serve that purpose, illustrating connections between elements or features of the innovation and its intended outcomes. Define the appropriate "type" of research . The Common Guidelines provide a typology of six purposes for research, ranging from Foundational Research contributing to basic understandings of teaching and learning; to Scale-up Research, examining if the innovation retains its effectiveness for a variety of stakeholders, when implemented in different settings "out in the wild" without substantial developer support. A skilled evaluator can help the client select the appropriate kind of research given the level of maturity of the innovation and other factors. Help clarify distinctions between "research" and "evaluation" purposes, roles, and functions. Clarity on the type of research required will inform study design, data-collection, analysis, and reporting decisions. A good evaluator should be able to help determine the expertise required for the research, requirements for external evaluation of that work, and the narrative explaining roles, responsibilities, and work plans required for a proposal. Rad Resource - If you work with education clients, become familiar with the Common Guidelines for Education Research and Development. Some complex conversations loom but they will be an important consideration in conversations about research and evaluation in education in the coming years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. Related posts: Kirk Knestis on Innovation Research and Development (R&D) vs. Program Evaluation Leslie Goodyear on Serving as a Reviewer for the National Science Foundation Lily Zandniapour and Nicole Vicinanza on The Social Innovation Fund Evaluation Plan Guidance Document: A Tool for Building Shared Understanding of Rigorous Impact Evaluation Designs

Hello fellow evaluators! I’m Sheila B Robinson, aea365’s Lead Curator and sometimes Saturday contributor, and I have to admit, I get lonely sometimes. After all, I’m the only program evaluator in my organization. Sure, there are other people who collect and analyze data, but no one who can sit down with me over lunch and discuss logic models, debate the merits of using a goal-based or goal-free  approach, prattle on about program theory, or compare favorite theorists on the Evaluation Theory Tree. Where’s the eHarmony or Match.com for evaluators? Thankfully, I have several options for going virtual to enjoy some good evaluation camaraderie. Strictly platonic, of course. Rad Resources: EvalTalk is the discussion list of the American Evaluation Association. It’s a listserv that has been going since 1995! There are many active members and many, many more readers. Discussions can get quite heavy and theoretical at times, and many contributors write lengthy responses to questions engaging in spirited debates. On the other hand, many people use the group to pose simpler questions, such as requests for recommendations of instruments, products or services. AEA’s LinkedIn group also hosts a number of interesting discussions on various evaluation-related topics. And while you’re on LinkedIn, look for other groups as well. I belong to a number of additional evaluation-related groups: The Evaluators’ Institute, The European Evaluation Society, Monitoring and Evaluation Professionals, Evaluators Group, RealWorld Evaluation, and Research, Methodologies, and Statistics in the Social Sciences. Some AEA Topical Interest Groups (TIGS) also have LinkedIn groups. And of course, some group discussions are more active than others. All of these discussion groups have featured conversations around topics such as systems thinking, definitions of terms (e.g. outputs, outcomes, indicators, metrics, measures, etc.), how to deal with different types of data (e.g. Likert scales), statistical analysis software, RFPs, research design, capacity building, evaluation approaches, job openings, and much, much more. Don’t forget to look for AEA, AEA TIGs, and AEA Affiliates on Facebook, and follow them on Twitter for even more evaluation conversation! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. Related posts: Susan Kistler on Getting Engaged in Environmental Evaluation Susan Kistler on Linking in to LinkedIn Poster Week: Anna Douglas on Using Asynchronous Discussion Groups for Evaluation

My name is Bonnie Richards, an analyst from Foresee and Chair of the Organizational Learning and Evaluation Capacity Building TIG. Welcome to the OL-ECB sponsored AEA365 week! This week our blog posts will cover a range of experiences discussing challenges and successes we have had sustaining learning or evaluation in our work with organizations or programs. Across our members’ varied experiences, you will learn more about their strategies and methods for facilitating learning and the challenges they have encountered. In my own role working with clients, one of my main goals is to help them understand where to prioritize improvements for their stakeholders. One of the challenges in doing this is navigating the different environments of organizations, companies, and government agencies. Each group is unique. For example, among government agencies, while there are some similar requirements or processes that consistently govern each, the mix of involved stakeholders who serve as the primary point of contact actually vary significantly. A primary contact could be a program analyst, or a director of the agency’s strategic planning and evaluation office, or technical director, or even a third party vendor. Understanding and acclimating to each client, meeting them at their "level" and working within their context is key because it helps you to learn the best ways for interacting with different stakeholder groups. This sets the stage for a successful relationship. Lessons learned: Ask questions. So, how does one get to the point of successfully meeting stakeholders in the appropriate context? Ask questions: Why are they beginning this process? Were they instrumental in initiating it, or are they tasked it as part of a directive from a director or committee? How do they intend to use the information? What are their goals? What information will be most useful? Take some time to ask questions. Stakeholders will appreciate your interest and the opportunity, and it exposes you to the thoughts, concerns, and values that are top of mind to the people you will be working closely with. The American Evaluation Association is celebrating Organizational Learning and Evaluation Capacity Building (OL-ECB) Topical Interest Group Week. The contributions all this week to aea365 come from our OL-ECB TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. Related posts: Michelle Baron on Gaining Support for Evaluations in the Military GOVT Week: David Bernstein on Top 10 Indicators of Performance Measurement Quality GOV TIG Week: Caroline DeWitt on Valuing Stakeholders in the Evaluation Process

Hi, I’m Joe Bauer, the Director of Survey Research & Evaluation in the Statistics & Evaluation Center (SEC) at the American Cancer Society (ACS) in Atlanta, Georgia. I have been working as an internal evaluator at the ACS for almost nine years, in a very challenging, but very rewarding position. Lesson Learned: Evaluation is always political and you must be aware of those cultural dynamics that are part of every environment. I came to the American Cancer Society to have an impact at a national level. I had envisioned evaluation (and still do) as a means to systematically improve programs to improve the lives of cancer patients. In the beginning, many were not ‘believers’ in evaluation. The perception was that evaluation could only lead to finding things that were wrong or that were not working - and that this might lead to politically problematic situations. We needed to navigate the cultural mine fields, even as we were acting as change agents. Over time, our Center worked hard to build a sense of trust. As internal evaluators, one must always be aware that we are being judged, as to how nice you are playing in the sandbox, even as we strive and push for higher quality, better data, and better study designs. Evaluators ask the tough questions - which at times cause ‘friction’. However, an internal evaluator must have a comfort level and the confidence with taking that role of asking the tough questions, which can be lonely. Hot Tips: As an internal evaluator, one must be willing to ‘stay the course’ and ‘weather the storms’ and to never compromise on your values. This is crucially important - because you always need to do the right thing. This does not mean you end up winning all these ‘battles’, because ultimately, you can and are over-ruled on many issues. However, you must keep your integrity - because that is something you need to own throughout your career. That is also what builds trust and credibility. Rad Resources: The American Evaluation Association’s Guiding Principles for Evaluators http://www.eval.org/p/cm/ld/fid=51 - which are intended to guide the professional practice for evaluators and inform evaluation clients and the general public about the principles they can expect to be upheld by professional evaluators. The Official Dilbert Website with Scott Adams http://www.dilbert.com/ - where there are many ‘real world’ examples of the cultural dynamics that occur in the world of work and the often absurd scenarios and dynamics that play themselves out. As an evaluator - you will not only need to have a good skill set and work hard at keeping your values and integrity - you will need to have a sense of humor and keep your perspective. The American Evaluation Association is celebrating Organizational Learning and Evaluation Capacity Building (OL-ECB) Topical Interest Group Week. The contributions all this week to aea365 come from our OL-ECB TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. Related posts: Internal Evaluation Week: Debbie Cohen on Working with External Evaluators Susan Kistler on Humor and Cartoons for Evaluators Internal Eval Week: Kathleen Norris on the Internal Evaluation Boa

My name is Lisa Richardson and I am the internal Improvement Advisor/Evaluator for the UCLA-Duke University National Center for Child Traumatic Stress (NCCTS), which in addition to coordinating the collaborative activities of the National Child Traumatic Stress Network (NCTSN), provides leadership in many aspects of child trauma policy, practice, and training. Online surveys are a […] Related posts: Jessica Foster on Maximizing Survey Response Rates Scribing: Summer Jackson on Insights Into Foundation Evaluation IE Week: Hilary Loeb and Kelly Bay on Enhancing the Quality and Quantity of Survey Responses at Program Events

Hi, we are Tom Archibald (Assistant Professor and Extension Specialist, Department of Agricultural, Leadership, and Community Education at Virginia Tech) and Guy Sharrock (Senior Technical Advisor for Learning with Catholic Relief Services). We believe one way to integrate and sustain learning in an organization is by intentionally promoting "evaluative thinking." Evaluative thinking (ET) is an […] Related posts: Jeff Sheldon on the Readiness for Organizational Learning and Evaluation instrument Tom Archibald and Jane Buckley on Evaluative Thinking: The ‘Je Ne Sais Quoi’ of Evaluation Capacity Building and Evaluation Practice OL-ECB Week: Jeff Sheldon on using ROLE to Determine an Organization’s Support of Evaluative Inquiry

Hello! My name is Michele Tarsilla, an independent evaluation advisor and evaluation capacity development (ECD) specialist with experience in over thirty countries. I have served as Chair of the International and Cross-Cultural Evaluation TIG at AEA and am currently in transition to become the new OL-ECB TIG Co-Chair. The idea for this blog sprang from the […] Related posts: PD Presenters: Michele Tarsilla on a new Framework for Evaluation Capacity Development Shar McLean and Esteban Colon on Tools for Organizational Assessment CPE Week: José M. Díaz-Puente on the Empowerment Evaluator’s Role