If you attended the 2015 SHRM Annual Conference & Exposition in June (or followed our SHRM bloggers), you probably heard a lot of buzz about leadership, coaching and developing great teams—especially from keynote speakers Mike Krzyzewski and Marcus Buckingham. While "Coach K" challenged HR to make every employee feel important and to implement guiding standards over rules, Marcus Buckingham emphasized the critical importance of the team leader in organizations. HR—Focus on Your Teams Traditional management models have always focused on the top-tier leaders. Progressive organizations in the new world of work, however, are directing their energy, attention...

Greetings! I’m Sheila B Robinson, aea365’s Lead Curator and sometimes Saturday contributor. Recently, Sara Vaca shared her 3 most remarkable evaluation moments and challenged the rest of us to share ours, so here are my 3 moments! Life isn’t a matter of milestones, but of moments. - Rose Kennedy 1. Program evalu-what? My doctoral work in Educational Leadership was focused on Professional Development. One day, a classmate mentioned something she learned in her "program evaluation" course. I had never heard that term but was intrigued. I enrolled in the course the following semester and was hooked from Day 1 (the first article we read was Building the Evaluation Capacity of a School District by Jean King). At the same time, my university created a Certificate in Program Evaluation and I’m proud to be among the first to have earned it. I’ve now been teaching Program Evaluation courses there for 8 years, one of my favorite and most rewarding experiences. 2. Tweet-endipity? Within a month of becoming a blogger and active tweeter, I answered a tweet from Kim Firth Leonard, who was looking for a collaborator to co-author blog posts on survey design. I didn’t know Kim, but was excited by her invitation and share her interests in both survey design and blogging, so I took a chance. That was over two years ago, and despite living on opposite coasts in different time zones, we’ve co-authored several blog articles (here, here, here, and here) and co-presented what turned out to be a standing-room-only skill-building session on survey design at Evaluation 2015 in Denver, CO where we met in person for the first time! We now have even bigger collaborative projects in the works. I’m truly honored to have the pleasure of working with Kim. 3. We’re go for launch! On December 31, 2009, John LaVelle welcomed loyal readers (well, there weren’t really readers quite yet) to aea365 Tip-A-Day By and For Evaluators. I just happened to look at AEA’s website that day and noticed it. Y’know that feeling you get when you see something new and you just know it will have an deep and lasting impact? Yeah, that’s the feeling I got. I became a daily reader, and have learned more about evaluation from these (nearly 2000!) brief articles and links to other evaluation-related sites than any other single source I can name. I couldn’t be more excited about the opportunity to be the Lead Curator (big shout out to Susan Kistler for that!) so that not only do I get an insider’s early peek at all posts, but it has put me in touch with some of the most interesting, gifted, and and generous authors in this field. When I tell people this is a labor of love, I mean it. What are YOUR 3 moments? Please consider sharing them in an aea365 post! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hey there! I’m Johanna Morariu, a Director of Innovation Network and the Co-Chair of the Data Visualization and Reporting TIG. The DVRTIG works to improve the quality of communications through better data visualization and improved approaches to reporting evaluation findings. What is Visual Reporting? Dataviz, data placemats, infographics, slide reports, and more! Visual reporting differs from traditional reporting by making increased use of shapes and colors to convey information.   Make Your Data Count: New, Visual Approaches to Evaluation Reporting from Innovation Network Why Visual Reporting? We use visual reporting to grab attention—they focus our audience on what matters. We use visual reporting to clearly communicate. By design, good visual reporting conveys a succinct message and reinforces it with data. Findings are front and center, instead of buried in long, dense text. And we use visual reporting to be engaging in the reporting phase, increasing audience participation when discussing implications and deciding on recommendations. What are the five elements of Visual Reporting? Pick a purpose: Clarifying the purpose of your report, including audience and delivery. What is the visual report meant to do? Who is the visual report meant for? Will the visual report be presented or will it speak for itself? Make a blueprint: Before we dive into putting titles, text, and data on a page, we need to know how it will all fit together. When you make a blueprint, you are sketching out the content and layout of information. Do good design: We need to pay attention to font, color, contrast, and white space. Good design helps you clearly communicate your data and findings. Use clean, uncomplicated sans serif fonts. Use a color scheme consistently throughout the visual report or visual element. Use headers to use font size consistently and to give the audience a navigation system. Test: Share your draft with colleagues or friends. Are they able to pick out the main points? Is there information that is unclear to them? What elements could be clarified or improved? Refine: Before sharing your report with your audience, make sure it is as clean and shiny as possible! Copy edit, format, align, and check all aspects of your design. Do all of your charts use the same size font? Is everything lined up that should be lined up? Is your spacing between elements or paragraphs consistent? Now you’re ready to share your beautiful, clear visual report! Rad Resource: Check out our visual report resource guide!   The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

Hello! I am Jill Lipski Cain with the Improve Group. This blog post promotes the use of a style guide to make a report both purposeful and aesthetically-pleasing. I think of a style guide as a thoughtful plan that considers best practices in data visualization, delivery tone, and highlighting the important information. I’ll be using the example of a report our team recently developed for MNsure, Minnesota’s marketplace for choosing health insurance coverage. This 100-page report includes qualitative and quantitative findings, special treatment to specific content, and a robust appendix. Because of the report’s comprehensiveness, I needed a plan for how all of these bits and pieces would create a semblance of a whole, while at the same time giving the reader breathing room through negative space and use of photos. So, before you find yourself sweating over font choices at the 11th hour, consider the following: Hot Tips: Choose a tone for the look and feel. This includes up to three colors and two fonts, header and body styles, spacing, portrait or landscape, margins, and use of columns (if any). For MNsure, I created two style guides: one with an earthy-soft feel and another with a bold, modern flavor of mint green, cherry red, and dark grey (we chose this option). Here is an example of how to layout these reporting elements: Determine how to distinguish content. Reports with multiple types of content (quotes, stories, findings, recommendations, etc.) need forethought as to how to inform the reader of the change in content and allow them quick access to specific content she or he is interested. Below is an example of how we treated text, quotes, and special callouts we captioned as a "Promising Practice" accompanied by a unique icon. Lessons Learned: TEAMWORK! Teamwork with roles and communication is paramount to pulling off a beautiful and functional deliverable. As the designer, it was essential to know the content and context; otherwise, the design and choice of images would have missed an opportunity to bolster the messages, or worse, misrepresented the content. As a team, for example, we brainstormed key words and situations that best represented the content of the report sections. From there, I was able to narrow my photo search and come up with a lot of good options for the team to choose from. And by communicating as a team, my colleagues were able to inform me as to which bars in the charts should standout (i.e. filled in a red among gray bars) to avoid the default of highlighting the most extreme bar on a chart. The design has to direct the reader to what is important, otherwise it falls short of the message. The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings! My name is Sarah Mason and I am a Ph.D. student in the Evaluation & Applied Research Methods program at Claremont Graduate University. My research focuses on the effectiveness of data visualization techniques, and my goal is to make sure that the communication strategies evaluators use have as much empirical support as the programs we evaluate. Last year a bunch of us got together at AEA to talk about our research and experience with data visualization (AEA 2014 session #DVR2). Here’s a summary of what we found. Lesson learned: We need more empirical data on the effects of data visualization in evaluation! One experiment with members of the public showed visual data reports were no more influential than text-based reports in shaping readers’ attitudes. But we need more research to understand how and why these findings occurred. Lesson learned: One data visualization does not fit all! This same experiment showed that members of the general public viewed visual reports as less credible and of lower quality than text-only reports. Tailoring your data visualization to your audience is essential if you want to get your message across. Rad Resource: Using MTurk to field-test your reporting strategies is a fast and low-cost way of trialing your visualizations before sending them to clients < https://www.mturk.com/mturk/welcome>  Lesson learned: One key to understanding the effectiveness of data visualization is knowing how it alters the cost structure of a task: whether it makes processing information easier or more difficult. Not all data visualizations are equal, and unless we’re careful our visualization efforts may make this processing task harder!  Rad Resource: Colin Ware’s book on Information Visualization offers a great overview of the theoretical foundations for data visualization techniques and strategies <http://www.amazon.com/Information-Visualization-Third-Edition-Technologies/dp/0123814642> Lesson learned: Adding interactivity to PDF documents is one way to tailor reports so readers can choose their own reading adventure. Rad Resource: Nicole Lobb Dougherty’s presentation at AEA 2014 was a great introduction to adding simple interactivity to PDFs http://comm.eval.org/viewdocument/?DocumentKey=cd2aa473-f749-40c7-a3ca-d943d79828d2 Lesson learned: Actively working with the media to develop visual materials they can use can help to make sure your message gets across to a wider audience.   Rad resource: Robin Kipke’s presentation at AEA 2014 offered an insightful overview on how the Tobacco Control Evaluation Center worked with the media to communicate evaluation findings to the general public. The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi! I’m Dave Shellard, the program evaluation manager at a professional association based in Washington, DC and Data Visualization and Reporting TIG (DVR TIG) Program Chair. As a founding member of the DVR TIG I’ve been a long advocate of incorporating meaningful and intentional data viz into communicating evaluation results. Like many of us, I’m faced with tight resources for data visualization tools and I’ve found the older posts on this blog on free or low cost tools helpful. However, I often hit a snag for data analysis and viz tools - my data would land in the public domain as part of the agreement to use the free version and in some cases that can be beneficial. However, depending on the sensitivity of my data it is not always acceptable. I have had to scale back my expectations and look to build something myself in Microsoft Excel or PowerPoint. Rad Resource: Using the Juice Analytics Chart Chooser I can filter through the free tool for ideas and download a template in Excel or PowerPoint. Most importantly I can keep my data off of a third party server. Their templates also put our data first and fancy design second, helping me tell my story. Rad Resource: I used the Speedometer Graph in Excel template to explore an idea for a project. My idea didn’t pan out but it was not due to my data needing to be uploaded to a third party server for presentation. Rad Resource: A recently used the Map of the United States Google Document template to create a very basic heat map. I was in a pinch and this template worked perfectly to help illustrate my story. I did not have to upload any data and the document was stored in my Google Drive where I control the privacy settings. Lesson Learned: Not every free template is created equal and some may lead you to a visualization that confuses your story may not score very high on the Data Visualization Checklist. With a little knowledge of Excel or PowerPoint you can convert any of these templates to match your style. Have a free downloadable template that you like? Leave a link in the comments. The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi! I’m Mandi Singleton, Research Assistant at Carson Research Consulting (CRC) in Baltimore, Maryland. Last October I attended Eval 2014 and gave a poster presentation on the process, results and implications of translating data findings into an infographic. My main focus when designing the poster was graphical content. I wanted to impress upon viewers the impact that data visualization can have on audience perception, recall and retention of information. My overall goal was to encourage people to think about the benefits of translating findings into an infographic. Hot Tips: Below are the steps I took to design my poster: Step 1: Identify the basics. Determine the purpose, audience and key message of your poster. These things will help inform the content of your poster. Step 2: Determine content. All content should be supportive of your key message and catered to your audiences’ level of knowledge. Be mindful of terminology, acronyms and icons used in your poster. Step 3: Sketch your design. I sketched the layout for my poster using pencil and paper, but you can use whatever medium you like. The goal here is to get an idea of how you want your poster to look. Hot Tip: Sketch your design on a paper that is the same size your poster will be. This helps with spacing and determining the placement of content. Step 5: Create! My poster was created using Microsoft Publisher 2013. Insert your content. Most of my visuals were directly inserted into Publisher as image files. You can also create charts/data visuals in another program (like Excel) and paste them straight into Publisher. Guide viewer’s eyes by creating visual paths of interest. This can be achieved through the use of color, varied font types and strategically placed shapes. Determine the color scheme. My posters color scheme was orange, gray, burgundy and blue. Orange and gray were chosen because they’re our company colors; burgundy and blue were selected because they looked nice with our company colors. Hot Tip: You can apply color schemes in Publisher; create your own or select a pre-made one. Rad Resource(s): For help selecting a color scheme, visit Adobe Kuler (provides RGB & HEX codes) or HailPixel (provides HEX codes). Step 6: Prepare poster for printing. Before printing, check for things like spelling errors, objects in non-printable regions and poor image resolution. Hot Tip: If you’re using Publisher you can run ‘Design Checker’ (found on the home tab) to find problems in your poster for you. Do you have any other tips for creating a poster presentation? Share by leaving a comment below! The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

Hi! We are Sara Vaca, independent consultant (EvalQuality.com) and Creative Advisor of this blog and Joseph Barnes (Senior Partner at ImpactReady). Today we are going to talk about the work ImpactReady is sponsoring to develop new ideas for using creativity and Dataviz to celebrate the International Year of Evaluation. Like many evaluators, we have been really inspired by the energy and events that have circulated around EvalYear, and we wanted to find a way to make our contribution to coming-together. After exploring several opportunities, we decided to combine our love of sharing the things we have learnt about practicing evaluation with exploring the exciting new frontier of visualization. Increasing access to broadband and awareness of how important user experience (UX) is to utilization means that there is a growing interest in Data Visualisation (Dataviz) in the evaluation community. But few of us are yet practicing it as much as we would like to. To help things along, we are attempting to create one new example of evaluation visualization each month throughout EvalYear. With each of these products, we are seeking to achieve three aims: To promote the dialogue within the evaluation community; To foster deeper knowledge of evaluation theories; and To have fun learning. Rad Resources: Our first releases include: Evaluation Metromap: Joseph put together this ‘underground map’ of evaluation where each line gathers the most representative of various evaluation aspects such as theories, approaches, techniques, actors, paradigms, etc. What type of evaluator are you? Test: we created this non-scientific multiple-choice test to help evaluators discover if they embrace the paradigm they think they do. We invite you to try it (and help us improve it)! What is good (and bad) evaluation (infographic): after having done many meta-evaluations, we have seen many common mistakes - including in our own work. We created this infographics as a way to provoke ourselves, and our fellow evaluators, to be as clear as possible about what we think ‘good’ really looks like. Ethical issues in evaluation: Ethics is often a forgotten element in evaluation. We wanted to highlight typical controversial moments in evaluation to acknowledge them. Rad Resources: You are welcome to visit the full project at EvalYear.com, and we have more releases coming soon. Here is a sneak peak: A decision tree for helping with the Evaluation design A motion picture explaining what evaluation really is. A Gender dashboard An evaluation coffee table e-book And others! Keep tuned to know more, and as usual, feel free to comment and contribute with your thoughts to help us learn more! The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

I have written about the benefits of Pinterest in past AEA365 posts, but now I want to help you get started! Pinterest is a great resource for identifying visual trends in evaluation—it’s also pretty fun to use for your personal life. First let’s understand the language: Pin: Pins are visual bookmarks that you collect on boards. You can save Pins you find on the Pinterest site or add new ones from your favorite websites or personal computer. Boards: Boards are where you save your Pins. You can make boards for anything and everything. Follow All: When you select "Follow all" you choose to follow all of someone’s boards, meaning all their boards are shown to you in real-time on your Pinterest Home Page. Follow Board: When you select "Follow board" you choose to follow and see updates on just one board from the individual’s profile. How to Join: Just go to Pinterest.com and click on the prompt to "Join." () You can use your Facebook login to create an account or just develop a user name and password. If you use your Facebook account, the two platforms will be connected and your Facebook followers will be able to see your pins. Once you create an account, Pinterest will ask you to create a profile, where you can add a photo. Create a Board: Go to your profile and select the icon that says "Create a board." This will open a form that prompts you for the name of your board, description, and category.  People make boards for all sorts of topics and interests.  Fo r example, data visualization info graphics, or even DIY all make great options for a board—the possibilities are endless.   Add a Pin: Now comes the fun part—you can start pinning images to your boards! To find interesting images, you can either search for a specific topic by keyword, or choose a topic from the drop down selection (located on the end of the search bar). Select "Home Feed" from the drop down box to see the pins of others you are following. Once you find a pin that you like, hit the "Pin it" button at the top of the image.  This will allow you to select the board you wish to pin the image to. Now if you visit your profile, you will see the image on your board and the source of the image for more information.   You can also add pins from the web, by adding the Pinterest button to your browser or upload your own images from your computer. If you want to save an image but not necessarily add it to a board, you can like the image. Dive right in: Pinterest can be a bit overwhelming at first, by just dive in. You will soon discover how intuitive it is to create boards, add pins, and browse the unlimited pins available on Pinterest. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.    

My name is Angie Ficek and I am a Program Evaluator at Professional Data Analysts, Inc. (PDA), a small firm in Minneapolis, MN that specializes in public health evaluation. Every once in a while I include timelines in my reports and/or presentations. I know there are many tools out there to create timelines, including Office Timeline and Timeline Maker, but recently I have been using a standard line chart in Excel. Say what?! Yes, I know, it’s crazy! But look at this lovely timeline! So how do you do it? Like I said, you will want to use a line chart in Excel. For this example, here is what my datasheet looks like: The first two columns are the year and month, and then I have a column for each timeline event - Intake data, Follow-up data, and Reports. Since I want these three events to appear in that order, from top to bottom, I typed in a 3 for each month for which there is Intake data available. I did the same for the Follow-up data with the number 2. And then for the Reports, since these happen at one point in time versus being continuous, I typed a 1 in the month in which a report is submitted, but left the cells in between blank. The exact numbers that you use aren’t as important as having equal spacing and the right amount of spacing between the events. For example, 2.5, 1.5 and .5 would have worked just fine too, but 3, 1.5, and 1 would have resulted in uneven spacing between events, which can be distracting to your audience and personally makes me twitch. Ok, so our data is set up, and our chart looks something like this: Not quite to the final product, but you’re close. From here we delete the vertical axis (and adjust the axis range if needed), the gridlines, and the legend, then reformat each data series. Change the marker type for the Reports to a sunburst, or circle, and adjust the color. Get rid of the markers for the Intake data and Follow-up data series and adjust the line color and width. Then, one at a time, select the first and last data points of these two series to add a sunburst, or circle, to the start and end of them. The final step is to label each event as needed with a text box. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

My name is Angie Ficek and I am a Program Evaluator at Professional Data Analysts, Inc. (PDA), a small firm in Minneapolis, MN that specializes in public health evaluation. It wasn’t too long ago that I was reviewing a co-worker’s report and came across a bunch of tables of descriptive statistics like the one below. I couldn’t help but think, "There has got to be a better way to visualize these!" Then I remembered learning about dot plots from Ann Emery (for a quick 5 minute tutorial on how to create them, click here) and reading a blog post from Stephanie Evergreen on dumbbell plots. I applied the same concepts and came up with this: It’s similar to a box or stock plot, but is actually created with an Excel scatterplot. In this example, I had three x-values (for the min, mean, and max), so I inserted three y-values which tell Excel where the data points should fall on the invisible y-axis. As for the oh-so-important formatting of the data points, I added a marker for the mean and a light gray line to signify the range. I added the automated data labels above each data point and formatted the font size and color so that the means stood out more than the min and max since they were the most important value. Then I manually added text boxes to label the min, mean, and max along the top, and the program names along the left side. I was really excited to find a simple, visual way for our client to see the pattern of these descriptive statistics a little more quickly. I cannot wait until I get to chart some descriptive statistics again! Said no one ever - until now. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! I’m Kate Tinworth, Program Co-Chair of the DVRTIG. I am excited to write about one of my favorite parts of my work as an evaluator— drawing. Last year at AEA I co-presented a session called, "Drawing Them In: Graphic Facilitation & Evaluation to Strategically Visualize Change" with my friend Chris Chopyak (a rock star who literally wrote the book on using visuals to help businesses address challenges and create strategies). We reminded our audience that we all think visually, images are key to memory and learning, and that you—yes, you—can draw. And you should! Visuals can resolve ambiguity, cut across language and cultural divides, help findings become more salient, and kick start action. Hot Tip: Find a Local Friend. If you want to explore the potential of teaming with a graphic facilitator, it’s time to make some new friends. Check out the International Forum of Visual Practitioners (IFVP): http://ifvpcommunity.ning.com/. There’s bound to be someone near you! Rad Resource: Go to a Class/Workshop. There are great opportunities to try your hand at graphic facilitation, whether you plan to incorporate it into your evaluation practice or just want to stimulate your visual thinking. Though it can feel intimidating, I highly recommend signing up for a class. Learning graphic facilitation techniques have helped me to sketch out graph and chart ideas, think through report layouts, and get far more creative with methodology and instrumentation. Cool Trick: Drink. Some graphic facilitation practitioners, including Chris, do "drink and draws" where you can get some drawing practice over a cocktail. Amazing! Hot Tip: Draw. All the Time. To become more comfortable with drawing I draw, all the time. Try covering your dining room/kitchen table with butcher paper and put crayons or colored pencils out. When you sit down for coffee or a meal, draw. Tape paper to the wall and "live capture" TED talks or your favorite podcasts. Carry a notebook and favorite pen everywhere. Commit to drawing for just 2 minutes a day. Cool Trick: Apps. More of a techie? Draw on your tablet! I like Notability and iMotion. Rad Resource: Get Inspired. The DVRTIG is a great place to find inspiration and make connect with colleagues who care about visual thinking. Check out the AEA365 blog posts, the TIG website, and resources on p2i. Hot Tip: Go Visual. Whenever you can, try to "go visual" in your projects. Try a visual logic model. Engage stakeholders in drawing. I often get my stakeholders to draw during a training or as I present preliminary data. Lately I have also been experimenting with data placemats, which I learned about through AEA (thanks @VeenaPankaj). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Jackson Hille, the Content Associate for FormSwift, a SF-based startup that helps organizations, entrepreneurs, and businesses go paperless. Recently, for a work project, I had to evaluate the current strengths and weaknesses of our content campaigns, and also decide what content avenues present our greatest opportunities and our greatest threats. As a professional evaluator, you might be familiar with the aforementioned planning concept, commonly referred to as a SWOT Analysis. I am new to the business world, as I just graduated from UC Berkeley in May 2014, so as any decent millennial would do, I Googled and searched guides to SWOT Analysis. While the majority of what I found was disappointing, there was one existing rad resource out there from the University of Kansas; nonetheless, there was no SWOT Guide that combined honest evaluations about the modern economy, usable templates and a user friendly design. So, I called up my college mentor and we created our own, "Essential Guide to SWOT Analysis." Rad Resource: The Community Tool Box’s SWOT Chapter: The Community Tool Box is a service of the Work Group for Community Health and Development at the University of Kansas. Their SWOT Guide is comprehensive, user friendly and a great resource for any evaluator within the public health or community development field. The SWOT Guide not only provides a thorough explanation of the appropriate evaluation process when conducting a SWOT Analysis centered around health and community development related issues, but it also provides readers with multiple forms of content. For instance, it contains relevant examples for health and community development professionals, a ready-made PowerPoint version of the guide’s highlights, and a handy checklist to ensure you are taking the correct steps when conducting your SWOT Analysis. Rad Resource: The Essential Guide to SWOT Analysis: The Essential Guide to SWOT Analysis is the end product of a unique collaboration. As previously noted, through my work at a startup, I realized that there were no great, comprehensive SWOT guides out there, especially for people in the business world. So, I enlisted my college mentor, Justin Gomer, a Lecturer at UC Berkeley, and we decided to make a comprehensive guide ourselves. The finished product is a guide to SWOT Analysis that is readily usable for either a professional evaluator hired as a consultant to help out with a company’s investment strategy, or for a volunteer at a non-profit, who needs help evaluating an organization’s goals for the upcoming summer. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, Veronica Olazabal, senior associate director of evaluation at The Rockefeller Foundation here with Karim Harji, director at Purpose Capital. As many of you already know, market-based approaches to poverty alleviation are gaining traction across the social sector. A range of innovative strategies are being used to finance these initiatives, including impact investing—an approach to deploy various types of capital to intentionally deliver social impact alongside financial return. Mapping of the Impact Investing Industry (courtesy of E.T. Jackson and Associates) There is clearly a strong need to strengthen impact measurement and evaluation in this space. Many evaluators have had limited engagement to date, but it is an emerging topic of interest, as demonstrated by the focus at the Evaluation 2014 AEA annual conference. Below, we outline a few ways the evaluation community can become more engaged with evaluating impact investing: Lessons Learned: Break out your Theory of Change facilitation skills! Like more conventional social sector program-designs, impact investors have theories of how they expect change to happen, including assumptions and goals.While they may use an ‘investment thesis/approach’ instead of  ‘theory of change,’ they often use these models to select, assess and monitor investments. Thus, it should not be surprising that theory of change has become a tool of emerging importance to impact investing. Rad Resource: Interrogating the theory of change. Use monitoring strategies to generate timely data: Monitoring can be an important strategy for tracking social performance, especially since investors already use financially-focused tools in this way. When designed and implemented well, monitoring data can provide timely and relevant information that can be used to adapt operational plans for investee enterprises or funds. For example, in the access to finance sector, monitoring has been used effectively to validate that target beneficiaries and clients are actually being reached. Rad Resource: Portfolios of the Poor Gain familiarity with emerging standards and approaches: There are a range of initiatives in this sector that seek to assess social change for a range of uses and users. For example, the Impact Reporting and Investment Standard System(IRIS) seeks to build a standard vocabulary/taxonomy at the output level, while Global Impact Investing Global Rating System (GIIRS) is a standards-based rating system that assesses impact funds and social enterprises. As this area continues to evolve, more evaluation capacity will be needed at every level, and particularly around moving from lives touched (reach) to validating lives impacted (depth) in the field. Evaluators will be important not only to assess the intended and actual outcomes from individual transactions, but also to critically analyze how the field is contributing to market-based approaches to poverty alleviation.  Rad Resource: Assessing Impact Investing: Five Doorways for Evaluators - Ted Jackson and Karim Harji. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings! I’m Sara Vaca, independent consultant at EvalQuality.com and Creative Advisor of this blog. As mentioned before, I consider creativity a key competence in life, and in evaluation. How so? If there is anything clear regarding evaluation it is that there are no rules. No rules. It is a totally adaptive discipline that can be taught and learned, but new situations can demand at any moment new answers. That is why creativity (plus knowledge and experience, of course) becomes handy. In my attempt to continue fostering our creative side and exercise it in case of needs, many things can be done. In this occasion, I collected some of my favorite quotes. I hope some of them inspire you and resonate with you: "Creativity is as important as literacy" Ken Robinson "The chief enemy of creativity is good sense." Pablo Picasso "I never made one of my discoveries through the process of rational thinking" Albert Einstein "Others have seen what is and asked why. I have seen what could be and asked why not. " Pablo Picasso, Pablo Picasso: Metamorphoses of the Human Form: Graphic Works, 1895-1972 "You can never solve a problem on the level on which it was created." Albert Einstein "You can’t use up creativity. The more you use, the more you have." Maya Angelou "Learn the rules like a pro, so you can break them like an artist." Pablo Picasso "The thing is to become a master and in your old age to acquire the courage to do what children did when they knew nothing." Ernest Hemingway "Absurdity and anti—absurdity are the two poles of creative energy." Karl Lagerfeld "Thoughts are like burning stars, and ideas, they flood, they stretch the universe." Criss Jami, Killosophy "When I am ….. completely myself, entirely alone… or during the night when I cannot sleep, it is on such occasions that my ideas flow best and most abundantly. Whence and how these ideas come I know not nor can I force them." Wolfgang Amadeus Mozart "If you’re not prepared to be wrong, you’ll never come up with anything original." Ken Robinson, The Element: How Finding Your Passion Changes Everything "Creativity arises from our ability to see things from many different angles." Keri Smith, How to Be an Explorer of the World: Portable Life Museum "Creativity is nothing but an intelligent imitation." Sandeep Kakkar "Creativity comes from applying things you learn in other fields to the field you work in." Aaron Swartz And my favorite of all times: "They didn’t know it was impossible so they did it".(Mark Twain). Do you have favorite quotes? Share yours: sara.vaca@evalquality.com Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Teresa McCoy and I work as an Assistant Director of University of Maryland Extension (UME) with responsibility for evaluation and assessment. Like many Extension colleagues, my evaluation department consisted of me until last year when I was able to hire an additional person. The two of us have responsibility across all program areas in our organization, and I am always looking for technologies that can help save time while adding quality to evaluation efforts. This past year, I needed to conduct about 25 interviews. I was faced with hours of conversation that would have to be transcribed and analyzed without assistants to help. Several people suggested, given that I’m at a university, to "Just hire an undergraduate student. It shouldn’t cost you very much that way." Well, I don’t know about you, but if I have spent countless hours preparing questions, designing a protocol, and contacting and scheduling interviews, I am not about to hand over transcription duties to the first student "off the street." Football solved my problem. I know that’s hard to believe, but while I was at a Baltimore Ravens football game party with friends, I was chatting with an education policy analyst. She told me about TranscribeMe!™ and her good experiences with the company and the product. Lesson Learned: Hot tips and rad resources often are found at unlikely places! Lesson Learned: After some investigation, I found out that TranscribeMe!™ and NVivo™ have a business partnership. I was able to upload my audio recordings from within NVivo™ (after setting up my account) and the transcripts were then sent back to me and into my NVivo™ project file. In the media options, there is a "purchase transcript" option. To clarify, you can use TranscribeMe!™ without having to use NVivo™. However, given that I was using NVivo™ for my coding, these two products made the initial work a lot easier and faster. I received some of the transcripts within 24 hours and almost all of them within 48. The transcript quality was excellent. And, as I am sure you’re wondering, the price was good (Price is negotiable depending on quantity of work, number of speakers, and other options.) You can use the app on your smart phone to record. No special equipment needed. Rad Resource: TranscribeMe!™ and NVivo™ partnership. TranscribeMe!™ at www.transcribeme.com Rad Resource: Information from QSR about using NVivo and TranscribeMe!™ http://www.qsrinternational.com/products_nvivo_transcription-services.aspx If you’re a football fan like me, now you have a great excuse to watch the games because you never know when you’ll find your new evaluation rad resource! The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

Hi! I’m Mary Arnold, a professor and 4-H youth development specialist at Oregon State University, where I spend the majority of my time in the area of program evaluation, especially in capacity building efforts. This is my second time preparing a blog post for the EEE-TIG, and the invitation came at a great time, because I have been thinking pretty obsessively these days on how we can do a better job of building Extension program planning and evaluation capacity. One of the conundrums and persistent late night ponderings that continues to rattle around my mind is how we can do a better job articulating what is suppose to take place in programs. If we are clear on of what is supposed to happen in a program, then we also should be able to predict certain outcomes and understand exactly how those outcomes come to be. This notion of prediction is what underscores a program’s theory. Because of the emphasis on program planning and that swept Extension in the early 2000s, most Extension educators are familiar with logic modeling. The good news is that   many educators understand the concepts of inputs, outputs, and outcomes as a result, so the groundwork is in place to think more deliberately about a program’s theory. But at the same time, there is scant evidence that logic modeling has resulted in better program planning practices, or led to the achievement of stated outcomes in Extension programs. And there is even less evidence that logic models are developed based on theory. Lesson Learned: Theory may be implied in logic models, but too often it is understated, assumed, or just hoped for. Program theory is what connects the components of a logic model and makes it run! Hot Tip! Did you know that there are two important parts to program theory? The first is the program’s theory of change, which is the way in which the desired change comes about. The second is the program’s theory of action, which refers specifically to what actions need to happen, at what level of success, for the program to reach its intended outcomes. Rad Resource! My favorite resource for understanding and developing a program theory of change and action is Purposeful program theory: Effective use of theories of change and logic models (Funnell & Rogers, 2011). This book has loads of great information and practical help on bringing logic models to life with program theory. Rad Resource! If you are looking for specific theories that are useful for Extension programs, The University of Maryland Extension has a terrific short guide entitled Extension Education Theoretical Framework that outlines how several well-developed theories can be useful for Extension programming. The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Scott Chazdon, Evaluation and Research Specialist with the Extension Center for Community Vitality, University of Minnesota. I have gained skills in a process known as Ripple Effect Mapping (REM) to document impacts of Extension community development programs. REM sessions often spur important thinking, connections and work. REM is a participatory group method that engages program and community stakeholders to retrospectively and visually map the chain of effects resulting from a program or complex collaboration. The REM process combines elements of Appreciative Inquiry, mind mapping, group interviewing, and qualitative data analysis. It is a powerful tool for documenting both the intended and unintended results of a program. It is also a way to engage and re-energize program participants and stakeholders around shared goals. Rad Resource: A more in-depth introduction to REM is at University of Minnesota Extension feature article on REM - "Ripple effect mapping makes waves in the world of evaluation" Lesson Learned: What started as a great method for evaluating community leadership programs morphed into a tool for a broad range of programs. In Minnesota, an effort to document the impact of urban Master Gardeners working in the neighborhoods became a more inclusive and community-driven project that showcased the many different outcomes of the program that may have been overlooked.   Here is a thumbnail graphic of the core section of the Ripple Effect Map from that project. Rad Resource: You can find full-sized REM graphics at this site University of Minnesota Extension REM Blog Lesson Learned: Recruiting the right number and mix of people is crucial in Ripple Effect Mapping. In terms of numbers, these are larger than focus groups, but if you go beyond 20 people you may not be able to include all voices in the process. I prefer groups of 12 to 20 people. You can invite both direct participants and non-participant stakeholders. This non-participant group can include funders, local elected officials, other influential figures, or representatives of the media. Lesson Learned: This mix of people creates an insider-outsider dynamic that sometimes leads to game-changing insights about efforts that have already happened, as well as efforts that could happen! That’s why Ripple Effect Mapping makes sense as a developmental evaluation tool. Rad Resources: To find out more about REM and approaches that can be taken, as well as if might be a tool you can use, take a look at these two articles: 1) Journal of Extension — Using Ripple Effect Mapping to Evaluate Program Impact: Choosing or Combining the Methods That Work Best for You and 2) Journal of Extension — Ripple Effect Mapping: A "Radiant" Way to Capture Program Impacts The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Salutations from the Land of the Midnight Sun. My name is Alda Norris. I am an evaluation specialist for the University of Alaska Fairbanks Cooperative Extension Service and webmaster for the Alaska Evaluation Network. There is a lot of activity packed into a single word when you say "evaluation" or "extension." Have you ever had someone stare at you blankly when you tell them your job title? My background is in the study of interpersonal communication, and I believe developing skills in providing effective comparisons will boost our ability to explain "what we do" to others.  Hot Tip: A three-step pattern I learned from speech class can be very helpful. Define the term. Give examples of what it is. Give examples of what it is not. Also, your audience will gain a deeper understanding if the examples you use are surprising. Here’s one from our state sport: Many people hear the term "sled dog" and think of a big fluffy Siberian Husky. However, many purebred Siberians are show dogs not used for mushing. Sled dogs are more commonly of a mixed heritage known as Alaskan Husky, and some are crossed with other breeds like Greyhound or Pointer! Lesson Learned: Clients may make demands that seem unreasonable because they misunderstand the scope of your expertise or duties. Even worse, they may not seek you out at all because they don’t see a link between your title and what they need. If you’ve ever had someone think evaluation is "just handing out a survey" or extension is "just agriculture stuff" then you know what I mean! Take the time to do some awareness-raising with your target audience. Hot Tip: Strip away the professional jargon and think about what words the public would use to describe you. Make sure those terms are included on your web page so that search engines will associate you with them. If you haven’t already, add an "About" or "FAQs" page that addresses what you do (and don’t) have to offer. Rad Resources: Books like Eva the Evaluator are great for providing examples and comparisons of what jobs like "evaluator" entail. Maybe someone will write an Ali the Extension Agent book someday! Also, search the AEA365 archives for related discussions on the difference between evaluation and research, and how to use metaphors to extend understanding. The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

My name is Brigitte Scott and I am the Evaluation and Research Specialist for the Military Families Learning Network (MFLN), which engages military family service professionals in high-quality, research-based professional development. The MFLN is part of the Department of Defense (DoD) - U.S. Department of Agriculture / National Institute for Food and Agriculture Partnership for Military Families (USDA/NIFA) and is also a part of eXtension—the online branch of America’s Cooperative Extension System (CES). Evaluation for the MFLN comes with a few challenges—leadership, PIs, and staff are spread out across the country; our cooperative funding agreement requires nimble and flexible programming (Hello, developmental evaluation!); and constituents in multiple institutions have different ways of communicating and varied reporting needs. Lesson Learned: When I first began working with MFLN, I drew heavily on my background in qualitative methods, and all of my mixed methods reports took on a narrative form. However, the reports weren’t getting read. With competitive funding forever at stake in an era of sequestration, this had to change. Enter data visualization. At AEA 2014, I took a two-day data viz workshop with Stephanie Evergreen. It was invaluable! My reports are still works in progress, but I know now they are being read. How? Folks are actually contacting me with questions! My reports are getting circulated at DoD, which has meant increased awareness of MFLN and a lot of kudos for our work. (It doesn’t hurt come budget time, either.) PIs and staff are utilizing the reports to discuss their progress against dynamic plans of work while focusing on the moving target of program innovation. Hot tip: CES just celebrated its 100th birthday last year, but make sure your reports aren’t dinosaurs! Your reports—your efforts!—need to be seen and heard to be actionable. I like to think of CES as power to the people. If you agree with me, then give data viz a try to get your points across and support CES in making a difference in counties across the nation. Hot tip: Data visualization isn’t all about Excel. Arrange key verbal points on a page with clean, clear data. Pull out a thread from a data story and expand it in a text box, or pick up qualitatively where your quantitative story said its piece. Hot tip: Font and color matter. Use your organization’s visual identities in your reports to let readers know that your report concerns them and their work. Rad resource: Check out AEA’s offerings on data visualization, including workshops, coffee breaks, and of course, the annual meeting data viz sessions. They really are amazing! Rad resource: Stephanie’s workshops are a must, but so is her book. Check them both out! The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Pam Larson Nippolt and I am a University of Minnesota Extension Evaluation and Research Specialist working with a team of program evaluators in 4-H youth development programs. Lesson Learned: Monitoring enrollment data is often a data-related activity that falls under the umbrella of program management.   Monitoring enrollment data enables program leaders to pay attention to some aspect of program implementation via inputs or outputs. What is monitored can be quite distinct, but it still can inform the focus of an evaluation or measurement of an outcome. When planning with program teams, I use the example that monitoring is similar to setting a metronome while playing piano-it keeps a steady beat going to help the pianist stay in tempo. Evaluation, on the other hand, is the assessment the pianist and audience make about the music created. Lesson Learned: Collecting, maintaining, and analyzing data for monitoring purposes are an investment of time and resources that can pay dividends for evaluation in the long run! Enrollment databases, used in many large youth development programs, are excellent data sources for program monitoring, but are often overlooked. For example, in 4-H, program data (shown below) revealed that a region with the largest Metropolitan area (Central Region) enrolled more youth from farms and small towns than what had been believed to be the case. This finding seemed to be counter-intuitive and led to further investigation of the data. We discovered that many youth living in the city and participating in the program were not in the enrollment database because of a particular enrollment practice. Monitoring the enrollment data led to an awareness about the need to make the process more accessible for all youth and families.   Program staff may not have identified the scale of this discrepancy without this type of monitoring. Hot Tip: Get started by "whetting the appetite" of your program partners for data use with available data about the program and participants. Build appealing and visually engaging graphics to make the using the data rewarding to staff who don’t typically attend to data. Ask questions and listen to how they make sense of the data. This practice will reveal what can be monitored "right now" for team learning. Rad Resource:  Consider investing in making your enrollment database more usable and accessible to staff with trend and comparison features. Interfaces can be designed for your enrollment software that provide a dashboard with menus to track changes over program years and geographic comparisons. Think like an interface designer to create tools and reports that will help program staff love their data! The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor, and I’m thinking deeply this morning. Really deeply. About unanticipated outcomes. It’s fascinating when I think back on what seem like tiny, trivial moments in my life - a sentence spoken, a question asked, an action taken - that had profound impacts on my life and the person I became. This makes me think about what we choose to measure when we engage in evaluation. But it’s not just about me. Unanticipated outcomes have profound effects on many lives. A friendship that develops in the workplace changes the trajectory of someone’s life. An otherwise unwelcome business in a neighborhood creates jobs and boosts the local economy. A drug originally developed to treat a relatively minor health condition is found to be effective for a much more dangerous one. Lesson Learned: Unanticipated outcomes have been a consideration in program evaluation for almost as long as evaluation has been a field of inquiry. Marvin Alkin wrote about them as early as 1969: In considering what to measure, …the evaluator may wish to point out the necessity for broadening the area of concern because of interrelated aspects of the … program, or to consider, as well, various areas of potential unanticipated outcomes. …There are evaluations necessary in providing information during the course of a program about the manner in which the program is functioning, en route objectives are being achieved, and what unanticipated outcomes are being produced. Such information can be of value in modifying the program (program improvement). Michael Quinn Patton addresses them in his 2015 text, Qualitative Research and Evaluation Methods: Integrating Theory and Practice: Image credit: Ripple Effect by sea turtle via Flickr An important contribution of qualitative inquiry is to identify and understand unanticipated consequences, including how unanticipated outcomes intersect with targeted outcomes in program evaluation. This potential of qualitative fieldwork to uncover anticipated outcomes deserves emphasis and is another reason why qualitative studies have become more valued. Statistical studies of outcomes can only imagine what has been thought of, conceptualized, and operationalized in advance. The open and nature of qualitative inquiry, in contrast, is especially useful in capturing unanticipated outcomes. Indeed, one can only turn up side effects, ripple effects, emergent outcomes, and unanticipated impacts through open-ended fieldwork. Hot Tip: The bottom line is, we may never truly know the reach of programs, interventions, or interactions, so it’s worth considering unanticipated outcomes when we design programs and evaluation plans. How can we prepare ourselves to recognize, document, and evaluate unanticipated outcomes? Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Bill Trochim and Arthur Blank here and we are delighted to introduce AEA’s newest Topical Interest Group (TIG) - the Translational Research Evaluation TIG - and this week’s theme. So, let’s start with - what is "translational research" (TR) and why is it so important? There are lots of definitions of TR. We prefer a broad and encompassing definition along these lines "the systematic effort to move research from initial discovery to practice and ultimately to impacts on our lives." In biomedical research, some refer to TR as "bench-to-bedside" suggesting that it links basic laboratory work to the practice of clinical medicine. Others (like us) tend to describe TR more sweepingly as "innovation-to-impact", emphasizing the entire research-practice continuum from initial new ideas to their ultimate application and effect on society. In one sense, TR is very new, and one of the hottest topics in contemporary research. But, in another sense, it is as old as the research-practice distinction itself. If research and practice were well integrated and functioning efficiently together, the emphasis on TR would be unnecessary. In most biomedical and applied social policy areas, research takes too long to influence practice; one well-known estimate is that it takes on average 17 years for a biomedical discovery to influence practice (and that is likely an underestimate). Some of this time is undoubtedly due to the inherent difficulties of translation. But there is considerable evidence to suggest that much of this time lapse may be due to other factors. Lesson Learned: In many fields, the problems being studied are complex and require multidisciplinary and transdisciplinary approaches. But researchers have not been trained in collaborative and team science methods that might enhance such work. In many fields, researchers develop innovations without considering the world of practice, only to find out later that their ideas won’t work in the real world. Learning how to involve the practice community as integral participants throughout the research development process could help avoid such costly errors. In many instances, research and practice realms are poorly managed and full of inefficiencies. For instance, we know that the process of reviewing and starting a single biomedical clinical trial can take years and involve hundreds of steps (many of them unnecessary or duplicative). If we learned more about how to manage the research enterprise better - something like a "science of science" or a "science of science management" - we might see significant progress. Rad Resources: This week you’ll be introduced to some of the members of our new TIG and to the kinds of issues we are addressing. Rad Resources: The National Institute of Health’s Clinical and Translational Science Award program provides support to professionals engaging in this work.  NIH also offers additional information about translational science in biomedicine. The American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

We, Arthur Blank and Julie Rainwater, are pleased to introduce a new Translational Research Evaluation Special Interest Group (SIG).  The SIG is part of the Association for Clinical and Translational Science (ACTS) and provides a forum for all aspects of evaluation related to clinical and translational science. The ACTS SIG recognizes organizations involved in translational science as diverse and their "evaluators" may not necessarily identify themselves as professionals in the field of evaluation.  Thus, our membership includes administrators, faculty, clinicians, librarians, biomedical scientists, and other stakeholders in translational research and workforce development.  The SIG, like its AEA partner TIG, offers its members the opportunity to share mutual interests, evaluation expertise, resources, and materials. Our work is closely coordinated with the AEA TIG and the diverse membership across both groups provides access to a valuable practice community that can share experiences and challenges. We are off to a great start.  The ACTS Translational Science 2015 meeting in Washington DC in April hosted a first-ever "evaluation" track sponsored by the SIG.  Judging by the high attendance and lively discussion at the two evaluation-relevant panels, this track is likely to be a feature of subsequent annual meetings. Lesson Learned: We walked away from this meeting with a few lessons to guide us moving forward.  The first panel, "Classifying Publications along the Translational Science Spectrum:  A Machine Learning Approach," provided an opportunity for us to learn state-of-the-art approaches for how Clinical and Translational Science Award (CTSA) organizations analyze publications to understand the progress of discoveries through the stages of translation to implementation.  The second panel, "The Role of Evaluation in Translational Science Organizations," was a unique opportunity for us to hear what current leaders of the CTSA Domain Task Forces and NIH National Center for Advancing Translational Sciences (NCATS) representatives think about the evolving role of evaluation.  The discussion about the future of CTSA evaluation was beneficial to all of us, including NCATS, as we consider how evaluation can help move the translational research enterprise in the right direction. Over the next few months we will transition to new SIG leadership and start planning for the ACTS Translational Science 2016 conference. We are looking forward to building on the 2015 meeting, as well as the opportunity to gather at the AEA conference in November 2015.  (See you in Chicago!) Rad Resource: For those interested in joining the Association for Clinical and Translational Science (ACTS), as well as learning about the various activities those engaged in translational science are involved with visit the ACTS web site. The American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.