We are internal evaluators Cath Kane, of Weill Cornell Clinical & Translational Science Center, and Jan Hogle, at the University of Wisconsin-Madison Institute for Clinical & Translational Research. The 62 Clinical and Translational Science Award (CTSA) hubs — along with community-based and industry partners — collaborate to advance translational research from scientific discoveries to improved patient care by providing academic homes, funding, services, resources, tools, and training. Lesson Learned: How do CTSA hubs define "translational?" Translation is the process of turning observations from the laboratory, clinic, and/or community into interventions that improve individual and public health. Translational science focuses on understanding the principles underlying each step of that process. Hot Tip: Ask "What are we evaluating?" Internal evaluators determine whether programs are efficiently managed, effective in meeting objectives, and ultimately impacting the process and quality of biomedical research — involving multiple variables in complex systems. Case studies use qualitative data to complement quantitative data on research and training productivity. Analysis of multiple case studies identifies factors that facilitate or impede successful translation. Retrospective or prospective? Retrospective analyses use an abundance of data to deeply study known successes. Prospective case studies can identify factors that influence translation in real time. Questions might include: What does ‘successful translation’ mean? Which operational process markers are most important? What are the ideal duration metrics? How does collaboration and the individual CTSA hub move translation along over time? How can we better support the translational process? The retrospective case study below tells a story of key operational markers in the development of Gleevec, used to treat leukemia. Public data such as publications, FDA approval and patenting events, and mortality data were overlaid with interview data from key informants. Lessons learned: An individual face-to-face interview is indispensable for obtaining an insider’s view of the research process. Record the interview and take notes. Often the research details are unfamiliar to the interviewer. Hand-written notes alone may not capture the interview accurately. Ask investigators for their interpretation of "translation." People have different ideas about the meaning of translation. What are the key moments or markers in translation? Case studies can serve many needs from short summary "stories" for public relations and newsletters, to more rigorous approaches paired with quantitative data for decision making by internal and external stakeholders. Perhaps the most important lesson learned is the value CTSA evaluation teams bring to their hubs, not only for long term objectives, but also in the shorter term through contributions on a daily basis to programming adjustments and course corrections using a mixed methods approach to understanding complex change. The American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Prevention research can have a huge impact on population health, but how do we evaluate the impact and translate the research into products for public health practitioners? We have been tackling that question at the Prevention Research Centers (PRC) Program at the Centers for Disease Control and Prevention (CDC). I’m Erin Lebow-Skelley and I work for the Evaluation and Translation Team that evaluates the impact of the PRCs, and I want to share our approach with you. The PRC Program directs a national network of 26 academic research centers, each at either a school of public health or a medical school that has a preventive medicine residency program (See figure, below). The centers are committed to conducting prevention research and are leaders in translating research results into policy and public health practice. All PRCs share a common goal of addressing behaviors and environmental factors that affect chronic diseases (e.g. cancer, heart disease, and diabetes), injury, infectious disease, mental health, and global health. Each center conducts at least one core research project; translates and disseminates research results; provides training, technical assistance, and evaluation services to its community partners; and conducts projects funded by other sources (CDC, HHS, and others). As a result, the PRC network conducts hundreds of projects each year. The Evaluation and Translation Team is tasked with the challenge of demonstrating the impact of this heterogeneous group of research centers. We have spent the last two years developing the evaluation plan for the current 2014-2019 PRC funding cycle, while engaging various stakeholders throughout the process. We started by developing the evaluation purpose, questions, and indicators, and now have a complete and piloted data collection system and qualitative interview guides. We plan to annually collect quantitative data from each PRC that reflects their centers’ inputs (e.g., faculty and staff), activities (e.g., technical assistance, research activities), outputs (e.g., research and practice tools, peer reviewed publications), and impacts (e.g., number of people reached) using a web based data collection system. Having a cohesive system that collects information allows us to link center activities to outputs and impacts (e.g., showing what partners were involved in X project, that contributed to Y impact), which provides a comprehensive understanding of elements that contribute to center and network impact. Hot Tip: Always start with program logic (after engaging your stakeholders!). No matter how complex the program, determining the overarching program logic will help guide the development of your evaluation indicators and provide a comprehensive picture of how the program is working. Hot Tip: Consider providing end-users an electronic means of systematically providing feedback within the information system itself pertaining to data entry problems, subject matter questions, and suggestions for improvement. he American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! Our names are Natalie Wilkins and Brandon Nesbit and we are both evaluators at the Centers for Disease Control and Prevention (CDC), in the National Center for Injury Prevention and Control (NCIPC). One of the projects we provide evaluation support for is the Injury Control Research Centers (ICRCs) program, funded through NCIPC. This has provided us with a number of important lessons learned around evaluating multi-site research center programs that are engaging in translational research and outreach. There are 10 ICRCs across the country, funded to conduct innovative research on the prevention of injury and violence.  These institutions serve as training centers for the next generation of injury and violence prevention researchers and act as information centers on injury and violence prevention for the public.  ICRCs are also pioneering innovative approaches to the translation of research to practice. They conduct translational research studies and engage in a variety of outreach activities to translate research on evidence-based injury and violence prevention strategies into practice settings. For example, one of the ICRCs works with partners to assess their capacity for using research findings in their work, and then provides tailored technical assistance based on each partners’ specific needs to ensure research is translated into practice.  In addition to these "research to practice" activities, some ICRCs are also employing a "practice to research" approach to their translational research, leveraging their outreach activities and partnerships in the field to inform their research priorities. As evaluators of this comprehensive, multi-site research center program, one of our challenges was to show the impact of the ICRCs’ translational research and outreach activities on bridging the gap between research and practice. To this end, CDC and the ICRCs developed a set of indicators to capture information on impact (e.g. studies, partnerships, outreach activities, development of research and practice tools, etc.). We display data on these indicators through Tableau, software that allows users to analyze, visualize, and share data in an interactive way. Hot Tip: Visually presenting evaluation data through interactive dashboards allows stakeholders to glean their own insights while still ensuring key messages are communicated.  Tableau enables us to showcase the approach and impact of each of these unique research centers, while also providing the option of presenting a "bird’s eye view" of the impact of the entire ICRC program as a whole. Lesson Learned:  Translational research and outreach can take many forms. Engage your stakeholders in the evaluation process early so you can ensure they have a clear understanding of the kinds of information you are looking for. Rad Resource: For more information on how evaluators have used Tableau, check out the AEA365 archives- http://aea365.org/blog/?s=tableau&submit=Go he American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello.  My name is Sally Thigpen and I work in the National Center for Injury Prevention and Control (NCIPC) at the Centers for Disease Control and Prevention (CDC).  As an evaluator, I often find myself encouraging scientists and stakeholders to think about evaluation from the very beginning of any study or project.  I do whatever I can to be included as early as possible so I can help build evaluation into every step of the process.  The value of inclusion is equally true for our practice partners.  They need to be included as early as possible in our scientific thinking because they are vital to the translation of research to practice.  Practitioners speak to the relevance and utility of the science and the value it has to current programmatic or policy efforts.  In today’s budgetary realities, understanding these practical aspects of uptake helps assure limited dollars have the maximum impact. The Division of Violence Prevention within NCIPC developed the Rapid Synthesis and Translation Process (RSTP) to systematize this communication loop between the research and the field of practice. This six-step process (in the graphic below) can help users facilitate the negotiation between the science and practical application. Hot Tips: Before engaging with a group of practitioners, do a gut check with the scientists of record.  Ask questions about what they see as the most valuable aspect of the study for practical application.  What is their biggest apprehension about how the science might be misinterpreted and used in ways it was not intended?  These answers not only help to focus the translation efforts, but also offer a little insight as you begin working with a selected group of practitioners. Work with the same group of practitioners from the beginning to the end of the translation process.  Begin this relationship with similar questions from above.  How do they anticipate using the science?  What is the significant contribution to the field?  What is least valuable? As the translational product moves through development, keep checking in with the group of practitioners and scientists.  Practitioners can guide you on relevance and balancing science with action.  The scientists can guide you in making sure you are keeping scientific integrity along the way. Lesson Learned: You’re not just a communicator/evaluator/researcher - you’re a negotiator.  In the role of translator, you are often negotiating between the details of pure science and the brevity of the practical world.  This is a critical role and takes finesse. Rad Resource:  My colleagues and I published an article in a July 2012 special issue of the American Journal of Community Psychology reviewing RSTP’s usefulness in the field, Moving Knowledge into Action: Developing the Rapid Synthesis and Translation Process Within the Interactive Systems Framework. he American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, my name is Jayne Corso and I am the community manager for American Evaluation Association and the voice behind the AEA Facebook page. If you manage a company Facebook page, you might have noticed a drop off of "likes" recently. Facebook has begun removing memorialized and voluntarily deactivated accounts from Pages’ like counts. This change ensures that data on Facebook is consistent and up-to-date—but could mean a drop for your analytics. Although some Pages might lose "likes," they could also gain a more accurate way to track their followers. I have compiled a few tips for tracking your analytics and gaining more visibility for your page. Rad Resource: Take advantage of Facebook "Insights" Facebook offers Page Insights after at least 30 people have liked your Page. Use this tool to understand how people are engaging with your Page. With this tool, you can see your Page’s growth, learn which posts have the most engagement, find demographic information about your audience, and identify when your audience is using Facebook.  This data is available for free and can easily be customizable for time frame and downloaded to excel. Rad Resource:  Use Google Analytics to track effectiveness Tracking your analytics through Google allows you to see how many people are coming to your site from social networks, understand the website pages they are most interested in, and gain a better understanding for how your audience is engaging with your web content.  To find this information, enter your Google analytics account and go to "Acquisitions". From here you can look at the performance of your social networks as an overview or look more specifically at referrals, activity, and user flow. All of this data allows you to gage the effectiveness of your social campaigns. Hot Tips: Increase your Facebook likes Finally here are a few simple tips for increasing the likes on your Facebook Page—hopefully you can make up for any followers you lost when Facebook made their changes. Add the Facebook icon to your website, so visitors know you have a presence on the social network (Place the icon high on the website page, near your navigation) Add the Facebook icon to your email communication or blog to reiterate your presence on Facebook to your subscribers Cross promote your Facebook page on your other social media sites. You may have followers on Twitter that have not liked your Facebook page or didn’t know you had a Page Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, I’m Stephen Maack, Owner and Lead Consultant of REAP Change Consultants, founded October, 2002. The Fourth of July is this week. Many become independent evaluation consultants because of the lure of freedom - freedom to choose projects, what you do, how you do it; freedom from the politics of organizations, bad bosses, and organizational hierarchies; freedom to set your own hours and work in your pajamas. It all sounds great. It’s all true, too - partly. Yes, you can choose which projects to take on but as Max A. Eggert and Elaine van der Zeil put it "Clients tend to be more rigorous and constraining than managers," and "Every client wants the right solution to his or her need without taking exceptional risks. Thus you have to have controlled creativity…."[1] Yes, work in your pajamas - but you also need to meet deadlines and juggle time for yourself, your family, loved ones, and friends. Try shutting down your computer and walking away at 5 p.m. when the final report is due tomorrow and you must get it done now because you spent time with your spouse earlier! Yes, you might be free of your own organizational politics and can walk away from those of other organizations once you finish a project. However, for good client relations, pay attention to the politics of your clients’ organizations! So what to do? Take hints from young America. Once the American Revolution resulted in independence, leaders of the young nation set about forming thirteen disparate states into the United States of America and created a new constitution. As you go independent you’ll figure out your own rules for operation, what you will do and how you’ll go about doing it. The young nation worked at developing allies and establishing markets for its goods and services. You’ll have to do that. Find business partners, learn from fellow consultants, explore and start marketing. Each of the states had others with whom to work. You do, too - AEA and the IC TIG. Go ahead. Declare independence. Just remember that bravery, responsibility, and hard work come with freedom. Like each of those who financed the revolution and founded this country, you are putting your fortune, fame and the welfare of your family on the line. Good luck! Hot Tips: Join the Independent Consulting TIG for community and advice. Take AEA training workshops. Rad Resources: Gail Barrington’s Consulting Start-up and Management: A Guide for Evaluators and Applied Researchers. Elaine Biech’a The Business of Consulting: The Basics and Beyond. Second Edition. [1] Max A. Eggert and Elaine van der Zeil (2011) The Perfect Consultant: All You Need to Get It Right First Time. The Perfect Series. Arrow Business Books, Random House Kindle edition.   The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Bernadette Wright, founder of Meaningful Evidence, LLC. As an entrepreneur, I noticed a proliferation of books, papers, courses, and other sources espousing conflicting ideas of how to succeed in business and entrepreneurship. For evaluators who are building independent consulting businesses, this makes finding the best ideas for our business difficult. In my and Dr. Steven E. Wallis’s recent research on entrepreneurship, we found a frequently mentioned problem was a lack of agreement on what entrepreneurship means. Some sources portray entrepreneur as those who start ventures, generally small businesses. Others present entrepreneurship as about being innovative, pursuing opportunity, or filling a need. Several studies discussed the need to integrate theories of entrepreneurship across disciplines and across levels of analysis (e.g. individual, organizational, and market levels). Studies have also recommended coordinating entrepreneurship theories across research, education, and practice. A 2006 U.S. Association for Small Business and Entrepreneurship (USASBE) white paper titled Are We Teaching Small Business Management to Entrepreneurs and Entrepreneurship to Small Business Managers? recommended making teaching more consistent with the literature. However, the fragmentation of theories makes choosing the best theories to teach and apply difficult. Integrative Propositional Analysis (IPA) is an emerging method to analyze and integrate theories across studies, strategic plans, and other documents. IPA draws on a long research stream related to systems thinking. This scholarship indicates we can improve a model by understanding its structure and making it more inter-connected. We can use IPA combined with empirical data and feedback from stakeholders to assess plans before we implement them. IPA is a six-step process. For details, see our white paper on "The Science of Conceptual Systems." Benefits for business leaders include: Lets you choose theories that are more likely to work in the real world (as compared to in the academic world) Gives you more "structured" knowledge, which research indicates increases managers’ chances of success Supports communication, collaboration, and shared understanding Avoids unanticipated consequences Rad Resource: "A Revolutionary Method to Advance Entrepreneurship Theories" by myself and Steven Wallis. In review, draft available upon request. Discusses our study on using IPA to integrate and improve entrepreneurship theories. Rad Resource: Casson, M. Entrepreneurship and the theory of the firm. Journal of Economic Behavior & Organization, 2005. This study creates an integrated theory combining insights from management and economics. Rad Resource: "Strategic Knowledge Mapping: The Co-Creation of Useful Knowledge," by Steven Wallis and myself. Developments in Business Simulation and Experiential Learning, 2015. Introduces a gamified version of IPA. Where do you find business/entrepreneurship information to consider for your business? How might your firm benefit from using theories that are measurably better than your competitors’? The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Susan Wolfe and I am a Senior Consultant at CNM Connect where I provide evaluation and other capacity building services to non-profit organizations. Two months ago I was the owner of Susan Wolfe and Associates, LLC, my own consulting firm; however, after five successful years I chose to close my doors and become an employee. People have asked why I made this choice to give up my independence. The main reason is that although I LOVED the work I was doing, I did not so much care for the business side of independent consulting. When I was offered the opportunity to join the non-profit CNM Connect as a consultant I easily accepted it without once looking back. Lesson Learned: Even if I am successful with something, if I don’t enjoy it then it’s OK to make a change when an opportunity that better fits my interests and needs. When I started as an independent consultant it fit well with my interests and needs, but things changed over time. When I was no longer enjoying my work, I allowed myself to be open to other possibilities, and because of this a more suitable situation presented itself. Throughout the past five years, I had a love-hate relationship with working alone at home. I liked the convenience of not commuting and having to dress up for work every day. I also liked being able to work with one of my dogs in my lap. On the other hand, after some time I started to feel the absence of co-workers. I missed belonging to an organization with other people but I knew I did not want to grow my consulting business. Hot Tip: If you like having colleagues that you work with daily while at the same time would like to be an independent consultant, consider forming a company or collective. Working at home alone can be isolating. Hot Tip: Networking with other evaluators, locally, statewide, and nationally, and teaming up for collaborative proposals and projects is one way to reduce your sense of isolation. I regularly scheduled lunch with colleagues or community partners, and ultimately teamed up with another local evaluator to form an informal local evaluator’s group that gathers bi-monthly for happy hours. Rad Resource: If you are thinking of transitioning from independent consulting to working for an organization, scan the AEA job listings to see what is available. It’s a great way to look around and see if there is something that would better fit your current needs and interests. The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Gail Vallance Barrington and I’m an independent consultant. It’s hard to teach novice clients about logic models. This year, instead of presenting a technical monologue, I came up with a story to capture their attention. It’s called My Rich Uncle and it goes like this: Icebreaker: You suddenly discover that you have a rich and eccentric uncle who lives far away and he wants to give you $30,000 to buy a car. Working in small groups, decide what your uncle’s name is and where he lives. To get the money, you will need to prove to him why you need the car. Use this worksheet and have fun coming up with the answers. [N.B. Do not provide the labels below until the exercise is over.] Activity: List three compelling reasons why you need the car and the difference it will make in your life. [Program Purpose, Long-term Outcomes] Your uncle wants to know that you are a good planner. What will you need to have in place before you get your car? List three or four important things you need. [Inputs] He wants to know how you will use your car on a regular basis. What will your main uses be? [Activities] Your uncle is pretty picky, and of course, he lives very far away. He wants to be sure that you actually spent the money on a car and that you actually use it. Every six months, he wants some concrete proof that you have been using it as planned. What evidence can you put in a brown paper envelope and mail to him? [Outputs] Finally, after three years, you suddenly hear from your uncle. He is coming for a visit! He wants you to show him what changes the car has made to your life. Can you start planning for it now? [Short- and Intermediate-Outcomes] Reflection: Once the groups have reported their answers, reflect on such topics as the group interaction, breakthrough thinking, general conclusions and the fact that everyone has just developed a logic model. Next Step: Launch immediately into developing the logic model for their program. Having set the stage in this relaxed and fun way, it is easy to make connections with their own context. Lessons Learned: I have used this exercise with novice clients, students and webinar participants. Every time the typical problem of differentiating between Outputs and Outcomes seems to melt away. Further, the accessible nature of the story provides clarity about a funder’s need for accountability. Next time you teach logic models, why don’t you try it? Let me know how it goes. The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi there! I am Ana García-Femenía, (@anagfemenia), PhD in Sociology and Political Sciences and independent evaluation consultant. I live in Spain and conduct evaluations for various international donors in different countries. Have you ever been in the middle of an interview in an evaluation and felt some specific emotion with no reason for it? What happens when you stop and ask yourself: Why do I feel this way? Over the years I have understood how important it is to take care of my emotional well-being, in order to do my job as evaluator. But one could say this holds true for any profession, doesn’t it? Well, yes it does. But it holds especially true for our profession where we as evaluators are confronted with constant decision-making requiring sound judgement and a special degree of calm, cool and collectedness. There is a list of situations and times when we need to exercise our best judgement not only during the evaluation process itself, but with clients and stakeholders in order to make the right decisions, to say ‘no’ when we need to, and to maintain our integrity. There are no purely rational decisions. Whether negative or positive, our emotions always let us know when we are or not on the right track. Lesson Learned: Make emotional care part of your self-care. Two Hot Tips to increase your emotional care: Meditating: for 15 minutes once a day. Meditation provides peace and serenity and opens your mind and your heart. It helps to improve your relationship with your emotions. It also provides a good preparation for full attention and deep listening. Dancing: A lot of stress and tension are relieved through dancing. Dancing helps to free our emotions. And it is a great way to socialize, which is very valuable for those of us who work alone! Rad Resources: A wonderful tool for the initiated in meditation and the uninitiated alike. It provides guided meditation ranging from 5 minutes to one hour. Online resources for dancing are huge. It also will depend on the music you like! Here are three links on the benefits of dancing for your emotional health: http://www.adta.org/ http://www.sciencedirect.com/science/article/pii/S0197455612001104 http://awakeningtothedance.com/2012/02/01/dancing-to-free-our-emotions/ And finally, for those interested, a beautiful video by one of the best world tango couples to get inspired! The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, I am Matt Feldmann, the principal researcher and owner of Goshen Education Consulting, Inc. and the chair-elect for the Independent Consulting TIG. My company focuses on educational evaluation and data support for clients in Southern Illinois. As an independent consultant, it is imperative to maintain a strong network of clients, colleagues, and advocates with whom you can grow and develop your business. The following are some of the "Soft skills" associated with working with clients that are frequently referenced in business literature. Lessons Learned: Form a business strategy and stick to it. Marriage of Skill, Enjoyment, and Resources. Jim Collins refers to this as the "flywheel concept" in his book, "Good to Great". After you have developed a well-considered business plan there is an additive effect to unyielding discipline to that plan. Learn more about the flywheel concept from Jim Collins here. Recognize you are in a service industry and focus on your client relationships. Harry Beckwith refocuses attention on the service industry in Selling the Invisible. The key point is that evaluation is not a commodity. From our client’s perspectives, our expertise with complicated evaluation approaches is secondary to our ability to communicate and relate the importance of our work. Jeffrey Gitomer says that you are more likely to receive your next consulting contract from an existing client in his book Little Red Book on Selling. Practice networking Karma. Recognize that your success is connected to the success of your network of clients, colleagues, and advocates. Your selfless work for others will return to you in unexpected ways. Keith Farazzi says you should not keep score with your networking relationships in his book Never Eat Alone; business development is not a zero sum game. Because we evaluation continues to be in a growth mode, evaluators should reach out to their competitors to learn from one another and to seek ways to develop cooperatively. Rad Resources: The following are the author websites for the four books referenced above and that provide excellent understanding for these and several more "soft skills." You can probably find these on audio CD, mp3, or download from your library and listen to them as you travel among your clients. Business Management - Good to Great by Jim Collins (see: http://www.jimcollins.com/article_topics/articles/good-to-great.html) Client Relationships - Selling the Invisible by Harry Beckwith (see: http://www.beckwithpartners.com/books/) Sales Strategy - Little Red Book of Selling by Jeffrey Gitomer (see: https://www.gitomer.com/Jeffrey-Gitomer-Little-Red-Book-of-Selling-pluLRB.html) Networking - Never Eat Alone by Keith Farazzi (see: http://keithferrazzi.com/products/never-eat-alone) The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! I’m Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor. It’s Independence Day in the US, the anniversary of the day our Declaration of Independence from Great Britain was signed in 1776. As I thought about what topic to tackle for this week’s post, I wondered what insights a word cloud created from the text of the Declaration of Independence might yield. Since I’m familiar with two word cloud generators, I tried each of them. Here is what I created in Worldle: And here is what I created in Tagxedo: Both generators allow for some creativity with regard to color, and font. Wordle allows you to choose directionality - whether you want all text horizontal, all vertical, or some mix of the two. Tagxedo is best known for its shapes, allowing you to choose from a bank of available shapes that can help you illustrate a point. Tagxedo can also use custom fonts from popular font sites. Lesson Learned: Word clouds can be fun and have a number of appropriate applications. Use them with caution, however, as they never substitute for analysis, and can potentially detract from important themes. Stuart Henderson and Eden Segal tackle word clouds in a chapter on qualitative data visualization in a recent issue of New Directions for Evaluation.  "Word clouds create a dramatic visual, which likely accounts for their popularity," the authors posit. "Despite concerns with word clouds, their ease of creation and striking visuals make them a useful tool for evaluators if they are used sparingly and their challenges are acknowledged" (p. 57-58).  Hot Tip: Henderson and Segal also mention another word cloud generator, TagCrowd, and I decided to give that one a try as well. Here is the result: While TagCrowd has fewer opportunities for creativity, it does give the user the option to see word counts after each word. I simply checked this option as I created this one. Rad Resources: A few other aea365 authors - Susan Kistler, Stacy Carruth, Sue Griffey, Jaquelyn Christensen, and Sarajoy Pond - have also shared their perspectives on word cloud generators. Did you know there is an advanced version of wordle? Learn about this and other Wordle tips, such as how to keep word phrases intact, and how to get numbers to show in Wordle here! Read the entire article Visualizing qualitative data in evaluation research, by Stuart Henderson and Eden Segal, to understand more about qualitative data visualization. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

We are Leah Neubauer, Dominica McBride and Anthony Heard of the Chicagoland Evaluation Association. Welcome to the Local Arrangements Working Group (LAWG) week. We look forward to Evaluation 2015 in the Windy City. In the Exemplary Evaluation conference year, our opening Chicago week offers issues relevant to our evaluation practice, environment, and policy. Today’s post notes 2015 Chicago happenings and themes for this week address Chicago people, politics, culturally responsive assessment, and evaluator learning. As CEA President and LAWG co-chair, I’m excited for our posts. Hot Tip: What’s Important to Know about Chicago? This is a great question with unlimited, potentially conflicting answers. Many might go on about sports, politics, corruption, weather, or traffic. Chicago celebrates and critiques. For many of us urban-entrenched evaluators, our Chicago-evaluation landscape demands a knowledge base that: 1) respects and challenges history, 2) keeps timely with rapidly changing, current/controversial events, and 3) ensures just implications for our city. One way to get "Chicago-ready" is to digest some news. A short highlight reel is shared below. This is not an exhaustive or representative 2015 list. It’s designed to be both serious and light-hearted. Ultimately, we hope that you find it aligned with the celebratory and critical nature of Chicago updates. The 2015 year started quite cold for the Midwest. January saw many days of Chicagoans and school policy-makers debate the merits of weather-related school closings and Governor Bruce Rauner, the first Republican Illinois governor in 12 years, took office. February marked the 5th largest snow in Chicago history. Freezing weather continued into March with the15th Annual Polar Plunge attended by locals, tourists, and celebrities. The annual event raised 1.2 million dollars for Special Olympics. Enough weather updates, right? April saw Chicago’s first ever mayoral run-off election between Jesús G. "Chuy" García and Rahm Emmanuel. Emmanuel won. Weeks later, thousands joined for the nationwide "Fight for $15" demonstrations. In May, the great King and Blues Legend B. King was honored, Chicago was selected for President Obama’s Library , and the Chicago Public Schools CEO resigned in the wake of a federal probe. June marked the Chicago Blackhawks securing The Stanley Cup , the 46th Annual Chicago Pride Parade, and the 20th anniversary of the Paseo Boricua Flags. And this weekend, fans gather for the Grateful Dead’s "Fare Thee Well" 50th anniversary at Soldier Field or head to Jackson Park for the Annual Chosen Few Old School Reunion Picnic. Hot Tip: What to Visit When You Come to Chicago. In addition to remaining Chicago news-updated, there are many other Chicago things to plan for. Check out events and museums . Be on the AEA lookout for the local guide! We’re looking forward to November and the Evaluation 2015 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hi, my name is Dominica McBride and I’m a member of the Chicagoland Evaluation Association and the Local Affiliate Working Group, along with being the CEO of Become, Inc., a nonprofit using culturally responsive program evaluation as a tool in realizing social justice. We’re excited to welcome you all to the Windy City in November! As you prepare to come to Chicago and connect with fellow evaluators, also prepare to learn from the city. This is both a wonderful and challenging place to be an evaluator. I want to take this time to share some of the recent local issues the city and state have come up against and for you think about and prepare to share your wisdom, talent, and skill in exploring solutions. Chicago is a city of many strengths and many problems. There is much wealth but also poverty, great diversity but also segregation. There are quality city services that don’t get allocated fairly throughout the area. There is also rich culture and history in all of its 77 neighborhoods. In doing evaluation in Chicago, I have come up against these problems but also been informed and inspired by the history, cultures, and my fellow residents. Some of the problems that I have witnessed in my evaluation work are the issues of quality education, employment, and violence (not just physical violence but the perception of violence). Even though we’re on a general upswing as far as history is concerned, this year, our movement towards equity has faced a barrier. On January 12, Bruce Rauner became governor of IL. Soon after he assumed office, he cut funding to programs, like youth development, violence prevention, and youth employment. These cuts drastically influenced some of our organizational partners, one having to serve the same amount of youth with less staff. These are all areas where evaluation can play a major role in making the case for reinstating and even increasing such funds. Lessons Learned: The political climate can have a profound influence on programs, participants, and evaluation. Be prepared for political shifts to influence your work, especially if you work with marginalized communities. This may include developing relationships with politicians or ensuring that you have the resources needed to complete your mission regardless of the politics. Hot Tip: As you plan your sessions and trip, look into Chicago and its history. Read about the assets and issues. Don’t let the media sway you one way or another. Rad Resources: Check out a February article about the budget cuts that were made Get a "taste of Chicago" at http://www.choosechicago.com/ We’re looking forward to November and the Evaluation 2015 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hello fellow culturally-responsive evaluation lovers! My name is Scott Tharp and I am the Associate Director for the Center for Intercultural Programs at DePaul University. It’s a simple fact that we live in a multicultural world where multiple values and experiences are present. While some might think evaluation is inherently "neutral", it never is. Social identities (e.g. race, gender, class), experiences, value systems, and beliefs will influence the assessment process in many ways. From how you word survey questions to the assumptions that inform your data analysis, social identities matter. Because not all people are the same, we must increase our cultural competence to ensure our assessment processes effective and respectful. To define it simply, cultural competence is the level of knowledge, awareness and skill you have to engage in effective cross-cultural interactions. Pope & Reynold’s wrote a great article back in 1997 entitled Student Affairs Core Competencies: Integrating Multicultural Awareness Knowledge and Skills. While they speak from their context of higher education, the insight applies to non-profits and corporations alike. They offer specific competencies related to each of the three domains for you to consider as you continue to enhance your culturally responsive evaluation practice. Because cultural competence in assessment begins with cultivating your own cultural competence, this is where we all must begin. This is a life-long process that takes time and dedication to self-work. Here are some useful questions to help you along the way. Hot Tip #1: Start with yourself. How knowledgeable are you about your own social identities beyond a label? Are you self-aware enough to articulate how your social identities influence your values and worldview? Do you have skills for critical self-reflection that you practice daily? Many people refer to this simply as your "self-work" Hot Tip #2: Learn About Others. How knowledgeable are you about social identities different from your own? Does your knowledge extend past material culture and into their values and histories? Are you self-aware enough to identify assumptions and biases towards others? Do you have the skills to challenge your own assumptions and biases before they become stereotypes? Hot Tip #3: Consider the Systemic. How knowledgeable are you about diversity and social justice concepts and the history of social identity categories? Are you aware of social inequalities around you, or do you dismiss them as "just the way things are?" Do you have the skills to address social inequality so that you do not replicate interactions that create harm to others? Taking time to increase our knowledge, awareness and skills related to ourselves, others, and the social environment we live in will sensitize us to social and cultural differences that will lead to better data, better conclusions, and better decisions. We’re looking forward to November and the Evaluation 2015 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hello, I am Matt Feldmann, the principal researcher and owner of Goshen Education Consulting, Inc. and a member of the Local Area Working Group. My company focuses on educational evaluation and data support for clients in Illinois, which also happens to include Chicago. J While I live about 4.5 hours south of Chicago, I am still a member of the Chicagoland Evaluation Association (CEA) because of my state-wide projects that often include Chicago Public Schools and districts in the suburbs of the city. Illinois is my home and the focus of my niche practice, a topic of a previous AEA365 post from October 2012. This post expands this on my experiences as a regional evaluator and ideas for how to continue to develop your evaluation practice. Lesson Learned: Focused geographical networking. I don’t readily take on contracts outside of my service area of Illinois…and I generally prefer projects in Southwestern Illinois. The goodwill created by my good work is more likely to result in new business when I focus my efforts on a distinct group of educational leaders who know each other. "Hit ‘em where they ain’t." This baseball quote attributed to Willie Keeler was Sam Walton’s strategy when he started Walmart. While Illinois is my broad service area, I am very focused on serving Southwestern Illinois, a community that is underserved by dedicated educational evaluators and that has more than $2 billion in annual educational expenditures. Get involved with your local affiliate (or start your own). Your local affiliate is the best resource for regular professional development, opportunities for partnerships, and (importantly) competitive intelligence. Through regular interaction with the Evaluation Association of St. Louis (EASL) and occasional programs with the Chicagoland Evaluation Association I have a wonderful understanding for my place in the local educational evaluation community. These professional colleagues have been instrumental in developing my regional strategy. For instance I won’t readily pursue projects in St. Louis or Chicago where there is more competition and I am not as familiar with the educational leaders.  Hot Tip: If you are focused on an underserved community of tightly networked potential clients that you intimately understand your growth may only be limited to your availability, marketing approaches, and the creativity for types of projects that best utilize your skill set. We’re looking forward to November and the Evaluation 2015 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hello! My name is Alicia Anderson and I am a student at The Chicago School of Professional Psychology in the Masters of Science in Applied Behavior Analysis Program. I am also a Leader Fellow with Become, Inc - a nonprofit team dedicated to community engagement, community involvement, and social justice. In the short time that I have been with Become, I have worked on projects consistent with Become’s core values of nourishing communities affected by poverty and injustice; most recently having the opportunity to assist with program evaluation and data collection for three Boys and Girls Clubs of Chicago (BGCC) sites in neighborhoods on the south side of Chicago in which economic disparity and resource gaps have been identified. My role as part of the evaluation team included disseminating surveys and conducting interviews and focus groups with students, teachers, and parents to assess whether students have access to the best programming possible; and whether any needs and deficits are or will be addressed. Lessons Learned: As a person new to evaluation and its methodologies, this has been an eye opening and humbling experience. Though evaluation plays a key role in determining the efficacy and impact of programing, it also teaches a lot about the drive and ingenuity of others. In places where there are resource gaps, where monetary compensation does not necessarily reflect effort and commitment, staff members work hard to ensure that students have a safe environment in which to learn, grow, and play after school. Students are engaged and learn important skills and constructs such as teamwork and respect. They receive help with their homework in addition to learning about being young men and women. They are given tools and encouraged to be their best as students, as children, and as people. Hot Tip: Be prepared to be inspired as you come to Evaluation 2015, where you will have contact with many people like those at Become, working with programs like tge BGCC through evaluation and seeing steps toward social progress. For me, building a framework for the future by reaching and motivating youth at present has shaped and molded my ideas and goals for the future. I want to be among the group of people who save lives every day and don’t realize it (teachers, school staff and administration, parents, after school program managers, etc.). I want to bring out the best in others. Rad Resources: A helpful book for working with and inspiring others: Bringing Out the Best in People by Aubrey C. Daniels. We’re looking forward to November and the Evaluation 2015 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

My name is Anthony Heard I’m a clinical social worker at the UI Hospital and Health Sciences System here in Chicago, and Secretary of the AEA’s local affiliate: the Chicagoland Evaluation Association (CEA). Today, I hope to encourage you to check out both your local affiliate and ours. Though I co-chair a department QI Committee, evaluation is not my primary role. I have been passionate about learning evaluation practice, building capacity, and implementing evaluations in the programs I’ve worked in since I completed my graduate degree, and our local affiliate has been an integral part of my evaluation journey. Lessons Learned: If your local affiliate resembles ours, it is likely to have a diverse membership: evaluators and evaluation boosters; students and degreed experts; internal and external evaluators; private consultants and organization-based evaluators. Members tend to come from many specific program areas such as public health, mental health, education, and more. Use this to your benefit. If you’re new or not sure where to start, like I was, there are many perspectives on evaluation to learn from, and make your understanding as rich and as accurate of the full spectrum as possible. If you’re a more seasoned evaluator, we each take similar lessons from our experience at gatherings. The knowledge and experience of other members can be useful when navigating common barriers to best evaluation practice: funding, (lack of) evaluation literacy, and identifying who your real stakeholders are. Hot Tips: Make contact: Affiliates can offer both networking and professional development, and will probably respond to the needs identified by members. Our affiliate is currently focused primarily on professional development, because that’s what our members wanted. Become a member: Not only are you supporting an organization of evaluators like you, but just like the AEA proper, your local affiliate likely has like-minded professionals actively engaged who will take note that you’re engaged in the community as well, and you can possibly be on the short list when opportunities come up. Get involved: Being a member is grand, but your local affiliate also provides the opportunity to show what you know and be a community leader. You can present your work, chair a committee, or run for office. Rad Resource: Stay in tune with the Chicagoland evaluation community leading up to Evaluation 2015 by checking out our website at www.evalchicago.org. We’ll have information available on Chicago-area evaluators who are presenting at Evaluation 2015 and local events closer to the conference. We’re looking forward to November and the Evaluation 2015 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hello AEA members! Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor with an important message about the AEA elections. The 2016 ballot is now open for our 2016 President-elect and Board Members-at-Large. AEA member voting has been historically low. Let’s change that this year! Hot Tip: Get to know your candidates and cast your vote! These candidates are accomplished leaders not only in AEA but in their own fields as well. Do your due diligence and give them each a few minutes of your time to learn about what they have to offer our organization. They’ve worked  hard to get themselves on the ballot through a rigorous nomination process that involved a great deal of time and effort. Once you click your personalized link in the email (all ballots are absolutely anonymous) you’ll find that the ballot includes extensive personal statements thoughtfully developed by each nominee to help you make informed selections. Many of us consider AEA to be our professional home and leadership is important to who we are as an association and what we have the potential to become. Rad Resources: Our 2016 nominees (in alphabetical order in each category): Image credit: tzu kwan valino via Flickr Candidates for President (Select 1): · Rakesh Mohan · Kathryn Newcomer Candidates for Member-at-Large (Select up to 3): · Thomas Chapel · Cindy Crusto · Huilan Krenn · Stephen Maack · Dominica McBride · Rebecca Woodland Please check your email from July 8 with information on the election and a link to your ballot. You may vote at any time between now and 11:59 PM Eastern Time on Friday, August 7. Reminder notices will be sent to those who have not cast their ballot by July 22. (Vote early to avoid the extra email!). Email info@eval.org or call the AEA office at any time with questions or concerns related to the ballot or any of your member benefits. Hot Tip: Play a role in this year’s election and make your voice heard. Let’s rock this vote! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Girija Kaimal and I am an Assistant Professor in the Department for Creative Arts Therapies at Drexel University. As an educator, evaluator, blogger and, artist, I’d like to share how my evaluation practice is informed by my artistic practice. The fields might seem unrelated but I think of art as metaphor. My colleagues and I published recently on how the arts can inform leadership practice (http://www.ijea.org/v15n4/). So then I wondered how the arts could inform evaluation as well. Lessons Learned: Tools: Each media option comes with its unique attributes. Oil pastels offer bright colors; watercolors require an absorbent base; felt-tip markers can provide detail but aren’t really useful if you want to cover large surfaces; and; if you need to erase and refine your work, then pencil or digital media are your best choices. Each media choice comes with its own set of strengths and challenges and I have to know these attributes to use the tool effectively. Choice of evaluation tools for data collection and analysis is no different. You might be skilled in a range of methods or you might be sought out for a specific specialized skill. Either ways knowing your tools is essential for artistic and/ or evaluation practice. Caring:If my paint brushes aren’t clean, my pencils not sharpened, paper not stacked and my supplies aren’t stored safely they will not be available or effective when I decide to use them. It is no different with evaluation tools. If my work files and software are not organized and saved safely, then neither my use of time nor my work will be efficient. Practice: Can I avoid doing art for months on end and then expect to be skilled when I decide to start drawing one fine day? No, like with any other skill, ongoing practice is essential to both sustain and improve skills in both artmaking and evaluation. Sharing: Artmaking is like visual journaling for me: it helps me think through problems and express complicated emotions and ideas. Sharing my work with others helps me see things that I did not or could not see on my own. It is no different in evaluation. I make it a point to share summary findings and/ or draft reports prior to any final submissions. Discovery: Starting a new project (in art or evaluation) is full of the promise of learning and discovery. At the end there is sometimes a thrilling insight or often just an incremental discovery. Regardless, each project’s process has meaning and relevance and offers lessons to be learned. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! I’m Matthew Von Hendy, owner of Green Heron Information Services, which provides research and information services to evaluation professionals.  I love helping connect people with information they need to solve problems or make decisions. A lot of time and effort goes into responding to Requests for Proposals (RFPs) so ideally you shouldn’t be spending precious hours just googling for RFPs opportunities. If you are searching for RFPs opportunities using Google make use of some tips and filters to make your searching as efficient as possible. Use double quotations for phrase searching. Employ the site filter (site:gov) to limit your searches to government websites since that is where you will find the majority of RFPs. Rad Resource: FedBizOpps is the central place to search for all government RFPs worth more than $25,000. It is a very complicated database; my suggestion is to use the advanced search function to get best results. The sheer number of state, county, tribal and other local government agencies makes tracking RFPs from these entities a truly daunting task. (Some estimates are there are approximately 70,000+ state and local government bodies in the US). While some of these agencies make their RFPs easy to find and search, many don’t. For this reason, paying a commercial RFP finding service may not be a bad idea depending on your needs. The commercial RFP finding service area is a crowded market place. You can find a lot of companies offering a range of services at differing price points. Some of the more well-known names in this area include Find RFP, The RFP Database, Bidsync, Government Bids.com, Bidnet, Onvia, Government Navigator, and Public Purchase. Each of the services have their strengths and drawbacks so it is worthwhile to explore which one might work best for your needs. Many government agencies have specific webpages where they post their RFPs. If you have a limited number of organizations that you are tracking for RFPs, software can help you save time by letting you know when any content is changed on a that page. Rad Resource: Change Detection is a free resource which can let you know when any content has been changed on a webpage. Organize the RFP notifications that you think might be good possibilities.   The organizational system could be a file or a folder whatever works for you. Just remember to look through it regularly as RFPs are time-sensitive. Citation managers that handle web documents and images can be good tools for this purpose as well. Zotero (https://www.zotero.org) is an excellent resource—easy to use and free. If you are interested in this topic, I will be offering my webinar on this topic twice again this year (late summer and December). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings! We are Rakesh Mohan, Director, and Lance McCleve, Principal Evaluator, of the Idaho legislature’s Office of Performance Evaluations. Recognizing the ongoing debate about evaluator credentialing/certification, here we discuss a study that demonstrates the value of credentials in evaluations but not necessarily credentialed evaluators. As the director, I manage high stakes evaluations. A poorly conducted evaluation could end my career and damage my office’s hard-earned credibility. For me, managing evaluations involves assembling the best team with the resources given. Hot Tip: The best team collectively has the total knowledge necessary to conduct and disseminate a credible evaluation. It includes evaluative thinking, ability to conceptualize complex issues, subject matter expertise, project management experience, research and analytical skills, political savvy, knowledge of evaluation standards, and communication skills. Depending on the type of evaluation, the team may benefit from having certain credentialed members. Lessons Learned: In 2008 the Idaho governor proposed a fuel tax to increase highway funding, but many legislators were not interested in raising taxes for a department that they thought was poorly managed. Through a resolution, the legislature asked us to evaluate the transportation department - a highly political assignment. By accepting this assignment, I took the biggest professional risk ever - I had no knowledge of transportation or engineering, the study had a big scope, and the report was due in six months. I contracted with a team of 11 consultants who brought expertise in transportation, engineering, construction, systems analysis, procurement and contract administration, capital project management, performance auditing, and performance measurement. This team included three licensed professional engineers and a certified project management professional. Additionally, I hired one consultant for financial analysis and two to provide quality assurance and help me manage the politics involved. Interestingly, no one on the team had the title of evaluator; the closest title was management auditor. The evaluation was a success—on time and under budget! We identified $30.6 million in potential savings the first five years and an additional $6.6 million annually thereafter contingent on implementation of our recommendations. The governor issued two executive orders: one required the department to implement our recommendations and the other established a task force to identify alternative sources of highway funding. The legislature passed bills raising $27.7 million for highways and appropriated about $10 million to implement the recommended asset management systems. Stakeholders found the evaluation useful because all parties saw it as accurate, thorough, and unbiased. They knew that I had hired highly qualified consultants to work on the evaluation. Clearly credentials have a place in evaluation but whether evaluators should be credentialed is not so clear. Hence, the discussion about the value of credentialing evaluators should continue. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Lisa R. Holliday, and I am a Data Architect with The Evaluation Group. One aspect of my job is to work with evaluation teams to create data collection tools and assess the quality of data received. I use data dictionaries as a way to establish quality expectations. Hot Tip 1: Create Data Dictionaries for All Projects Data dictionaries are used with databases as a way to maintain records about the data being utilized, including type, format, and information use. Data dictionaries can also be used for evaluations. I include the following areas in evaluation data dictionaries: Data Name Data Description Data Type Data Format Precision Acceptable Values Data Collection Cycle Data Collector Responsible Hot Tip 2: Define What’s Acceptable Be as specific as possible when it comes to what is acceptable. What is the data type (numeric, text, date)? How should data be formatted? For example, should dates be entered as mm/dd/yy or dd/mm/yyyy? For numeric data, how many decimal points do you want? If collecting names, what are acceptable values (first and last name, or first initial last name)? Also, are there any data that are required, such as identification numbers? Once you know what you want your data to look like, determine the minimum amount of non-conformance you can accept. For example, you might establish a threshold that 85% of data submitted must be formatted correctly, and 100% of required data must be submitted. Hot Tip 3: Enforce Data Quality Expectations Use your data dictionary to create your data collection tools. Enforce data type, format, precision, and acceptable values where possible. Also, provide instructions for data entry and training to data collectors. Hot Tip 4: Profile your Data Regularly At least twice a year, profile your data: how well do the data you collect align with the rules you specified? If you find that the data you are receiving aren’t meeting your expectations, consider modifications to the data collection tools you are using or the use of another collection method. Also, provide follow-up training to data collectors. Rad Resource: Data Cleaner This is a free data profiling tool that works with a variety of data sources, including MS Access and MS Excel, as well as numerous relational database management systems. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

What do one doctor, three nurses and an MBA have in common? More than you might think . . . We’re Pam Senesac, Anita Morris, Sai Cherala, Joan Johnston, and Ruth Aboagye, members of the Healthcare Innovation & Quality unit at UMass Medical School’s Center for Health Policy and Research, dedicated to individual and organizational Performance Improvement (PI) in our global efforts to evaluate our work. In a PI project, it’s tempting to jump ahead to "fixing the problem." STOP, take the scenic route, explore different avenues, meet the challenges head on, and enjoy the journey. The results will be a marked improvement and more sustainable. During our PI journeys, we’ve acquired some nuggets of wisdom we’d like to share with you. Hot Tips: Plan things out well from the start. Don’t be afraid to revisit and change plans if needed. Come to the table equipped with data. It’s key to tracking progress and measuring success. Take the time to research and think through your implementation plan. It makes monitoring and sustaining improvement effortless. Lessons Learned: When pondering a performance issue, use these 4 field-tested tools to get through the journey. Get the Facts: In today’s fast-paced environment, the impetus to problem-solve in real time sometimes inhibits long-term organizational process improvement. Gather as much data on your current process as possible; understand what you have and why you have it. This provides a good baseline and helps measure the impact of interventions. There is a metric for every process and outcome. Chart the Flow of the Process: A picture of the current process in all its glorious complexity helps everyone understand the process, identify any gaps or other issues, and informs your improvement plans. Use a Fishbone Diagram to brainstorm potential causes: A Fishbone helps describe the context. What is the environment? What is this process trying to achieve? What conditions impact or influence the condition being addressed? Why are we doing it? How do we put it together and stay true to our core values? With these questions in mind, list potential causes that may be contributing to the situation at hand. Get to the Root Cause, not just the first cause or pause, with Five Whys: Thoroughly understand why your system is the way it is. Armed with potential causes, ask WHY as many times as needed to help you identify the root cause and where to target your improvement intervention. You’re looking for the diagnosis, not the symptom. Finally, remember, it’s OKAY to fail. Knowing what doesn’t work hones your skills and focuses future efforts. Fail fast, learn from the experience, and move on. Rad Resource: Use this link to learn more about these 3 PI tools and many others. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.