Hello! I’m Matthew Von Hendy, owner of Green Heron Information Services, which provides research and information services to evaluation professionals.  I love helping connect people with information they need to solve problems or make decisions. A lot of time and effort goes into responding to Requests for Proposals (RFPs) so ideally you shouldn’t be spending precious hours just googling for RFPs opportunities. If you are searching for RFPs opportunities using Google make use of some tips and filters to make your searching as efficient as possible. Use double quotations for phrase searching. Employ the site filter (site:gov) to limit your searches to government websites since that is where you will find the majority of RFPs. Rad Resource: FedBizOpps is the central place to search for all government RFPs worth more than $25,000. It is a very complicated database; my suggestion is to use the advanced search function to get best results. The sheer number of state, county, tribal and other local government agencies makes tracking RFPs from these entities a truly daunting task. (Some estimates are there are approximately 70,000+ state and local government bodies in the US). While some of these agencies make their RFPs easy to find and search, many don’t. For this reason, paying a commercial RFP finding service may not be a bad idea depending on your needs. The commercial RFP finding service area is a crowded market place. You can find a lot of companies offering a range of services at differing price points. Some of the more well-known names in this area include Find RFP, The RFP Database, Bidsync, Government Bids.com, Bidnet, Onvia, Government Navigator, and Public Purchase. Each of the services have their strengths and drawbacks so it is worthwhile to explore which one might work best for your needs. Many government agencies have specific webpages where they post their RFPs. If you have a limited number of organizations that you are tracking for RFPs, software can help you save time by letting you know when any content is changed on a that page. Rad Resource: Change Detection is a free resource which can let you know when any content has been changed on a webpage. Organize the RFP notifications that you think might be good possibilities.   The organizational system could be a file or a folder whatever works for you. Just remember to look through it regularly as RFPs are time-sensitive. Citation managers that handle web documents and images can be good tools for this purpose as well. Zotero (https://www.zotero.org) is an excellent resource—easy to use and free. If you are interested in this topic, I will be offering my webinar on this topic twice again this year (late summer and December). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings! We are Rakesh Mohan, Director, and Lance McCleve, Principal Evaluator, of the Idaho legislature’s Office of Performance Evaluations. Recognizing the ongoing debate about evaluator credentialing/certification, here we discuss a study that demonstrates the value of credentials in evaluations but not necessarily credentialed evaluators. As the director, I manage high stakes evaluations. A poorly conducted evaluation could end my career and damage my office’s hard-earned credibility. For me, managing evaluations involves assembling the best team with the resources given. Hot Tip: The best team collectively has the total knowledge necessary to conduct and disseminate a credible evaluation. It includes evaluative thinking, ability to conceptualize complex issues, subject matter expertise, project management experience, research and analytical skills, political savvy, knowledge of evaluation standards, and communication skills. Depending on the type of evaluation, the team may benefit from having certain credentialed members. Lessons Learned: In 2008 the Idaho governor proposed a fuel tax to increase highway funding, but many legislators were not interested in raising taxes for a department that they thought was poorly managed. Through a resolution, the legislature asked us to evaluate the transportation department - a highly political assignment. By accepting this assignment, I took the biggest professional risk ever - I had no knowledge of transportation or engineering, the study had a big scope, and the report was due in six months. I contracted with a team of 11 consultants who brought expertise in transportation, engineering, construction, systems analysis, procurement and contract administration, capital project management, performance auditing, and performance measurement. This team included three licensed professional engineers and a certified project management professional. Additionally, I hired one consultant for financial analysis and two to provide quality assurance and help me manage the politics involved. Interestingly, no one on the team had the title of evaluator; the closest title was management auditor. The evaluation was a success—on time and under budget! We identified $30.6 million in potential savings the first five years and an additional $6.6 million annually thereafter contingent on implementation of our recommendations. The governor issued two executive orders: one required the department to implement our recommendations and the other established a task force to identify alternative sources of highway funding. The legislature passed bills raising $27.7 million for highways and appropriated about $10 million to implement the recommended asset management systems. Stakeholders found the evaluation useful because all parties saw it as accurate, thorough, and unbiased. They knew that I had hired highly qualified consultants to work on the evaluation. Clearly credentials have a place in evaluation but whether evaluators should be credentialed is not so clear. Hence, the discussion about the value of credentialing evaluators should continue. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Lisa R. Holliday, and I am a Data Architect with The Evaluation Group. One aspect of my job is to work with evaluation teams to create data collection tools and assess the quality of data received. I use data dictionaries as a way to establish quality expectations. Hot Tip 1: Create Data Dictionaries for All Projects Data dictionaries are used with databases as a way to maintain records about the data being utilized, including type, format, and information use. Data dictionaries can also be used for evaluations. I include the following areas in evaluation data dictionaries: Data Name Data Description Data Type Data Format Precision Acceptable Values Data Collection Cycle Data Collector Responsible Hot Tip 2: Define What’s Acceptable Be as specific as possible when it comes to what is acceptable. What is the data type (numeric, text, date)? How should data be formatted? For example, should dates be entered as mm/dd/yy or dd/mm/yyyy? For numeric data, how many decimal points do you want? If collecting names, what are acceptable values (first and last name, or first initial last name)? Also, are there any data that are required, such as identification numbers? Once you know what you want your data to look like, determine the minimum amount of non-conformance you can accept. For example, you might establish a threshold that 85% of data submitted must be formatted correctly, and 100% of required data must be submitted. Hot Tip 3: Enforce Data Quality Expectations Use your data dictionary to create your data collection tools. Enforce data type, format, precision, and acceptable values where possible. Also, provide instructions for data entry and training to data collectors. Hot Tip 4: Profile your Data Regularly At least twice a year, profile your data: how well do the data you collect align with the rules you specified? If you find that the data you are receiving aren’t meeting your expectations, consider modifications to the data collection tools you are using or the use of another collection method. Also, provide follow-up training to data collectors. Rad Resource: Data Cleaner This is a free data profiling tool that works with a variety of data sources, including MS Access and MS Excel, as well as numerous relational database management systems. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

What do one doctor, three nurses and an MBA have in common? More than you might think . . . We’re Pam Senesac, Anita Morris, Sai Cherala, Joan Johnston, and Ruth Aboagye, members of the Healthcare Innovation & Quality unit at UMass Medical School’s Center for Health Policy and Research, dedicated to individual and organizational Performance Improvement (PI) in our global efforts to evaluate our work. In a PI project, it’s tempting to jump ahead to "fixing the problem." STOP, take the scenic route, explore different avenues, meet the challenges head on, and enjoy the journey. The results will be a marked improvement and more sustainable. During our PI journeys, we’ve acquired some nuggets of wisdom we’d like to share with you. Hot Tips: Plan things out well from the start. Don’t be afraid to revisit and change plans if needed. Come to the table equipped with data. It’s key to tracking progress and measuring success. Take the time to research and think through your implementation plan. It makes monitoring and sustaining improvement effortless. Lessons Learned: When pondering a performance issue, use these 4 field-tested tools to get through the journey. Get the Facts: In today’s fast-paced environment, the impetus to problem-solve in real time sometimes inhibits long-term organizational process improvement. Gather as much data on your current process as possible; understand what you have and why you have it. This provides a good baseline and helps measure the impact of interventions. There is a metric for every process and outcome. Chart the Flow of the Process: A picture of the current process in all its glorious complexity helps everyone understand the process, identify any gaps or other issues, and informs your improvement plans. Use a Fishbone Diagram to brainstorm potential causes: A Fishbone helps describe the context. What is the environment? What is this process trying to achieve? What conditions impact or influence the condition being addressed? Why are we doing it? How do we put it together and stay true to our core values? With these questions in mind, list potential causes that may be contributing to the situation at hand. Get to the Root Cause, not just the first cause or pause, with Five Whys: Thoroughly understand why your system is the way it is. Armed with potential causes, ask WHY as many times as needed to help you identify the root cause and where to target your improvement intervention. You’re looking for the diagnosis, not the symptom. Finally, remember, it’s OKAY to fail. Knowing what doesn’t work hones your skills and focuses future efforts. Fail fast, learn from the experience, and move on. Rad Resource: Use this link to learn more about these 3 PI tools and many others. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! We are Bill Trochim, of Cornell University, and Arthur Blank, of the Albert Einstein College of Medicine. We are the Chair and Program Chair, respectively of the Translational Research Evaluation (TRE) topical interest group. There is a growing recognition in many fields that the problems associated with the translation of research to practice are among the most important and costly of our modern era and that our society needs to address these issues. Many U.S. federal agencies such as the Centers for Disease Control and Prevention (CDC) and the National Institutes of Health (NIH) have been mounting a variety of efforts to enhance research translation and address major translational barriers. For instance, in 2006 the NIH started the Clinical and Translational Science Awards (CTSAs), one of the largest programs at the NIH. Administered by the newly formed National Center for Advancing Translational Science (NCATS), the CTSAs now encompass a network of 62 "hub" organizations (academic medical centers, medical schools, community organizations, etc.) in a national research-practice network. In the past year a variety of AEA members joined together to start the Translational Research Evaluation (TRE) Topical Interest Group. The purpose of the TIG is to provide a community for all evaluators interested in the evaluation of translational research initiatives to enable them to share the specific and unique challenges they face in this evaluative endeavor. The TIG provides a forum for addressing all aspects of evaluation related to clinical and translational sciences including (but not limited to) education, frameworks and models, innovative applications, novel methods, data collection techniques and research designs. This TIG will offer its members - evaluators, practitioners, program managers and other stakeholders - an opportunity to share mutual interests, evaluation expertise, resources and materials. The over-arching goal of the TIG is to explore current, state-of-the-art evaluation approaches and applications, foster communication among TR evaluators and provide opportunities to discuss existing and emerging techniques to evaluate translational research efforts. Furthermore, this TIG will encourage its members to identify and disseminate successful strategies to overcome challenges associated with translational research evaluation. Rad Resource: The TR TIG welcome professionals and evaluators looking to connect practice with research. Check out our TIG page and see if you’d like to become a member. We also look forward to seeing you at our TIG-sponsored sessions at Evaluation 2015 in Chicago! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Happy Saturday everyone!  Liz Zadnik here to start the weekend off with a recent struggle of mine as a practitioner and evaluation enthusiast. For a few years now I have been honored to be conferred with and asked to speak on evaluation within the anti-sexual violence movement.  I also have had the chance to help curate this blog and communicate with all of you, the blog’s brilliant readership.  I have worked with talented and experienced evaluators who have described me as "a collaborator" and "coauthor." Truth be told though, I quietly wait for someone come along and finally say, "Hey!  You don’t belong here!" I heard of "Imposter Syndrome" a years ago;  I didn’t want it to resonate quite as much as it did.  For folks unfamiliar, Imposter Syndrome is the individual belief that accomplishments, jobs, and/or reputation are not due to intellectual ability or skill, but to luck.  While it doesn’t discriminate, it does tend to impact women professionals more than their male peers. ‘Luck, Error and Charm’, Hannah Kent at TEDxLittleLonsdaleStWomen, 1 December 2012 from Emily Hehir on Vimeo. Hannah Kent’s TEDxLittleLonsdaleStWomen talk sounds like my own.  Her experience sounded like mine; professional achievements and academic honors, but still questioning whether or not she earned them.  The successful women she mentions tend to attribute their honors, accomplishments, reputation "to luck, to being in the right place at the right time, to factors other than ability.  They live in fear that eventually some significant person will discover that they are, indeed, intellectual impostors." Lesson Learned: It’s OK to not know it all.  Learning something new is fun and exciting and keeps work interesting!  I try to tell myself, How will I ever get better/stronger/more confident if I don’t try something new or ask questions?  I have come to see my hesitations and second-guessing as anchors, keeping me in the same place so they can always be right.  Learning new things and connecting with more knowledgeable and experienced evaluators is the only way I can move forward. Hot Tip: Find your power position.  For me, it’s standing with my feet hip-distance apart with my hands clapped in front of my chest.  I feel like I am honoring myself and everyone else in the room.  It also helps me become present and remember I deserve the respect of the people in the room, just as they deserve mine.  I am hear to share and learn. Have you ever felt like a fraud or imposter?  What are some of the ways you help yourself get back to the present? Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Maryann Durland, an independent evaluator and Social Network Analysis (SNA) practitioner. In this post I will address the requirements for doing an SNA application, particularly in evaluation, and which we could also call the standard for an application. I will use the early literature that formed and grounded SNA thinking and that continues to be relevant. Early on in the history of defining SNA, Linton C. Freeman, described four requirements for completing a social network analysis in his book, The Development of Social Network Analysis: A Study in the Sociology of Science: A structural perspective Empirical data Graphics Mathematical models with analysis I believe and promote, particularly in evaluation applications, that these are still the requirements for meeting the standard for doing SNA. Evaluations using SNA are distinct from research on SNA theories and measures, which may have different requirements. In evaluation applications structural perspective means that we can define relationships within the program and these relationships create a structure through which information flows, resources are found, barriers are identified, spaces are found that need connections, and so on. Data is the existence or non-existence of a relationship between two elements. Empirical data refers to verifiable data collected on the relationship between any two elements, also called the members of a set. Just like traditional data collection, we collect relational data through a variety of methods from observations to surveys about experiences. The data we collect populates matrix cells, indicating the presence or degree of a relationship between two members. Graphics indicate that we can visualize the network, results and/or analysis in graphs such as sociograms. Mathematical models with analysis allow us to calculate SNA measures which are measures of the network, not attributes assigned to individuals. Models called algorithms or a set of procedures, are as much a description of the relationship as they are the algorithm for how to calculate a measure. Lesson Learned: Clearly, these four requirements delineate a specific methodological basis that is different from traditional quantitative and qualitative analysis. These requirements mean evaluators must think differently, ask questions for a different purpose, and conceptualize an evaluation differently. Rad Resource: Early literature on SNA sought to develop what we could call standards for applications and one of the most important resources is the work of Linton C. Freeman. Freeman’s work continues to set the standards for SNA applications and the reference for the requirements. The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Bethany Laursen, an independent consultant and evaluation specialist for several units at the University of Wisconsin. I fell in love with social network analysis (SNA) as a graduate student because SNA gave me words and pictures to describe how I think. Many evaluators can relate to my story, but one of the challenges to using SNA in evaluation is identifying what counts as a "good" network structure. Hot Tip: Identify keywords in your evaluation question(s) that tell you what counts as a "good" outcome or goal for your evaluand. An example evaluation question might be: "How can we improve the collaboration capacity of our coalition?" The stated goal is "collaboration capacity." Hot Tip: Use published literature to specify which network structure(s) promote that goal. Social networks are complex, requiring rigorous research to understand which functions emerge from different structures. It would be unethical for an evaluator to guess. Fortunately, a lot has been published in journals and in gray and white papers, Continuing our example, we need to research what kinds of networks foster "collaboration capacity" so we can compare our coalition’s network to this standard and find areas for improvement. You may find a robust definition of "collaboration capacity" in the literature, but if you don’t, you will have to specify what "collaboration capacity" looks like in your coalition. Perhaps you settle on "timely exchange of resources." Now, what does the literature say about which kinds of networks promote "timely exchange of resources"? Although many social network theories cut across subject domains, it’s best to start with your subject domain (e.g. coalitions) to help ensure assumptions and definitions mesh with your evaluand. Review papers are an excellent resource. Lesson Learned: Although a lot has been published, many gaps remain. Sometimes the SNA literature may not be clear about which kinds of networks promote the goals you’ve identified for your evaluand. In this case, you can either 1) do some scholarship to synthesize the literature and argue for such a standard, or 2) go back to your evaluation question and redefine the goal in narrower terms that are described in the literature. In our example, the literature on coalition networks may not have reached consensus about which types of networks promote timely exchange of resources. But perhaps reviews have been published on which types of brokers foster diversity in coalitions. You can either 1) synthesize the coalition literature to create a rigorous standard for "timely exchange of resources," or 2) reframe the overall evaluation question as, "How can brokers improve diversity in our coalition’s network?" Rad Resources: This white paper clearly describes network structures that promote different types of conversations in social media This short webinar reports a meta-synthesis of which networks promote adaptive co-management capacity at different stages of the adaptive cycle Different network structures promote different system functions. This is the take home slide from the Rad Resource on ACM capacity. In this case, the evaluand’s network goal is timely social learning, collective action, and resilience. The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m John Burrett, of Haiku Analytics Inc., Ottawa. One serious problem with logic models is that they usually leave out external influences and feedback effects, even when they may be important, because they make the model "too complex". It is good to simplify, but ignoring important influences on program success when planning an evaluation may lead evaluators to fail to collect important data and to misinterpret results. Trying to embrace complexity by drawing a web of boxes and arrows is not helpful: it’s too complex to use and explain and will drive your audience away. This will probably come only from the mind of the evaluator or program manager, thereby easily missing important external influences and other complexities. Hot Tip: I recently stumbled onto an alternative approach during a mapping of factors of cause and effect related to a complex policy problem. Data was obtained from an expert panel, developing a matrix linking a number of factors with an estimate of strength and direction of relationship between them. Mapping this with network analysis software helped the panel to visualize what they had created. It followed that this form of data could generate outcomes chains and logic models. Here’s a simple example: a program supporting trades training by providing grants to students and developing state of the art teaching materials in collaboration with trade schools drives the immediate outcomes of… Students gaining the ability to take training and Currency and quality of the training being improved, in order to achieve The ultimate outcome of increased employment. Exogenous effects influencing these results include cost of living, demand for skills and technical changes affecting the training’s currency. The size of the nodes indicates betweenness centrality, identifying those factors that connect many influences, thus propagating certain effects. The width of the edges indicates the hypothesized strength of influence. Possible unintended effects and a feedback loop are also shown. Lesson Learned: A key advantage of this approach is that that it creates a logic model using expert knowledge, rather than simply an evaluator/manager’s understanding of a program. This could also include other sources of information like findings from literature and program stakeholders’ experiences. Importantly, you could do this without imposing any prior idea of the logic model on those providing the cause-effect data other than including the program/outputs/activities and specifying the immediate/intermediate and ultimate intended outcomes. A second major advantage is that the logic model utilizes network metrics generated from the data, so how the program and influences are expected to be related can be analyzed. For instance, factors that are thought to have an important role in propagating effects across the system would show high betweenness/eigenvector centralities. The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Rebecca Woodland, an Associate Professor of Educational Leadership at UMass Amherst. If there is one thing that I know for certain it’s that relationships matter and how we are connected influences the quality and outcomes of our shared endeavors. Social Network Analysis (SNA) has had a profound influence on my evaluation work. I want to introduce and encourage evaluators (who may not know much about SNA) to consider integrating it into their own practice. Simply put, SNA is all about telling the story of how "ties" between people or groups form, and how these "links" may influence important program objectives and outcomes. With SNA you can mathematically describe and visually see connections between people. You can use SNA to explain and predict how ties between "actors" influence the attainment of program goals. Hot Tips: Evaluators can use SNA to address a wide-range of pressing program evaluation questions such as these: Want to know whether a program has the capacity to spread a new or novel intervention? SNA was used to evaluate school-level capacity to support or constrain instructional innovation. Want to know how large, inter-agency partnerships develop and how inter-agency collaboration correlates with intended program outcomes? Evaluators used SNA to track the development and impact of a Safe Schools/Healthy Students inter-agency community mental health network. Want to know who influences the budgeting and disbursement of funds for advocacy programs in fragile environments? SNA was used to map the flow of resources and funding patterns for new-born survival activities in northern Nigeria. Lesson Learned: Possibly the biggest wow factor is that SNA enables the creation of illustrative visuals that display complex information, such as intra-organizational communication flow and the location of network "brokers," "hubs," "isolates" and "cliques", in user-friendly ways. Image via under Creative Commons Attribution 3.0 License Rad Resources ®Visualyzer is an easy to use program (with a 30-day free trial) that enables you to create socio-grams on any network of interest to you. The SAGE Handbook of Social Network Analysis. An outstanding guide to all aspects of SNA, including Chapter 38 which provides a comprehensive overview of SNA data collection and analysis software. The International Network of SNA (INSNA) has links to journals and to topical areas of interest. The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.