I’m Lisle Hites, Chair of the Needs Assessment TIG and Director of the Evaluation and Assessment Unit (EAU) at the University of Alabama at Birmingham (UAB). Today’s posting is about the use of data visualization to enhance your needs assessment. Recently, my team worked with a state agency to help them identify potential sites for a pre-k development initiative. We used ArcGIS 10.2 Geographic Information Systems (GIS) technology to geocode and map all child care centers and grant applicants within the state. In turn, these data were displayed on an interactive, web-based map using ESRI’s ArcOnline platform. Supplemental data regarding percentage of people in poverty were added to the map to enhance the decision making process for policy makers (American Community Survey Census). Displaying these multiple sets of data visually allowed state representatives to see the highest concentrations of four year olds in the state as well as potential gaps in service coverage by existing pre-k programs. In other words, these data were used to reduce the potential for duplication of services and to identify areas of greatest need. Lessons Learned: Needs assessments can be conducted in a variety of ways using existing data in new and innovative ways. While state representatives had ideas of what they wanted to know, data visualization led them to refine their questions and identify additional sources of information to support their "data-driven" decision. Hardcopy paper maps of each county did not provide enough geographic detail of childcare facilities. To maximize the large amount of disparate data, an online interactive mapping platform was critical to the success of this project. Rad Resources: ArcGIS Online (n.d.). The mapping platform for your organization. ESRI. ArcNews (2013, Summer). ArcGIS 10.2 brings transformational capabilities to users. ESRI. Azzam, T., & Robinson, D. (2013). GIS in evaluation: Utilizing the power of geographic information systems to represent evaluation data. American Journal of Evaluation, 34(2), 207-224. doi: 10.1177/1098214012461710 Evergreen, S. (2013). Presenting data effectively: Community your findings for maximum impact. NY: Sage Publications. United States Census Bureau (2015). American community census. The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

Editor's Note:  This post originally appeared on Social Security Matters blog, the official blog of the Social Security Administration. We’re turning 80 this year, and boy, are we proud! We’re proud of our history, and we proudly remain committed to serving America, especially our most vulnerable people, including retirees, children, and individuals with disabilities. We’ll be celebrating this milestone by declaring July 19 through 25 as National my Social Security Week. ...

I am Ryan Watkins from George Washington University. Needs have been described and defined in many different ways of the years (see the December issue of New Directions in Evaluation for elaboration). One consequence of this often perplexing medley of definitions is that the word need has lost much of its meaning. Here I will try to help clarify some important relationships with a little additional precision to our language around needs we can greatly improve our results. Lesson Learned: When conducting a needs assessment… Differentiate Needs from Solutions. It is easy to get tangled up in the distinction between needs and solutions to needs. Don’t confuse what you want to accomplished (closing needs) with the activities and resources used to achieve those results (such as, homeless shelters, mobile phones, and even money). Use Need as a Noun, not as a Verb. You do NOT need to buy a new car. Nor do they need Internet access. These are options that may (or may not) help satisfy needs. Yet, by using need as a verb (or in a verb sense) we commit ourselves to one solution (a car, or Internet access) before we define the need. Rather, use need as a noun (50% reduction in gender-based violence this year) so that you have a basis for comparing potential solutions and guiding decisions.    Don’t Confuse Needs and Wants. Really strong wants or desires are frequently elevated to the status of needs through our choice of words. Just ask any 3-year-old in a toy store if they want or need a shiny new toy. Don’t confuse the two. Expand to Include Individual, Group, and Societal Needs. Maslow’s Hierarchy of Needs popularized the concept that needs are individual. Nevertheless, groups (such as, teams, organizations, and institutions) have needs, as do societies.   Quality needs assessments recognize and align needs across three levels. Balance Needs and Felt-Needs. Felt-needs are often described as those perceived by the community rather than defined by an external expert. Both views on needs can be valuable. Recognize however that while people have perceptions of needs, their perceptions may not be an accurate reflection of reality. News reports, for example, may distort peoples’ perceptions on crime rates in a city. Therefore, integrate externally verifiable measures of needs in all assessments. Rad Resource: This blog is based on: Watkins, R., & Kavale, J. (2014). Needs: Defining what you are assessing. In J. W. Altschuld & R. Watkins (Eds.), Needs assessment: Trends and a view toward the future. New Directions for Evaluation, 144, 19-31. The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Managing multiple social media channels for your business or personal use can be difficult because each social channel is on a separate site. Managing these sites can take a lot of time out of your day, that’s why I use Hootsuite to manage AEA’s social channels, including Twitter and Facebook. Hootsuite is a social media management tool that helps you monitor your social channels and track what people are saying in the field. I have compiled a few ways you can benefit from Hootsuite! Rad Resource: Monitor multiple channels in one place The best feature of Hootsuite is that it allows you to manage multiple social media streams on one dashboard.  The tool allows you to view: Twitter Facebook Profiles, Events, Groups, and Pages LinkedIn Profiles, Pages, and Groups Google+ Pages (currently not personal profiles) Foursquare You can post and monitor your social media pages all from this one tool. You can even post the same content across multiple platforms. However be careful here—your Facebook fans and Twitter followers may have different needs. Also, Twitter only allows 140 characters whereas Facebook allows much longer and richer posts with photos and videos. Rad Resource: Schedule Posts The scheduling feature on Hootsuite is very beneficial especially for the busy professional who still wants to have a presence on real time social conversations. Hootsuite allows you to determine the time, date, and channel for your post. We recommend not posting too far in advance in order to stay relevant with your followers. Hot Tool: Customize your dashboard Hootsuite allows you to customize the information you see about each of your social media channels.  For example, if you add your Twitter account to Hootsuite, you can customize the dashboard to view your newsfeed, mentions from other twitter users, your tweets, new followers, retweets, scheduled tweets, and the list goes on. This allows you to see all the pieces of information that are truly relevant to your needs. Rad Resource: Monitor topics and hashtags In addition to creating streams for your social media channels, you can create streams for keywords and hashtags which allow you to follow conversations in the field. By simply choosing "add stream" then select "search" or "keywords" you can enter keywords, phrases, or popular hashtags. Follow words such as evaluation, eval, data visualization, or #dataviz. Hootsuite will show you all of the tweets and posts related to this theme or topic. This is a great way to stay on top of the latest conversations in the field. Click here to learn more about getting started on Hootsuite. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

We all know that the Americans with Disabilities Act makes its unlawful for an employer to discriminate against an individual on the basis of his or her disability. But, the Act has even broader protections for employees. The ADA expressly prohibits "excluding or otherwise denying equal jobs or benefits to a qualified individual because of the known disability of an individual with whom the qualified individual is known to have a relationship or association." That’s one type of of associational discrimination, the most recent allegation of which came in a lawsuit the EEOC filed last week against a medical facility. The...

We are Zhao Min, Deputy Director of the Asia-Pacific Finance and Development Institute (AFDI) in Shanghai, China, and Ningquin Wu, Coordinator at AFDI. AFDI is a member of the CLEAR Initiative (Centers for Learning on Evaluation and Results) and hosts the East Asia CLEAR Center. CLEAR promotes evaluation capacity building in regional centers across the globe. This week’s blogs are by CLEAR members. Effective governance and spending is important in a city like Shanghai, with a population of 24 million residents. To that end, in 2011, the Shanghai government established a performance-based budgeting (PBB) system. Performance budgeting aims to improve the effectiveness and efficiency of public expenditure by linking the funding of public sector organizations to the results they deliver. It uses systematic performance information (indicators, evaluations, program costings, etc.) to make this link. The effect of performance budgeting may be felt in improved prioritization of expenditure, and in improved service effectiveness and/or efficiency. Our partners at the Shanghai Municipal Finance Bureau (the Bureau) are working to establish a sound management approach in applying performance-based management concepts to their public budgeting framework. We’ve been researching their PBB efforts. Here are some lessons from their PBB experiences, useful to others who are working on using performance to better manage budgets and spending. The key ingredients in setting up Shanghai’s PBB system are below, which drew on advanced theory and international practices of PBB management. A strong framework for running the performance-based budgeting system Sound legal systems in place Signing of performance contracts by public managers to improve accountability Public managers having sufficient discretion to achieve results Use of accrual accounting to do comprehensive accounting of government costs Institutional arrangements and manpower There were many monitoring and evaluation elements that were also part of the PBB arrangements, described below. An ex-ante budget review system was established, with an institutional framework for the evaluation of earmarked fiscal funds. A mid-term monitoring system of fiscal budget funded projects was established. Ex-post evaluations are planned for all public departments. Self-evaluations are being phased in among budget departments The application of evaluation findings to budget management has been enhanced. Additionally, the Bureau is implementing these processes: Releasing of evaluation findings to all public departments Providing recommendations for improvement on performance related problems Monitoring the implementation of action plans by public departments  Rad Resources: Learn more about performance-based budgeting and how to do it with these resources. Performance-based Budgeting Manual Program Classification for Performance-Based Budgeting: How to Structure Budgets to Enable the Use of Evidence (also available in Spanish). The American Evaluation Association is celebrating Centers for Learning on Evaluation and Results (CLEAR) week. The contributions all this week to aea365 come from members of CLEAR. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

  This morning I had a great conversation with a group of people who I consider to be recruiting masterminds. One person had worked for a national, venture funded job board for the previous decade. Another has run a staffing agency for two decades. Another had helped create the employment brands for some of the nation’s most respected places to work. Needless to say, it was a diverse collection of people with unique perspectives on today’s recruiting landscape. The conversation could be summarized to one talking point: workplace culture shouldn’t be a crapshoot for today’s recruiters. The former job board...

I’m Neha Sharma from the CLEAR Global Hub at the World Bank’s Independent Evaluation Group. A key Hub role involves facilitating learning and sharing knowledge about evaluation capacity development. So I often think about how people learn. In this context, I’ve been reading a lot of behavioral science literature, and reflecting on what makes people learn to change behaviors. Richard Thaler, University of Chicago Economist and Behavior Science Professor, recently wrote about how he changed his class’s grading scheme to minimize student complaints about "low" grades when he administered difficult tests (to get higher dispersion of grades to identify "star" students).  His trick was to change the denominator in the grading scheme from 100 to 137, meaning that the average student now scored in the 90s and not in the 70s. He achieved his desired results: high dispersion of grades and no student complaints about "low" grades! Thaler’s blog made me wonder what effect this change in grading scheme had on student learning and the lessons it carried for communicating tough evaluation results. The relationship between performance and learning holds critical lessons for evaluators - does a 70 disguised as a 90 have an effect on learning? Like classroom tests, evaluations that are seen as overly harsh or critical are often questioned and lessons are underused by the evaluated agency. This doesn’t mean that poor results should not be communicated - they absolutely should - but evaluators need to keep in mind that receiving and then learning from bad performance is not easy when there is a lot at stake - future funding, jobs, professional growth, and political stability. On the other hand, evaluations that reaffirm stakeholder-biases are futile too. This balance between communicating actual performance and encouraging learning may be key to determining evaluation use. If evaluations are to fulfill their learning mission the "how to" learn is just as, if not more, relevant as the evaluation itself. Cognitive science research about behavior change could teach us a lot about how to encourage learning through evaluations. For instance, we see that easy is better than complicated, attractive is better than dull, and social is better rather than teaching in isolation when trying to change behaviors. Behavior science is an interesting field of study for evaluators - to help us demystify the relationship between evaluation performance and learning. Rad Resources: Thaler is one of many behavioral scientists (and psychologists, economists) writing about what influences our behavior. Here are more. Nudge: Improving Decisions About Health, Wealth, and Happiness Thinking Fast and Slow Scarcity: Why Having Too Little Means So Much Dan Ariely’s website and research The Paradox of Choice Behavioral Insights Team (UK Nudge Unit) The American Evaluation Association is celebrating Centers for Learning on Evaluation and Results (CLEAR) week. The contributions all this week to aea365 come from members of CLEAR. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

                         July 26th marks the 25th anniversary of the Americans with Disabilities Act of 1990 (ADA) - a comprehensive civil rights law for people with disabilities. The ADA is often called the Emancipation Proclamation for people with disabilities because it ensures that people with disabilities, like all Americans, have "equality of opportunity, full participation, independent living, and economic self sufficiency." The Act prohibits discrimination against people with disabilities in employment (Title I), in public services (Title II), in public accommodations...

I’m Urmy Shukla, Capacity Building Manager at the CLEAR South Asia Regional Center, hosted by J-PAL South Asia at the Institute for Financial Management and Research. Since our 2011 start with CLEAR we’ve developed a wide-range of activities aimed at improving monitoring and evaluation (M&E) capacity throughout the region, including 90 trainings for partners such as the Indian civil services, state governments, NGOs, donor agencies, and academic institutions. Each training requires a significant amount of planning and preparation, including a needs assessment to assess skills and the partners’ role in evaluation, the development of customized content, and delivering the course itself. As such, we want to ensure that are trainings are meeting their objectives.  How do we know if our trainings are ‘working’? As evaluators, we know that there are several steps to plan for, and later assess, effectiveness of our activities. Most importantly, we need to: define a theory of change and/or results framework for program activities, focusing on desired outcomes measure/assess the desired outcomes For evaluation capacity development, these aren’t always easy to design and implement. But we’re taking several steps to assess the effectiveness of our trainings, including developing an organization-specific results framework and tracer surveys to track past training participants. We’re testing our approach as we’re going, and below are sharing some practical and strategic tips. Hot Tips: For training tracer studies: Clearly define training objectives from the outset. These objectives should go beyond skills gained, but should also include what you hope the participants will do after the training, within what is reasonably feasible during that timeline. Develop a way to systematically organize your multiple objectives. This will make it easier for you to design future tracer surveys and needs assessments. We categorize our objectives by (a) partner type (those who either do evaluations, use evaluations for decision-making, fund evaluations, and/or commission evaluations) and (b) knowledge, attitude, or behavior (KAB). From this, we have developed a database of tracer survey questions, which can be easily filtered for each type of training. Get partner buy-in early. Getting people to participate in a tracer study a year or two after the training can be hard, so give advance notice at the training that a tracer study will occur. Then have some contact with trainees - through newsletters, announcements, listservs - after the training to keep contact info current and so they remain familiar with you. Rad Resources: While not M&E focused, this Helvetas Measuring Education’s Path to Prosperity: A practical Toolkit for VET Tracer Studies guide is a useful resource for tracer study planning. Check out our own Interactive Course Manual: A Manual on Organizing and Running a Successful Training Course on Evaluation Methods. The American Evaluation Association is celebrating Centers for Learning on Evaluation and Results (CLEAR) week. The contributions all this week to aea365 come from members of CLEAR. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Lycia Lima, the executive coordinator of the newest CLEAR center- for Brazil and Lusophone (Portuguese-speaking) Africa. We’re formally joining CLEAR later this year and are planning our inauguration in October 2015. I was also one of the organizers involved in the formation of the Brazilian M&E Network - Rede Brasileira de Monitoramento e Avaliaçã - which has become a very active association. We’re based in Brazil, at the Sao Paulo School of Economics at Fundação  Getulio Vargas and work jointly with the school´s Center for Applied Microeconomics. Through CLEAR we’re looking forward to expanding into new areas and building bridges with the M&E communities in Brazil and elsewhere. In particular, we’ll be working to advance evaluation capacity development services and products in Portuguese for use in Lusophone (Portuguese-speaking) countries, all to foster evidence-based policy making in these countries. Historically, our team in Brazil has had a lot of experience in carrying out impact evaluations in all sectors. Though we specialize in impact evaluation, we have experience in and appreciate the broader range of M&E approaches, and think that an integrated approach will make our work better. In this post, I have put together a few tips about impact evaluation that you would not learn in conventional econometrics books. This is advice I’d give to impact evaluators. Lessons Learned: Know well the theory of change of your intervention! If you don´t know the theory of change well, you might not fully understand the causality channels and might leave important impact indicators out of the analysis. Get your hands dirty! Go to the field, talk to project managers, talk to beneficiaries and make sure you fully understand the intervention you are trying to evaluate. Also, be careful with the quality of your data. Make sure you spend some resources on hiring and training qualified staff to supervise data collection. Good quality data is crucial for your study. Lessons Learned: Even if you are an empiricist and believe mostly in quantitative methods, do not underestimate the value of mixed methods. In particular, qualitative approaches will help you understand "why and how" things happened. Importantly, get to know M&E "foundational" literature from Patton, Scriven, Bamberger, and others. Rad Resources: While in general M&E materials available in Portuguese are limited in numbers, there is a very useful impact evaluation book that I have co-authored with other Brazilian experts. The book may be obtained free at http://www.fundacaoitausocial.org.br/_arquivosestaticos/FIS/pdf/livro_aval_econ.pdf We look forward to contributing to the M&E literature base in Portuguese, so please check back with us on this. The American Evaluation Association is celebrating Centers for Learning on Evaluation and Results (CLEAR) week. The contributions all this week to aea365 come from members of CLEAR. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.    

I’m Nidhi Khattri from the CLEAR Global Hub at the World Bank’s Independent Evaluation Group.  As a member of the team that got CLEAR up and running, I’ve been interested in how countries make evidence-based decisions, and the role that evaluation plays in that process. Coming from a research-based background where I was more concerned with producing evaluations, I began reading more about how evaluations can be used systematically.  This book on how governments use evaluations to inform budget decisions was especially informative. My understanding grew about the ecology around the production and use of evaluation that is grounded in public sector (or indeed organizational) management.  For evaluation evidence to be used, it’s not enough for the evaluation to be technically sound.  It must be timed correctly and connected closely to the different points of decision in the policy cycle - policy design/budget allocation, program design, implementation, review, and back to budget allocation (both within and across sectors and programs) - and to the fundamental questions that policymakers (and program implementers) must contend with at those specific points in the cycle.   Furthermore, the set of evaluations an organization or a government conducts or commissions must also be based on principles of effective and efficient use of resources, helping guide the choice of evaluations. Many countries (and large organizations) have developed different institutional mechanisms and arrangements to deal with these issues.  They attempt to address the different points in the policy cycle, but they close the loop only partially.  Some focus predominantly on budget decisions.  Others are far more robust in considering and solving implementation issues.  Still others focus much more on accountability at the end.  In part it’s because of issues of coordination and capacity across the range of ministries and departments.  It’s also is due to differences in management philosophy and use of monitoring rather than evaluation.  Similarly, there are quite a few differences in governments (and organizations) making rational decisions regarding the set of evaluations, ranging from somewhat formulaic approaches to letting "…a thousand flowers bloom." This subject - use of evaluation as a tool for public sector management - intrigues me, and I wonder how it’ll evolve in the future with greater access to technology and multiple sources of information, collected and analyzed on an ongoing basis by non-evaluators.  Will it be tied less to decisions at specific points in time and far more to real time decision-making? Will evaluations become less "evaluative" and more "facilitative" along the entire cycle?  In which case the question regarding the choice of a set of evaluations may become moot. Rad Resources: World Bank Independent Evaluation Group’s case study series on M&E systems The American Evaluation Association is celebrating Centers for Learning on Evaluation and Results (CLEAR) week. The contributions all this week to aea365 come from members of CLEAR. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

On July 22, @shrmnextchat chatted with the EEOC Commissioner Chai Feldblum (@ChaiFeldblum) and Jonathan Segal (@Jonathan_HR_Law) about the 25th Anniversary of the Americans with Disabilities Act. In case you missed this important chat you can read all the tweets, many which include great links to ADA resources, here:   [View the story "#Nextchat RECAP: The 25th Anniversary of The Americans with Disabilities Act " on Storify]  ...

Hi everyone!  Liz Zadnik here, aea365 Outreach Coordinator and occasional Saturday poster.  Today I’d like to chat about a few people who have shaped the way I approach and appreciate evaluation. Lesson Learned: Feminism is for everybody (…thanks bell hooks).  "The soul of our politics is the commitment to ending domination." I carry this message with me as I engage in my primary role as a trainer and preventionist in the anti-sexual violence movement, as well as when think about evaluation.  How am I dismantling forms of oppression while both engaging in evaluation and talk about evaluation to others? Hot Tip: Don’t be afraid to think with things.  I learned so much working with Sharon Wasco on some community readiness assessment and pilot projects.  Sharon introduced me to so many new ideas as a trainer and evaluation enthusiast.  One of the tools she gave me - and I still use today - is bringing objects and crafts to evaluation training to help adult learners activate new learning styles and energies.  Sharon is also responsible for my deep appreciation for community readiness and the power of qualitative methodology! Rad Resource: Maria Popova of Brain Pickings.  How has this blog helped me with evaluation, you ask?  Well, it’s Maria’s approach to the blog and how creativity shapes our thinking, I think of it as LEGOs — if the bricks we have are of only one shape, size, and color, we can build things, but there’s a limit to how imaginative and interesting they will be. The richer and more diverse that pool of resources, that mental library of building blocks, the more visionary and compelling our combinatorial ideas can be. Lesson Learned: Evaluation is for everybody (…thanks Stephanie Townsend).  Stephanie was the first professional evaluator I worked with; she brought an extensive background in the anti-sexual violence movement and a passion for evaluation I had never seen before.  I owe a lot of "click!" moments to her.  I LOVE spreadsheets and data analysis in large part to her.  She is a brilliant evaluator and patient teacher. This last person is really…well…a group of people.  In no way monolithic, but still someone I carry with me: survivors of trauma and violence.  I started my career about seven or eight years ago working in a community-based domestic violence organization.  I worked with youth in schools and our shelter.  Since then I have talked to countless women, children, and young men who have experienced (and resiliently moved forward from) some of the worst this world has to offer.  My approach to evaluation always has them in mind and I consistently challenge myself to find ways to honor them in this work. Do you have questions, concerns, kudos, or content to extend this aea365 contribution?  Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip?  Please send a note of interest to aea365@eval.org.  aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

  Today's talent pool is better educated and more tech-savvy than ever. They use various technologies as consumers and in their personal lives, and they expect employers to have the same level of comfort with technology as they do. If you talk to someone who was managing an older baby-boomer workforce during the late 1980s or early 1990s, you will likely hear tales of employees who just could not seem to adapt to new technology. You might hear about workers who had difficulty grasping the difference between formatting a floppy and formatting a hard drive, whose typewriters had to be...

My name is Michael Quinn Patton and I am an independent evaluation consultant based in Minnesota but working worldwide. I have had the honor and privilege of participating in and presenting at every Minnesota Evaluation Studies Institute since it began 20 years ago. A lot has changed in evaluation over the last two decades but one thing remains constant: Evaluation is a political activity. The Social Justice theme of this year’s conference highlighted the political nature of evaluation, but politics plays some part in all aspects of evaluation. Lesson Learned: Evaluation is NOT political under the following conditions, all of which must be met: No one cares about the program. No one knows about the program. No money is at stake. No power or authority is at stake. And, no one in the program, making decisions about the program, or otherwise involved in, knowledgeable about, or attached to the program, is sexually active. (Patton, M.Q., 2008, Utilization-Focused Evaluation, p. 537) Hop Tip: Be prepared to deal with politics as a professional The Joint Committee Standards call on evaluators to be politically sophisticated. "Contextual Viability: Evaluations should recognize, monitor, and balance the cultural and political interests and needs of individuals and groups." The AEA Guiding Principles call for evaluators to exercise "Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation." Lesson Learned: Beyond Neutrality Enter the political fray from a strong values base. In a classic article distinguished evaluation pioneer Bob Stake articulated what evaluators care about: We often care about the thing being evaluated. We, as evaluation professionals, care about evaluation. We advocate rationality. We care to be heard. We are troubled if our studies are not used. We are distressed by underprivilege. We see gaps among privileged patrons and managers and staff and underprivileged participants and communities. We are advocates of a democratic society. Rad Resource: "How Far Dare an Evaluator Go in Saving the World?" Bob Stake. American Journal of Evaluation, Vol. 25, No. 1, 2004, pp. 103-107. Lesson Learned: Everybody’s got to serve somebody.  Know whose interests you serve in an evaluation. Not sure about this? Minnesota native son Bob Dylan’s evaluation anthem makes it clear. Check it out. Rad Resource: Bob Dylan singing "Gotta serve somebody" (music and lyrics) Rad Resources: Where Politics and Evaluation Research Meet, Carol H. Weiss, American Journal of Evaluation, February 1993; vol. 14, 1: pp. 93-106. Politics and Evaluation: More Than Methodology, Lois-ellin Datta, American Journal of Evaluation, June 2011; vol. 32, 2: pp. 273-294. Politics and Evaluation, Michael Quinn Patton, American Journal of Evaluation, February 1988; vol. 9, 1: pp. 89-94. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

It is no secret that today’s Human Resources, or Human Capital, industry made its way onto the scene as The Personnel Department. In a time long before sophisticated networks housed human resources information systems, when employee information was maintained on 3×5 index cards in a file cabinet down the hall, we were signing and cutting physical checks for payroll, walking through rows of offices and cubes distributing typewritten memorandums, and a variety of "other duties as assigned" (aka, the admin work no one else wanted to do). ...

Hi, I’m Nora F. Murphy, a developmental evaluator deeply committed to social justice. I recently attended the Minnesota Evaluation Studies Institute (MESI) Spring Training and Donna Mertens’ workshop on Weaving Social Justice and Evaluation Together. Stimulated by the concepts and conversations I have been reflecting on how social justice appears in my practice and had the following insights: Lessons Learned: #1: I actively choose evaluations of projects related to systems change to increase social justice and equity and assumed this was enough. Mertens challenged us to go a step further by placing human rights and social justice at the center. While these elements are always present in my evaluations they are not always at the center. #2: Where people are working towards social justice and equity there is trauma—individual and community, past and present. Evaluators can ignore this and, I suspect, often do. I realized that my most meaningful evaluations did not ignore this but rather recognized and honored this aspect of people’s experiences. #3: AEA’s Guiding Principles For Evaluators (2004) states that evaluators bear responsibility for general and public welfare. When designing an evaluation I can choose to ignore the trauma or design an evaluation that creates the space to recognize the trauma and promote healing as a way to benefit both individuals and society as a whole. I will ask myself these questions and commit to the following as I explore the intersection of evaluation, social justice, trauma, and healing: What gets placed at the center? Mertens suggests we place human rights and social justice at the center. I will be more intentional about doing so. How do I attend to what’s in the center? I will consider methods that promote healing through deep listening, bearing witness, and creating opportunities for people to connect to their inner selves and to each other. For what purpose and to what ends do we evaluate? Bob Williams suggested recently in an EvalTalk post (4.4.15) titled "Evaluation’s Warrant" that one possible purpose is to serve humanity. I will deepen my thinking about this idea. Who is evaluating? Educator Parker Palmer (2009) asks himself: "How does the quality of my selfhood form— or deform— the way I relate to my students, my subject, my colleagues, my world?" In a similar vein I will ask this question of myself as an evaluator and do the inner work needed to bring my best self to my work. Rad Resources: Palmer’s The Courage to Teach: Exploring the Inner Landscape of a Teacher’s Life. AEA365 MESI Week: Donna M. Mertens on Evaluation’s Contribution to Solving Wicked Problems AEA356 CP TIG Week: Karen Countryman-Roswurm and Bailey Patton on Qualitative Research Methods as an Empowering Practice with Marginalized Youth Chilisa’s Indigenous Research Methodologies Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Robotic food service is replacing lower level workers. I am currently reading Rise of the Robots by Martin Ford. Martin is a software development entrepreneur who has been in computer design and software development over 25 years. Working in Silicon Valley he has seen the effect of automation and robotics on the world of work. The subtitle of his book is Technology and the Threat of a Jobless Future. As you might imagine he does not have an optimistic outlook....

I am Rebecca Reznik-Zellen from the Lamar Soutter Library at the University of Massachusetts Medical School. In my work, Open Access (OA) is an issue that I encounter regularly. But OA is becoming a familiar term to many. We hear it from publishers, funders, and research institutions about scientific and scholarly publication. For some, OA conjures ideals of democratizing access to research by removing subscription barriers that keep important findings away from public, industry, and government entities. For others, OA inspires concerns about article processing charges, predatory publishers, and inconvenient processes for bureaucratic mandates. Which inclination is correct? By definition, OA is the free and unrestricted online access to scientific and scholarly work. It enables broad and rapid dissemination of research, informs the public, and expedites the scientific process. When research articles are available to be downloaded, read, and reused, they demonstrate greater impact than articles locked behind subscriptions. OA has been growing steadily since the Budapest Open Access Initiative and the Berlin Declaration defined and formalized it in the early 2000s. The Directory of Open Access Journals lists over 10,000 individual titles that publish research openly, comprising approximately one-third of all scholarly publications. Members of the Open Access Scholarly Publishers Association agree, by virtue of membership, that OA is an established approach to scientific publishing. But while OA has become more established, it is still a communication model undergoing transformation. Determining the actual costs of publication is more difficult than it might seem, so the article processing fees designed to recoup these costs are highly variable. Unethical publishers who take advantage of an author’s desire to publish openly have emerged. And the complexities of complying with the NIH’s Public Access Policy can detract from the perceived value of OA. So, both characterizations of OA are correct: OA IS a democratizing force in an otherwise unbalanced scholarly publishing system, AND the practicalities of implementation are challenging. OA will improve only if the producers and consumers of scholarship participate. This means actively engaging in the publishing process, calling out predatory practices, and taking the time to comply with funder mandates. Open Access is a goal worth pursuing. Rad Resources to learn more about Open Access: UNESCO’s OA Curriculum for Researchers Lamar Soutter Library’s OA LibGuide Guide for Evaluating OA Journals SPARC Resources on OA Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! I’m Siobhan Cooney, an internal evaluator at the College Board and member of the Eastern Evaluation Research Society’s Board of Directors. In this post I share information on one of my favorite features of Excel - the RAND function - and how it can be used in a random selection process. Evaluators at times need to randomly select individual study participants or groups of participants. In random assignment studies, we select teachers, schools, neighborhoods and the like to form our treatment and control groups. Or, we might randomly select individuals from a large population in an attempt to achieve a smaller but representative sample. And, when offering incentives to study participants, we might give them each an equal chance of winning one of five iPads, for example; in this case, we want to randomly select five names from a full list of participants. Hot Tip: Sophisticated software is not required to complete a random selection process. When you need to choose one of two options, I recommend flipping a coin. For many other cases, Excel works well. Cool Trick: Let’s suppose we want to randomly select two AEA past-presidents for a research study. Eleven past-presidents have agreed to be part of our research, if selected. (1) In an Excel file containing the list of 11 AEA presidents, we first create a column in which we will tell Excel to generate random numbers. (2) We then type "=RAND()" into the first empty cell in this new column. This function will return a random number between 0 and 1. We then click the cursor on the bottom right corner of that cell and pull down to apply the same function to other cells in the column. (3) We have decided a priori that we will select the two AEA presidents with the lowest random numbers. Therefore, we want sort our cases from the smallest random number to the largest one. However, Excel’s typical set-up tells it to automatically recalculate our random values when we sort cases. For this reason, we first need to copy our list of names and numbers and paste it in a new sheet. Importantly, we need to "Paste Values" so that only our values are pasted, not our formulas as well. (4) We can then select our list of names and numbers and sort them (Data > Sort > Sort by Random # on Values with Order Smallest to Largest). Our end result indicates that Nick Smith and Mel Mark have been randomly selected for our research. Rad Resources: YouTube and Ann Emery’s Excel blog contain a wealth of resources on using Excel’s features to your advantage. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Tweet   The FLSA and the move to workplace flexibility will be tugging at workers and management.   In early July the US Department of Labor announced changes in the Fair Labor Standards Act that will radically alter the number of workers who will no longer be considered exempt. As of the moment the projected minimum salary level is anticipated to be $50,440. (You can learn more here.) At the...

¡Hola! from Kirsten Büchner of Insight Evaluation Services in Fairfax, Virginia. I am a bilingual evaluation consultant working with museums and informal learning organizations in the cultural sector. As a native Spanish speaker, I get asked to participate on projects that have components designed for Spanish-speaking audiences, so part of my job includes making sure Spanish-language consent forms and instruments accurately reflect the original English-language versions. Here are a few things to keep in mind when converting documents to another language. Hot Tip: Whether you use a free service like Google Translate or engage a translation firm, it is always a good idea to have a native speaker with professional level experience or education review your documents and verify they are correct and make sense. Check out these examples from a consent form I recently was asked to look over: Lesson Learned: The client had already received IRB approval for both English and Spanish forms, but the latter had to be resubmitted given the number of revisions needed. Considering the energy we evaluators put into developing valid and reliable instruments, making sure they are clear and understandable in all target languages is worth the extra effort. Hot Tip: Translators should have current knowledge of the region(s) your respondents are from. Spanish is spoken in 20+ countries, and while we all understand the same basic language (what I like to call "broadcast Spanish", such as what is spoken on Univision and other Spanish-language television stations), there can be important differences in vocabulary, phrasing, or how we address someone. For example, in some Central American countries and parts of Colombia, "usted" (formal you) is used when speaking with someone we are meeting for the first time or do not know well, as well as with people who are older or of higher rank; however, in Spain, Mexico and Puerto Rico, "tu" (informal you) is more commonly used. Lesson Learned: The AEA Statement on Cultural Competence in Evaluation defines culture as "the shared experiences of people, including their languages, values, customs, beliefs, and mores. It also includes worldviews, ways of knowing, and ways of communicating." Taking into account the way a given cultural group communicates helps us earn respect and establish trust among the members of that group, which in turn allows us to produce a higher quality evaluation. Rad Resource: The American Translators Association lists translators by the language combination desired with advanced options to help narrow your search by specific criteria, such as education, experience and areas of expertise. Google Translate AEA Statement on Cultural Competence in Evaluation American Translators Association Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Tom Wolff, independent consultant with Tom Wolff & Associates.  A recent Boston Globe article, "Trials aim to keep recurring patients from hospitalization" by Sabrina Tavernise, described the work of the $10 billion Innovation Center that is part of the Affordable Care Act. As they attempt to manage the most expensive patients the article notes, "They raise a new question for health care systems: What is its role in tackling problems of poverty?" As a community psychologist this is hardly a new notion. That people’s behavior and health is a function of their environment is basic to our field. We are all interdependent and interconnected. However, this premise makes for very complex program development and evaluation question; when we consider the full environment that impacts people’s health (often labeled Social Determinants of Health) we are dealing with a myriad of variables and multiple sectors of the community. Rad Resource: The New England Racial Justice and Health Equity Collaborative (NERJC) is an action and learning network across 12 communities in three states. These communities are working to promote racial justice and health equity. The goals of the coalitions and programs are to create policy changes to address systemic racism in the social determinants of health (e.g. food access, educational disparities). How does the history of systemic racism in housing, education , transportation impact people’s health? As the brilliant film "Unnatural Causes" notes, our zip code may be more important for our health than our genetic code. You can learn more about the model NERJC used in this article. Although the link between health disparities and racism may seem obvious, in practice most efforts to address health disparities in the U.S.do not focus on racism. Thus, if they are addressing diabetes rates in Black men they will offer classes on nutrition to Black men, rather than work to change the food desert where Black men reside. The NERJC meets regularly through the year on a totally unfunded voluntary basis to provide support (both emotional and skill based) to these social justice activists who brings this racial justice perspective to their work in addressing health disparities and reaching for health equity. Creating an ongoing learning environment where participants can gain new techniques to take back to their communities which will lead to creating changes in local programs, policies and practices consistent with a racial justice perspective Lesson Learned: As a white man who has experienced all the benefits of white privilege I have felt a responsibility to help the NERJC start up, and thrive and survive. I see our work on this frontier as an example of community engaged scholarship. What do you think? What can you do? Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.