Hi there, Liz Zadnik here, bringing you another Saturday post focused on practitioner experiences and approaches. Today I’m going to focus on my personal journey to stay up-to-date and relevant in all things evaluation. I was not formally trained as an evaluator - everything I know has been learned or gained through on-the-job hands-on experiences and mentorship (I’m very lucky to have been able to work with a few brilliant evaluators and researchers!). Self-study, reading, and ongoing training have been intentionally incorporated into my personal and professional schedule. Rad Resource: Coursera is an excellent resource for online learning. You can even get certifications in concentrations after completing a set of courses in sequence. They have a number of courses around data analysis and data science! Rad Resource: iVersity coordinates and archives some really interesting and innovative massive open online courses (MOOCs). The "Future of Storytelling" course gave me a number of ideas and skills for crafting accessible and engaging trainings and resources, as well as some insights for capturing stories for program evaluation. Recent and future courses focus on idea generation methods and gamification theory. Lesson Learned: Follow your gut! At first I thought I needed to select courses, books, and resources that were explicitly "evaluation-y," but found it was those courses that made me say "Oooh! That looks interesting!" helped me think creatively and find ways to enhance my evaluation and program development skills. Rad Resource: MIT Open Courseware is much more structured and academic, as these are courses held at MIT. These require - for me - a bit more organization and scheduling. Rad Resource: edX is another great chance to engage in online courses and MOOCs. Right now they have two courses on my "to-take" list: Evaluating Social Problems and The Science of Everyday Thinking. Are there other online course providers or resources you rely on to stay current? How do you stay up-to-date and innovative as you balance other obligations and projects? Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I wrote my first conference recap post - then called "Rants and Raves" - in March of 2010. This has become my favorite blogging experience of every conference I attend, because it really causes me to think about what is happening around me and whether I love it or hate it - or if I am just momentarily in a mood. This is why I always wait a day or two until conference end. And by focusing...

I’m Kylie Hutchinson, independent evaluation consultant and trainer with Community Solutions Planning & Evaluation. I also tweet regularly at @EvaluationMaven. Have you ever wondered what evaluation recommendations and SpongeBob SquarePants have in common? Well, in my opinion, a lot. Think about why we make recommendations. We want stakeholders to take action on our evaluation findings. But we all know this doesn’t happen by magic. And it doesn’t occur as soon as we submit our final report either. In fact it can be months or years before managers and policy-makers are actually in a position to make decisions based on our findings. In order for utilization to happen, I think recommendations need to be three things: easily absorbed (at the time of first reading) sticky (so they stay in the minds of decision-makers) have ‘legs’ (so they prompt action). Hmmm…now think…what has good absorption, is sticky, and has legs? Exactly! SpongeBob SquarePants!   Rad Resource: Here’s a tip sheet on Recommendations That Rock! Hot Tip: Well-written recommendations don’t have to check every tick in the box, but they do deserve significant attention. Don’t leave them to the end or the last minute. Instead, keep a running list of your initial ideas as soon as they occur, even if it’s at the beginning of the evaluation. And always run them by your stakeholders to increase ownership and the chances of implementation. Better yet, develop them collaboratively during a data party. Rad Resource: You can find a Pinterest page with other resources for writing better recommendations here. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

  I attended a great event at #SHRM15.  A few social media influencers Rayanne Thorn, Dwane Lay, and Jason Lauritsen organized a fundraiser for "No Kid Hungry".   Here’s the deal: They pulled it off in about 3 weeks. Found a venue (shout out Public House at the Luxor, Las Vegas) that worked with budget. Got cos they work for to sponsor (Quantum Workforce/Dovetail- maybe some others?). Actually Quantum chose to skip the #shrm15 vendor booth to sponsor this social...

Hi! I’m Madhawa "Mads" Palihapitiya, Associate Director at the Massachusetts Office of Public Collaboration at UMass Boston. We recently concluded the first phase of a statewide municipal conflict resolution needs assessment study commissioned by the state Legislature. Hot Tip: The term "need" can mean many things to many people. For needs assessment purposes, needs are defined as Gaps in Results. Organizations don’t often think of aligning their institutional needs with the societal bottom-line, but working towards this alignment is crucial.  We needed to investigate if our institutional mission, which is to help government and other entities address public conflict, was perfectly aligned with the needs of Massachusetts municipalities and their constituents. This alignment was particularly important to us as a statutory state agency and would add measurable societal value. Overtime, this alignment can also increase the institutional bottom-line. Rad Resource: Roger Kaufman’s Needs Assessment for Organizational Success. See also Bethany Pearsons’ talk at Evaluation 2014 on the Triple Bottom line. People don’t usually talk about societal results when they talk about organizational needs. How do we define societal results? We first developed an Ideal Vision that contained a series of societal results and indicators to measure them. Lesson Learned: We had to resist the impulse to focus on immediate institutional needs like organizational inputs and processes. Imagining an ideal future or vision can tell us where the journey should end. Rad Resource: Kaufman’s Ideal Vision. Cool Trick: To help the organization and others being engaged understand the difference between different results, consider developing a visualization like the DoView chart below. Assessing societal results while assessing the institutional bottom-line requires access to valuable data both within and outside of your organization. A Needs Assessment Committee (NAC) was established as the ‘public face’ of the process and to provide advice and guidance on assessment design, participant selection etc. Cool Trick: Set-up a website to communicate the purpose of your needs assessment. Use social media whenever possible. There are 351 cities and towns in Massachusetts. Multiple organizations were involved. We had limited resources to collect the data we needed. We had to get creative! A series of regional focus groups and telephone interviews were held. To reach the rest, we launched an online survey. Hot Tip: Online surveys are a great way to involve more people. Keep survey questions close-ended and completion time to 10-15 minutes. Plan ahead so that you keep the survey open for as long as you can. Cool Trick: Get creative with survey dissemination by using contact databases, newsletters, list servers, Facebook and twitter. Ask people you know to invite others to take the survey. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

#SHRM15 the conference is a wrap! We had a lot of fun, it was really hot, I met great people, hung out with old friends, and had a classy time!   Now that we’ve learned lots of great tips and tools and received some wonderful advice, it’s time to get to work! Where should we start? Well, let’s talk about engagement. Engagement is something we learned a LOT about this week. Let me recap: ·         Throw employee engagement surveys away ·         Use these tools to increase employee engagement ·        ...

Greetings! My name is Catherine Cooper, and I am the Faculty Director of the Educational Partnership Center and Professor of Psychology at the University of California, Santa Cruz. I invite you to explore and use the resources from the Bridging Multiple Worlds Alliance (BMWA). The BMWA is a growing network of researchers, educators, and policy makers - including evaluators - in the U.S. and other nations who work with P-20 (preschool through graduate school) partnerships to support low-income, immigrant, and ethnic minority youth.  These partnerships support youth in building pathways from childhood to college and careers without giving up ties to their families and cultural communities. We work in collaboration with alliance partners, including youth themselves and evaluators of programs and partnerships. Rad Resource: In the BMWA, we offer three resources that evaluators tell us are especially useful: Aligning models and measures to build a common language among partners. Tools for research, policies, and practice, including formative and summative evaluation. Longitudinal data tools for qualitative and quantitative evaluation and research The Bridging Multiple Worlds (BMW) Model (shown below) taps five dimensions for opening pathways: Demographics—students’ age, gender, national origins, race/ethnicities, languages, and parents’ education and occupation Students’ aspirations and identity pathways in college, careers, and cultural domains Students’ math and language academic pathways through school Resources and challenges across students’ cultural worlds of families, peers, schools, community programs, sports, and religious activities, among others Partnerships that reach across nations, ethnicities, social class, and gender to open pathways from preschool through graduate school (P-20)   Rad Resource: Bridging Multiple Worlds Tools include: Survey measures of these five dimensions for middle/high school and college students Activities for middle and high school students for building pathways to college and careers, with pre- and post-activity surveys (in English and Spanish) Logic model template for programs and alliances among programs Longitudinal case study templates Rad Resource: I invite you to join BMWA partners- students, families, schools, community programs, and universities-in using these tools to ask your own questions and build common ground among evaluators, researchers, educators, and policymakers. The tools and other resources are available at www.bridgingworlds.org. Rad Resource: Bridging Multiple Worlds: Cultures, Identities, and Pathways to College (Cooper, 2011) describes BMW and related models, supporting evidence, tools, and applications in P-20 research, practice, and policy work. Hot Tip: Healthy partnerships are learning communities where "everyone gets to be smart". Focus on questions and indicators partners are interested in and display data in clear and meaningful formats.  This increases enthusiasm, engagement, and cooperation. Examples of such questions, indicators, and formats are on our website. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest toaea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, I am Rupu Gupta, Analyst at New Knowledge Organization Ltd. and Co-Chair of AEA’s Environmental Program Evaluation Topical Interest Group. My evaluation work focuses on learning about the environment and conservation in informal settings. As we celebrate Earth Day, I would like to share some reflections on evaluating these experiences. Lessons Learned: Informal learning settings are critical to learn about the environment and actions to protect it. Informal learning settings offer opportunities for "free-choice" learning, where the learners choose and control what they learn. They are typically institutions such as zoos, botanic gardens, aquariums, and museums, distinct from formal educational settings like schools. With hundreds of millions of visits to these institutions annually, they are prime settings to engage the public in thinking about the environment. Conservation education is often a key aspect of these institutions’ programming, where visitors can learn about different forms of nature (e.g., animals, natural habitats), threats they face (e.g., climate change), and actions to address them (e.g., reducing energy use). Educational experiences here are often referred to as informal science learning for their connection with understanding natural systems. Learning about the environment in informal learning settings can happen through a variety of experiences. Informal learning is socially constructed, through a complex process that includes oneself, close others (friends, family) and more distant others (institution staff). Specific experiences, like animal encounters, hands-on interactions with flora in botanic gardens, or media-based elements (e.g., touch screens) enable visitors to engage with information about nature and the environment. Docents play an important role in helping visitors ‘interpret’ the message embedded in the experiences and exhibits. Evaluators assessing the impact of the different experiences in informal settings, need to be mindful of the multiple pathways for visitors to engage with the environmental information. Informal learning manifests broadly. Learning experiences in informal settings encompass outcomes, beyond learning traditionally associated with school-based education. In the process of making meaning of the various experiences, learning is tied to the multiple aspects of the human experience. They can be cognitive (e.g., gaining knowledge about climate change impacts), attitudinal (e.g., appreciating native landscapes), emotional (e.g., fostering empathy towards animals) or behavioral (e.g., signing a petition for an environmental cause). A mix of qualitative and quantitative methods are best to capture the complex learning experiences. By considering the range of learning possibilities, evaluators can design and conduct effective evaluations to understand how people engage with the multi-faceted topic of the environment. Rad Resources: The following are great to get acquainted with evaluation in informal learning settings: "Free-choice Learning and the Environment" by John H. Falk, Joe E. Heimlich, and Susan Foutz "Framework for Evaluating Impacts of Informal Science Education Projects" edited by Alan J. Friedman Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, my name is Catherine Nameth, and I’m the Education Coordinator for an NSF- and EPA-funded research center at the University of California- Los Angeles. As Education Coordinator, my primary job is not evaluation, so I have to act creatively in order to integrate evaluation into my work and balance the need for internal evaluation with my other administrative and research responsibilities. Hot Tip: Be an active learner and an active listener. Get to know your colleagues and their areas of expertise. Go to meetings, listen, and be open to learning about your colleagues and what they do. Your understanding of them and their work will inform your understanding of your organization as well as its people and programs/research. This understanding can then inform how you design surveys and collect evaluation data. People who know you are more likely to respond to your surveys and other "official" evaluation requests, and when they respond, you get the information you need! Rad Resource: Map it out! Use Community Solutions’ map for "How Traditional Planning and Evaluation Interact." This map displays how an evaluation logic model (inputs-activities-outputs-outcomes) situated horizontally interacts with program planning (goals-objectives-activities-time frame & budget) which is modeled vertically. In using this map, you’ll see that the "activities" of each model intersect, and this cohesive visual aid also serves as a reminder that program planning goals and evaluation outcomes should- and can- inform one another. Use this map to keep yourself focused, which is really important when your primary responsibilities include many aspects other than evaluation, and to help you show your organization’s leadership what you are doing and why you are doing it. Hot Tip: Have an elevator pitch at the ready. When your work includes evaluation but is not entirely about evaluation, you need to be able to explain quickly and concisely what you are evaluating, why you are evaluating it, what information you need, and how your colleagues can help you by providing this needed information . . . which they will be more willing to do if they know you! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Dr. Michael Woodward, a member of the #SHRM15 Blogger Team, chats with Ernie Anastos about what employees fear most at work. The three fears discussed: Not knowing where they stand, automation and being too digitally connected.          ...