Hello from Florent and Margaret in Sydney! We are two seasoned evaluators from ARTD Consultants, an Australian public policy consultancy firm providing services in evaluation, research and strategy. As more Australian government services and programs are delivered through partnerships, evaluators need to find better partnership evaluation methods. Faced with the challenge of evaluating partnerships, we quickly realised that there are a number of methods out there: partnership assessment surveys of varying types, social network analysis, collaboration assessment, integration measure, etc. But which one should we choose? Having looked at a number of these we felt that choosing one would not enable us to see what was really happening at all levels of the partnership. So, in our most recent partnership evaluation, we combined some of these methods to get a more complete picture of the partnership. The three we chose were: a partnership survey (adapted from the Nuffield Partnership Assessment Tool), an integration measure (based on the Human Service Integration Measure developed by Brown and colleagues in Canada) and Social Network Analysis (using UCINET). The diagram below represents our conceptual framework, with each method looking at the partnership at a different level: overall, between organisations and departments, and between individuals. Lesson learned #1: A key benefit of combining partnership assessment methods is that it enables you to look at the partnership at different levels. Adding in-depth interviews or other qualitative methods to the mix will allow you to explore further and drill down into underlying mechanisms, perceptions of what works for whom, experiences of difficulties and suggestions for improvement. Lesson learned #2: Partnerships are abstract/ intangible evaluation objects and evaluations of partnerships often lack data about what is happening on the ground. Adding methods to quantify and substantiate partnership activities and outcomes will make your evaluation more robust and the findings easier to explain to stakeholders. Lesson learned #3: Combining methods sits within the good old mixed-methods tradition. Various metaphors are used to describe the benefits of integrated analysis in mixed-methods research (see Bazeley, 2010). In this case, the selected methods are combined ‘for completion’, ‘for enhancement’ and as ‘pointers to a more significant whole’. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

  "Find buttons we can push to make a change overall." - Ben Eubanks (@beneubanks), Analyst, Brandon Hall Group  Whether your business goals (or concerns) are centered around alignment, sales or even revenue, HR most likely has a solution that can help solve the problem. For many business problems, there’s a solution from HR that can fix the root of the problem. All you need to do is integrate HR practices into the business strategy and include them as an integral part...

  With over 600 vendors, about 200 sessions, daily Smart Stage, and thousands of attendees, SHRM 2015 was successful in educating HR pros and novices in trending ideas and practices that can give HR a seat at the table. Between sessions with Marcus Buckingham and the more intimate ones with those presenting on the Smart Stage like Blake McCammon and Stacey Carroll, the conference this year revitalized attendees. Reviving New Thoughts Sharing experiences with other professionals in your field dusts off the excitement around HR. Kari Chinn, Human Resources Manager at...

My name is Eva Guenther, I am a Project Manager for a for-profit, employee-owned US Government (USG) contractor in Washington, DC. I am responsible for the successful implementation of USG funded projects in the Middle East and North Africa (MENA). While my heart beats for evaluation it is not the primary focus of my role and responsibilities. Here are a few tips on how I support the local project teams from afar and ensure excellent data driven decision making and monitoring and evaluation happens on the projects. Hot Tip: Establish a culture of data driven decision-making. Evaluation is often an afterthought and treated as if it were the sole the responsibility of the evaluation team rather than the entire project team. I try to set the expectations from the start that we should measure the outputs and outcomes for of all activities so that the team wants to know this data and proactively seeks it. Hot Tip: Get involved early and set dates for revisits. After award I help operationalize our evaluation plan with the local team, work with them on regular data quality assurance activities and review evaluation data in regular reports to the client(s). Hot Tip: Use web-based tools for data sharing. All project team members, often dispersed, should know the outputs and outcomes of project activities. Web-based tools make that easier. The company I work for has a proprietary project management tool that includes an evaluation module that facilitates easy capturing and sharing of evaluation data. This makes checking progress easy for everybody easy, including for myself from the US. Hot Tip: Use data visualization. USG funded projects are heavy on narrative centric reports. I help our implementation teams on the ground to better tell the project’s story through data visualization and web-based interactive maps. This has led to deeper conversations as the information is more accessible. It also has led to outcome and impact data being shared outside the immediate project circle as it did not require a deep background on to understand it. Rad Resource: Get inspired by others. There are a lot of brilliant minds out there so looking at other’s infographics has given me great ideas for how to explain a project output or outcome better with the help of an image. I have drawn lots of inspirations from these blogs: http://dailyinfographic.com/ http://www.visualizing.org/ http://www.coolinfographics.com/ Try out one of the free resources for how to create infographics: http://www.creativebloq.com/infographic/tools-2131971 Lessons Learned: Keep up-to-date with new developments in data collection. Mobile data collection, sentiment analysis and other approaches and tools can be helpful for the local teams where resources are often scarce. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am still pumped up after watching @IAMJHUD’s performance last night. She is absolutely an amazing artist. Her music touches your soul and will bring you to tears. Her connection and interaction with the crowd was so natural and appeared effortless. Everyone that knows me understands that I have a heart for Talent Management. If @IAMJHUD would have asked me to join her team after the concert, I would have hopped in her limo and never turned back. Not even for my luggage! I would pour everything I had...

Greetings! I am Linda Cabral from the University of Massachusetts Medical School’s Center for Health Policy and Research. A big part of my job uses qualitative methods to evaluate different health and human services programs. Our data collection processes can include utilizing one-on-one or group interviews and as well as focus groups. With this type of narrative (i.e., first person data collection), decisions must be made up front as to the ultimate format of your data set. One of the decisions is whether or not to audio record these data collection events and another is whether these audio files will be transcribed. Transcribing data can be a tedious process requiring several hours for each recorded interview. A general rule is that the text of a 30-40 minute interview takes about 1-2 hours to type and results in about 15-20 pages of text. Recently, we have been faced with a myriad of projects requiring decisions on how formal our transcription process should be. Let me offer you some of our thinking and lessons learned! Lessons Learned: Decisions are needed as to the level of detail needed from each qualitative data collection event, which can range from a verbatim transcript to a less formal write-up of notes. While transcribing your own data can have significant analytic benefits (getting close and personal with the material), it may not be practical for everyone - particularly if you’re time-strapped. Transcription of interviews allows for each evaluation team member to go through the transcript carefully, providing an easily readable document from the study. Having a transcript can facilitate working together in a team where the tasks have to be shared. Agreement about data interpretation is key. When considering outsourcing transcription: Realize that a fully transcribed interview will result in pages and pages of data to sift through. There will be a lot of "noise" in there that could potentially be eliminated if the transcription was done in-house by evaluators familiar with the project’s evaluation aims. You have choices as to the type of transcript that would be most helpful to you, including: word-by-word verbatim; clean verbatim (removing ‘hmm’ and ‘you know’); or one with improved grammar and readability. You have options ranging from independent contractors to large firms that specialize in transcription services. Transcribers can be paid by the word, the hour, or the length of time of the recording. Hot Tips: Always have your evaluation aims drive your decision about whether to transcribe or not. Plan ahead for how notes, audio recordings, and transcripts will be stored and how personal identifiers will be used in order to main data integrity. Budget the necessary time and resources up front whatever your decision is! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

  As an American living abroad, I often get asked 'why did you choose to build a company in the UK'?  My answer is simple -- I believe that finding top talent is the number one priority for building a successful company, and being based in the UK allows me to access the best from all over the world. Let me expand on this further. Today, most executives would agree that the ability to attract and retain top talent is a key competitive differentiator for their business, but more than 63% of CEOs are concerned about the availability of key...

Greetings! I’m Galen Ellis, President of Ellis Planning Associates Inc., which has long specialized in participatory planning and evaluation services. In online meeting spaces, we’ve learned to facilitate group participation that - in the right circumstances - can be even more meaningful than in person. But we had to adapt. Although I knew deep inside that our clients would benefit from online options, I couldn’t yet imagine creating the magic of a well-designed group process in the virtual environment. Indeed, we stepped carefully through various minefields before reaching gold. As one pioneer observes, Just because you’re adept at facilitating face-to-face meetings, don’t assume your skills are easily transportable. The absence of visual cues and the inability to discern the relative level of engagement makes leading great virtual meetings infinitely more complex and challenging. Assume that much of what you know about leading great meetings is actually quite irrelevant, and look for ways to learn and practice needed skills (see Settle-Murphy below). We can now engage groups online in facilitation best practices such as ToP methods and Appreciative Inquiry and group engagement processes such as logic model development, focus groups, consensus building, and other collaborative planning and evaluation methods (see our video demonstration). Lessons Learned: Everyone participates. Skillfully designed and executed virtual engagement methods can be more effective in engaging the full group than in-person ones. Some may actually prefer this mode: one client noted that a virtual meeting drew out participants who had been typically silent in face-to-face meetings. Software platforms come with their own sets of strengths and weaknesses. The simpler ones often lack interactive tools; but the ones that do allow interaction tend to be more costly and complex. Tame the technical gremlins. Participants without suitable levels of internet speed, technological experience, or hardware—such as microphoned headsets—will require additional preparation. Meeting hosts need to know ahead of time what sorts of devices and internet access participants will be using. Participants should always be invited into the meeting space early for technical troubleshooting. Don’t host it alone. One host can produce the meeting (manage layouts, video, etc.) while another facilitates. Plan and script it. Virtual meetings require a far more detailed script than a simple agenda. Indicate who will do and say what, and when. Practice, practice, practice. Run through successive drafts of the script with the producing team. Rad Resources: Settle-Murphy, N. (2015). Making Virtual Meetings Come Alive: It’s Everyone’s Job! Some great tips from the Virtual Facilitation Collaborative (we have taken their eight-week course) Tessie Catsambas and Hallie Preskill’s (AEA members) amazing book "Reframing Evaluation through Appreciative Inquiry" (I refer to this all the time.) Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

  Ahh, Vegas - how we love to hate you! You're hot, you're sleazy, you're crowded, you're expensive and you cheat...of course we hate you! The annual SHRM conference was held in Las Vegas this year and over 15,000 attendees gathered to learn more about their profession. I have been to Vegas for conferences before but this is the first time in which I gambled...and by gambled I mean I played PaiGow. The PaiGow table is a generous, social and helpful table. The game itself is easy to learn and even if you're a bit slow on the uptake, the dealer and all the others offer their...

Hello, we are Marissa Szabo and Humberto Reynoso-Vallejo, members of the Chapter 224 research team evaluating the Health Care Cost Containment Law from the Office of the State Auditor in Massachusetts. This is another in a series of posts on lessons learned from the ongoing evaluation of Chapter 224 published during the March 8th week. Given the wide scope of the Chapter 224 evaluation project, the research team needed to tap into the healthcare community for its advice, perspectives, and feedback. To this end, we assembled an advisory committee comprised of members from the health insurance industry, health care advocates, academia, labor, and professional organizations to gain diverse input. At the first meeting of this advisory committee, the research team presented a draft of the evaluation plan for their consideration. Members of the board provided suggestions related to methodology, potential data sources, and provided valuable information on the overall logistics of the evaluation. Since that meeting, the team has been in regular contact with members of the committee for specific feedback on different aspects of the evaluation including reviewing sections of the evaluation plan, and providing additional support as needed. In addition, stakeholders will be included as research participants in qualitative interviews complementing the quantitative component of the study. Stakeholders who belong to task forces, councils, commissions, agencies, boards and other groups associated with the enactment of Chapter 224 will be asked to participate in a brief on-line survey to assess initial observations and concerns related to the roll out of Chapter 224 legislation. Two follow-up interviews will be conducted with selected participants to explore their reactions to our quantitative findings As the Chapter 224 evaluation project continues, the research team will continue to consult with the advisory committee and to engage key stakeholders to ensure the quality of this project. Hot Tip: After an Advisory Committee meeting, follow up with members that provided specific feedback on aspects of the evaluation to keep them engaged. Hot Tip: Include Key Stakeholders as participants in the research including taking part in surveys and participating in in-depth structured or semistructured qualitative interviews. Hot Tip: Ask for their guidance in identifying data sources and other resources valuable to the evaluation process. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.