Empowerment Statistician....What I Strive to Always Be

As I sat in a dissertation defense today of one of my evaluation students the conversation turned towards how we as evaluators/assessment professionals assist our clients with becoming informed consumers. As we talked about strategies for this and the usefulness I thought more and more about Empowerment Evaluation (see all the great work by Fetterman) and how I when I wear my statistician hat (I seem to have many hats) how important it is for me to help my clients with their understanding of the methodology that I've used to analyze their data.

As conversations after the defense (he passed by the way...woohoo Dr. Barlow!) came back to this topic many of us on the committee talked about the number of unfortunate times that we knew of/experienced when a "statistician" assisted a client/student with their analyses and failed to fully explain what, how, or why they conducted those analyses and more importantly what it all meant. As an evaluator I always try to empower my clients to know more about the evaluation process and how it can benefit their organization. As a statistician we need to do the same thing. Maybe when I'm in this role I should call myself an "Empowerment Statistician" because my goals are the same when I'm in this role as well.....to educate and increase my client's capacity to understand the statistical analyses used to explore their data and lead to their utilization of those statistical results. I want my clients to not only understand it when I report the statistical significance of the analyses that I perform but also how they can use these results to inform practice and enhance their organization. As a statistics teacher I also need to encourage my students to do this as well as it is not enough to just conduct the analyses and report the significance/non-significance of findings but to also empower our clients to learn how they too can develop meaningful questions, select appropriate data collection instruments, collect relevant data, and analyze that data in a way that is meaningful to their organization.

So...now to come up with strategies to empower my clients in this regard and to also train my students to become empowerment statisticians.  :)

Some of My Favorite Evaluation Resource Websites

Every few months I try to clean out my website bookmarks...getting rid of outdated ones and adding new ones to my category folders. I was going through my bookmarks in evaluation and I thought it would be beneficial to others if I posted some of my favorite websites that contain a wealth of information about program evaluation. This is not an exhaustive list but rather ones that I tend to use and recommend to my evaluation students. Enjoy!

Program Evaluation Websites:

http://ctb.ku.edu/en/tablecontents/sub_section_main_1338.aspx
http://en.wikipedia.org/wiki/Program_evaluation
http://extension.arizona.edu/evaluation/
http://extension.psu.edu/evaluation/
http://extension.wsu.edu/pd/Pages/ProgramPlanningandEvaluation.aspx
http://omerad.msu.edu/meded/progeval/index.html
http://preventionhub.org/training/evaluation-tools
http://sammie.osu.edu
http://www.cdc.gov/eval/resources/index.htm
http://www.epa.gov/evaluate/resources/index.htm
http://www.icbl.hw.ac.uk/ltdi/cookbook/contents.html
http://www.mapnp.org/library/evaluatn/fnl_eval.htm
http://www.nagc.org/index.aspx?id=1838
http://www.plannedparenthood.org/resources/program-evaluation-tools-23517.htm
http://www.wkkf.org/knowledge-center/resources/2010/w-k-kellogg-foundation-evaluation-handbook.aspx
http://www.wmich.edu/evalctr/
https://www.bja.gov/evaluation/evaluation-resources.htm

Cultural Comptence and Attending the AES Conference in Brisbane, Australia

As I begin packing for a trip to Australia for the Australasian Evaluation Society conference I realize I have no idea what the local cuisine is in Australia. What is Australian food? About the only thing that I know is Australian is vegemite, and only because of an 80s pop song. So I feel a bit undeprepared for my trip. It got me thinking about conducting evaluations in locations/areas that you are unfamiliar with. We all learn in graduate school how to utilize the methods of evaluation and we all are aware of the various evaluation theories.....but how many of us are prepared (and competent enough) to conduct evaluations in settings that we are unfamiliar with? Are we culturally competent in all situations?

You can't just ask someone if they are culturally competent....I mean who is going to say "
no I'm not"! So how do we gauge who is culturally competent and who is lacking in this area? When we are hiring an evaluator how can we ensure that they will be able to conduct a competent evaluation in our specific setting with our program clients?

For you evaluators how do you decide if you are able to conduct an evaluation in a setting you are unfamiliar with? If you take the job, how do you prepare yourself? Can we truly conduct evaluations in any setting....just by doing some research to understand the setting that it's being conducted in? I don't know...

So I need to do some research on what is Australian food and customs. While I'm not conducting an evaluation while I'm there I will be conducting a workshop at AES so I need to make sure I understand enough about my setting in order to successfully interact with my workshop attendees.

For those attending AES I can't wait to meet you all. Happy evaluating!

Importance of Dealing with Outliers

I just finished my second day of conducting a webinar (estudy) for AEA on data cleaning. I just covered outliers....one of my favorite topics! As I was preparing my activity for the attendees I was manipulating variables in a bunch of datasets and comparing what happens when you ignore outliers (leave them as raw variables), delete outliers (delete any outliers from your variables), winsorize outliers (change the outlying value to the value that is 3sd from the mean), or you modify the outliers (change the outlying value to one unit lower or higher than 3sds from the mean....as discussed in Tabachnick & Fidell). I will say when I'm looking at all this output and seeing how even a few number of outliers can greatly change the values of my variable...it is a powerful argument to show how important it is to not ignore this step of the data cleaning process.

I've gotten into many debates with colleagues about what to do with outliers. Some are adamant about leaving them alone...keeping them as is. While others are quick to toss them out of the dataset no matter what it does to their sample size and mean value of the variable. I tend to modify my outliers so I can keep those individuals values in the dataset but with less of an impact on my mean value.

Students always ask me "what is the best/correct way to deal with outliers?" I give them my famous "it depends". There really is no agreed upon way to deal with outliers. You can find a citation to support any of the above mentioned ways to deal with outliers. THIS CAN BE FRUSTRATING! My goal is to always present to my client the most accurate, reliable data I can.

So....what does everyone else do with their outlying values? What is common in the Evaluation field? What do we teach our novice evaluators/students to do? I'd love to hear others opinions on this

Great Resource for Novice Evaluators - SAMMIE (http://sammie.osu.edu/)

In less than a month fall classes will start and I again will be teaching Program Evaluation I (not quite sure where the summer went but that's a whole different story). One of my favorite resources that I encourage my PE1 students to use is SAMMIE (http://sammie.osu.edu/), which was created by the Ohio State University Extension Office of Program Development and Evaluation.

SAMMIE is a web portal that links you to 22 different modules on a variety of evaluation topics. Some topics include: Developing a logic model, Ethical issues, Data collection and analysis strategies, Web based surveys, and Reporting your results. Each module links you to readings on the internet and includes a self-test to gauge what you learned from those readings. You can also create a personal page to jot down ideas about your own evaluation projects and receive feedback from other SAMMIE users. SAMMIE gives a great overview of the field of evaluation. I highly recommend it!

SAMMIE: http://sammie.osu.edu/

Conference Etiquette - How to Get the Most Out of the AEA Conference

I just booked my travel to the American Evaluation Association (AEA) conference that is in DC in October. I remember attending my first AEA conference in 1999 in Orlando, FL. I was still in graduate school at that time and still pretty green about how one should behave at a conference. I would go to the sessions that sounded interesting, eat lunch with people I knew and spend any free time taking in the local sites. Boy....lots of missed opportunities there! I wish I had had someone to pull me aside and tell me about all that conferences have to offer besides just stimulating sessions. Networking, networking, networking....I try to drill this concept in my students. Conferences are more than just a fun vacation away from classes and hanging out with your graduate school friends.

I've presented at hundreds of conferences over the years and I'm amazed by some conference goers, em...how do I say this...less than appropriate behavior at the conference. From walking in late to a talk and disturbing the speaker to answering a phone call during the middle of a presentation! My favorite is when the presenter reads his/her paper...verbatim, sitting down, looking down and not engaging the audience. What, I paid $1000 to travel to listen to someone read their paper? Ugh!

So as I prepare for AEA this year I plan on reviewing a few suggested Dos and Don't's with my students before we all travel to DC. If anyone has any additional suggestions I'd love to hear them! Hope to see many of you at AEA!  :)

Some Conference Do's and Don't's:

Do

• Prepare your presentation/poster well in advance of the conference
• If creating a poster, make it look professional and inviting to your audience (tacking up a few 8.5 by 11 sheets of paper on a board is not what we want to see)
• If creating a paper presentation have slides and/or a handout prepared and practice your speech beforehand
• SHOW UP FOR YOUR PRESENTATION/POSTER SESSION! Throwing up the poster and going out to explore the town doesn't count as doing a presentation. And putting up a piece a paper to capture emails of those interested in your poster doesn't count either
• Go to a TIG meeting or two.....great place to network and meet new colleagues. And lots of times there is free food!
• Talk to people you don't know....introduce yourself, make small talk. I've met the most interesting and diverse people at AEA...people I never would have gotten to know if I just hung out in my hotel room all day
• Get involved! Join a TIG and offer to help out. It's a great way to meet others in your area and it looks great on your vita!
• Bring business cards....and get cards from those you meet. Follow up with those you meet right after the conference. You never know what opportunities for future projects/presentations there could be with those you meet there
• Attend as many sessions as you can....you can learn from others strategies for presenting, learn about new directions in our field, and meet future colleagues

Don't

• Skip the morning plenary sessions so you can get that extra hour of sleep. Those sessions are some of the most well attended. So many networking opportunities...as well as listening to a top notch speaker
• Text your friends/significant other during the presentations. A quick hi is one thing but completely ignoring the presenters and texting the entire time....why even attend if you do that?
• Talk loudly to your neighbors during a presentation...even if you are in the back, guess what...the presenter can hear you....and boy it is distracting and rude
• Walk into a presentation after it has already started loudly and continue to make noise as you find a seat up front. Yeah, that is not distracting or anything
• Go into a presentation and grab a handout and leave. Why listen to what they are saying if you already have the handout right? Geez
• Stick like glue to your advisor/classmates and go to every breakfast/lunch/dinner with just them and no one else. You don't need to network at all
• Go out at night with your friends and overindulge in alcohol (boy have I seen this too many times). You never know what future employer you are going to run into when you are sloshed

Are You a Competent Evaluator?

As a faculty member in an evaluation program I'm constantly trying to ensure that our students become competent evaluators prior to graduation. The question is, "What makes an evaluator competent?" I've struggled with this quite a bit lately. Can I really ensure in 4 years (sometimes more) that my students are capable of conducting successful and ethical evaluations on their own? I'd like to think that we can. My colleagues and I spend countless hours going over our curriculum and making sure it covers the various competencies discussed in both the ECPE and CES frameworks....but is that enough?

One thing that constantly bugs me is that anyone can call themselves an evaluator. A person takes one evaluation class (or sometimes none), a few research methods/statistics courses and they label themselves as an evaluator. This drives me nuts!!! Many potential clients are counting on their evaluators to know what they are doing....if we don't have a certification or a license to ensure that the evaluator is competent in their field how do we know when someone is a competent evaluator?

Should we all have to apply for certification or a license to practice? If yes, what would the test(s) look like?

I'd love to hear others suggestions and comments on this.