Great Resource for Novice Evaluators - SAMMIE (http://sammie.osu.edu/)

In less than a month fall classes will start and I again will be teaching Program Evaluation I (not quite sure where the summer went but that's a whole different story). One of my favorite resources that I encourage my PE1 students to use is SAMMIE (http://sammie.osu.edu/), which was created by the Ohio State University Extension Office of Program Development and Evaluation.

SAMMIE is a web portal that links you to 22 different modules on a variety of evaluation topics. Some topics include: Developing a logic model, Ethical issues, Data collection and analysis strategies, Web based surveys, and Reporting your results. Each module links you to readings on the internet and includes a self-test to gauge what you learned from those readings. You can also create a personal page to jot down ideas about your own evaluation projects and receive feedback from other SAMMIE users. SAMMIE gives a great overview of the field of evaluation. I highly recommend it!

SAMMIE: http://sammie.osu.edu/

Conference Etiquette - How to Get the Most Out of the AEA Conference

I just booked my travel to the American Evaluation Association (AEA) conference that is in DC in October. I remember attending my first AEA conference in 1999 in Orlando, FL. I was still in graduate school at that time and still pretty green about how one should behave at a conference. I would go to the sessions that sounded interesting, eat lunch with people I knew and spend any free time taking in the local sites. Boy....lots of missed opportunities there! I wish I had had someone to pull me aside and tell me about all that conferences have to offer besides just stimulating sessions. Networking, networking, networking....I try to drill this concept in my students. Conferences are more than just a fun vacation away from classes and hanging out with your graduate school friends.

I've presented at hundreds of conferences over the years and I'm amazed by some conference goers, em...how do I say this...less than appropriate behavior at the conference. From walking in late to a talk and disturbing the speaker to answering a phone call during the middle of a presentation! My favorite is when the presenter reads his/her paper...verbatim, sitting down, looking down and not engaging the audience. What, I paid $1000 to travel to listen to someone read their paper? Ugh!

So as I prepare for AEA this year I plan on reviewing a few suggested Dos and Don't's with my students before we all travel to DC. If anyone has any additional suggestions I'd love to hear them! Hope to see many of you at AEA!  :)

Some Conference Do's and Don't's:

Do

• Prepare your presentation/poster well in advance of the conference
• If creating a poster, make it look professional and inviting to your audience (tacking up a few 8.5 by 11 sheets of paper on a board is not what we want to see)
• If creating a paper presentation have slides and/or a handout prepared and practice your speech beforehand
• SHOW UP FOR YOUR PRESENTATION/POSTER SESSION! Throwing up the poster and going out to explore the town doesn't count as doing a presentation. And putting up a piece a paper to capture emails of those interested in your poster doesn't count either
• Go to a TIG meeting or two.....great place to network and meet new colleagues. And lots of times there is free food!
• Talk to people you don't know....introduce yourself, make small talk. I've met the most interesting and diverse people at AEA...people I never would have gotten to know if I just hung out in my hotel room all day
• Get involved! Join a TIG and offer to help out. It's a great way to meet others in your area and it looks great on your vita!
• Bring business cards....and get cards from those you meet. Follow up with those you meet right after the conference. You never know what opportunities for future projects/presentations there could be with those you meet there
• Attend as many sessions as you can....you can learn from others strategies for presenting, learn about new directions in our field, and meet future colleagues

Don't

• Skip the morning plenary sessions so you can get that extra hour of sleep. Those sessions are some of the most well attended. So many networking opportunities...as well as listening to a top notch speaker
• Text your friends/significant other during the presentations. A quick hi is one thing but completely ignoring the presenters and texting the entire time....why even attend if you do that?
• Talk loudly to your neighbors during a presentation...even if you are in the back, guess what...the presenter can hear you....and boy it is distracting and rude
• Walk into a presentation after it has already started loudly and continue to make noise as you find a seat up front. Yeah, that is not distracting or anything
• Go into a presentation and grab a handout and leave. Why listen to what they are saying if you already have the handout right? Geez
• Stick like glue to your advisor/classmates and go to every breakfast/lunch/dinner with just them and no one else. You don't need to network at all
• Go out at night with your friends and overindulge in alcohol (boy have I seen this too many times). You never know what future employer you are going to run into when you are sloshed

Are You a Competent Evaluator?

As a faculty member in an evaluation program I'm constantly trying to ensure that our students become competent evaluators prior to graduation. The question is, "What makes an evaluator competent?" I've struggled with this quite a bit lately. Can I really ensure in 4 years (sometimes more) that my students are capable of conducting successful and ethical evaluations on their own? I'd like to think that we can. My colleagues and I spend countless hours going over our curriculum and making sure it covers the various competencies discussed in both the ECPE and CES frameworks....but is that enough?

One thing that constantly bugs me is that anyone can call themselves an evaluator. A person takes one evaluation class (or sometimes none), a few research methods/statistics courses and they label themselves as an evaluator. This drives me nuts!!! Many potential clients are counting on their evaluators to know what they are doing....if we don't have a certification or a license to ensure that the evaluator is competent in their field how do we know when someone is a competent evaluator?

Should we all have to apply for certification or a license to practice? If yes, what would the test(s) look like?

I'd love to hear others suggestions and comments on this.

Importance of Double-Checking Data in Reports

This summer I am swamped with evaluation projects....three separate grants...three sets of deliverables...and never enough graduate student support it seems to get things done. Over the  years I've gotten much better at giving up some control of my projects to my advanced students....so they can learn project management and I can work under 70 hours a week.  :)

However, even though I may give up the reigns from time to time I can't stress enough the importance of having multiple eyes review report data prior to a report going out to a client. Recently I had a student draft a report for a very important client (i.e., one who actually pays us and hopefully will hire us again!) and as I was reviewing the tables something seemed off. The means and standard deviations for some of the items just didn't make sense. Since I keep copies of all versions of the datasets (so important!) I went back to the one used to generate the report and reran some of the analyses. SO GLAD THAT I DID!!!! The numbers in the table were wrong....not just a little wrong or small typos but completely wrong, copied incorrectly from the output files. Rating scales were reversed, numbers were transposed...it was a mess. After my slight meltdown, since the report was due that day and there was no way that this was going to get fixed quickly...I reran all analyses and redid the tables. Yes, the report was a couple of days late....but it was correct and ultimately that is what I want to be able to give to my client.

So....the morale of this post. First, always have multiple people review the data in the tables...you never know when mistakes could be made. Second, don't wait till last minute to get a draft of the report to review.....you never know how much editing might be needed. Lastly, keep all versions of all data....so much easier to detect mistakes along the way.

Oh, and always have emergency chocolate handy in your office. Definitely helped me get over my meltdown a lot quicker.

:)