Right Wing Nation

Let’s talk evaluations, starting with this article from Inside Higher Ed:

But what if the much derided Web site’s rankings have a high correlation with markers that are more widely accepted as measures of faculty performance? Last year, a scholarly study found a high correlation between RateMyProfessors.com and a university’s own system of student evaluations. Now, a new study is finding a high correlation between RateMyProfessors and a student evaluation system used nationally.

A new study is about to appear in the journal Assessment & Evaluation in Higher Education and it will argue that there are similarities in the rankings in RateMyProfessors.com and IDEA, a student evaluation system used at about 275 colleges nationally and run by a nonprofit group affiliated with Kansas State University.

What is notable is that while RateMyProfessors.com gives power to students, IDEA gives a lot of control over the process to faculty members. Professors identify the teaching objectives that are important to the class, and those are the measures that count the most. In addition, weighting is used so that adjustments are made for factors beyond professors’ control, such as class size, student work habits and so forth — all variables that RateMyProfessors doesn’t really account for (or try to account for).

The study looked at the rankings of 126 professors at Lander University, in South Carolina, and compared the two ratings systems. The findings:

• Student rankings on the ease of courses were consistent in both systems and correlated with grades.
• Professors’ rankings for “clarity” and “helpfulness” on RateMyProfessors.com correlated with overall rankings for course excellence on IDEA.
• The similarities were such that, the journal article says, they offer “preliminary support for the validity of the evaluations on RateMyProfessors.com.”

It has been my experience that those who discount evaluations are bad teachers. In fact, I can think of many examples of faculty I know who discount evaluations and who are bad teachers, but I cannot think of even one counterexample. I do not know even one faculty member who is a good teacher and discounts evaluations. Not one. But I know lots of bad teachers who discount evaluations.

The most common statement about evaluations is that students care only about how easy the course is and what grade they get. But it takes little more than a quick perusal of professor ratings on RateMyProfessors.com to see that the assumption is a fallacy.

Let’s look at the categories RateMyProfessors.com uses to evaluate professors (1-5):

Easiness - Some students may factor in the easiness or difficulty of the professor or course material when selecting a class to take. Is this class an easy A? How much work do you need to do in order to get a good grade? Please note this category is NOT included in the “Overall Quality” rating.

Helpfulness - Helpfulness is defined as a professor’s helpfulness and approachability. Is this professor approachable, nice and easy to communicate with? How accessible is the professor and is he/she available during office hours or after class for additional help?

Clarity - A professor’s organization and time management skills can make a great difference on what you get out of the class. How well does the professor teach the course material? Were you able to understand the class topics based on the professor’s teaching methods and style?

Overall Quality - The Overall Quality rating is determined by the average rating of the Helpfulness and Clarity given by all users. An overall rating of 3.5 to 5 is considered good (yellow smiley face). An overall rating of 2.5 to 5 is considered average (green smiley face). An overall rating of 1 to 2.5 is considered poor (blue sad face). The Easiness rating is NOT included when calculating the Overall Quality rating.

Rater Interest - There is always that one class everyone recommends taking before graduating. As a student, how interested were you in the class, BEFORE taking it? Or how interested were you in taking this course from this specific professor.

So easiness is included, but crucially, it is not included in the overall rating. There goes the cornerstone of that major assumption about evaluations, that they are nothing more than evaluations of the ease of a course. My university evaluations are admittedly superior as rating tools: There are more questions, half of which relate to the course and the other half to the instructor, and they are more detailed. RateMyProfessors.com is by no means a rigorous tool for rating teacher effectiveness, but then, it’s a web application, and if there were twenty questions on it, nobody would use it.

I said above that a quick perusal of the evaluations would destroy the major assumption about RateMyProfessor.com, and here is an example:

We may assume that this student is a history major. Note that the student calls the professor “demanding,” and gives him only a 2 on easiness. Yet, this professor gets a glowing review. Here is another, from a different professor, at a different university, in a different field:

The first evaluation was (presumably) from a major, and an upperclassman. This evaluation is from a student who dislikes the course subject, and admits that he has a hard time with it. It also looks like this student is most likely a freshman or a sophomore, to judge from the course number (I replaced the letters, but not the first course number, because it indicates the level of the course). This student gives this professor the lowest possible score for easiness (1), yet like the last evaluator, gives the professor a glowing review. How is this possible, if evaluations are nothing more than a popularity contest, and students rate primarily on how easily they can get As?

But what this article misses is that one hears exactly the same complaints among faculty about university student evaluations as RateMyProfessors.com. It’s one of the Laws of the Faculty Lounge that if there are tenured faculty present and student evaluations come up, at least one professor will make a sneering comment. As a group, tenured faculty give little concern to teaching (which renders the whole “I want a real professor to teach my class!” argument ludicrous.). Don’t believe me? Here it is from the Chronicle of Higher Education:

Colleges are quick to argue that a college education is more about enlightenment than employment. That may be the biggest deception of all. Often there is a Grand Canyon of difference between the reality and what higher-education institutions, especially research ones, tout in their viewbooks and on their Web sites. Colleges and universities are businesses, and students are a cost item, while research is a profit center. As a result, many institutions tend to educate students in the cheapest way possible: large lecture classes, with necessary small classes staffed by rock-bottom-cost graduate students. At many colleges, only a small percentage of the typical student’s classroom hours will have been spent with fewer than 30 students taught by a professor, according to student-questionnaire data I used for my book How to Get an Ivy League Education at a State University. When students at 115 institutions were asked what percentage of their class time had been spent in classes of fewer than 30 students, the average response was 28 percent.

That’s not to say that professor-taught classes are so worthwhile. The more prestigious the institution, the more likely that faculty members are hired and promoted much more for their research than for their teaching. Professors who bring in big research dollars are almost always rewarded more highly than a fine teacher who doesn’t bring in the research bucks. Ernest L. Boyer, the late president of the Carnegie Foundation for the Advancement of Teaching, used to say that winning the campus teaching award was the kiss of death when it came to tenure. So, no surprise, in the latest annual national survey of freshmen conducted by the Higher Education Research Institute at the University of California at Los Angeles, 44.6 percent said they were not satisfied with the quality of instruction they received. Imagine if that many people were dissatisfied with a brand of car: It would quickly go off the market. Colleges should be held to a much higher standard, as a higher education costs so much more, requires years of time, and has so much potential impact on your life. Meanwhile, 43.5 percent of freshmen also reported “frequently” feeling bored in class, the survey found.

Ha! Students are idiots! What do they know? Read on:

College students may be dissatisfied with instruction, but, despite that, do they learn? A 2006 study supported by the Pew Charitable Trusts found that 50 percent of college seniors scored below “proficient” levels on a test that required them to do such basic tasks as understand the arguments of newspaper editorials or compare credit-card offers. Almost 20 percent of seniors had only basic quantitative skills. The students could not estimate if their car had enough gas to get to the gas station.

Unbelievably, according to the Spellings Report, which was released in 2006 by a federal commission that examined the future of American higher education, things are getting even worse . . .

There follows a lengthy discussion of what to do about the problem. But one thing that can be done, the one thing that is relevant here, is to start taking student evaluations seriously, and not just for teaching faculty.

Student evaluations are by no means perfect. Yes, there are students who care about nothing but how easily they can get As. But at least our evaluations, and the ones at RateMyProfessors.com, can be filtered. Our evaluations are done on a scantron, and on the back are four short answer questions:

1. What did you like most about this course/instructor?
2. What did you like least about this course/instructor?
3. What could the instructor do to improve the course or his or her teaching effectiveness?
4. Other comments?

The comments are linked to the evaluation scores on the other side of the scantron. Non-serious comments indicate a non-serious evaluation. In other words, one can safely ignore the evaluation of a student who answered, say, the second question with something like, “Give us all As.” And yes, you do get non-serious comments, because as I admitted, there are students who care about nothing but how easily they can get As.

However, that some students care about nothing other than the ease with which they can get As does not imply that all or even most students care about nothing other than the ease with which they can get As, nor does it discount the student evaluation as a rating tool. One would assume that somebody with a PhD would have enough basic intelligence to grasp that, but far too many do not — or they do, but would rather discount evaluations rather than face up to the fact that they might not be the brilliant teachers they think they are.

Tenured or teaching faculty, if you are a good teacher and get good evaluations, you have no reason to sneer at them. In fact, if you get good evaluations, yet claim that evaluations are nothing more than a popularity contest, then you are admitting that you are a poseur who does not take your career seriously and that you hand out As like candy. And how stupid would you have to be to do that?

Again, some evaluations are crap. But many of them can be filtered, because they are accompanied by crap comments. This is a useful, but sloppy filter, since not all students bother to write comments on the back. Still, one may filter some of the, ahem, less than useful evaluations.

Note that the same is true on RateMyProfessors.com: Each evaluation is accompanied by comments, and non-serious comments indicate non-serious evaluations. But also note that by no means are all comments non-serious. Neither are the comments on our evaluation scantrons.

At the university where I worked, student evaluations were required to be given in all classes. However, they were largely ignored for tenured (and tenure-track) faculty, but used as a major employment criterion for adjunct, or teaching, faculty. There is a certain amount of logic to this, except that it relieves tenured faculty of any review of their teaching skills. And unsurprisingly, it is always tenured faculty who sneer at evaluations. Teaching faculty do not, because if they get bad evaluations, they are not re-hired — and they shouldn’t be.

Of course, neither should the university harbor tenured faculty who consistently get bad evaluations, but that’s another topic, and addressing that problem would require at least a major redefinition of tenure, if not abolishing it altogether.

But back to student evaluations. They serve two major functions: They provide teaching effectiveness feedback for the department, school/college, and university, and they provide teaching effectiveness feedback for the faculty member. I, at least, would hold that both are equally important. At the university, they provide yet a third function.

Evaluations as school/university feedback

If evaluations are to be used administratively as teaching effectiveness rating tools, a number of issues must be addressed. Evaluations are subjective, after all. As I said earlier, some students care about nothing other than ease. Also, a statistician should design the evaluation, and not, say, a committee of teachers or the school board. Some outside agency should administer, or at least design and process, the evaluations.

Because of the nature and purpose of student evaluations, there should be no “undecided” option available to students. That is, the available options should force students to make either a positive or negative response of some degree. Student evaluations are not political polls, where “undecided” can often give useful information. Student evaluations are rating tools, and as such, should force a rating. After all, a student who, after a whole semester, is “undecided” about the difficulty of the course or some aspect of the teacher’s effectiveness obviously doesn’t have much of value to say about the course.

Evaluations have to be controlled for non-serious ratings. As I said above, one method (admittedly sloppy) is to use the comments as a context for the evaluation. This is not really possible when the administration is using the evaluations to rate teacher effectiveness, however. Another option would be to trim the evaluations, that is, remove the top and bottom scores from a teacher’s evaluations when calculating overall scores. The problem with this method is trimming too much, easy to do if one is teaching small courses of, say, only forty students (removing the top two and bottom two from forty evaluations is the maximum). The problem is that with smaller classes — say again forty students — you aren’t trimming enough data to control for the variable (but for larger classes, like 240 students, this works well).

Perhaps the best solution is to rate all faculty after all of the evaluations are in and analyzed. Calculate population means and confidence intervals for the different criteria, and compare faculty member scores not to other faculty member scores, but the population means. This will neutralize the “lazy/disaffected student factor” and rate all faculty members on the same scale.

But because evaluations are subjective, it is also important not to use them as a sole measure of teaching effectiveness. Use them, yes, but use them along with another, more objective, measure, such as test scores. If you are an administrator and you do this, I suspect you will find, as our department did, that there will be a high correlation between student evaluations and test scores.

Evaluations should be processed individually by class, and should not be processed by instructor. John Smith may be teaching Algebra and Geometry, and may get very different evaluations in the two classes. He should therefore receive evaluations for each of his classes, instead of an overall evaluation. Even if he is teaching three Algebra courses, and all are the same content, the three should be processed individually. Excessive aggregation masks differences in data, and obscures the results of the evaluations. Aggregating evaluations by class also allows the administration to spot faculty members’ strengths and weaknesses.

Evaluations as faculty feedback

Student evaluations can provide faculty with crucial information. Remember that perceptions differ on either side of the desk. You may believe that you are being clear, but that doesn’t mean your students find you clear. The same holds for almost any criterion you can put on an evaluation.

The biggest problem with student evaluations is their infrequency. Only once at the end of the semester really isn’t helpful for your current classes, although you can use the information to improve your teaching for future classes. Another problem is that they provide too little data. As an example, let’s say you read your evaluations and find that students don’t think you’re as clear as you could be. The evaluations do not tell you much about how you could improve.

For these reasons, I administered informal, short-answer only, evaluations three times throughout the semester: a quarter of the way through, around midterms, and three-quarters of the way through. I explained to students that the evaluations were only for me and them, so I could find out from them how I could better present the material, and I encouraged them to be as detailed as they felt they needed to be. I got some crap, certainly, but I also got some valuable feedback about where and how I was being unclear or unhelpful, and adjusted my teaching accordingly.

I also always asked my students when they came to office hours (after we had dealt with their problems) how I could have better presented the material they had come to ask about. Never miss the opportunity to ask your students how you’re doing. Your students are not idiots. If you ask, they will have some important things you need to hear. Stop and listen.

Also, take evaluations into account along with your students’ test (or course assignment) scores. If your students’ grades are consistently low, even if you teach a difficult course, you could be more effectively presenting the material. Poll your students even more frequently, and find out what they think you could be doing. Use the test or assignment results to tell you what your students are not getting. Keep your thumb on the pulse of the class — and you can’t do that if you’re not paying close attention.

Evaluations as consumer information

The university differs in one crucial way from the elementary, middle, or high school: University students have the luxury of choice. The student can choose his major and degree program. The student can choose his courses based in part on which faculty members are teaching them. After all, if you have to take M125 and three faculty members are teaching it, why not take it with the best of the three, or at least not take it with the worst?

When I was an undergraduate, that choice was mitigated by a lack of technology, and we were limited to word of mouth. Now we have the web, and RateMyProfessors.com, and many universities have their own forums for rating faculty. Today’s university students is far more informed about faculty than we were, and can make better choices.

Of course, there is that silly objection to students as consumers, but face it, that’s exactly what they are. The course is for them, after all, not your ego. That alone makes them consumers, and you, the provider. If you’re a godawful teacher, but your colleague is a really good teacher, then students have every right to avoid you and take your colleague’s class instead, and you would do exactly the same.

Finally, I think that student evaluations are, if anything, underused. They are certainly underused in public schools, particularly where teachers’ unions block any kind of teacher evaluation, and just as underused by universities with respect to tenured faculty who should be seriously evaluated on their teaching skills, but even more importantly, they are underused by well-meaning teachers, who could be using them as a powerful tool for improving their own effectiveness. If you teach, and if you care whether you are a good teacher, start asking your students for guidace, and take their answers seriously.