Right Wing Nation

When you teach a high-enrollment, mandatory course to highly competitive, goal-oriented students, fairness is paramount. One student can cause a great deal of pain by going to the proper university office, and we all want to treat our students fairly. For those who teach smaller courses and have as few as 40 or so students, personal integrity and professionalism should be enough. In our case, it was not.

The course (actually, a two-semester course sequence, but the assessment is identical in both) is a mandatory data-analyis course. In the first semester course, enrollment averages 1700 in the fall semester and 1200 in the spring semester (because a student must get a C or better in the first semester course to take the second, enrollments in the second-semester course average 1200 and 800 for the fall and spring semesters, respectively). There are one fifty-minute lecture and two fifty-minute labs per week. Lectures are held in one of the several lecture halls, and average around 240 students per section. Labs are held in computer clusters, and range from 24-45 students per section, depending on the size of the computer cluster in which the lab section is held (do the math–that’s a lot of computer clusters across campus).

Because the course has such a large enrollment, and because so many lab sections must be taught, over half of the instructors are graduate students, nearly all of whom have no teaching experience, and no interest or investment in teaching. Also, since this is a university and not the real world, we–meaning those of us in charge of the two-semester sequence–have no control over which graduate students teach the class or whether they are allowed to continue to teach, no matter how poorly they perform (the university sees it as financial aid instead of employment). This presents a set of problems, the relevant ones here being how we control both the lack of teaching (assessment) experience and teacher (assessment) subjectivity.

Students are the ultimate control for fairness. Even in such a large course, when one instructor doesn’t cover something other instructors do, it not only gets around, but it gets back to us, usually within 24 hours, and often, sooner. If one instructor were grading a project one way and another were grading another way, or if one instructor bumped up Jeannie’s grade because he felt sorry for her, or if one student got credit for an answer while other students did not, students would be lined up in front of our offices to raise hell–and justifiably so. Subjectivity cannot be justified because it is inherently unfair, and autonomy does not protect the unfair, inconsistent assessment of students.

The solution was to develop a system of assessment that was wholly consistent and wholly fair. This is how we did that.

The assessment system is exclusively performance-based, and no criteria are subjective. Here is the grade breakdown for the class (1000 points possible):

Each written exam is 100 multiple-choice questions (five distractors), and is written to test abstract concepts covered in the lecture portion of the class, as well as the student’s ability to extract abstract information from a problem (as are the ten unannounced quizzes, each of which is five multiple-choice questions with five distractors). Anything covered in any of the materials for class is fair game for assessment. This seems to be a foreign concept to students, who believe for some reason that there is only a small subset of what the course covers that will appear on an exam. The questions most students miss are those taken from the reading assignments but not overtly covered in the lecture (in other words, students don’t do the assigned reading, and don’t believe you when you tell them that anything from the material can appear on the exam). An example question which tests the student’s ability to analyze a problem and abstract crucial information from it:

“SomeCorp, Inc. produces widgets. Each widget costs $1.1942 to produce, and markup is 113%. Customer demand is 8498, 7742, 9023, and 8936 for the next four weeks, no backlogging is allowed, and excess widgets must be warehoused. Warehousing costs are $0.0943 per widget per day, and maximum warehouse capacity is 652. What crucial piece of information do we need to supply to SomeCorp, Inc so they can minimize their total costs over the next four months?”

A. The number of customer orders over the next four months
B. The cost of widgets produced over the next four months
C. The gross profit margin over the next four months
D. The number of widgets to produce over the next four months
E. The warehousing costs over the next four months

(D, by the way, is the correct answer.)

Practical exams, like the projects, assess problem-solving skills in Excel (the ability of a student to solve a problem similar to the above is assessed on the practical exam). Students are given the Excel exam file on a flash drive, a printed copy of the questions, and have two hours to work through the exam. All exams are administered in a closely monitored, timed environment (any student who does not hand in his written or practical exam when time is up is given a zero). Testing administrations are tightly monitored, with at least two proctors in the small rooms, and as many as six in large lecture halls. No cell phones or electronic devices are allowed in the room, students are staggered in every other seat of every other row, and given alternating versions of the exam. The administrations of practical exams are similarly secure. The only application that can be running on any student’s machine is Excel. All infractions of testing policy are considered academic dishonesty. The student’s materials are collected, he is ejected from the room, and academic dishonesty charges are filed against him with the Dean of Students. The projects are the only component of the course worked outside of class or a supervised environment.

No criterion is subjective. There is no participation component, no attendance component, no self-esteem component. Of course, grading itself, even on objective criteria, can be subjectivized by the instructor, but even that has been purged from the assessment system. (In a sense, the quizzes can be thought of as partially an attendance grade, since they are given at the end of lectures and only those who attend can take the quiz, but only partially, since the quizzes are scored. Also, while students are not required to attend class, they are expected to attend, and know that if they skip lecture, they may be skipping a quiz. No makeup exams or quizzes are given, and no late project submissions are graded.)

Included in each practical exam and project Excel file are two VBA modules. One grades the file, logs in to the central gradebook, and uploads the score (grading is a process of downloading a zip file of student files, opening Excel and running the grading module for that project or exam). The other secures the file against cheating. If any instance of cheating (copied and pasted cell contents, one student turning in another student’s file, changed times and dates in files, etc.) is caught by the module, the instructor is alerted, the file is flagged, and the student is automatically emailed a message including the university statement on academic dishonesty, stating that the student’s file was flagged for possible academic dishonesty, and “requesting” that the student make an appointment to see the coordinator within the next five days (if the incident includes some other student or student’s file, for example if one student turns in another student’s file, both students’ files are flagged, and both are notified). There are numerous beartraps in the file in case a resourceful (ahem) student cracks the VBA password and hacks into the code. In the case of practical exams, the security module also prohibits a student from opening a file not on the flash drive.

Every point of every Excel assignment is graded by program. Instructors are locked out of the system. At no time and in no way does any instructor have the ability or opportunity to add, subtract, or change points for any student.

Certainly, errors creep into written exams, even though each exam goes through an extensive review process before it goes to be printed and copied. For this reason, written exam results are rigorously analyzed statistically before exam scores are recorded in the gradebook. Any bad question is discarded from the exam before student scores are recorded.

Here is the grading scale (the total points can vary if exam questions are tossed, in which case the total point grade cutoffs are recalculated):

The scale is strictly points-based with no rounding. If Susie totals 729.75 points at the end of the semester, she gets a C-, even if she is only 0.25 points away from a C. Letter grades are calculated and reported centrally. The instructor has no access to the central grades, and cannot add or subtract points, or otherwise adjust grades. The instructor may, of course, petition the faculty in charge of the course to adjust grade, which new grad student instructors sometimes do, but the answer is always no. (Students do have emergencies, and they are dealt with as they arise, but no student’s grade is adjusted at the instructor’s whim.)

Education is not golf. There are no handicaps. Each week, if the student does an additional exercise, goes online and logs in, uploads his file, and answers a five-question quiz about the exercise, he can earn one bonus point, up to a maximum of fifteen bonus points. In order to get bonus credit, the Excel file uploaded must be worked, the file must be uploaded and the quiz worked, both from the same machine in the correct computer cluster during the student’s class, and three out of the five quiz questions must be correct (if the student tries to upload the file or answer the quiz from a computer other than one in his lab section cluster or at a time other than his assigned lab, he will get a message that tells him he may only upload files or answer the quiz during his lab section). So while there are bonus points available (1.5%), there is no way to give Marjorie a handicap to boost her self-esteem, or because she’s a disadvantaged, pigeon-toed, transgendered lesbian of color, or because she was abducted and anally-probed by space aliens. Likewise, there is no way for an instructor to adjust Billy’s grade down because he didn’t like him, or because Billy asked questions in class that made the instructor feel uncomfortable, or because the instructor didn’t like Billy’s politics. Each student is assessed exclusively on his performance and nothing else. And because each assessment is graded by program and not by a human instructor, every item on every assessment for every student is graded in exactly the same way and given exactly the same weight.

The only way to remove subjectivity from assessment is to remove all opportunities for subjective assessment. And the only way to remove instructor bias from assessment is to take assessment entirely out of the hands of the instructor. The system is “cold” in the sense that it makes no allowances for circumstance (note that if a student really does have a death in the family, or some other excusable problem, we do make allowances for it–but not in the grading system, and not by accepting late projects) and students do sometimes complain about that, but no student has ever complained that the system is unfair.

Just as importantly, all element of reward has been removed from the assessment. Students are assessed based solely on how they perform, and are not “given” a grade to reward them for being responsible students. We don’t reward students for being responsible; we expect students to be responsible. Whether or not they went to class or participated has no bearing on their assessment. Whether they liked or were liked by their instructor has no bearing on their assessment. Nothing but how well they learned the material determines their assessment–and that’s just as it should be.

We do not “teach to the test.” Assuming that students have the arithmetic and algebraic fluency required, teaching the material prepares students for the test. Practical exams follow the same format as the problems we do in class and the project problems. The last lab before the administration of the practical exam is a review day, and students are supplied with review files to work in class (there is too much material to cover to allow any more than one review day). We used to do reviews for the written exams, but dropped them. Students did not prepare for the reviews, and review days became wastes of valuable time.

Is this high stakes testing? Given that this is a mandatory class, and given that the exams comprise 80% of the class grade, yes. Does this testing or assessment somehow get in the way of learning? So far, no student or faculty member has come to us to complain that anyone’s creativity or “higher-order thinking” was impaired, but we have had many students thank us later for teaching them the skills they need, and quite a few faculty colleagues tell us how well prepared for their classes the students are. Teachers who think that the only way to prepare students for an exam is to emulate it in class lack imagination or intelligence or most likely both. Bear that in mind the next time you hear some teacher whining about having to “teach to the test.”

The best way to ensure fairness is to use performance-based assessment. While implementing the VBA code in every project and exam is extremely work intensive, removing all instructor subjectivity and catching all occurences of cheating make it well worth the trouble. But a subjective assessment system can only return a subjective–and unfair–assessment.

Other education articles here.