Hurdle rates

Topics/tags: Rants, Grinnell, assessment, liberal arts curricula

Today I was reading through the CS Department’s Self Study, a huge [1] document prepared for our decennial external review. The Self Study contains a narrative (about 25 pages), faculty CVs [2] (about 80 pages), course syllabi and schedules of topics (about 160 pages), survey data (about 35 pages), a letter from the SEPC (a page), and enrollment data (about 25 pages). The last part of the enrollment data is a table of so-called hurdle rates, giving the percentage of students who complete eight-or-more credits [3] in each division, twelve-or-more credits, and sixteen-or-more credits.

Reading those data reminded me of the dangers of bad metrics. And yes, these kinds of hurdle rates are a bad metric. Let’s see why.

The data tell me that, over the six years studied, 88.9% of Grinnell students had 8+ credits in each division, 73.2% had 12+ credits in each division, and 50% had 16+ credits in each division. CS is doing slightly better. Of students majoring only in CS [4], 91.8% had 8+ credits in each division, 81.2% had 12+ credits in each division, and 54.1% had 16+ in each division. Surprisingly, our dual majors did slightly worse; 90.1% had 8+, 78.6% had 12+, and 50.4% had 16+.

What division is the low outlier? The Humanities Division. Over those same six years, only 79.1% had eight or more credits in each of the three divisions. That is, a smaller percentage of Humanities majors had eight credits in each division than pure CS majors had twelve credits in each division [5]. You might be saying something to yourself like, Those Humanities students are more likely to be narrowly focused. Or perhaps you’re thinking something about advising in the Humanities.

But you’re wrong to do so.

Here’s the thing. Consider a common six core elements of a liberal education: Foreign languages and cultures, arts [6,7], traditional humanities [8], the scientific method, reasoning in formal systems [9], and the study of societies [10]. Three of those elements fall in the Humanities division. So a science student who takes, say, a year of a foreign language can meet the 8+ hurdle rate in the humanities, even though they have not studied two of the six core elements. Of course, a CS major who has 16+ credits in each division could also miss taking any natural science courses that emphasize the scientific method. Saying that a humanities major who has failed to take 8+ credits in some division has automatically had a less good education is inappropriate.

For reasons like these, I consider divisional hurdle rates a vacuous measure. It worries me that the College continues to use them. And, every time I see them, I have to fight my inclination to treat them as useful data. They are not.

What are the alternatives? Course tags are supposed to provide one option. However, they suffer from other problems, including some too-narrow definitions of areas [11]. We could look at individual transcripts. However, transcripts often lack context. For example, a student who chooses not to take a foreign language may already speak two languages fluently. Similarly, a student who has not taken computer science may have demonstrated mastery, say, by developing a moderate-sized application to support her scholarship in another discipline.

I may not be a humanist, but I believe Grinnell is small enough that we could take a humanistic approach. For example, we might require each student to write a retrospective essay evaluating their liberal arts education. The folks who need more quantitative results could then hire a cadre of faculty to read those essays, identify representative examples [12], and assign categorical scores [14].

Faculty colleagues: Stand with me! Fight to discontinue the use of divisional hurdle rates as a reported measure anywhere on campus! [15]

Postscript: I would like to see more administrators develop statistical literacy. Too often, this group seems to treat numbers as a panacea. But you can tell that they don’t have a deep understanding of quantitative data when you start to probe them on their results, asking questions like, How were those ratios computed? or What majors are included in the interdisciplinary majors category? In my experience, folks who have developed such skills have already asked those questions and therefore know the answers.

[1] 323 pages!

[2] I’ve seen bout Curricula vitarum (records of lives) or Curricula vitae (records of life) used. I’ll stick with CVs.

[3] Approximately two courses.

[4] That is, excluding double majors.

[5] Boy, that was an awful sentence. I’m not going to rewrite it.

[6] Fine art, music, theatre, dance, etc.

[7] I’d prefer to separate the creation of art from the analysis of art, and require both. But most places combine them.

[8] E.g., philosophy, classics, and religious studies.

[9] Reasoning in formal systems is my new name for what we call quantitative reasoning and use to mean Math, Statistics, or CS. I’ve chosen the different term because many common uses of the term quantitative reasoning focus primarily on statistical reasoning.

[10] You might also call this Social Science or Social Studies.

[11] For the TL;DR crowd: The Quantitative Reasoning definition for course tags focuses only on statistical thinking, making it appear that neither CS nor Math courses should use the QR tag.

[12] For example, to provide to accreditors.

[14] I’d go with a simple scale of Excellent, Adequate, and Inadequate. But I’d be willing to debate other rating strategies and ways to build inter-rater reliability.

[15] I may no longer be on FOC, but I can still try to organize.

Version 1.0 of 2018-10-11.