Do professors care whether college students are actually learning?

The lead article in the 25th anniversary issue of the peer-reviewed Journal of Public Affairs Education , by Heather Campbell, is a deep-dive review of research that throws significant shade on student evaluations of teaching (SETs). SETs do not measure student learning, and may actually have the wrong sign (not to mention that their inherent gender, age, and racial/ethnic bias means their use for personnel decisions is probably illegal). Universities like my own, in which SET scores are the main, or often the only, teaching evidence used in promotion and tenure decisions are systematically damaging student learning.

From the abstract:

In many if not most colleges and universities in the United States,
raw scores from Student Evaluations of Teaching (SETs) are the
primary tool of teaching assessment, and teaching evaluations
often have real consequences for promotion and tenure. In 2005,
JPAE published an article on teaching evaluations, and this article
added to what was at that time a somewhat thin literature indicating
that SETs are systematically biased against female faculty,
and probably against older and minority faculty. Since that time,
this literature has swelled and grown and now the evidence that
SETs are invalid and systematically biased is too strong to ignore.

Why do we keep doing this? I have four hypotheses:

(1) A body of research of which I am unaware and can’t find refutes the findings Campbell summarizes. This is certainly the most generous conjecture, but JPAE is peer-reviewed and reviewers would have rejected this paper if they knew about such material. Not to mention reviewers for all the publications Campbell cites.

(2) Confirmation bias. We used to think SETs were valid indicators, and we subconsciously reject evidence that would change our mind. This mode of inference has recently been validated by our US president, who “just knows things” that are contradicted by every kind of expertise and evidence.

(3) Fecklessness and laziness. We assert our commitment to good teaching if asked, but actually we just don’t care enough to do anything that would actually advance it. The joke is, “Teaching is the tax you pay to do your research; tax evasion is a crime, but tax avoidance is the duty of a citizen” and the corollary is, “why are you talking about whether students are learning, when I have a research article to finish writing for my academic peers to admire?” SETs take a distasteful task (collegial responsibility for better teaching) off my desk and load it onto an unpaid, docile labor pool (students); what’s not to like?

(4) Fear. I have never received evidence I can respect as a scholar, or any other way, that I am any good at teaching or could become so (I do have evidence of this kind that I can do research OK and that I’ve gotten better at it over the years). Teaching is affectively fraught, and like everyone I know, I’m sure my emotional intelligence is not what it should be. The ego hit of talking while a roomful of people write down everything I say (lecturing, just as an example of a dubious pedagogical habit) is a lot to risk by trying to learn a new skill. Anyway, improving my teaching will take a lot of time and my job depends on publishing.

We would be a lot better off if we could shift our attention more generally from summative evaluation (at promotion time) to formative methods (coaching and experimentation between these high-anxiety moments). I suppose one could believe that college teachers only respond to money and status incentives, so if we reward good ones and ding or fire bad ones, we will eventually have only good teaching, but one would be wrong (ask any successful manager whether you can fire (or bribe) your way to success). One would be especially wrong if your filter for “good teachers” doesn’t measure student learning.

What we need is not a cheap, lazy way to pretend we are improving our teaching, but a real quality assurance program that a Google or Toyota manager, for example, would recognize as such. Got kids choosing a college? on your junior year visit, ask what their QA program is, be sure it doesn’t depend on SETs, and don’t be distracted by the fancy athletic facilities. Are you a student, paying through the nose with your time and money for the best possible education? Do the same, and if you don’t get good answers, recruit your classmates to go in the quad with pitchforks and torches.

Author: Michael O'Hare

Professor of Public Policy at the Goldman School of Public Policy, University of California, Berkeley, Michael O'Hare was raised in New York City and trained at Harvard as an architect and structural engineer. Diverted from an honest career designing buildings by the offer of a job in which he could think about anything he wanted to and spend his time with very smart and curious young people, he fell among economists and such like, and continues to benefit from their generosity with on-the-job social science training. He has followed the process and principles of design into "nonphysical environments" such as production processes in organizations, regulation, and information management and published a variety of research in environmental policy, government policy towards the arts, and management, with special interests in energy, facility siting, information and perceptions in public choice and work environments, and policy design. His current research is focused on transportation biofuels and their effects on global land use, food security, and international trade; regulatory policy in the face of scientific uncertainty; and, after a three-decade hiatus, on NIMBY conflicts afflicting high speed rail right-of-way and nuclear waste disposal sites. He is also a regular writer on pedagogy, especially teaching in professional education, and co-edited the "Curriculum and Case Notes" section of the Journal of Policy Analysis and Management. Between faculty appointments at the MIT Department of Urban Studies and Planning and the John F. Kennedy School of Government at Harvard, he was director of policy analysis at the Massachusetts Executive Office of Environmental Affairs. He has had visiting appointments at Università Bocconi in Milan and the National University of Singapore and teaches regularly in the Goldman School's executive (mid-career) programs. At GSPP, O'Hare has taught a studio course in Program and Policy Design, Arts and Cultural Policy, Public Management, the pedagogy course for graduate student instructors, Quantitative Methods, Environmental Policy, and the introduction to public policy for its undergraduate minor, which he supervises. Generally, he considers himself the school's resident expert in any subject in which there is no such thing as real expertise (a recent project concerned the governance and design of California county fairs), but is secure in the distinction of being the only faculty member with a metal lathe in his basement and a 4×5 Ebony view camera. At the moment, he would rather be making something with his hands than writing this blurb.

3 thoughts on “Do professors care whether college students are actually learning?”

  1. Interesting that there is a claimed bias against female teachers when the overwhelming majority of students are female.

  2. “If you don’t get good answers, recruit your classmates to go in the quad with pitchforks and torches.”
    Addressing an angry crowd is perhaps the ultimate test of public speaking. The record is held by the Emperor Claudius’ Greek freedman Narcissus who broke a mutiny of four legions at Boulogne in AD 43 just by addressing them. (One despised foreigner and ex-slave against 20,000 angry and heavily armed professional soldiers in one of the best armies in history).

    Wikipedia says it was done by humour:

    In 43, during the preparations for the Roman conquest of Britain, he headed off a mutiny by addressing the troops. Seeing a former slave in their commander’s position, they cried “Io Saturnalia!” (Saturnalia was a Roman festival when slaves and masters switched places for the day) and the mutiny ended.

    You have to suspect that Narcissus staged the intervention to break the ice and gain a hearing. He still had to win the legionaries over and get them on to the invasion ships with some mixture of flattery, bribes and threats. He couldn’t use threats much, as the legions were themselves the Empire’s ultimate threat.

    The analogy with teaching is imperfect. The object of rhetoric is persuasion, in a good or bad case. The teacher’s goal is to elicit thinking. There are common elements: gaining attention and a hearing is one.

  3. I will say that, to the extent you’re right, it applies mostly to research-oriented institutions. My career was spent at teaching-oriented institutions, and my experience was that not only were factors other than CTEs important in the assessment of teaching, the importance of non-CTE factors has increased over time.

    (Just a reminder, if it’s really needed) than not all of us–indeed, likely not even most of us–word in research-first institutions.

Comments are closed.