A paper for the next year or two: The Long Hard Slog of Improving Student Learning Outcomes

A paper (or more likely a letter) that we will get around to writing at some point is the narrative of the long arc of how we went about making changes to our algebra-based physics course.

The short version of the story is that over the course of a decade, the department implemented many popular research-based physics teaching strategies with no measurable effect on student learning gains as measured by the FCI. Our normalized learning gains held strongly in the 0.20–0.25 range (with no individual class ever scoring above ~0.3), despite efforts to use research based teaching strategies such as

• Collaborative problem solving (white-boarding)
• Peer Instruction (clicker questions), and
• Learning Assistant led Tutorials

In response, a more comprehensive curricular overhaul was developed over a 2-3 year period. Early piloted versions with developers and select instructors achieved normalized gains in the 0.40 – 0.60 range. Our first year of full implementation (with ~10 different instructors across 20 different sections) achieved an average normalized gain of 0.46.

The perhaps interesting / challenging part of the story is that pretty much the exact same teaching strategies listed above (for the older course) also describe the new course structure. From a bird’s eye view, nothing about the course changed. Perhaps not surprising, but worth unpacking, is how improved learning outcomes were maybe achieved through a combination of efforts to (1) develop/adapt instruction to local conditions, (2) design for curricular coherence, (3) and provide on-the-job professional development.

Our story I think has relevance to broader physics education community for a few reasons:

1. Our experience with several failed efforts to improve learning outcomes through popular research-based strategies is probably common just under reported. I do not take this to mean that these strategies are “ineffective”, but it probably means that we don’t have a comprehensive picture of what factors allow them to be effective. More nuance is needed to understand what actually works and how, when, why.

2. The population of students we work with is understudied in physics education research. (See here). How this does or doesn’t contribute to failed efforts and successes is important to start to address.

3. We did this across the entire department, not just for select classes and instructors. I don’t know how common this is, nor how much of the research has been done in this context. There are likely different threads here related to implementation, buy-in, and professional development.

4. Given the overall sameness of the course structure, it could be valuable just to unpack more of the details of what was done, and to construct plausible stories of what details may have mattered and why. When Hake published his original 6,000 survey, they spent some time documenting and discussing cases where IE methods were used but high learning gains were not achieved. Having a documented case of the journey from an ineffective IE course to more effective could contribute to better understanding.

All of that would be too ambitious for a paper about our curriculum development efforts, but I think it helps frame why I think trying to document and report about our experience is worthwhile.

Update: here are some links to what our curriculum overhaul looked and felt like:

• Images of student work
• A lesson on UCM described