Participations in this summer’s conferences has really got me thinking about a many thing about physics education research. Here I’m going to begin writing about one of those things. To start this conversation, I want to talk about a poster from the 2013 Physics Education Research Conference.
Steve Kanim from New Mexico State University
Steve analyzed Physics Education Research publications from the American Journal of Physics and Physical Review–Special Topic in PER. The analysis was limited to publications that included actual student data (i.e., no discussions, opinions, sharing of best practices, dissemination only papers, etc). Steve finds that 75% of the students we study are enrolled in calc-based physics. This is disproportionate to distribution of classes we teach–only 33% of the students we teach take calc-based physics. The population of students least studied are those in two-year colleges, which comprise 25% of the students we teach (less than one percent of the students we study). Students in algebra-based courses are also under-represented in our research.
Steve is careful not to overly criticize our community’s beginnings. Our field has grown, in part, due to the fact that our research has focused on how even our “best” students struggle to develop functional understandings of basic physics concepts. Rather than blaming our past, Steve’s analysis points to a gap we need to address now.
Steve also looked at this data by disaggregating studies based on the SAT MATH distribution. From this perspective, it still appears that we are studying students on the high end. For me (Steve did not say this), this is especially critical due to the correlations that exist between achievement tests like the SAT and poverty and correlations between poverty and race. It could easily be said that we have been focusing more of our efforts and resources on the privileged. Steve also mentioned some research, which I can’t remember right now, that has found that a SAT Math score of 600 is a threshold for achievement in upper-level physics.
Oh, Force Concept Inventory
Steve’s poster also referenced some research about the FCI, which has also got me thinking again about the Force Concept Inventory (FCI), and how the FCI relates to our field’s focus on the upper end. If you don’t know, the FCI is the most widely used assessment / evaluation instrument in physics education. When using the FCI, normalized gains are the most widely used method to report student learning outcomes.
The idea behind normalized gain is to “take into consideration” students pre-test scores. Normalized gains can be interpreted as the “fraction of gain that could have occurred.” For example: a student who starts with a score of 40% and ends with 70%, gains +30% out of possible +60% gain, thereby having a normalized gain of 50%.
Despite measuring scores this way it appears that normalized gain (can be) strongly correlated with pre-test score. (Coletta and Phillips, 2005).
Underlying this correlation is additional findings that normalized gains on the FCI are strongly correlated with student scores on the Lawson Test of Scientific Reasoning Test, and they also correlated with students’ SAT scores (Coletta and Phillips, 2007).
A potentially huge problem we have as a community is that we report normalized FCI gains with out disaggregating these scores along such measures. I’d argue that this tendency is potentially dangerous, because it can lead us to make claims and offer implications for instruction that are distorted. For examples of how failures to disaggregate student achievement with measures of poverty lead to trouble, see Michael Marder’s prezi on Education and Poverty.
What can we do?
#1 We need Steve to publish his analysis of the mismatch between who we teach and who we study. This will enable those seeking funding to study under-represented populations to point to Steve’s research on the immense need for such research. It will also enable us to press funding institutions to create more parity in funding priorities. I emailed Steve this morning to offer encouragement and any help in making sure this happens.
#2 We need to begin as a community to publicize our own FCI normalized gains along with accompanying data that aides with meaningful disaggregation. This is true not only for publications about research. It should also include standards of reporting to funding agencies, and even standards of reporting on blogs. For example, right now, my own institutions reports normalized FCI gains from our algebra-based physics course to PhysTEC, and PhysTEC shares back data from all PhysTEC supported sites scores without disaggregation. I’ll start this process here: Our normalized gains at MTSU for algebra-based physics hover just below 0.3, and our SAT MATH scores are 460-570 range, with SAT Reading being 460-510 range. Note that this falls nicely in line with the graph above. Along this issue, we should really support the PER user’s guide. Although not on the site yet, they are working hard to create an Assessment Database and Analyzer tool that will make it easier for everyone to upload, use and interpret data in meaningful ways.
#3 Physics Education Researchers as individuals need to go out of their to engage with more research concerning students who aren’t just down the hall. The disproportionate focus on calc-based physics and severe shortage on two-year colleges is not malicious–it comes from convenience and a desire to improve our own local educational settings. Research-intensive universities are more likely to have students at the higher end of preparation and opportunity, and are also likely to have professors who have time and resources to do research. Instructors at two-year colleges have the opposite situation–no time, resources, or support to conduct research, and more likely to have students with less preparation and opportunity. I emailed three community college physics instructors this morning to begin that conversation.
What say you? (Feature Comments)
Eric Brewe: “We should think about the use of normalized gain. It over values gains made at high end schools.”
Gasstation without pumps: “One question remains—why are students taking algebra-based physics? … Is the FCI the appropriate measure?”
Teach. Brian. Teach. 
This is excellent. Thank you. Much of my interest in PER has been spurred by these kinds of questions. Every other year, I teach both our calc-based and algebra-based intro courses, and the populations of students are very different. If PER is going to be as valuable as possible to as many people as possible, we need to address these differences.
Are there specific issues/topics/concerns that you would like PER to address?
I want to see a comparison of learning between the different populations. Much of what I have access to that was developed by the PER community was tested on students in calc-based classes at R-I schools. I don’t have much comparative data on what effectiveness the different methodologies might have with students in different educational settings.
In my work, I’m starting to address this by collaborating with researchers at an R-I school, so that we can perform the same studies with our students.
I say: Let’s write a grant! 🙂
Truth. I’m curious what kind of grant do you see here?
Simultaneously we should think about the use of normalized gain. It over values gains made at high end schools. I will happily get a plot of effect size vs sat math at fiu in the coming weeks. Either way great post, and Steve’s work was really cool, I agree
Agreed. Effect size is likely the way to go here. I also think we need to report raw (achievement) scores in addition to whatever gain measure.
One question remains—why are students taking algebra-based physics? It is almost certainly not for a physics or engineering degree. What do they gain? What do they hope to gain? If it just a check-off box on a list of requirements with no pedagogical value to the students, is there any point to measuring how well students learn? If there is a point to their learning the subject, is the FCI the appropriate measure for their objective?
I took a algebra-based course in HS and then took AP. So at least some students go on to be physics majors, etc. But your bigger question is, what should purpose and goals of these physics courses be? If the purpose and goals are different, how would we know we were doing a good job? Those questions, I agree, are extremely important. What are your thoughts?
I don’t know their purposes or goals. I agree that high-school algebra-based physics is often a good entry point for students to go on to physics in college (my high school did not offer calculus-based physics, and no high school in Santa Cruz County does).
Algebra-based physics at the college level is almost never a preparation for further study in physics and engineering, though, and teaching it as if it were could result in poor choices of content or pedagogy. For most students it will be the last time they learn physics, not the first.
I had a similar problem in designing a circuits course for bioengineers, and ended up emphasizing the things that they were being taught electronics for: to get design experience (thinking like an engineer) and to build circuits that were actually relevant to bioengineers. This meant that the course centered on the labs, not on the math (the way traditional EE circuits courses are taught).
I tell my students the purpose of the algebra-based physics course is to teach them about problem-solving. Physicists approach problems differently than folks in other disciplines, and I think that’s why they take it. Secondarily, there are plenty of people who need an introduction to things like forces, kinematics, etc (athletic trainers, often biologists, theater technicians, etc), who will never take a calculus course. We have to meet them where they are, I think – expecting people who struggled to pass algebra to make it through calc is unlikely.
I’m not sure that the “problem solving” techniques really show much transference—the math folks have been arguing that one for decades, without much in the way of experimental validation. More direct problem solving instruction might be more valuable for that goal.
The better argument is that some people need some understanding of forces, without needing (or being able to master) calculus. Identifying why they need to know this, and what forces are of interest to them, could go a long way towards selecting more compelling demos and lab exercises.
One way way researchers can respond to these important issues is to take NSF’s “Broader Impacts” pressures seriously. NSF now asks us to emphasize diversity issues more strongly in our proposals (Broader Impacts is its own section of the narrative), gives us incentives to serve underserved populations (such as extra money if your proposal includes TYCs), and requires reviewers to evaluate proposals in terms of their broader impacts. NSF’s self-study of past measures along these lines suggests that they have made a big difference.
That’s a good point. Do you think grants focusing on TYC just haven’t made their way to publications where Steve is looking?
Brian. Is the algebra-based population being larger than the calculus-based population the norm across the US? Of the schools at which I have taught or been a student, only one of the three was a situation where the number of algebra-based students outnumbered the calc-based ones.
I’ve not been able to find the size of the algebra-based physics market, but the calculus-based one is about 300,000 a year (http://www.twmresearch.com/pdfs/calcphys.pdf)
Our biology students are required to take calculus-based physics, so we have 2 streams (one for physicists and engineers, the other for biologists). There is an algebra-based physics, but it is primarily for humanities students. The physicist/engineer course is taught once a year, as is the algebra-based one, but the calculus-based one for biologists is 3 times a year.
I think the US percentages were 33% calc, 26% algebra, and 12% conceptual for 4 year institutions, and rest were two-year colleges. I’d assume more algebra-based physics and conceptual in TYC than calc. But unsure. We have around 200 ABP and around 40 CBP each semester.