What I Wanted to Share at PERC 2019

I am not making it to AAPT or PERC this year, but this is a summary of what I would have liked to share had I been able to go. I’ll start with the abstract I submitted to PERC:

Leveraging FCI–ACT Correlations to Communicate the Impact of Course Reform

Conceptual inventories such as the FCI are widely used to assess the impact of course reform efforts in physics education. Gains on these instruments, however, are known to correlate with other student variables such as students’ SAT score. I make use of this type of underlying correlation to communicate the impact of course reforms to broad audiences outside of physics education for which the FCI is not a useful benchmark. To illustrate this approach, I present an analysis of FCI learning gains that are disaggregated by ACT score before and after a major course reform in an introductory physics course. In this sample, normalized learning gains shifted from 0.28 to 0.46 were found to be equivalent to a shift in average ACT score from 24 to 29. Implications for the communicating research and evaluation results with different audiences are discussed.

Here’s the gist. Whenever we in physics education are trying to communicate the impact of some course reform on student learning, we all have a sense that it is important to consider and understand the audience. Who are you talking to? What do they know? What can they relate to? What are useful ways of articulating impact to them?

So what is this post job about? This post is about how I have recently changed the way I talk about the impact of course reform for folks not strongly connected to physics education.

So, consider the Force Concept Inventory. With this instrument, it is common in the physics education community to talk about the impact on student learning in terms of normalized gain and/or effect size.

  • Normalized gain is familiar to a large swath of researchers and teachers in physics education for historical and political reasons. As a result, many physics educators have developed a sense of what different scores might mean in a normative way — How does this compare to “typical” outcome? What does a “good” score look like? What does a “bad” score look like? What range of scores should we be aiming for? That said, normalized gain is not always a useful tool for communicating outside of physics education, and is a tool with  some drawbacks in terms of its utility for making scholarly claims.
  • Effect sizes (e.g., Cohen’s d) are more likely to be reported by researchers for a variety of reasons that I don’t want to get into here. It will suffice to say that there are fewer drawbacks than normalized gain, and that effect size is a tool better designed for characterizing the degree of “impact”.  As such, effect sizes are valuable tool for making scholarly claims and also for communicating with persons outside of physics education. On there other hand, effect sizes have the disadvantage of being less familiar to the broad physics education community.

OK, so there are certainly other ways one might choose to communicate impact of course reforms using the FCI, but I want to keep this discussion focused on just these two, because they are the most common and it will keep the discussion focused.

So what’s the deal? I have personally found that neither normalized gain nor effect size has been very effective for talking with certain audiences, which for me include:

  • Physics Faculty, especially those who may be less familiar with (or skeptical) of Physics Education Research
  • Administrators
  • Faculty and educators from other disciplines
  • Education researchers not familiar with physics education or effect sizes.

In this post, I want to introduce a way that I have recently shifted to in talking about the impact of our course reform to certain audiences that I think is quite useful. I will use our recent efforts at course reform at our introductory as an example:

OK, so here is part of the timeline for our course reform efforts:

Screen Shot 2019-07-22 at 10.08.01 AMI joined the department in 2011, when active learning was implemented department-wide in our introductory physics courses. Slightly before and in the year following my arrival, our department tinkered with various reforms. Here is what our data looked like in terms of normalized gain.

To locate these normalized gain within a comparative landscape, it’s useful to consider our results as compared to typical results.

Screen Shot 2019-07-22 at 10.13.11 AM.png

Dissatisfied with these results are more comprehensive overhaul was undertaken

Screen Shot 2019-07-22 at 10.14.42 AM.pngAt the time, we were engaged in the pilot, I had tried communicating the results to my faculty in the following way —  showing raw gain, and providing percentages of students who met certain thresholds.

Screen Shot 2019-07-22 at 10.18.19 AM.png

Roughly, these corresponded to normalized gains for the pilot that were 45-55%, which put our results more on the “good” side of the comparative/ normative landscape:

Screen Shot 2019-07-22 at 10.21.00 AM.png

After the pilots, we scaled up the new course to the entire department, and we were able to maintain the gains, which you can see here.

Screen Shot 2019-07-22 at 10.24.15 AM.png

If you are someone in the physics education community, there’s a good chance that I’ve provided sufficient information for you to have a sense of what the outcomes we achieved might mean, even if there’s certainly more you would like to know. This is mostly because, I have focused on using normalized gain as the main lens.

So where to go now? For me, in talking about this with various audiences who are not connected to the physics education community, I have found it useful to translate these results in a different way. To do so, I leverage that fact that normalized gain on the FCI are often correlated with SAT /ACT scores, which I have written about before generally. and also more specifically here.

The basic idea is to disaggregate the data by ACT score. For our reform courses, you get the following graph:

Screen Shot 2019-07-22 at 10.38.25 AM

This shows that, on average, students with higher ACT scores end up with higher normalized gains. Our average student has an ACT score of 24, and a learning gain of 44%.  Students with ACT scores above 30 tend to achieve learning gains above 65%. Students with ACT scores below 20 tend to below 30%.

If we see similarly disaggregate our data from before the course reform you get the following:

Screen Shot 2019-07-22 at 10.37.46 AM.png

So, the basic gist here is that our new course tended to improve outcomes for everyone, regardless of ACT score, and it tended to increase scores by about the same amount, say about 17% gains.

But there is another way that we could have achieved a similar boost of 17% without making any changes to the course. We could have somehow been more selective about what students we allow to take our course. The question then becomes, how much more selective would we have needed to be?

This is the graph that I have been building up to when I am speaking with audiences that are less familiar with the FCI and physics education.

Screen Shot 2019-07-22 at 10.45.21 AM.png

I turns out, that for our course reform efforts, relative to the old course, students are now performing as if they had an ACT score ~ 5 points higher than before. Or, in order to achieve the same result with the old course, we would have needed a population of students with an average close to an ACT of 29.

[For reference, the ACT benchmark score for college readiness across STEM fields is ~ 26, and for biology specifically it is ~23. The average ACT score for students at MTSU is 22 ]

I don’t think what I have done here is a big deal, but I think it’s definitely worth sharing. So, here are some of my take-aways:

First, at my institution, faculty and administrators are much more likely to be familiar with the ACT.  Thus, when engaging those audiences, it makes sense for me to translate into a measure related to the ACT. Maybe this will be useful for others?

Second, disaggregating data is really useful, in general. I hope to motivate more physics educators to disaggregate their data in various ways. Beyond what I’ve talked about here, disaggregating your data gives you a much more nuanced view of what’s going on.

Anticipating based on Patterns (Kinematics)

Here is a task I developed to engage students in thinking about kinematics. I’d run this as a think-pair-share type activity. I could see using this early on before students know much kinematics or after students know quite a bit. Lots of different places to go, focus, emphasize, depending on where your students are at, including distinguishing kinematic quantities, assumptions, developing models, changing your thinking as you accumulate more evidence.

This slideshow requires JavaScript.

Two “Does Tension Change” Questions?

This is a question set I’ve developed that is targeted for pre-service physics teacher, first to engage them in qualitative reasoning and argumentation, and then to push them to do the algebraic modeling. One of the questions has friction and the other doesn’t.

This slideshow requires JavaScript.

Practice with Position vs. Time Graphs

Just sharing resources: This is a graph I had my students working with in class after struggling with a similar homework problem.

Screen Shot 2019-07-21 at 8.44.06 AM.png

  • I first had students identifying whether points A-I were moving to right, to the left, or not moving.
  • I then had students identifying whether sections 1-7 were speeding up, slowing down, or moving with constant speed.
  • I then had students work on constructing a velocity vs. time graph.

Lots of other questions could be asked.

 

Canaries in the Classroom

In my readings over the past year, I keep coming back to the following:

“[they] may be telling us – loudly, visibly, and memorably — that the arrangements in our classrooms are harmful to human beings. Something is toxic in the air… each teaches us something important, powerful, and worthy about how to reimagine classrooms in the image of being fully human… They offer lessons on power and authority, loneliness and belonging, creativity and conformity. Their experiences and insights draw our attention away from the confinement of pathology and toward the complexity of goodness; away from blame and toward understanding, away from evaluation and toward curiosity.”

— Carla Shalaby In Troublemakers

In that spirit, I’ve been spending a bit of my summer looking into some equity-related data for the classes I teach. One aspect I’ve been looking into this week is grade assignments by gender and race.

Over the past three years of teaching first-semester introductory physics:

  • I have assigned a grade of A to 42% of white students compared to only 12% for black students.
  • I have assigned a grade of B or better to 72% of white students compared to only 32% for black students.
  • I have assigned a grade of C or better  to 84% of white students compared to only 53% for black students.
  • I have assigned a grade of D or better 94% of white students compared to only 83% for black students.

The median grade for white students is a B, while the median grade for black students is a C.

The modal grade for white students is an A, while the modal grade for blacks students is a D.

Needless to say, black students as a whole are not thriving in my physics classes.

There is not a significant nor consistent difference in grade distributions by gender.

some demographic context

At MTSU, black students comprise 18% of the student body; in my introductory physics classes that percentage has been somewhat lower, closer to 15%. With class sizes ranging from 24 to 32 students, this has translated into my classes having as few as 1 black student and as many as 7 black students. There are even smaller numbers of Asian and Latinx students, in my classes about ~ 5% each.

Separate from those statistics, Muslim students make up about 10%, with students being a mix first and second generation in America and also some international students.

What about the Physics and Astronomy department faculty?

  • Full professors (6): all male, 1 faculty of color (Ethiopian)
  • Associate professors (2): both white males
  • Assistant professors (2): one male faculty of color (Nepalese) and one female faculty member.
  • Full Time Instructors (4): 3 white females and one male faculty of color

So we are a lot less diverse than the students we teach.

Some plans for making plans:

  • There is a faculty learning community on inclusive teaching I will apply to be a part of. I’m excited about this.
  • I have at least one other colleague who will be helping me to gather and analyze more of the institutional data needed to get a more comprehensive picture. I
  • Over the next two months, I am consulting with several different colleagues and critical friends about some sensible starting places.

On General Education (Part II)

In my first post on general education, I emphasized the importance of faculty becoming more aware of their own conceptions of general education. In that post, I introduced one common view of general education — the view that the purpose of general education is to provide both foundational knowledge and breadth of knowledge. So, what other views are out there?

In this post, I want to discuss just two different views on general education that are NOT radically different from the common view. Rather, each view can be understood as merely a pivot from the common view– to a slightly different vantage point. With our attention shifted slightly, I hope we can step back and see general education anew, and I also hope we can gain perspective on the common conception, including a better sense of its limitations and liabilities.

So here it is– a different perspective:

General education should aim to support students in making connections between different domains of knowledge.

That’s it. Simple enough.

But I’ll suggest it is radically different. Here’s why:

Models of general education that adhere to the common conception aim to provide students with foundational knowledge (that theoretically should be applicable across domains), and also provide students with breadth of knowledge in many domains (across which connections could theoretically be sought). In reality, however, students are often left on their own to make these connections — that is, to transfer their more general knowledge and skills into specific domains, to synthesize across specific domains of knowledge in order to produce more general / power understandings, and to hybridize their knowledge across domains to generate new ways of knowing that may not have existed before. That is, general education courses may provide students with the raw materials, but they often do nothing to help students integrate those into a meaningful experiences or understandings.

Here is another way to say a similar thing. At a recent AAC&U conference, a faculty participant made the following analogy that has stuck with me:  This participant likened the most common general education structures as akin to only ever getting to taste ingredients separately–ingredients that in theory could go together into making delicious food. Students get to taste flour in one course, salt in another, eggs in a third, and perhaps even butter or sugar in a fourth course. Unsurprisingly, no one very much enjoys these separate tastings.  Alone, each ingredient can seem either quite bland and boring or too singular in their flavor and texture. Each of the ingredients is experienced as disconnected from each other, and so it’s hard to see how they might go together.

There’s a lot more to say about this perspective of general education. But for now, it’s worth pausing to consider: How might general education need to be different if the goal is not to ensure adequate distribution of knowledge across domains, but to ensure integration of knowledge across domains?

  • Would we need different types of courses or would the same ones work?
  • What would you want / need to know about what’s happening in other general education courses?
  • How would the types of assignments (for student learning) and assessments (of student learning) need to change in order to support this new goal?

Final words: You could be reading this and thinking that the two perspectives so far are not in fact different. I want to argue that they are different. In the common conception, generalized foundational knowledge  and distributed specific knowledge is the goal; whereas in the second conception, courses that focus on on generalized knowledge and distributed specific knowledge are necessary means for getting students to develop integrated / connected understanding.  That is, the goal of the first is a means of the second.


OK, so now I’m going to introduce a second perspective on general education, and to do so, you are going have to forget about the first one for a moment. Here it is:

General education courses (individually and collectively) should support a shared vision for student learning (whatever it may be) — one that is specific, explicit, and can be instantiated in an overarching way across different domains.

That’s really wordy and abstract, so I’ll tell a story to help. I was recently talking with a colleague about his general education astronomy course, and the colleague said something like, “I just want students to leave my class understanding why the earth is not flat.” This learning goal is very “astronomy specific”, but it’s not too hard to find kernels of “generality”.  I offered that perhaps if he was the overlord of all general education, he would want the goal to be something like “for students to confront deeply held but mistaken beliefs” or, “to grapple with seemingly-right ideas by understanding both what makes them so seductive and also flawed.”  Now, I’m not suggesting this as a goal (or non-goal) for general education. What I am putting forth is the idea the goals of general education need to be made explicit and they need to offer some footholds to transcend disciplinary boundaries.

The question I then posed to my colleague was, “What do you think it would look like for all general education courses to be about that goal – the goal of helping students confront deeply help but mistaken goals?”  How do you think that would inform the content of our courses? How would that inform the pedagogy of our courses? How would that inform the types of assignments and assessments we administer in our general education courses?

To step back from this specific claim, this third conception of general education says that it’s important for everyone teaching across their disciplines to knows and be aware of common threads courses need to be driving students along. These threads could relate to aspects of critical thinking, or inquiry, or ethics, or intercultural competence, or information literacy, etc. The point here is that everyone is working towards teaching the same set of something(s) that are more “general” than any one discipline, and that these something can be instantiated meaningfully in disciplines in ways that contribute more generally to that learning.

So how is this different than the common conception of general education? The common conceptions treats the word “general” as the general list of things we want students to be generally educated about. Whereas the “word” general, in this conception, refers to the specific ways of understanding, reasoning, or being that we help to generalize through a process of individually and collectively nurturing it across our courses. From the common conception, foundational knowledge is something we see as intrinsically generalizable, and we expects students to bring this knowledge with them to other courses.  Whereas in this third conception, foundational knowledge is generalizable through a process of carefully structured learning opportunities. Instructors should therefore have a better sense of what we are all “generally” trying to accomplish, independent of (or perhaps interdependent with) our own specific disciplinary aims.

Again, it’s worth asking: How might general education need to be different to support this view?


OK, so now I’ve written my 2nd post about general education, here is some food for thought:

  • First, I’ll put forth the claim that the common conception of general education tends to silo disciplines from each other. It does not require that instructors / departments seek out a common sense of purpose, nor that they coordinate their efforts. In contrast, the two conceptions of general education introduced above seek to connect disciplines in one way or another; and thus they place more responsibility on instructors / departments to seek out common purpose and to coordinate their efforts. Siloing is perhaps the norm for academia, so it should not be surprising that this is common for general education. 

   

  • Second, the common conception of general education does not require much self-awareness about general education. Everyone teaches their own courses, fulfilling their role of either providing foundational knowledge or breadth of knowledge. In contrast, the two conceptions here require that everyone become much more aware of their goals and their values and how those goals and values overlap with others. All of this ties into my first post about the importance of becoming increasingly aware of our conceptions of general education. For many of us, our working in general education did not require conscious awareness of it, and so it is hard work to begin this process of awareness.

 

  • Finally, I want to say (again) that I’m not necessarily advocating for the two views above, nor am I saying that I’ve gotten these two views perfectly right. There are, of course, other views of general education that depart even more widely. The point of this post is to begin helping readers to open the door on their own understanding of general education. To do so, I introduced two ways of thinking about general education that are only slightly different than the common conception. Yet, still, that shift is significant and has important implications for how we think about our current model of general education and any potential changes going forward.

 

In my 3rd post,

  1. I will provide details about the current model of general education at MTSU, and how that fits with the common conception.
  2. I will unpack what we have learned about faculty goals, aspirations, and concerns regarding Gen Ed from focus groups and surveys we have completed.
  3. I will also begin to introduce a few components of alternative models of general education, but the details of that will probably end up in a 4th post.

General Education Brain Dump (Part I)

I am writing about some of things I want the faculty in my department to know about in terms of the process we are undertaking in redesigning General Education at my institution.


First, I’d like you to know that General Education (at MTSU and more generally) is more nuanced, complex and multi-faceted than any one person’s, department’s, or even college’s perspective on it. To be sure, I’m not saying that our own perspectives about Gen. Ed. from Physics and Astronomy are wrong or irrelevant, but they can be quite limiting due to our own experience being just one slice of very broad landscape. That is, our limited experience not only affects our understanding of the structures and processes of General Education at MTSU, but also its broader aspirations & goals, and the operating constraints that different parts of the system work within.

Second, it is important to recognize that faculty, students, chairs, and administrators often have very strong feelings about General Education based on their own perspective and experiences, and this can make it difficult for well-meaning people to communicate effectively about both the current model of General Education and any potential changes. For this reason, it is imperative to be a bit modest, humble, and willing to learn in order to be an effective participant in the process.

This first post is intended to be an overview of how to begin thinking about thinking about general education. I apologize for it being so abstract, but I’ll try to make it as concrete as possible.

Beginning: Faculty and Student Conceptions and Perspectives on Gen Ed.

How we conceptualize the question of “what General Education is supposed to be?” influences a lot of our downstream thinking about general education. It is important for individuals participating in this process to recognize that their own views regarding the purpose of general education is not universal and it is likely to be differ from person to person. Certainly, there is a range of views in our own department, but the range of views beyond our department is quite a bit larger. I would be so bold to say the following: our own individual and collective views are quite “naive” in some regards. Most of us have not spent an appreciable part of careers learning about general education in the broad sense. The one thing I have learned is that everyone has a lot of learn.

Common Conception of Gen Ed

I’d like to start with one common conception of what General Education is — which while overly simplistic, is a useful starting place. Consider this: many faculty come to think of general education in terms of requirements that aim to guarantee two outcomes —

(1) that all students further develop certain foundational skills (e.g., writing, mathematics, communication) that ideally should apply broadly across all disciplines,

and

(2) that all students develop some breadth of knowledge across a variety of disciplines that are different than their primary focus (i.e., major).

[Remember, this is not THE definition of general education. It’s a common perspective]

What about in Physics?

If we are thinking about General Education this way (courses are foundation + breadth), our own courses in Physics and Astronomy tend to fall into the second category — fulfilling breadth in the sciences, specifically. So, how did we end up with the general education courses that we did? It is important to recognize that we, like most science departments, did not seek out to develop courses from scratch as “exemplary general education courses”. Rather, we more so developed courses that could serve as appropriate introductions to our disciplines (often, but not always, for majors). We then modified these courses a bit in some directions to qualify them for general education. This approach to course development made sense. The reasoning might be: our courses, already being high-quality foundational introductions to the discipline for majors (or other science students), should easily meet the lesser requirement of engaging the broader populations of students in the beginnings of science / physics for the purposes of cultivating breadth of knowledge. This approach is practical and efficient, as it required little reworking of existing courses / curriculum.

There are a couple points I’d like to make here:

(1) Even though this approach to Gen Ed course design is efficient, it may not be great for producing courses that actually foster the goals of general education,

(2) deciding whether such an approach (or any other approach) does or doesn’t produce high-quality general education courses requires the specification of goals for general education. That is — we need to able to identify what it is we are trying to accomplish.

(3) which goals each of us chooses to foreground or background for general education is strongly influenced by our unique experience, but also the overall conception of general education we’ve adopted from our own cultural and academic communities.

As a flow chart this might be:

“our overall conceptions of general education”    –influences–>

” goals we choose to foreground for general education”    –provides–>

 “goal we would need to evaluate success.”

You might be able to see both the logic and the problem here. In order for us to know whether general education courses are doing a good job, we need to know what the goals are, but to understand where our own goals come from, we need to understand how it is we conceive of general education. If we believe that are conception of general education is correct and others is wrong or that ours is universal, people are going to talk past one another.

Pause: So what about student perspectives and experiences? (how do they conceive it)

There is certainly a lot more to say about this later, but the main thing I’ll say is this. Students often experience general education courses as disconnected. For foundational courses, students often come to see them as overly “generic”  and thus to students they feel  “disconnected” from any meaning or skills that students see as relevant for their majors, their careers, their lives.  Similarly, students often experience breadth courses as overly “specific” and thus similarly disconnected from their majors, careers, and lives. These common experiences with Gen Ed Courses then serve as the kernel from which students come to see Gen Ed courses as “something to get out of the way” and as “irrelevant and a waste of time,” which actually get reinforced through various cultural and academic communities that students participate in.

So what about Faculty? How do faculty feel about general education?

Out of this common conception of Gen Ed. (again that general education is foundation + breadth), faculty in physics and other disciplines may find themselves falling into one or more of the following specific judgments toward general education courses and teaching.

  1. General education is a structure that dilutes and waters down (to varying degrees) what would otherwise be rigorous introductory courses.
  2. General education consists of a few (slightly annoying) “hoops” we have to consider, but it doesn’t influence much (if any) of our decision-making regarding curriculum and pedagogy.
  3. General education consists of remedial courses in the sense that they are filling gaps of foundational and breadth knowledge that students should have acquired already. If students were better prepared, we wouldn’t need general education.

I’d like to step back again and recognize that everyone’s actual ideas about general education are much more complex and nuanced than any of the above. I’m not accusing anyone of having any particular view above. My hope is, instead, that as you read the above perspectives and judgment, you can recognize them. Perhaps you will recognize them in yourself or perhaps so in others. And I do hope that you recognize them whether or not you agree with them. And finally I hope that you can recognize them even if you think I haven’t perfectly described all their nuances.

Trying to Wrap up things up in this First Post: What the big idea?

I am hoping to help draw a line between these specific judgments that we often hold about general education (i.e., that Gen Ed. may be see as damaging, inconsequential, or remedial) and the common conception of general education as having a primary purpose to provide foundational and breadth knowledge. That is, foundation and breadth are both things that we can perceive to be damaged, in need of remediation, or we can choose not to worry the foundations outside our immediate community.

Flow Chart:  

our conceptions of general education –>

color our experiences of general education –>

together our conceptions of and experiences with gen. ed. –>

result in judgments, judgments that can be powerful.

So, when most people start talking about or thinking about general education, these judgments are often the first thing that comes to mind. People *feel* strongly one way or the other about general education. But they are often much less aware of what conceptions of general education they even have that give rise to those judgments. Furthermore, even if they can articulate a conception of general education, it is often tacitly assumed that this view is universal (i.e., “this is what general education is”).

Thus, for any of us to begin productively engaging in conversations about general education, we will need to strive to distance ourselves away from our immediate judgments and experiences of general education; and instead strive to become more curious about and aware of two things — our own conceptions and others’ conceptions of general education. Often times this will at first involve recognizing our own judgments (and emotions) and then trying to set them aside. We may need to hear about the judgments of others people, and not initially react to them based solely on our own judgments. The broad goal needs to be to get to the point where we can see general education more objectively and multidimensionally — to recognize different perspectives on general education (different from our own), and to try on these different perspectives in order to getter a better understanding of the beast that general education is.

Thanks for hanging in there on the abstraction.


In the next post, I hope to invite the reader into different view points of what general education is (or might be) and to compare and contrast that with the common view point. I do not, at this point, want to advocate for any particular view. The goal will be to explore different view points that will allow us to expand the possibilities of how we imagine general education. It will also, in turn, help us to better understand some of limitations and liabilities of the common conception of general education.

 

Blog at WordPress.com.

Up ↑