Teaching Evaluations: One semester to the next

After my first semester here, I gave the break down of my teaching evaluations. Here I’m at it again.

More importantly, I’m going to follow up this post with another post that is more reflective on this past year has gone (in terms of teaching) and how I see this year’s experience informing goals for improvement next year. But I wanted to get the data down first.

So, here it is. Below is a summary of my teaching evaluations for both my inquiry and physics courses–course that I taught both fall and spring semester. In each category, I show the average rating for inquiry, physics, and the department. In parentheses, I show the change in my ratings from last semester to this semester. In every category, the ratings either improve or stay the same. With respect to department averages, my evaluation for my physics class are near ceiling. In inquiry, there are two categories that I am still below departmental averages, but both showed significant improvement.

Presentation:

Inquiry 4.7  (+0.3 from fall to spring semester)

Physics: 4.9  (+0.0)

Department: 4.2

Organization:

Inquiry: 4.3  (+0.4)

Physics: 4.8 (+0.0)

Department: 4.1

Assignments/ Grading

Inquiry: 4.3 (+0.4)

Physics:  4.9 (+0.1)

Department: 4.4

Scholarly Approach

Inquiry: 4.2 (+0.2)

Physics: 4.8  (+0.1)

Department: 4.2

Student Interactions:

Inquiry: 4.6 (+0.1)

Physics: 4.9 (+0.0)

Department: 4.0

Motivating:

Inquiry: 4.6 (+0.4)

Physics:  4.7 (+0.0)

Department: 4.1

Effectiveness/Worth

Inquiry: 3.5 (+0.5)

Physics: 4.6  (+0.1)

Department: 3.8

For better or worse

From “Lessons From the Physics-Education Reform Effort” by R.R Hake

As previously indicated, the data of Fig. 1 show that seven of the IE courses (717 students) achieved <g>’s close to those of the T courses. Five of those made extensive use of high-tech microcomputer-based labs (Thornton and Sokoloff 1990, 1998). Case histories of the seven low-<g> courses (Hake 1998b) suggest that implementation problems occurred. Another example of the apparent failure of IE/high-tech methods has been described by Cummings et al. (1999). They considered a standard physics Studio Course at Rensselaer in which group work and computer use had been introduced as components of in-class instruction, the classrooms appeared to be interactive, and students seemed to be engaged in their own learning. Their measurement of <g>’s using the FCI and the Force Motion Concept Evaluation (Thornton & Sokoloff 1998) yielded values close to those characteristic of T courses (Hake 1998a,b,c). Cummings et al. suggest that the low <g> of the standard Rensselaer studio course may have been due to the fact that “the activities used in the studio classroom are predominately ‘traditional’ activities adapted to fit the studio environment and incorporate the use of computers.” Thus the apparent “interactivity” was a product of traditional methods (supported by high technology), not published IE methods developed by physics-education researchers and based on the insights of cognitive scientists and/or outstanding classroom teachers, as for the survey courses.

This quote and similar ones from Hake have been on my mind these past few weeks as undergraduate students I am working with are grappling with the question, “Why are our learning gains on the FCI lowered than expected and desired?” This question emerges out of our learning about the FCI, analyzing our local data, and comparing to other research and outcomes that have used the FCI.

So, in our courses, we use peer instruction methods, collaborative-problem solving with whiteboards, and computer facilitated learning. Our class is mostly “flipped” (where students read lecture at home and practice problems in class). Most of it takes place in a studio setting. It sounds and often looks very interactive. On the other hand, all the content students are interacting with– from the textbook, to the laboratory activities, to the questions and problems students work on in groups–is home-grown. For better or worse, these home-grown materials would probably be characterized as “traditional activities adapted to fit the studio environment and incorporate the use of computers.”

As a group of researchers, we are approaching our question in a variety of different ways:

(1) One students hopes to ask instructors in our department to take the FCI, not by answering how they would answer, but by choosing the answer they think would be the most common incorrect answer chosen by students, and to estimate the percentage of students answering each questions correctly after the course. He hopes to compare instructor expectations to reality, in order to answer question like, “How knowledgeable are instructors of specific content difficulties students have and how aware are they of the prevalence of those difficulties in our courses?”  We have been reading a lot of pedagogical content knowledge.

(2) Another student is interested in examining student learning in relationship to our home-grown textbook. Does the book explicitly address specific difficulties we know about from our own data and research in physics education? Does the book implicitly reinforce any difficulties? Does it provide opportunities for developing conceptual understanding as well as problem solving? He is also interested in questions like, “Do students actually read the text? How much? How deeply? In what do they engage with text? What do they actually take away from the reading the text, and how does that play out in relationship to classroom instruction?” … We have been reading a lot about self-explanation, preparation for future leraning, refutation texts, and the influence of prior knowledge (e.g., misconceptions) on reading comprehension.

(3) Another student is interested in examining structural factors of instruction that might be contributing to lower than expected FCI gains, including

  • Student background and academic preparation
  • The prevalence of exam questions that probe (and hold students accountable) for conceptual understanding
  • The quality of apprenticeship and training that undergraduate TAs and new faculty receive for teaching using interactive engagement methods
  • The strategies that instructor use to motivate and cultivate a classroom culture in which IE methods are taken seriously.

We have been reading papers about the kinds of background that correlate with FCI scores, as well as papers about programs that have successful or unsuccessful implementations of reform-physics-curricula.

This work, for better or worse, treads on a sensitive arena–a close examination of ourself.  The fact that this work is being carried out by students, I think, could be perceived as making this endeavor even more sensitive, but in an odd way it makes it authentic. All of these students are really interested in improving instruction here, doing research that is valid but also relevant to local stakeholders. They have no axe to grind or hidden agenda. We are also just genuinely intrigued by the puzzle, and curious to pursue its potential solutions. Some of that solution, will not doubt in my mind, need to be geared toward improving the curriculum at the content-level–the content as embedded in all the tasks we ask students to engage with, from the text, to the labs, to the questions and problems they work on. Some of that solution will no doubt be about getting our department on board with the continued renewal of that content based on assessment, feedback, analysis, and ongoing revisions. In that sense, the work we are beginning serves a launching point for what will need to become an ongoing endeavor.

 

What Galileo already knew about physics education?

As I’ve been digging further into Galileo’s Two New Sciences, I’m realizing just how well Galileo understood how difficult it was for humans to learn and understand complex ideas, and how necessary it was to explicitly discuss necessary prior knowledge, subtle mis-interprations, misconceptions, arguments, epistemology, evidence, etc. In this sense, Galileo already knew much of we know about physics education.

Can you imagine if a physics textbook we wrote today, paused to have a character say things like what’ I’ve quoted below?

On the importance of prior knowledge and cognitive load

“Your demonstration proceeds too rapidly and, it seems to me, you keep on assuming that all of Euclid’s theorems are as familiar and available to me as his first axioms, which is far from true…”

On acknowledging common difficulties

“This is one of the difficulties which I also at the beginning, experienced, but…”

On “misconceptions” in process of learning

“I am one of those who accept the proposition, and believe that a falling body acquires force [vires] in its descent, its velocity increasing in proportion to the space, and that the momentum [momento] of the falling body is doubled when it falls from a doubled height; these propositions, it appears to me, ought to be conceded without hesitation or controversy.”

On argument, epistemology, and the role of evidence:

“Your words are very plausible; but I hope by experiment to increase the probability to an extent which shall be little short of a rigid demonstration.”

or

“I would like to have been present at these experiments; but feeling confidence in the care with which you performed them, and in the fidelity with which you relate them, I am satisfied and accept them as true and valid”

On tentative nature of scientific knowledge (even definitions)

“We must add to the old definition (which defined steady motion simply as one in which equal distances are traversed in equal times) the word “any,” meaning by this, all equal intervals of time; for it may happen that the moving body will traverse equal distances during some equal intervals of time and yet the distances traversed during some small portion of these time-intervals may not be equal, even though the time-intervals be equal.”

And this is just one of my favorites:

“You present these recondite matters with too much evidence and ease; this great facility makes them less appreciated than they would be had they been presented in a more abstruse manner.”

Ramblings on Velocity and Galileo

We don’t explicitly teach the notion of a velocity at an instant in our introductory algebra-based course, although the term instantaneous appears in the text every now and again. The concept is, of course, implicit in everything we do beyond chapter two. I have noticed this past year, the negative consequences for students’ understanding of velocity, acceleration, forces, and energy.

When I have mentioned to several colleagues this lacking in our introductory curriculum, a common question that arises is how to think about teaching velocity at an instant (in an intellectually honest way) without calculus.  In this post, I don’t intend to describe how to teach the concept, but to illustrate about how clear the ideas of constant, average, instantaneous, and change in velocity were in Galileo’s mind even without with the mathematical machinery of calculus. Here is a excerpt from a translation of Galileo’s Two New Sciences.

“When I think of a heavy body falling from rest, that is, starting with zero speed and gaining speed in proportion to the time from the beginning of the motion; such a motion as would, for instance, in eight beats of the pulse acquire eight degrees of speed; having at the end of the fourth beat acquired four degrees; at the end of the second, two; at the end of the first, one: and since time is divisible without limit, it follows from all these considerations that if the earlier speed of a body is less than its present speed in a constant ratio, then there is no degree of speed however small (or, one may say, no degree of slowness however great) with which we may not find this body travelling after starting from infinite slowness, i. e., from rest. So that if that speed which it had at the end of the fourth beat was such that, if kept uniform, the body would traverse two miles in an hour, and if keeping the speed which it had at the end of the second beat, it would traverse one mile an hour, we must infer that, as the instant of starting is more and more nearly approached, the body moves so slowly that, if it kept on moving at this rate, it would not traverse a mile in an hour, or in a day, or in a year or in a thousand years; indeed, it would not traverse a span in an even greater time; a phenomenon which baffles the imagination, while our senses show us that a heavy falling body suddenly acquires great speed.”

OK. So while Galileo didn’t have the mathematical machinery of calculus, he certainly had many of the ideas:

“Time is divisible without limit”

“There is no degree of speed however small with which we may not find after starting from infinite slowness”

“So that if that speed which it had at the fourth beat was such that, if kept uniform, the body would..”

“As the instant of starting is more and more approached, the body moves so slowly that, if kept moving at this rate, it would not…”

A key idea is the hypothetical, “So that if that speed which it had was such that, if kept uniform, the body would…” Another words, Galileo is specifically thinking about speed at an instant by considering how far it would go in a measure of time if the speed it had at that moment was not allowed to change. Of course, this idea is really the same as slope of the tangent line idea. We zoom in on a moment, fix the rate of change, hypothetically extend a line with that rate, and measure how far that line extends vertically in a fixed measure of horizontal change.

A Second Excerpt

Here Galileo describes ideas about the relationship between velocity at instant, displacement, and average velocity:

Let the line AI represent the lapse of time measured from the initial instant A; through A draw the straight line AF making any angle whatever; join the terminal points I and F; divide the time AI in half at C; draw CB parallel to IF. Let us consider CB as the maximum value of the velocity which increases from zero at the beginning, in simple proportionality to the intercepts on the triangle ABC of lines drawn parallel to BC; or what is the same thing, let us suppose the velocity to increase in proportion to the time; then I admit without question, in view of the preceding argument, that the space described by a body falling in the aforesaid manner will be equal to the space traversed by the same body during the same length of time travelling with a uniform speed equal to EC, the half of BC.

lf0416_figure_052


Here Galileo has essentially constructed a velocity vs. time graph for an accelerated object (turned on its head), and is arguing that distance traveled during an interval of time is equal to the distance an object with uniform speed would cover if the speed was half the speed the accelerating object acquired at the end of that same interval. Not only did Galileo have in place many of the ideas for thinking about limits and rates of changes, but also the beginnings of integral calculus.

Hypothetical Velocities

The claim I want to make here is that Galileo made sense of instantaneous rates of change and accumulation via two different hypothetical constant velocities he had to imagine.

Instantaneous velocity at a given moment of time was construed as the distance an object would travel in a measure of time if the velocity it had at that moment were kept constant.

Accumulation was conceived as distance traveled by an object with a hypothetical constant velocity, here described as the velocity half of the velocity obtained at the end of the time interval.

And this is what makes instantaneous velocity so difficult to comprehend… it is a discussion of a hypothetical moving object and its relationship to a real moving object. Instantaneous velocity is thus a huge imaginative leap of faith–one in which we imagine a differently moving object and aim to establish some relationship between the imagined object and our real one. Instantaneous velocity requires a suspension of reality–an acknowledgement that you aren’t going to talk about what actually happened, but to make an explicit analogy between reality and an imagined one.

If I’m right, that instantaneous and average velocity are merely analogies to imagined hypothetical objects moving at constant velocity, then it seems that pinning down the meaning of constant velocity becomes even more important. Galileo, refined his definition of uniform motion over time:

Definition

By steady or uniform motion, I mean one in which the distances traversed by the moving particle during any equal intervals of time, are themselves equal.

Caution

We must add to the old definition (which defined steady motion simply as one in which equal distances are traversed in equal times) the word “any,” meaning by this, all equal intervals of time; for it may happen that the moving body will traverse equal distances during some equal intervals of time and yet the distances traversed during some small portion of these time-intervals may not be equal, even though the time-intervals be equal.

Reading Quiz 2.0 Continued: Day 2

I’m not as happy with this… not as coherently packaged as first, trying to do too much, etc. But here it is.

Question 1:

In class we discussed that 60 mph means that you travel 60 miles in each and every 1 hour of travel. What does it mean for a car to be traveling at 60 mph if it hasn’t yet gone for an hour? What does the “60” mean, for example when you’ve only traveled for 30 minutes, or 15 minutes, or 1 minute?

Question 2:

You and a friend are driving along the highway at 60 mph, exactly 9 miles away from your exit. Your friend announces, “It should take about 9 minutes to travel those 9 miles, because every mile takes one minute.”

How do you think your friend arrived at this conclusion? Is he right? Explain.

Question 3:

Your friend is back at it again, keeping track of where you are one the highway, but this time he jots down the mile-marker every ten minutes.

Clock Reading

Mile-Marker

6:00 AM

15

6:10 AM

25

6:20 AM

36

6:30 AM

48

6:40 AM

61

6:50 AM

75

7:00 AM

 

 

 

Are you guys driving with constant speed, increasing speed, or decreasing speed? Explain how you can tell from the data above.

Question 4:

What’s your best estimate for where your car will be at 7:00am? Explain how you came to your answer and why it makes sense to you.

Question 5:

During your drive, you and your friend start arguing about whose car is better at speeding up. Your car can go from 0-60 mph in 4 seconds. Your friend’s car that can go from 0-80 mph in 5 seconds.

Whose car would you say is better at speeding up? Explain how you reached your conclusion.

Reading Quizzes 2.0: Day 1

I’m drafting my pre-class reading quizzes for the summer session. Here is day one. Please comment.

Question 1:

Imagine you are tutoring an 8th grade student in physics. You notice that the student is a little confused about the difference between what is meant by 3:00 pm and what is meant by 3 hours. How would you explain to this student the difference between 3:00 pm and 3 hours?

Question 2:

You are still tutoring this same student, and notice the student is having a similar problem, except now they are confused about the difference between being at mile-marker 72 and having traveled 72 miles. How would you explain to them the difference between being at mile-marker 72 and having traveled 72 miles?

Question 3:

Here is a physics problem! Imagine you are driving a car along a long straight. Your friend in the passenger seat is recording where you are every hour of the trip.

Clock Reading Mile-marker

2:00 PM

20

3:00 PM

72
4:00 PM

124

5:00 PM

???

6:00 PM

228

Explain why your friend might reach the conclusion that the speed of the car at 3:00 PM was 24 miles/hour.

Then explain why that answer is probably not correct.

Question 4:

You might have noticed that the table above does not have information about where the car was at 5:00PM. This information is missing because your friend was asleep. What do you think is a good estimate for the mile-marker of the car at 5:00PM? Explain how you determined your answer and why you think this is a good estimate.

 Question 5:

Here is one final question to consider. How much total distance did you and your friend travel between 2:00PM and 5:00PM? Why is this answer different than your answer to question 4?

Paper: Are All Wrong FCI Answers Equivalent

A paper that an undergraduate researcher and I were reading this week is this:

Are All Wrong FCI Answers Equivalent?” by Helena Dedic, Steven Rosenfield, and Nathaniel Lasry, published in the 2010 Proceeding of the Physics Education Research Conference.

Essentially the paper examines patterns of student responses to the 4 Newton’s law questions in the FCI in order to identify different classes of students. Using a statistical method called Latent Markov Chain Modeling, they group students together based on how they respond to the four questions–not just in terms of correctness, but which answer they pick specifically. Their analysis identified 7 groups of students. One surprising thing from their analysis is that the groups form a natural hierarchy.

Basically, the authors looked at how students transition from one group to another, by calculating the probability of transitions occurring between classes. The cool result is that there is a strong directional bias to the transitions. In other words, there exists an ordering of the groups (e.g., C1-C7), such that transitions are always toward C1. After instruction, students are likely to either stay put or move up the hierarchy, but they are very unlikely to move down the hierarchy.

What’s even more interesting is that the classes do not form developmental stages. This means that C7-C1 is not a progression of learning. Certain forward transitions do not happen, or are very unlikely to happen.

It’s been helpful for me to think of the situation quantum mechanically. There are discrete cognitive states which can be measured, and there are certain probabilities of transitioning into different cognitive states; some transitions are very unlikely to happen and can perhaps even be forbidden. The fact that they fall in a hierarchy also reminds me of quantum mechanics, in that we can order QM states by there energy levels, and this ordering implies a bias in how perturbed and unperturbed systems such as the hydrogen atom will transition.

What I especially like about this paper is the meaning they give to each of the classes (in terms of student schemas), and how those interpretations build on much of what we know about student thinking about Newton’s 3rd Law. Each of the classes is generated from the data, and are more complexly defined than what I write below, but the gist is this

Class 1: They get all the questions right, essentially. This class is intended to represent a class that is highly likely to be a Newtonian Thinker about these Questions

Class 2: This class essentially gets the all the questions right except for the the question about the Newton’s third law when two objects are speeding up together. The interpretation given here is that these students are thinking of the Newton’s third law pairs within a Net Force (or competing forces) schema.  ** In my teaching physics class, we had long arguments about this question **

Class 3: What I can make of this class, is that the students essentially get either 1 question wrong (but not the speeding up question), or two questions wrong (but not in a pattern similar to Class 4 or Class 5). I wasn’t really quite sure what to make of this class, exactly. Students had about 65-75% of answering any question correctly, implying that there is an assortment of 1-2 answers wrong. But the Modal answer for each question is the correct answer, where as Class 2 the Modal answer for one of the question is wrong.

Class 4/5:  These two classes are almost the same. They both answer consistently that the more “dominant” object exerts more force. Class 4 says that when maintaining speed neither object is more dominant (so forces are same); whereas Class 5 says that the one of the object is more dominant (forces not same).

Class 6: I think this less strongly defined (with lots of scatter), but I gathered they included students who said that objects can be obstacles that are merely in the way without exerting a force.

Some cool comments about transition probabilities: I think a really cool gem in this paper is that there is a fairly low transition probability from Class 2 to Class 1. In other words, students who start in Class 2 only have a 37% of transitioning up to Class 1. This is in contrast with Class 3, where students have a 65% of transition up to Class 1 (and low probability of going to Class 2). This means that the cognitive state of thinking about Newton’s 3rd Law Pairs in terms of Net Force is a fairly stable state. This mean that being high in hierarchy doesn’t necessarily imply you’ll ever get to the top! It’s like there a trajectory of learning that gets you nearly to the top, but never quite there. Other things are this: Students with a dominance schema (class 4 and 5) have a high probability of basically staying put, and also true of Class 6, for which students are 58% likely to stay put.

It’s kind of weird, but only Class 3 makes significant movement to Class 1. Classes 2, 4, 5, and 6 have limited mobility, at least within the time frames they are looking.

Anyway, I’m curious what other people think, and what they see in their FCI data.

Duckworth on Teacher Preparation

I’ve been thinking about teacher preparation, and keep coming back to Duckworth:

“First, teachers themselves must learn in the way that the children in their classes will be learning… Second, the teachers work with one or two children at a time so they can observe them closely enough to realize what is involved for the children. Last,  it seems valuable for teachers to see films or live demonstrations of a class of children learning in this way, so that they can begin to think that it really is possible to run their class in such a way. The fourth aspect is of a slightly different nature. Except for the rare teacher who will take this leap all on his or her own on the basis of a single course and some written teachers’ guides, most teachers need the support of at least some nearby co-workers who are trying to do the same thing, and with whom they can share notes. An even better help is the presence of an experienced teacher to whom they can go with questions and problems.”

From the Essay ‘The Having of Wonderful Ideas’ by Eleanor Duckworth

Notes to Self about Oral Assessments

One question I asked a lot of the future physics teachers during their oral assessments was to discuss the following expression in terms of what it could mean (i.e., the story it tells) and if it seems to be a valid statement or not.

Δx = vf Δt – 1/2 a (Δt)²

Follow up question were targeted at asking students to identify correspondences between various terms and features of graphs.

With some students, this led to a conversation about all three of equations below–how they were similar and different etc, and how they corresponded to features of graphs.

Δx = vi Δt + 1/2 a (Δt)²

Δx = vavg Δt

Δx = vf Δt – 1/2 a (Δt)²

The growth that I’ve seen in many of the students is this. At the beginning of the semester, most students would have either said that the equation was wrong by pointing out that the wrong velocity and the minus sign, or for some, their only way of determining its validity would have been to substitute another equation into that equation until it looked like an equation from a textbook that they already knew. Thus, by the authority of the textbook, the statement must be true as well.

Most students by the end of the semester were able to look at this brand new equation–one they have never seen or considered before–and tell both a verbal story and a graphical story that explains and validates the equation without need for algebraic substitutions. I am impressed.

In other areas, growth has come in fits and starts, and there were many relapses. Many students during oral assessments this last week exhibited one of more of the following difficulties:

  • Confusing instantaneous velocity with constant or average velocity (e.g., claiming that an object thrown upward with a speed of 20 m/s goes 20 meter in the first second)
  • Confusing change in velocity for average velocity–the two are subtly related in constant acceleration case. Interestingly, this mistake came up both mathematically and conceptually for the same student in the same assessment.
  • Confusing acceleration, velocity, and displacement–claiming that when an object falls 10m, it will take 1s to do so, and be up to 10 m/s of speed.
  • Confusing change in speed with change in velocity–claiming that a bouncy ball experiences no change in velocity upon rebounding off the floor.

To be fair, these relapses often came up within much more complex physical situations and tasks that were more cognitively demanding–bouncing balls, oral assessments, deriving a result, etc. It’s good to know that, distinguishing these concepts, is not stable and automatic for them. It requires specific cognitive attention to maintaining the distinctions. They can do it, but are prone to making these mistakes when there are many balls to juggle all at once. I discussed with students how the mistakes they are prone to making here are similar to the mistakes we encountered all semester in looking at student work and how they are similar to the ones they are going to encounter in the classroom, so it is best to understand these mistakes well by making them yourself in as many different situations as possible, and doing the hard work of finding your way out of them.

Doing oral assessments for standards makes the assessments spontaneously grow into interesting conversations. Sometimes, what I thought I was initially assessing became something else entirely. Sometimes during assessments, I would respond to wrong things student said by saying, “OK. Tell me about that again, and then tell me why what you just said can’t be right.” Some students would be able to recover, and others would need more time, and I’d give it to them. Other times, I would send a student off, saying, “There’s a mistake in here. I want you to go figure out what the mistake is. When you come back, I don’t just want to know what the right answer is, I want to know why you made that mistake and why it’s a big deal.”

It’s been interesting to watch the varied resiliency of students in the face of my refusal to accept their work as proficient. The only times I know I was doing a less-than-good job was when they would come back and say, “Is this what you want?” Fortunately, it wasn’t often, but it did happen a few times. Most of the times they would come back saying, “OK. I got it now,” or, “I still don’t get this, can we talk about it some more?”

Next year, I’m putting a 2-week rule on my standards, so that students have to turn in an initial draft within 2 weeks. I had many students flying in last minute trying to work through all the standards.

 

Blog at WordPress.com.

Up ↑