Douglas Anderson Professional Portfolio: Personal: Improvement Cycle

How do I evaluate and improve my performance as a scholar, colleague, and teacher?

The school year provides an institutional time cycle conducive to frequent evaluation. For the past twenty or so school years, I have been comfortable thinking of change as a conscious and continuous process. Given the nature of a faculty job, it is also an individual process. It has been over fifteen years since I was last reviewed formally by the institution, when the Promotion and Tenure Committee recommended me for tenure. In most of the years since, I have written an annual self-evaluation, but it was never a document that a chair or dean discussed with me other than a perfunctory "Doing great! Keep it up!" Changes in my performance have been initiated by me and evaluated by me. I am reluctant to call them "improvements" because some of them don't improve learning. Even if they did, I am likely to change them again soon.

In upcoming years, I will be reviewed twice, once by my department, the periodic professional review scheduled for fall 2007, and once by the Promotion and Tenure Committee for promotion to full professor, probably in the following school year. This portfolio will be the focus of both reviews, which will provide yet another evaluation cycle.

As you will read below, evaluation, I do all the time. But improvement? I'm not so sure.

Scholarship

Process

In terms of my scholarship, the appropriate cycle is the project, each of which has its own time demands. Because scholarship is "my" time and there is so little of it, I reflect often on the project's process. How can I make it more efficient without sacrificing the opportunity to explore and discover, notoriously inefficient yet productive processes?

One measure is clicks. Can I do the same thing in a fewer number of clicks? If so, then I tend to improve my process in that manner. I am very conscious of clicks as I make more videos. The demands of file management for thousands of media files and the demands of the fancy editing and effects software drive me to find ways to reduce my number of clicks, and thus save time.

For the first three years that I had Ricci Street, I analyzed the server logs with WebTrends, the industry-leading software. It told me fascinating things about traffic at the site. As the company's name implies, the interest is not so much in a snapshot of one month's traffic. It is in the trends over time. Such sophisticated web analytic techniques have many problems and trade-offs, but the trends have a high degree of validity.

As an example, based on the patterns I saw in the server logs, I rearranged my course webs and their navigation bars so that users could get to more popular information in one or two fewer clicks. If successful, this reduces site traffic, oddly enough. Learn much more about my site analytics.

Product

The projects' outcomes -- the products -- I evaluate differently from the process. There, the concept of improvement is ridiculous, a variant of the post hoc fallacy, a glaring example of the pernicious factory model, rationalizing all human activity adding value to raw materials. A factory can't prosper without such thinking. However, it is inimical to scholarship. Is the fifth novel I finished, the one that Random House published, an improvement over the third and fourth, which remain in manuscript? Most emphatically: No! Nor is there any reason consistent with human psychology why there should be any expectation of improvement in a creative work that comes later in time than another creative work by the same person. Do we expect that a third child will be an "improvement" over the first two?

I have learned over the decades to solicit feedback on my creative work in ways that do more with less. I do more by listening better, and I do it with less by doing it in such a way that the person giving the feedback isn't burdened by it. I have also learned where in the process such feedback will do me the most good. Finally, I have learned to trust myself more, lessening the need for feedback.

Thus, for my creative projects, the improvement cycle is internal, the ongoing trial and error process, adding, deleting, rearranging, and trying to experience each incremental revision with a fresh eye.

Service

In terms of service, the appropriate cycle is the school year. Over the years, I have become a better colleague as I gained experience and perspective. As the faculty has grown, I have been on fewer committees and task forces. As a senior faculty member, I try to pick and choose the spots where I feel as though I can do the most good, for example, Promotion and Tenure Committee, or where I have the most interest, for example, the internationalization initiative.

My expectations of what can be accomplished in higher ed in general and at Medaille in particular have steadily dropped. From bruising political wars, I have learned the value to society of academic freedom, tenure, shared governance, and due process. I have learned that efficient administration of a higher ed organization and prudent management of its resources according to the factory model are restraints on learning. Learning wants to diverge, not converge on a final exam or shrink itself to the one-dimensional line of a rank-ordered grading scale or chop itself into thirty-eight sixty-minute segments over fifteen weeks. The business survives in spite of the faculty, and learning happens in spite of the administration.

Teaching

Am I a better teacher than I was thirty years ago? Different, yes. Better, I'm not so sure. What I consider better will not necessarily be considered so by the students.

I am "better" in the sense that I am more comfortable in the classroom, more confident every year -- as the students get older and I get younger -- that they have a lot to learn. Their reliance on received wisdom and their peers shackles them to the past, and my enthusiastic embrace of the unknown, perilous future challenges them to cast off their restraints.

I am "better", perhaps, for having adapted to emerging technologies rather than waiting until they have proven themselves. However, for most students that is a challenge because I know more about computer technology than they do. It doesn't often happen to most of them that a teacher knows more about computer technology and the Internet than they do.

In terms of teaching, each school year contains two similar cycles - semesters. And each semester has four or five course cycles. A cycle overlapping that is the number of times I have taught the course.

How am I doing?

The standard quality improvement cycle breaks into four iterative phases: plan, do, check, act.

Let me enter my teaching cycle at the check phase. How am I doing? I don't think it makes any sense to ask the students that question directly. They aren't teachers and I've been doing it for over thirty years. What makes sense is to ask them what they are learning.

Of more value than the official surveys (2006 and 2007) are the other methods I use to get active feedback from students.

As part of my pedagogy, I am continually asking students what they learned and how they learned it. I ask the same in emails and private conferences.

Rather than ask the students' opinion about the course, I use four methods to learn about their learning:

These methods have their own time cycles which ensures that I am continually getting information about student learning and how my performance affects it.

in-class

At the end of almost every class session, while the students are beginning to pack up, I ask several of them, in turn, questions like the ones on the right. I expect everyone to listen to the answers.

The students don't know whom I will ask on any given day. Over the whole course, I try to ask everyone several times. I don't respond directly at the time; instead, I listen and prompt for more.

weather report

From 1997 until this fall, every course I taught had an online discussion forum that I call the "weather report" (Tom Angelo's phrase). I start a new topic for each class session, which I link to from the course web's syllabus page. Most often, I post the starter message on the right.

Some classes use this more than others. It takes off when the students start talking to each other and more or less ignore me. When that happens, this ongoing forum is a rich source of course evaluation for me. I often make night-by-night adjustments in a course based on the night-by-night feedback the students are giving me. Also, this forum lets students learn how others are perceiving the course, which is a great way to build the sense of community that I consider the main advantage of the classroom. Without the open Internet connection in class and the forums, the students would be more isolated.

mid-course review

In most courses, I ask the students to write a mid-term review for everyone to see at the Bistro on Ricci Street. I post the starter message on the right.

The public nature of this mid-course review is, I think, its most important feature. Students are talking to each other as much as to me.

Not that I want to rush you, but we're starting the second half of this course.

You've probably learned more than you realize because much of this learning takes time to sink in. Especially since you probably have adjust several mental models about how organizations and industries compete, it will take a period of time using the models to fully learn what you can do with your business plan.

This is a good time to reflect on what you've learned so far and its value to you.

In response to this message, list the half-dozen most important or useful things you've learned so far in this course. Explain one in detail, the more detail the better, especially about your mental models. Many of you haven't had the time for more than very short Bistro messages. This one is your opportunity to write looooooong.

I recommend that you compose your reply off-line and save it before you copy-and-paste it into the Bistro reply box. Over the course of the next few weeks, add more messages to this discussion by responding to other students' messages. The greatest value of the Bistro is for you to speak to and listen to each other, not me.

student self-assessment

As the final requirement of every course, I ask for a self-assessment. The course disclosure typically has the statement on the right.

When I remind the students of this self-assessment requirement during the final or next-to-final class session, I try to leave enough time to go all the way around the room once, asking each student, "What is the most important thing that you learned in this whole course?" I often record the answers on the projected computer screen. The purpose is to start their thinking about the self-assessment.

I ask for the self-assessment as a private email between each student and me. In my annual self-evaluations (2006 and 2007), you will find excerpts from some of those responses.

More formal data collection

As part of my scholarly teaching (as distinguished from the scholarship of teaching), I used the Lab section of Ricci Street to collect data from students. In the overview of the teaching section of this portfolio, I discuss how collecting student data over time has helped me understand the limits of my pedagogy. Here, I will elaborate on one example out of dozens of how I used this data.

MBA 504 eSkills for Managers was the recommended first course for every on-campus MBA student. It was designed to give them more control over the post-industrial age's means of production: their Internet-enabled laptop computers. MBA 504's central question (all my courses have one, and only one):

How geeky do you have to be to prosper in today's globally networked business environment?

As you can see on the screen shot below, the form asked for a self-assessment. The categories were general enough and expressed in operational terms to account for the self-assessment aspect. It asked about several dozen software skills, from the more common seen in the screen shot to the more arcane such as:

First, I aggregated the responses for all the members of a course and displayed the results at the beginning of the next class. This addressed the course's most common anxiety. For those who felt they were the least geeky in the room, it told them that they were not alone and most probably were not the least geeky. For those who felt they were so geeky that they didn't need the course, it told them they still had a lot to learn.

Second, the course addressed all these skills in half a dozen groups. At the end of each section, I asked the students for an addition to their individual Communications Improvement Plan (CIP). The CIP was not for them to tell me what they had learned. That was in their self-assessment. The CIP was for them to tell me what they hadn't learned yet, what they didn't have time to get to before we had to move on to the next topic.

Third, at the end of the course I had a personal conference with every student. We discussed their CIP and then we looked at the Basic Skills form that they had sent to me after the first class night.

The form then became a tool for them to measure their learning. It was also a tool for me to change what we did in class. Because MBA 504 was designed to move the students closer to a cutting edge that was always receding beyond their horizon, the course had to change, too. This Basic Skills form, folded into the students' self-assessment, was a valuable tool to help me make rational changes.

What should I do differently?

I reject the factory supervisor mind-set. A rank-ordering of students along a one-dimensional line provided by a multiple-choice test:

does not tell me anything useful about an individual student's learning. It is, however, administratively very convenient and provides a comforting illusion.

However, faculty have since William Farish (see right) used the exam to distinguish between a 89.45 B+ and an 89.55 A- course grade. The same statistical methods applied to that faculty's scholarly research would not pass the peer-review requirements for valid evidence. Yet applied to teaching, such faulty reasoning has great impact on students' lives. And what does the multiple-choice test tell the teacher to do differently? If everyone answers the same test question incorrectly, does that mean the teacher should spend more time lecturing about it next semester? Or just teach to the test by emphasizing that it "will be on the final exam"?

In short, applying scientific methods unscientifically and applying widget-production methods to human beings are not processes that I find helpful, truthful, or ennobling of the human being. To the extent that every student is a one-off, teaching is a craft or trade. The changes I have made over the years have come from that entrepreneurial cottage-industry mind-set.

I support the College's outcomes assessment effort because it will provide me with aggregate information without doing further damage to individual students.

Not all students adapt well to my pedagogy because I put so much of the responsibility on them for learning rather than on me for teaching. Because my classroom is so relatively student-centered and because there are so few students, the mix of personalities and my management of their conflicts can dramatically affect individual students' perceptions. A lecture-and-test classroom has a standard, consistent personality, a class culture determined mostly by the instructor who is presenting most of the material. My courses, centered on student projects and presentations, have a different personality from mod to mod, indeed from day to day and night to night.

The information collecting methods described above come class session by class session rather than course by course. Thus, I can easily make mid-course changes to better adapt to the individual students in the class. Eloquent testimony to these changes come from the end-of-course student surveys, where the students give me a score that is consistently among the low scores on question #4: The course was well organized. For almost all courses in the students' experience, this means organized prior to the semester's starting. What I see as flexibility and adaptability caused by their continual on-going feedback, some students see as indecision and poor planning. (Please note that those low scores are still closer to 1 Strongly Agree than to 2 Agree and even further from 4 Disagree.)

This continual gathering of feedback from students creates a class culture where students can express their needs and frustrations without having to turn it into a complaint or a personal crisis or a power struggle over points on an exam.

This web, offered in fulfillment
of the requirement in the handbooks of
Medaille College, Buffalo, NY,
Volume IV: Faculty Handbook,
section 4.5.4.3 Faculty Portfolio,
is © 2007 and licensed under a Creative Commons License.

web established: February 2007
page last modified: October 2007
by Douglas Anderson
http://toLearn.net/portfolio/personal/cycle.htm