February 24, 2010 « End of Course Surveys, Part I: Why Do Them? »
This multi-part Executive Briefing was prepared exclusively for senior decision-makers. The focus assumes progressive organizational experience leading to a senior position in a college or university setting.
Part I looks at the typical objections to end-of-course assessments and suggests they can be addressed in ways that achieve benefits outweighing risks. Part II will outline key executive considerations in revising an end-of-course assessment system to achive stated benefits.
As with other Executive Briefings where multiple issues of some complexity are involved, I will focus on assertions of potential value to you. Additional detail and justification is always available via email or a phone conversation.
Background
Since 1985, my research team and I have developed, validated, deployed, analyzed, and reported findings on more than ten million end-of-course assessments administered in more than 500,000 courses. For more than half of these assessments, we have captured, taxonomized, and analyzed the open-ended responses to open-ended questions about instruction, curriculum, other learners, the learning environment, and support services. In all, we have retained more than three million comments in a knowledge-base used to profile new comments. Most of these rich data sets and knowledge bases have been derived from the “other half” of higher education’s market, working adult students, and most have been analyzed for whatever secrets they can reveal about the relations between educational processes, outcomes, and impact. These meta-analyses include examination the relations between end-of-course and other process measures and educational outcomes including test scores, panel ratings, performance metrics and rubrics, professional licensure outcomes, and grades.
This 25 year span of work has brought me face-to-face with more practical problems, more challenges to validity and usefulness, and more inferential dead-ends than I could have imagined at the outset. The experience has also demonstrated, and replicated well past any claims regarding validity, the important benefits that accrue to a well-designed, well implemented, and well used end-of-course assessment process. Equally important is the fact that these benefits mature and deepen with use.
A Bad Rap
End-of-course assessments have received a bad rap. Among the claims typically leveled at these assessments are that they: (a) are nothing more than a popularity contest, (b) cater to and reward negativism, (c) represent isolated, atypical perspectives, not the mainstream, (d) correlate highly with grades awarded (good evaluations are easily “purchased” by awarding good grades), (e) provide scant valid information of use to instructional, program, or operations personnel, (f) are not taken seriously by students, (g) show little or no correlation with learning, and my favorite (h) they are invalid because students don’t have the capacity to make informed judgments about their educational experience.
Is the Bad Rap Justified?
Are any of these claims true and, if so, to what extent? The answer rests on the specific assessment process in question, including not only its scientific and technical merit but its institutional context, including setting conditions and patterns of use. In principle, end-of-course assessments can be implemented so poorly as to make some of these claims true. Poor instrument development can produce skewed findings. Poor administration procedures can reduce student confidence in the process. Poor reporting procedures can stifle the development of skills essential to interpretation and use. A lack of senior administrative support (read: use) can undermine the seriousness of efforts required to implement and manage a good process.
The good news is that even a modest application of necessary expertise and conscientious administration will vitiate the negative outcomes and replace them with information of significant benefit to the institution. In multiple settings, we have proven every one of these negative claims to be false under standard conditions of long term administration and use.
Why Assess?
The easy answer is that regional, national, and professional accrediting bodies generally require end-of-course assessments. However, these oversight bodies lack the expertise and political will to set the kind of standards that would require assessments to produce and use valid information. For this reason, and because faculties have mixed feelings about student assessments, end-of-course assessments suffer a low institutional priority. In the typical setting, instruments are poorly designed (usually by faculty committee with no collective skill in measurement science or decision-support), comments are not gathered or are inappropriately treated, analyses are sophomoric, reports are not timely nor are they individually customized for stakeholders, and there are no requirements that the findings be used in an intelligent way. In such cases, one should expect to hear some of the objections I identified above.
If I were limited to offering one reason for implementing a sound end-of-course assessment system it would be that such assessments provide the first and best line of decision-support for effective process management.
For most of the 20th century, higher education defined quality based on inputs. We were of high quality because our faculties had the right degrees and our laboratories and libraries had the right stuff. Driven largely by external forces, we are gradually if begrudgingly accepting the idea that outcomes in relation to defined standards should be given more attention in our definitions. This gradual shift is taking us closer to rational definitions of quality, all of which rest on the notion of “suitability to purpose.” Were we in any other service industry, this new focus on outcomes would take us all the way to the mid-1950’s. While outcomes (and impact) are arguably the sine qua non of educational service, other service industries have learned that the most effective way to achieve outcomes that suit the purpose is via process management based on carefully designed measures that are aligned to performance standards. In a process managed environment, outcomes are measured but are confirmatory rather than surprise factors.
Well-designed and appropriately used end-of-course assessments mitigate much of the risk in outcomes assessment, where it is too late to effect remediation.
Like a turn-by-turn GPS, end-of-course assessments provide critical information that can be used to ensure that students, instructors, curriculum, and support services remain harmoniously on course, working in unison to achieve the desired outcome.
In Part II, I will discuss key executive issues in making your end-of-course assessment program work.
Robert W. Tucker is President and CEO of InterEd, Inc.
He can be reached through this forum.
The expression of other views by leaders in higher education is welcomed.
Reader Comments (4)
I loved how this started -- validity problems, etc., are foremost in my mind when I think about student surveys.
I once devised my own instrument for my classes, throwing the useless pre-printed student survey supplied by the college in the trash. I understand that they had another decade's supply in a closet somewhere. And the administration was so iron-fisted and iron-clad that the suggestions that we needed something better glanced off their sides like useless arrows. No one could get anywhere, and committee after committee deadlocked on the issue.
After I collected my make-shift survey, I eagerly sorted through the results with a colleague, but, to his dismay, he found himself named as one of the worst/most hated teachers on campus. (Yes, I actually had a question like that.) But when we were finished, we looked at each other and agreed that, yes, the highest ranked teachers were the easiest graders, with the least challenging (most pedestrian) presentations and course work. It was sad but true -- for these students right out of high school, at a community college, this was what their expectations were -- grade 13, no homework, no work at all.
Now, I look forward to more on this important topic.
But I was also somewhat confused -- EOC also refers to end-of-course exams, like the State of New York's Regents Examinations in high school.
Florida is now doing away with its FCAT proficiency exams, seeking this legislative session to replace them with EOC exams. For the record, New York Regents exams started in 1866, but I may be a year of two off.
I wanted to add that I long noticed a bi-modal pattern (a dumb-bell shape) in my student surveys, both in those issued by the institution and the one that I devised (see above).
The populations that seemed to correspond were (at the low end) those just-graduated high school students, mainly interested in getting an "A" (more so than actually learning anything). I wouldn't say that they were disengaged, it was just that they were too young to understand that learning means growing and changing oneself in response to the course, and assignments, etc.
At the high end were the adult learners, those eager to explore the world through the classroom and what I was offering them there. Of course they wanted the grades, but they were not young credentialists just out of high school.
These adult students were also not as negatively impacted by the institutional arrangements as were the younger students; that is, the fact that I was in the role of the teacher and they were students did not seem to get in the way of our interaction. Rather, many times, I had the distinct pleasureable sense that we were engaged in a collaboration, in a learning partnership, and not a power struggle. And it is this sense that I miss most, having now left the classroom.
Corresponding to Mr. McGhee's observations with respect to his teaching experiences, we have extensive data suggesting that working adult students are somewhat more likely to see the relationship between themselves and the instructor as an implied contract which they take seriously in accordance with the amount of money they are spending. The operative word is "somewhat."
The base case is that most students, irrespective of age, will take the classroom seriously, including it evaluative components, in response to what they perceive to be authentic interactions between themselves, their fellow students, and the instructor. There are outliers in any system. The proportion of students who behave irresponsibly, even in the context of effective teaching, etc. is very small. Unfortunately, faculties tend to talk about outliers as if they were the base case. In doing so, they are neither lying nor telling the truth.
In an upcoming Executive Briefing, we will discuss how the 3,000,000+ records in the 20 year comment knowledge base authenticates the intelligence of students when evaluating their courses. The sneak preview I can offer here is that it will convince most individuals that students are indeed critical consumers of educational services who remain focused on the central elements of the learning environment and the value delivered to them. They are not incompetent and their judgments cannot be bought with grades.
As much as we may dislike acknowledging it (we include ourselves in the consideration), bad evaluations (including evaluations not taken seriously) are generally the result of ineffective teaching and classroom management. When we say, "generally" we mean often enough so as to suggest that it is unproductive and perhaps irrational to focus on the outliers when the base case is so clear.
- InterEd Staff
This article by Ray and Michelson starting me thinking in another direction, having more to do with the very different economic / social contexts of adult-students and teenage students, and how this impacts expectations as well as retention. For the latter, prospects of low wages, limited hiring opportunity, and no benefits were found to result in "attitudinal, intellectual, and often physical disengagement from school". I have no doubt that this context shows up in student surveys; but the big question is, to what extent?
http://majorsmatter.net/schools/Readings/RayMickelson.pdf
If anything, the negative economic factors described here have significantly worsened, and students have little choice but to go back to school, whether they like it or not -- and this is probably an unacknowledged driver of credential inflation, since every time students graduate, they find themselves competing with other students that have just graduated, albeit with a more advanced degree. So, it is back to school, again. While high rates of return are good for the schools, they also drive credential inflation.
I also read "high school" students as today's "college" students in the article.