Student Evaluation of Instruction Report

University of West Georgia Department of Management
Faculty Perceptions of Student Evaluation of Instruction (SEI)

Keywords

Course Evaluation, Student Evaluation of Instruction, Assessment, Program Evaluation, Accreditation.

Abstract

The Student Evaluation of Instruction (SEI) is a required part of evaluating a faculty member’s teaching, and is the basis of career decisions such as promotion, tenure, or merit raises. There have been concerns raised about the validity and reliability of SEI instrument in the department, as well as in published literature. A study was conducted to determine perceptions of Department of Management faculty about the Student Evaluation of Instruction (content and process). It was found that majority of faculty respondents were not satisfied with SEI forms and processes. There were concerns expressed about validity, reliability, use of the same instrument to measure the quality in different types of courses such as undergraduate, graduate, small and large enrollment classes, and online courses. Although many respondents were open to having the evaluations administered online, several raised issues regarding efficacy of implementing such a system.

Introduction

Student Evaluation of Instruction (SEI) plays an important part in not only the assessment of teaching, but also being tied to promotion, tenure, and merit raises. Often SEI is considered to be the primary vehicle for evaluating teaching performance. Although some instructors may be able to challenge students, motivate them to work hard, reward them with appropriate grades, and still receive favorable ratings, most instructors recognize that demanding workloads do not help raise evaluation scores and may well lower them (Kleeberg, 1993).

There had been informal discussions among some Department of Management faculty in the Richards College of Business on the reliability of SEI forms. The question raised was whether the SEI process could be considered an unbiased, valid, and reliable instrument for determining the effectiveness of an instructor's classroom teaching. As a result, a study was undertaken to further investigate this perception. In October 2007, a questionnaire was made available to 19 department faculty provide input on SEI that all instructors are required to administer to students at the end of each semester. After a second reminder a week later, ten respondents completed the survey giving a response rate of 52.6%.

1. Are you satisfied with the current Student Evaluation of Instruction (SEI) forms and process?

● Yes ● No

2. Please comment on strengths and weaknesses of the current system.

3. If offered, will you use an Online version of Student Evaluation of Instruction (SEI) instead of conducting the evaluation by distributing paper forms usually on the last day of class? The Online version would have a student go to a web site to complete the evaluation form. This would replace the paper-based version currently in place.

● Yes ● No

4. Please comment on your response for #3 above. What do you see are advantages/disadvantages of a completely electronic system?

5. To capture more focused responses specific to a discipline, should Student Evaluations be conducted at the School level rather than at the University level?

● Yes ● No

1. Are you satisfied with the current Student Evaluation of Instruction (SEI) forms and process?

Faculty comments:
#1 While the current instrument has shortcomings, having seen other evaluations, I am afraid it changes for worse rather than better.

#3 The on-line survey from distance education does not adequately capture instructor performance.

#4 The items on the SEI seem to lack quality question design from a basic perspective of validity and reliability.

Also once completed by students (per UWG 103.06) the policy suggests that the SEI files should only be provided by the administration to the review committee, and not submitted by the individual faculty under review (such as 3rd year, or P/T).

Weaknesses--not particularly relevant to the goals and purposes of the course; takes the focus away from academics and places the emphasis on "popularity"; instrument weights all areas equally (for example: how the instructor speaks carries the same weight as whether the course was challenging); makes it much harder to make students responsible and accountable because the instructor then will be "chastised" through the evaluation instrument.

#6 The system is very weak. The fact that faculty get to take own evaluations back to the departmental office does not look good. The system must be more transparent. Folks at many other institutions use teaching and research assistants and/or secretaries to collect and deliver the forms to the department office.

#7 It asks the students to comment on aspects of courses that are far beyond their knowledge. It asks wholly subjective questions such as "The instructor seems to enjoy teaching" Many of the questions are colored by the grade that the student has earned and offer a ready internal excuse for any of the student's shortcomings. Many other questions assume that the students have no responsibility for asking questions when something is unclear much less reading the instructions posted to WebCT and elsewhere. The system does not allow for any rebuttal - the student voice is jury and the department chair hears the verdict - only there is no provision for a defense attorney.

#9 It doesn't take into account variations in purpose in courses - it weights all factors the same, and not all may be of equal importance to all courses.

4. Please comment on your response for #3 above. What do you see are advantages/disadvantages of a completely electronic system?

#1 I don't see any advantages/disadvantages one way or other. This would be just alternative method. Outcome will be the same.

#2 Having used online evaluations at several postsecondary institutions, students forget to complete the online evaluation. To counter this problem, I include the online evaluation as a part of the grade and instruct students to forward the confirmation showing completion of the evaluation. Otherwise, students who participate are usually the "unhappy" students; therefore, the online evaluation is skewed negatively.

#3 I don't believe my students will actually complete the evaluation if offered online. At least in class they are essentially pushed into doing the evaluation.

#4 The only problem is actually motivating students to complete this. My understanding is that this form is already available to be placed on WebCT, but students do not complete it.

#5 No, the threats from possible contamination, control of access to both the raw and completed data, and loss of administrative control are very high. We as faculty have the responsibility ability to administer the instrument when we deem to have a sufficient sample. With an on line environment we lose control of sufficiency of volume. To transfer this task externally from the faculty would require a complete new definition of faculty expectations and responsibilities.

#6 Fewer students (except those who wish to vent) will complete this unless it is monitored (as with distributing paper copies during class). Thus feedback would be reduced to a few negative comments rather than from the majority of students.

#7 I believe it would be a more transparent process. Each student must be given a unique identifier so that completion of form is done only once.

#8 Student responses would be truly anonymous. As it is, a professor might recognize the writing of a student. It would also reduce the chances of students colluding to downgrade an instructor (I realize this sounds paranoiac, but when nearly identical verbiage appears on several forms from the same class, one cannot help but wonder)

#9 I believe more people would complete the survey if it were online.

#10 It would be impossible to ensure that the person filling out the form is a registered student in the class, and the results would be completely untrustworthy.

5. To capture more focused responses specific to a discipline, should Student Evaluations be conducted at the School level rather than at the University level?

Additional Analysis

Because of the small sample size, a “member check” which would provide additional insight on the responses was conducted. The use of member check has been recommended in research especially with small samples. Member check involves sharing the researcher's interpretations of data obtained from surveys with the participants from whom they were gathered and determining if they feel that the results are credible (Creswell, 1998; Merriam, 1998).

Based on random selection, one faculty member each with the rank of Full Professor, Associate Professor, Assistant Professor, and Instructor was interviewed. A summary of responses provided during the interview were as follows:

#1 The SEI form is too complicated and students go through it quickly during the last class session. They do not have time to reflect on their responses. A simpler form is needed which could be administered online.

#2 Reduce the current 27 questions with maybe 5 questions that capture teaching attributes

#3 Students should be involved in a focus group session where they share their comments with administrators.

#4 Graduate students tend to be more critical in their comments (because they are older, and have more experience) so the graduate course evaluations tend to be lower. This should be taken into account.

#5 Some instructors are playing the game of lowering standards so they can get higher evaluations.

#6 There needs to be a common way of administering the evaluations e.g. by using graduate assistants or administrative assistants rather than the instructor who administers the evaluation.

#7 Instructors can artificially inflate the scores by giving out the evaluation forms after an easy test.

#8 Instead of relying on SEI forms for evaluating teaching, there should be more peer observations, seminars on improving teaching etc. that contribute to helping faculty be better teachers.

It was found that these responses were consistent with original responses gathered from the online survey.

It was clear from above survey and member check responses, there is concern regarding validity and reliability of SEI instrument and the process by which it is administered. Key issues concern validity, reliability, use of the same instrument to measure the quality in different types of courses such as undergraduate, graduate, small and large enrollment classes, and online courses. The author also contacted faculty members in other colleges as well as Institutional Research and Planning to investigate if any work had been done in the past to address changes to the SEI. The following response was received:

“About 10 years ago an Assessment Committee in the COE developed another instrument that was pilot tested. It had 4 parts to it to accommodate the various types of classes that were taught. Every class answered the basic questions and then specific classes selected the other types of questions according to the type of class:(a) online classes (b) activity classes and (c) lecture classes that were face to face. Unfortunately the instrument never got adopted. I believe it was successful in evaluating student performance”

In contacting the Institutional Research and Planning, the author was informed that this is something that the Faculty Senate would have to address either through the work of a sub-committee or ad-hoc committee.

Summary

This study examined faculty perceptions in the Department of Management, Richards College of Business. Although the study was conducted at the department level, with a small sample, several issues emerged as to the reliability and efficacy of the system currently in place. Responses received are consistent with research that shows student rating forms are unreliable and invalid (Aleamoni, 1994), grades and evaluations are positively correlated (Millea & Grimes, 2002), and evaluations are used by the administrators more for faculty career decisions (raises, promotion, and tenure) rather than as a tool for improving teaching and learning (Hmieleski & Champagne, 2000). Further studies that include other departments and colleges are needed to identify action steps that address faculty concerns about the format and content of SEI as currently being administered at the university.

1) Aleamoni, L (1994). Typical Faculty concerns about student evaluation of instruction, (ERIC Document Reproduction Service No. ED 113995).

2) Creswell, J. (1998). Qualitative inquiry and research design: Choosing among five traditions. California: Sage.

4) Kleeberg, R. N. (1993, September). Letter to the Editor. The Chronicle of Higher Education, B4.

5) Merriam, S. B. (1998). Qualitative research and case study: Applications in education. San Francisco: Josey-Bass Publishers.

6) Millea, M. & Grimes, P. W. (2002). Grade expectations and student evaluation of teaching. College Student Journal. Retrieved on November 14, 2007 from http://www.findarticles.com