Evaluating Online Courses using SELT

Background

Evaluating online learning is, in principle, no different from evaluating learning in other contexts, such as lectures, clinical work or problem-based learning. That is to say, teachers in both contexts are likely to be interested in such matters as learning outcomes, students' preparation, their experience of learning and the learning environment.

Certainly, the technology supporting online learning gives us the potential to evaluate in different ways. For example, we can use online surveys, collect written feedback from students electronically and monitor learning behaviours. Some of these possibilities still have technical and management problems associated with them. These problems are somewhat too complicated to discuss here.

For the moment, we want to describe resources that are available to you online or by using SELT. We emphasise that there are other methods of gathering information, such as discussion groups, observation, interviews and document analysis.

Instructions

The following material is intended to help you design your evaluation in a framework that emphasises good principles of online learning. Examples of the kinds of questions you can access from the Student Experience of Learning and Teaching Service (SELT) are listed in relation to each principle.

The questions are available in the SELT Manual (Q1 – Q526) or from a more extensive database held by Planning and Performance Reporting (Q 527 – Q3140). New resources, questions and instruments are being designed and added to the SELT service progressively.

The SELT Manual and forms needed for administration and evaluation are downloadable from the SELT system.

Evaluating online learning and teaching: some principles

The 'Seven principles for good practice in undergraduate education', originally published in 1987, are a popular framework for evaluating teaching in traditional, face-to-face courses. The principles are based on 50 years of higher education research.

A follow-up article is available online Implementing the Seven Principles: Technology as Lever. The article originally appeared in print in the AAHE Bulletin, October 1996, pp. 3-6.

A team of evaluators from Indiana University's Center for Research on Learning and Technology (CRLT) used the seven principles to evaluate four online courses at a large US university. The evaluations were based on analysis of online course materials, student and teacher discussion-forum postings, and interviews.

The following text material on the seven principles is reproduced from an article which was originally published in The technology source. The article is:

Charles Graham, Kursat Cagiltay, Byung-Ro Lim, Joni Craner, and Thomas M. Duffy 'Seven principles of effective teaching: a practical lens for evaluating online courses'. The technology source, March/April 2001.

The material is reproduced here (with minor editorial changes to adapt North American terminology to that common in Australia) with the permission of the publisher.

The following are links to the principles within this page.

• Principle 1: Good practice encourages student-staff contact
• Principle 2: Good practice encourages cooperation among students
• Principle 3: Good practice encourages active learning
• Principle 4: Good practice gives prompt feedback
• Principle 5: Good practice emphasises time on task
• Principle 6: Good practice communicates high expectations
• Principle 7: Good practice respects diverse talents and ways of learning

Principle 1: Good practice encourages student-staff contact

Lesson for online teaching: Teachers should provide clear guidelines for interaction with students.

Teachers wanted to be accessible to online students but were apprehensive about being overwhelmed with email messages or bulletin board postings. They feared that if they failed to respond quickly, students would feel ignored. To address this, we recommend that student expectations and staff concerns be mediated by developing guidelines for student–teacher interactions. These guidelines would do the following:

• Establish policies describing the types of communication that should take place over different channels. Examples are:

  Do not send technical support questions to the teacher; contact the Departmental Computing Officer first or send them to the MyUni Help Desk.

  The public discussion forum is to be used for all communications except grade-related questions.

• Set clear standards for teachers' timelines for responding to messages. Examples:

  I will make every effort to respond to email within two days of receiving it.

  I will respond to emails on Tuesdays and Fridays between three and five o'clock.

Relevant SELT items

Principle 2: Good practice encourages cooperation among students

Lesson for online teaching: Well-designed discussion assignments facilitate meaningful cooperation among students.

In our research, we found that teachers often required only 'participation' in the weekly class discussion forum. As a result, discussion often had no clear focus. For example, one course required each of four students in a group to summarise a reading chapter individually and to discuss as a group which summary should be submitted. The communication within the group was shallow. Because the postings were summaries of the same reading, there were no substantive differences to debate, so that discussions often focussed on who wrote the most eloquent summary.

At the CRLT, we have developed guidelines for creating effective asynchronous discussions, based on substantial experience with staff teaching online. In the study, we applied these guidelines as recommendations to encourage meaningful participation in asynchronous online discussions. We recommended the following:

Learners should be required to participate (and their grade should depend on participation).

• Discussion groups should remain small.
• Discussions should be focussed on a task.
• Tasks should always result in a product.
• Tasks should engage learners in the content.
• Learners should receive feedback on their discussions.
• Evaluation should be based on the quality of postings (and not the length or number).
• Teachers should post expectations for discussions.

Relevant SELT items

Principle 3: Good practice encourages active learning

Lesson for online teaching: Students should present course projects.

Projects are often an important part of face-to-face courses. Students learn valuable skills from presenting their projects and are often motivated to perform at a higher level. Students also learn a great deal from seeing and discussing their peers' work. While formal synchronous presentations may not be practical online, teachers can still provide opportunities for projects to be shared and discussed asynchronously.

Of the online courses we evaluated, only one required students to present their work to the class. In this course, students presented case study solutions via the class web site. The other students critiqued the solution and made further comments about the case. After all students had responded, the case presenter updated and reposted their solution, including new insights or conclusions gained from classmates. Only at the end of all presentations did the teacher provide an overall reaction to the cases and specifically comment about issues the class identified or failed to identify. In this way, students learned from one another as well as from the teacher.

Relevant SELT items

Principle 4: Good practice gives prompt feedback

Lesson for online teaching: Teachers need to provide two types of feedback: information feedback and acknowledgment feedback.

We found during the evaluation that there were two kinds of feedback provided by online teachers: 'information feedback' and 'acknowledgment feedback'.

Information feedback provides information or evaluation, such as an answer to a question, or an assignment grade and comments.

Acknowledgment feedback confirms that some event has occurred. For example, the teacher may send an email acknowledging that they have received a question or assignment and will respond soon.

We found that teachers gave prompt information feedback at the beginning of the semester, but as the semester progressed and teachers became busier, the frequency of responses decreased, and the response time increased. In some cases, students got feedback on postings after the discussion had already moved on to other topics.

Clearly, the ideal is for teachers to give detailed personal feedback to each student. However, when time constraints increase during the semester's busiest times, teachers can still give prompt feedback on discussion assignments by responding to the class as a whole instead of to each individual student. In this way, teachers can address patterns and trends in the discussion without being overwhelmed by the amount of feedback to be given.

Similarly, we found that teachers rarely provided acknowledgment feedback, generally doing so only when they were behind and wanted to inform students that assignments would be graded soon. Neglecting acknowledgment feedback in online courses is common, because such feedback involves purposeful effort. In a face-to-face course, acknowledgment feedback is usually implicit. Eye contact, for example, indicates that the teacher has heard a student's comments; seeing a completed assignment in the teacher's hands confirms receipt.

Relevant SELT items

Principle 5: Good practice emphasises time on task

Lesson for online teaching: Online courses need deadlines.

One course we evaluated allowed students to work at their own pace throughout the semester, without intermediate deadlines. The rationale was that many students needed flexibility because of full-time jobs. However, regularly distributed deadlines encourage students to spend time on tasks and help students with busy schedules avoid procrastination. They also provide a context for regular contact with the teacher and peers.

Relevant SELT items

Principle 6: Good practice communicates high expectations

Lesson for online teaching: Challenging tasks, sample cases and praise for quality work communicate high expectations.

Communicating high expectations for student performance is essential. One way for teachers to do this is to give challenging assignments. In the study, one teacher assigned tasks requiring students to apply theories to real-world situations rather than remember facts or concepts. This case-based approach involved real-world problems with authentic data gathered from real-world situations.

Another way to communicate high expectations is to provide examples or models for students to follow, along with comments explaining why the examples are good. One teacher provided examples of student work from a previous semester as models for current students and included comments to illustrate how the examples met her expectations. In another course, the teacher provided examples of the types of interactions she expected from the discussion forum. One example was an exemplary posting while the other two were examples of what not to do, highlighting trends from the past that she wanted students to avoid.

Finally, publicly praising exemplary work communicates high expectations. Teachers do this by calling attention to insightful or well-presented student postings.

Relevant SELT items

Principle 7: Good practice respects diverse talents and ways of learning

Lesson for online teaching: Allowing students to choose project topics incorporates diverse views into online courses.

In several of the courses we evaluated, students shaped their own coursework by choosing project topics according to a set of guidelines. One teacher gave a discussion assignment in which students researched, presented, and defended a current policy issue in the field. The teacher allowed students to research their own issue of interest, instead of assigning particular issues. As teachers give students a voice in selecting their own topics for course projects, they encourage students to express their own diverse points of view. Teachers can provide guidelines to help students select topics relevant to the course while still allowing students to share their unique perspectives.

Relevant SELT items