2008 Learning and Teaching Implementation Grants

Project: Online Role Play
Project Team: Ms Elizabeth Hutton (Professional and Continuing Education)

Online role-plays are a powerful teaching tool providing students with the opportunity to communicate in a safe environment, the anonymity allowing them to participate more actively in discussions. International students, who generally find it difficult to express their views in face-to-face discussions, will have the opportunity to develop their critical thinking skills, negotiation, decision-making skills and problem solving skills in a more comfortable environment. Online role-plays provide a learning environment that helps to develop in students a situational awareness and increase their engagement. By participating in online role-plays, students develop graduate qualities, team work and communication skills.

Project: An integrated web-based assignment submission, assessment and feedback system
Project Team: Dr Alfred Brown; Mr Brad Alexander (Computer Science)

The school of computer science has over recent years developed in an ad-hoc fashion a number of web based applications to directly support learning and teaching. The most useful of these is a web based assignment submission system that requires students to use a version control system to look after their assignment work. The aim of this proposal is to significantly enhance this system by carefully designing the integration of the many useful components to better serve the needs of academic staff and students. The specific aim of this project is to implement an integrated web based assignment submission system that:

1. Supports all University campuses. At present Computer Science teach courses at both the North Terrace and Singapore campuses. The system will be available 24x7 lo any student with an internet connection using freely available software that runs on all common operating systems. Therefore, the same secure, authenticated access will be available whether a student is working in a computing laboratory on North Terrace, working at home in Singapore or working in an internet cafe in Paris. An online assessment cover sheet with appropriate links to University policies will be used to simplify the online assignment submission process.
2. Is easy to understand and use. The system will be able to be configured by academic staff via a web interface that performs a number of validation steps. This should simplify the task of setting up new assignments and testing that they work. The simple web interface will also allow students to quickly locate their assignment submission pages, easily make new submissions, receive marks and receive feedback. Where appropriate the marking and feedback will be immediate.
3. Supports Computer Science programming assignments. The system will be able to run immediate tests over student programming assignments in a secure manner that can quarantine any erroneous or malicious code that is submitted. This permits immediate feedback that may include assignment marks and it allows students the opportunity to resubmit to correct errors and/or misunderstandings. The system will also accommodate group work where any group member can make a submission on behalf of the entire group. This area of support is essential in a number of assessment and curriculum initiatives being pursued in Computer Science.
4. Supports industry best practice. Professional programmers and software engineers rely on version control systems to manage their projects, record all changes they make and record reasons for the changes. The proposed system will require students to use the version control system, subversion. Subversion uses an internet accessible database to store all components of a project, to record logs of why changes are made and provides the ability to retrieve any previous versions when required. Given the use of subversion, assignment submission is then simply a matter of requesting that a particular version be submitted for assessment. The efficient use of the network by subversion means that remote access from Singapore can be fast and efficient despite the large distances involved. Software projects include all kinds of materials in addition to programs. Therefore, it is also appropriate to require all essays, reports and theses to be stored in and be submitted from subversion too, in line with industry best practice.
5. Supports improved feedback to students. The web interface of the assignment submission system provides a convenient secure location where personal student feedback can be posted. In addition to any automatic feedback that may come from immediately run tests, the system will provide academic staff with a template mechanism that can efficiently generate detailed feedback to each student. Student feedback presented via the web interface becomes a permanent that record students and staff can revisit at a later date. This will be supplemented by per assignment records of deadlines, any extensions that have been given and any late penalties that have been applied. Automatic processing of marks, extensions and late penalties will simplify collation of marks entry at the end of a course.
6. Supports more effective use of discussion forums. A common problem in answering student forum questions is that academic staff do not have direct access to a student’s work. The system will provide a mechanism for matching student names to their subversion database and their assignment submissions. This is necessary because names no longer uniquely identify our students and all user accounts are now identified by student number. The linkage permits answers to discussion questions to be better informed allowing for better guidance without giving answers to problems that are being assessed.
7. Supports practical examinations. A key component of Computer Science education is learning how to program. Finding ways to encourage students to invest sufficient practise is difficult, as is the task of knowing whether or not a student really wrote the programs that they submit for assessment. The system’s ability to automatically test submissions and give immediate feedback in combination with the requirement to use subversion, provides the ability to run effective practical examinations. A laboratory can be set up with just the required software, network access can be restricted to subversion and the assignment submission system, student identified can be checked and all network access can be logged. If plagiarism is suspected a complete audit trail is then available in the assignment submission system, in each student’s subversion database and in the network traffic logs. The ability to run effective practical examinations is critical to a number of recent curriculum initiatives such as our problem solving curriculum project funded by Google.

It should be noted that, most of the component technology is well understood by the applicants and in most cases has been reliably used by students both in Adelaide and Singapore over a number of years. For example, the basic web based submission system has been used but has significant limitations in how submissions must be processed, there is no way of adding feedback, marks collation is very primitive, the mechanisms to contact misbehaving programs are too heavy handed, group work requires manual intervention behind the scenes, there is no way of easily relating student names to their submissions, the ad-hoc nature of the current systems means that mistakes are easily made and the student experience is not always what we would like. The proposed system will address these issues through careful redesign and integration of the system components. It will also address many other issues we have identified over the years so that all users, both staff and students, will be better served. Historical concerns have included technical issues regarding how the components interact whilst maintaining secure authenticated access for authorised users only policy issues regarding the extent to which ITS have to be involved in maintaining the systems and educational issues regarding how best to guide staff and students new to the systems. Tried and tested solutions that address all of these issues will be implemented as part of this project.

Project: An enhancement implementation of Maple TA in Mathematics Service Courses
Project Team: Dr Adrian Koerber; Mr Peter McCann (Mathematical Science)

Purpose: (a) To use Maple TA to enhance student learning in Mathematics service courses by implementing new modes of summative and formative assessment, providing students with instant, tailored feedback on their work. (b) To encourage and facilitate the further use of the Maple T.A. environment amongst Mathematics staff by producing exemplary assessment materials, extensive documentation on how those materials were created, and through the provision of peer mentoring and consulting services.

Background: Maple TA is a web-based teaching and assessment tool which allows online content delivery and assessment to be achieved in a flexible and mathematically literate manner. At the University of Adelaide we have implemented Maple TA as a building block inside of MyUni, the University of Adelaide’s online learning environment. Maple TA has been used in large courses for summative assessment tasks by the School of Mathematical Sciences since Semester 1, 2006.

Why is the project necessary?: The student cohort within service courses in Mathematics continues to diversify in many ways, but of particular import and concern are the variation we see in students’ backgrounds and abilities. Such differences need to be addressed urgently and effectively, and constitute a significant driver for this project. We see alternate assessment modes – particularly formative tasks such as flexible, self-paced learning modules that are accessible on demand – as valuable tools in helping to fill conceptual gaps that may arise from differing backgrounds, and in allowing students to develop a mastery of work they have previously found challenging. The significant advantage of developing these materials in Maple T.A. is that students can receive instant, directed feedback on their work, and can immediately request new assessment tasks based on the same concepts until they have been mastered. This establishes a positive feedback loop without requiring the intervention of a tutor or lecturer. For large service courses this free-standing capability greatly enhances the range and the flexibility of the help that can be offered to students, and is a considerable improvement on waiting for submitted material to be marked and returned manually.

Our rationale is echoed by international academics employing similar Computer Aided Assessment (CAA) packages. For example Siew notes that "allowing the student to rework the resubmit and resubmit an answer is important in the learning process", a conclusion reached through personal reflection and student feedback. This view is repeated by Gibbs, who states that "feedback [is] the variable that has most impact on students' achievement" and that "skills are not learnt in one go but through practice and feedback in repeated cycles of improvement". Our ability to offer repeated work and feedback is dramatically enhanced by CAA. Sangwin argues that CAA provides new opportunities to encourage higher levels of mathematical learning through question modes that have been traditionally infeasible, such as the "construction of instances" (Where students must construct their own example that satisfies the question requirements, which can potentially result in an infinite number of possible correct solutions).

Results from a pilot study: During 2007 we conducted a trial in a level II course servicing 500 students, in which a re-workable assignment provided simple but instant feedback regarding correct/incorrect results. A customised SELT was conducted for this new aspect of the course. In an open ended question that asked students what was good about the re-workable assignment by far the most common comments were an appreciation of the instant feedback and the ability to be able to diagnose their own mistakes. Some students noted that this capability was improving their mastery of the material. One of the unexpected outcomes of allowing assignments to be re-worked that we observed was the overwhelming tenacity exhibited by many students in acquiring the last few available marks for a task, even when this component was worth relatively little of their overall grade for the course.

Invariably there was also some dissatisfaction with this mode of assessment, primarily due to the ambiguity of some questions, but also because of the possibility of guessing at answers (eg multiple choice) and the difficulties encountered with using correct input syntax. These issues will be addressed in the current project through amore considered construction of individual questions, and by producing a standardised guide for syntax. The syntax issue will also be helped by a winder use of Maple TA (both within and across courses).

The positive aspects of the instant-feedback re-workable assessment model are well expressed in an unsolicited email received (before the SELT) from on e of the students in the trial, who found these tasks: "...to be of much use, obviously in terms of improving my mark, but more importantly in improving my understanding of the course work. As I knew when I had got an answer wrong, I could work much harder to improve it, picking up mistakes which I never would've had I not been certain the answer was wrong. This was of much more use to me than finding out two weeks after I did a question that I got it wrong, and then getting worked solutions which often as not used a slightly different method to solve the problem, and as such didn't help me."

Project: Development and evaluation of the Script Concordance Test as a test of clinical reasoning in high stakes examinations in the MBBS program
Project Team: Dr Paul Duggan (Obstetrics and Gynaecology); Associate Professor Anne Tonkin (Medicine Learning and Teaching Unit)

Satisfactory performance in clinical reasoning is a key outcome for graduates of the MBBS program and is relevant across all clinical courses in the program. The current MBBS curriculum has, since 2000, incorporated the teaching of clinical reasoning early in the program and has assessed students in clinical reasoning in high stakes examinations from Year 2 through to the final barrier examination that is currently held at the end of Year 5. The assessment of clinical reasoning is not straightforward and places high demands on resources, in particular the academic and support staff charged with running these assessments. The MBBS program is unable to sustain the existing methods of assessment of clinical reasoning in the earlier years of the program due to the nature of the current assessments and the 25% increase in student load. This applies, in particular, to the sequential reasoning assessment called the SRQ, in which students are asked to analyse clinical data, develop hypotheses and utilise additional information to assist in their understanding of the case. This is currently done in written format, resulting in a marking burden of about 12 minutes per student. This will not be sustainable with up to 180 students.

The SRQ is replaced in the later years of the program with modified essay question (MEQ) papers that are sat at the end of Year 4 and Year 5. The faculty has made substantial efforts to improve the quality of the MEQ paper and our discipline-based standard-setting has included representatives from teaching hospitals and the Flinders University to assist with standard-setting and benchmarking. In 2007 this process involved 50 senior clinical academics from 7 broad discipline groups with affiliations to the University of Adelaide or Flinders University and who participated in structured question development and standard-setting using the well-established, modified Angoff method. An analysis of the 2007 Year 5 MEQ paper (Duggan, Paul et al: University of Adelaide Assessment: Processes and Issues. Interdisciplinary Workshop, University of Adelaide, 4 March 2008) indicates that this paper was significantly flawed, both in terms of excessive testing of low level function (memory recall – 77% of the total questions) and in a substantial number of item writing flaws and marking scheme flaws. These data, whilst disappointing, must be considered in context – they represent our best possible effort in an examination format that has been in use for many years and are consistent with evaluations in other institutions of the quality of their examinations (Ware, James: The impact of flawed high stakes examinations, The 13th Ottawa International Conference on Clinical Competence, Melbourne 6 March 2008). Thus, we have identified that a major component of our clinical assessment appears to be unsuitable to its purpose. This raises important questions of fairness particularly with pass/fail decisions. The risk to the university of legal challenges to flawed assessment is considerable.

As we must continue to assess clinical reasoning in high stakes assessments, and from Year 2 of the 6 year program we need to develop an alternative method of assessment of clinical reasoning that has good reliability, construct validity, fidelity and that is able to be administered and marked with our existing resources. A very important bonus would be that the new assessment could be shared with other medical schools both for benchmarking and efficiency of resource utilisation. The Script Concordance Test (SCT), initially developed by Charlin of the University of Montreal (Charlin, Bernard et al: The Script Concordance Test: A Tool to Assess the Reflective Clirucian. Teach Learn Med 2000, l2(4) 189-195), satisfies all of these criteria. A key feature of the SCT is that a minimum of l0 and, preferably, 20 experts are required to develop each question, which creates challenges especially for smaller disciplines and encourages resource sharing with other universities and professional colleges. Thus, development of the SCT should be web-based, given the wide geographic location of the experts who are required to develop it. Another key feature is that the SCT, although requiring substantial effort in its development, can be marked by computer. This is very important to our university as the current methods of assessment that the SCT is intended to replace are all manually marked. There is no prospect of continuing to manually mark the assessments and meet criteria for timely return of results. Elimination of manual marking will also eliminate the intra-rater error that we have documented in our MEQ analysis, a problem which we know is of great concern to our students. Another key feature of the SCT is that it appears that high reliability (alpha > 0.8) can be achieved with 1-2 hours of testing time, or approximately 100 questions. All our present assessments require 3 hours of testing time to achieve the same reliability.

We have experience in SCT question development in the Discipline of Obstetrics and Gynaecology. (Duggan Paul. Development of a Script Concordance Test using an electronic voting system. Ergo 1, 1 December 2007; 35-41) The Discipline of Obstetrics and Gynaecology (O&G), is allocated approximately 25% of the teaching in Year 5 of the MBBS program. Students are assessed at the end of each of the four, 9-week clinical terms in O&G in addition to a significant component in the final (barrier examination) assessment at the end of Year 5. To improve the reliability, validity and economy of the end of term assessments the Discipline of Obstetrics and Gynaecology plans to replace one of its two end of term assessments with the Script Concordance Test. This explains why there are a relatively large number of O&G questions in Year 5 in the table below.

We are in negotiation with the University of Montreal regarding access to their questions and leasing of software for web-based development of our own questions in all clinical disciplines that teach in the MBBS program. We are leading a group of educators attached to key medical schools throughout Australia in a proposed Carrick Grant (Priority Project) application this year. The success of both initiatives will be dependent on adequate funding.

Project: Implementation of an electronic audience system (EARS) to enhance student learning through direct formative and summative feedback
Project Team: Mr David Foley; Mrs Sue Sharrad; Mr Frank Donnelly (Population Health and Clinical Practice); Dr Abdallah Salem; Associate Professor Alan Nimmon (Medical Sciences)

This project will introduce an electronic audience response system (EARS) to all levels of the BNurs program so that students can answer questions anonymously in class with immediate feedback, Student responses will enable formative feedback and will also be stored for tracking of performance, Students will be asked to evaluate the system and compare it to similar didactic lectures where it is not used.

Audience response systems have been used in teaching since the 1960's (Freeman et al, 2007) however they have not become practical until more recently. Physiology in the Faculty of Science purchased a system in the 90's. This system is attached to a plywood backboard and is consequently cumbersome to use, time consuming to set up and does not always work.

CLPD has a system with 30 ResponseCard RF keypads, but this is heavily booked and has insufficient keypads for the BNurs program. CSIRO mobile phones systems are currently inadequate.

In traditional didactic lectures involving groups of 50 or more students student interaction is limited by a fear of feeling foolish and public performance anxiety, consequently students are reluctant to ask or answer questions. EARS is able to overcome this by allowing students to answer questions and give opinions anonymously via an electronic keypad. Currently we are unable to get real time indicators of student engagement in class. EARS enables this, Further more Freeman et al (2007) claims that electronic response systems:

• foster a more student centred approach to learning
• allow improvements to formative learning
• allow student responses in classes of almost any size
• give immediate feedback in the next PowerPoint slide
• allow students to see if their answers are correct and also to see how other students answered
• can be used to gauge opinions
• start discussions
• can be used by academics to inform lecture content and complexity
• can be used for summative assessment with immediate feedback

EARS has already been trialled and shown to be of use by improving student satisfaction with lectures
(Lectures given by David Foley in 2007 with student response and SELT feedback).

Benchmarking
The EARS project will establish a benchmark for immediate student feedback, this is inline with the AUQA audit focus on assessment of 2007 – 2008.

This project will enhance lectures with EARS by enabling immediate student feedback. It is also likely to improve student satisfaction, participation and attendance. By linking this technology with summative assessment, students will be able to complete more assessment in class time leading to a reduction in out of class assessment, in a program that is already very full (approximately 36 contact hours per week).

Project: Implementation of a program evaluation mechanism
Project Team: Dr Chris Medlin; Associate Professor Bryan Howieson;Associate Professor Edward Lin; Dr Jill Thomas; Dr Sam Wells (Business School)

The Business School is developing a Program Evaluation Mechanism (PEM) as a part of its Accreditation Plan submitted to the AACSB (www.aacsb.com) in December 2007. In 2008 the Business School must complete the groundwork for this major project with significance for ongoing curriculum renewal over the next three to five years. This timeline is required if the Business School is to achieve accreditation with the AACSB. All accredited business schools are undertaking similar projects.

By articulating Program Learning Outcomes, the value of the PEM is that it clearly articulates for students, academics, employers and accrediting bodies the analytic and professional skills identification of assessable degree level Learning Outcomes also offers a logical alignment between all aspects of the curriculum, including graduate attributes and assessment, and the possibility to evaluate the strengths, weaknesses or gaps in a program.

In 2007 the Faculty of the Professions provided funds to prepare the curriculum development tools for the PEM based on PLOs. The work was undertaken during the last six months of 2007 by Dr Barbara Wake on two degree programs: BCom (Management) and MCom (Applied Finance). A report of the project was submitted to the last meeting of the Faculty of the Professions Learning and Teaching Committee and is available on the intranet of the school. Dr Wake has developed key curriculum design materials and group processes that can focus course coordinators on the central curriculum design issues as espoused in PLOs. Continuity in Dr Wake’s employment on the project is strategically critical to the AACSB accreditation plans of the Business School. The Business School seeks funds to employ Dr Barbara Wake at 0.5 time for the 2008 academic year to oversee the further development of the PEM across all degree programs of the Business School.

The success of this project will reflect positively on the Faculty and the Business School. Further, the success of this project will see the Business School at the forefront of the Australian AACSB business schools, with the potential for staff to report the project at AACSB conferences and in peer reviewed journals.

Project: Adaptation of a successful, medically-orientated, online formative learning tool for use in Science based practical courses
Project Team: Dr James Botten (Molecular and Biomedical Science); Mr Edward Palmer (CLPD); Mr Peter Devitt (Surgery)

Over the last decade in tertiary education there has been a major shift to what is referred to as ‘Online learning’. The perceived advantages of this educational tool include ease of delivery and production of self-assessment materials with built-in marking systems. From the student perspective, online learning resources can be accessed in their own time, are flexible, and interactive, allow for control over learning, and allow them to build skills in using technologies. Additionally, online learning can have benefits for teachers, including: flexible delivery of materials, potentially reduced marking loads, and (once resources are developed) reduced time demands. Online learning is an ideal medium for assessment purposes, both formative and summative – once suitable software and content have been developed.

Although isolated instances of online and technology-based education have been successful within individual tertiary institutions, most of these initiatives have been confined to a single component of one curriculum and have had limited application outside the field of the individual or individuals who developed the original concept. In general this represents a poor return for what has often been a costly (in terms of time and intellectual effort) exercise. Ideally, broader cross-disciplinary and cross-institution collaboration should be promoted in order to help design and provide appropriate online resources, which can be used across disciplines, faculties and institutions. In other words, effort and resources are best put into projects where it can be shown that the end results will have use outside the departments or faculties where they originated. Furthermore, such a philosophy is essential for the broader dissemination of effective online resources to improve learning outcomes nationally and internationally. Expertise in one discipline is rarely transferred to other areas because of lack of efficient lines of communication.

A software package called Medici has been developed by project members and is currently used at the University of Adelaide as an online formative learning tool in the School of Medicine. Primarily designed to produce adjunct material to supplement the clinical learning experience, the software and its educational content are used by undergraduate medical students and have been carefully evaluated and reported in educational journals. The lecturers in charge of providing this package were awarded the Stephen Cole prize for teaching excellence in 2007. The Medici package contains a series of more than 70 cases covering the core curriculum of the Discipline of Surgery. It is used by students primarily in their clinical teaching years. The material is used for self-directed study and for formative and summative assessment, providing instant feedback to students. The resource can be viewed at http://www.emedici.com and is also available on CD, as many students prefer this latter medium for their learning. Students are receptive to the software and the content, reporting they find the formative nature of the program and the content of considerable value during their studies. In a structured setting, the resource has also been shown to enhance students’ cognitive skills and to stimulate learning around the subject material. Results of these evaluations have been reported in the international literature.

The project team plan to adopt the successful strategies used in the Medical School and extend this online resource to the Faculty of Science, in particular the School of Molecular & Biomedical Science. In this School, as in many science disciplines, practical work is of vital importance, yet it is difficult to provide students with appropriate preparatory tools, largely due to lack of resources. A case based learning tool, tailored to the discipline, supported by multimedia resources and using a formative assessment strategy similar to the successful model working in Surgery is likely to provide students with a solid learning platform, which they can use to effect in the laboratory. In situations where practicals cannot be run, due to resource issues, a Medici case can be used to provide students with some experience in the use of techniques and equipment.