Short Paper Review Criteria
A short paper format of 1500 to 2000 words (excluding references) was used for both the research stream and practice stream.
The review criteria assessed whether papers:
- used, adapted, connected together or critiqued one or more of the MELT frameworks
- specified the practice or theory gap
- drew on and synthesised an appropriate literature base that was broader than just the RSD literature
- contained well-reasoned methods/methodology for research or approach for practice
- for posters or presentations, work in progress or results/outcomes were presented in ways that fit the data or information *
- for workshops, there was a clear sense of the activities the audience would engage in
- were communicative and succinct
- consistently used one citing/referencing format.
* Qualitative, quantitative and mixed-methods data were welcome, with the expectation that typical standards for each type of data should be adhered to. It was acceptable for for outcomes of practice to use a blend of student evaluation data and research vignettes or stories.
Support for your research
The conference organising committee wanted to provide information on the Short Papers as early as possible, so that presenters who wanted to submit a research-stream paper had time to work on researching their MELT approach before the submission was due.
If you have been using one of the MELT frameworks, such as the RSD, you may want to treat use till now as a pilot and begin to plan research. For example, you may want to apply for ethics approval in the next 3 months, so that you can gather research data in Semester 1 or Summer Semester 2017.
A good starting point is the various research papers under evidence of effectiveness.
We invite you to consider research approaches for determining the effectiveness of your use of MELT. You can email John Willison and state:
- the ways in which you have already used one or more of the MELT
- aspects you are thinking of researching with ethics approval
- gaps this may address
- preliminary research questions
- ideas you have on research methods
Some possible research methods for the timeframe include:
- Pre- and Post-Questionnaires
- Exist for research skill development and problem solving development
- analysed for statistically and educationally significant changes over, say one semester in Likert Scale questions.
- Open-ended questions may provide student attribution to what caused any changes noted in Likert scales
- Online response rates are typically so low they are not worth conducting unless...
- Ethics/questionnaire committee approval allows surveys to be completed in class time
- need to be part of the educational program, e.g. for student reflective practice, as well as to improving the course from info gathered in the pre-questionnaire.
- need to be administered by someone not involved in the teaching of the course
- We found pre-and post-questionnaires were not useful outside of a specific context, such as a subject
- Individual Interviews
- good to capture student experience
- we relied on individual interviews when determining RSD efficacy across programs by interviewing graduates and students after they completed three years of their degree
- rich data requires complex analysis
- can be used in conjunction with pre-post questionnaires, especially when students make and attribution of what caused any changes
- Focus groups
- good for improvement of approaches
- students may also make attribution of what caused change
- Observation data
- person external to the course/program
- ethics required
- observer needs to be skilled in what to observe
- Mixed methods
- potential 'triangulation' of data
- quantitative and qualitative aspects need to satisfy their own determinant of quality, such as 'reliability' and 'trustworthiness' respectively
Reliability and trustworthiness
An upfront consideration of 'reliability' (for quantitative research) or 'trustworthiness' (for qualitative research) is worthwhile in the planning stage.
Likewise, specification of 'disconfirming evidence' is worthwhile in analysis stages i.e. data that does not fit into your standard pattern, and that may not complement your main inferences. Specifying disconfirming evidence is particularly encouraged for the research stream.