Auditory-perceptual (AP) judgments are essential in determining the success or failure of many speech pathology treatments of communication disorders world-wide; the trained human ear is one of the most important tools in measuring speech pathology outcomes. Auditory-perceptual rating is considered the most common form of voice and resonance assessment, and the âgold standardâ due to its high ecological validity as a measure of the voice (Oates, 2009). Similarly, speech intelligibility is impacted by a range of features including prosody, pitch, speech rate, syllable segregation and fluency as well as voice and resonance. These AP judgments are needed for initial diagnosis and is directly linked to accurate final diagnosis and management.
The great challenge in teaching and assessing the AP judgments of students is it is a highly subjective task; there is no one correct answer. Human perceptual measures are subject to variation and bias. As students learn, they change their own ratings of the same recording overtime but overall they have similar ratings to other students. Experienced, expert listeners conversely have high agreement with themselves but often do not agree with other (less experienced listeners).
Research into the validity and reliability of perceptual judgments, tells us that training in auditory-perceptual discrimination improves identification of auditory features and training in auditory-perceptual judgments improves reliability of ratings.
We therefore need the students to practice listening to and rating speech and voice recordings so that they become more consistent raters, and therefore more reliable.
Encouraging students to practice in groups, discussing and comparing their responses, to establish a consensus answer has been one solution to the lack of a correct answer. This solution, demands either a lot of scheduled in-class time or is difficult for the students to arrange themselves. This discourages engagement and delays feedback on their performance.
Oates, J. (2009). Auditory-perceptual evaluation of disordered voice quality: pros, cons and future directions. Folia Phoniatr Logop, 61(1), 49-56.
1) To develop a mobile responsive website (CAPTAIN) that integrates the auditory-perceptual rating activities currently undertaken across Speech Pathology, including voice, resonance and speech, into one site that includes:
a) Voice and speech samples (plus support materials including background, clinical scenarios and client information) for current cases being taught in the curriculum in each of the units of study
b) High quality recordings taken in real-life, clinical settings
c) Internationally recognized, best practice assessment tools to be used when rating samples
d) Immediate, online feedback on ratings, provided by experts, and by other students.
e) 24/7 access on mobile devices to encourage engagement and promote frequent practice and use across teaching and learning activities and assessments
f) Utilisation across all countries that teach Speech Pathology to develop, for the first time, a global solution to a learning and teaching challenge.
g) Collecting data on student learning and auditory-discrimination in relation to AP judgements across voice and speech to further enhance learning and teaching tools in this domain.
2) To use the resources across different units of study for the same students and clinical cases to promote a deeper understanding of these judgments and promote enquiry and integration across units of study.
3) To design a user interface that allows global use, i.e. instructors can load their own recordings and analyse their own data.
4) To collect large data on rating behaviors across different countries, regions, language groups and time. This will allow a deeper understanding of learning behaviors so that more targeted teaching approaches can be utilized to support student learning.
5) Develop an online Assessment platform that can be used in conjunction with other forms of assessment.
Improving reliability of AP judgments in Speech pathology students will ultimately lead to improved assessment and treatment of communication disorders world-wide.
The learning and teaching website is optimized for mobile learning and student centered enquiry. Such resources bridge the divide between students’ digital needs, the university curriculum and readily available educational tools (Akour, 2009). The resource currently contains multiple cases with background information and high quality audio files for students for students to rate which can be used across units for different aspects of practice. The rating scales are considered to be gold standard and internationally recognized to ensure they are best evidence in each area of practice. The scales are engineered to allow mobile access and utility on a range of devices for ease of use by students. All rating forms have multiple expert responses as well as student responses for instant feedback. CAPTAIN was built using Meteor and MongoDB.
CAPTAIN allows the student to:
• complete the rating anytime, anywhere using a mobile device with access to the internet
• receive immediate feedback as to their rating compared to the expert (thereby appreciating that there will be a difference between their rating and the expert)
• view their rating compared to their peers
• track their rating over numerous attempt
• see the consensus response emerging over time
Other features include
• the functionality for the teacher to upload new listening samples that can demonstrate the effect of different tasks, accents, speech disorders etc on rating behavior as the students need it in their course
• Analytics that allow teachers to track student rating behaviors, identifying possible problems with a specific feature, task or recording.
• Allow online assessment to be developed and rating behavior analyzed, discovering more about how student learn this task over time and in different contexts.
Akour, H. (2009). Determinants of mobile learning acceptance: An empirical investigation in higher education. (PhD), Oklahoma State University.
1) Engaged enquiry: Engagement is achieved through: a) ready availability of the website on mobile devices and b) the use of clients that are taught in class as case studies across the Speech Pathology case-based curriculum. Enquiry is supported by the availability of a range of samples and rating tools for each client. Students can rate their client in one or many areas of communication. Many of the rating tools will enable students to decide which features are of interest which this gives them the opportunity to explore and test the specificity and sensitivity of nominated features with their peers.
2) Feedback and Assessment: Students receive immediate, on demand feedback on their rating of each element of communication. Once they have completed and submitted their rating, they can view the expert consensus. Students can also keep a record of their performance over successive practice sessions so they can measure their progress over time (i.e. semesters and years). Teachers can collect de-identified data from the website on both formative or summative activities and feedback student performance to the class.
3) Effective class and peer interaction: This website is available for use in and out of class. Students can share and discuss their ratings in relation to specific class cases or in relationship to their developing skills in auditory-perception.
4) Creative use of existing technologies or approaches: The use of a web-based platform that contains all necessary content in one place that is interactive and provides feedback, and can be shared with peers and used in classroom and assessment activities, is innovative and unique in Speech Pathology internationally.
This resource will impact on student’s learning experiences in the following ways:
a) Content will be related explicitly to current curriculum content - students will rate cases they have seen in class.
b) It will encourage constructive alignment and transfer of skills across units of study. This discourages learning content in silos and promotes enquiry and cyclic engagement with cases across the curriculum.
c) Use of internationally used rating tools will ensure students are exposed to both local and international practice protocols, raising awareness of cultural diversity of clinical practice.
d) Availability of the resource on mobile devices, 24/7, will promote student-centred learning.Students will be able to practice alone or in groups, at a time of their choosing. The website can be used in a range of teaching activities including online quizzes, in class activities, and homework tasks (Lametti et al., 2014).
e) Provision of immediate feedback on ratings (provided by experts and other students) enables the tool to be used either as formative (primary use) and summative feedback (in specific assessment tasks). Such feedback is proven to enhance auditory-perceptual judgments (Chan et al., 2012).
A prototype of one element of the rating website (CAPTAIN) has been built and tested on 2 student cohorts the University of Sydney. At the time of submitting this application, students had completed 6567 form submissions across two cohorts (n = 180). Student engagement with the prototype has been 100% and feedback from students has been 100% positive.
Chan, K., Li, M., Law, T. Y., & Yiu, E. (2012). Effects of immediate feedback on learning auditory perceptual voice quality evaluation. IJSLP, 14(4), 363-369.
Lametti, D., Krol, S., Shiller, D., & Ostry, D.(2014). Brief Periods of Auditory Perceptual Training Can Determine the Sensory Targets of Speech Motor Learning. Psychological Science, 25(7), 1325-1336.
Now that proof of concept has been tested locally using AP rating of voice samples, the next steps for the project are:
Phase 1) : To build other rating portals (e.g. resonance and speech sound discrimination) incorporating a range of speech recording and different rating tools. These will be tested with student cohorts at the University of Sydney in Semester 1, 2016. At this stage, we will analyse student use in relation to curriculum design, learning outcomes and student engagement across the different portals and different subjects to see if they are used differently. We will also design an assessment platform that can be incorporated into different assessment activities and link to the rating activities the students have been practicing.
Phase 2) : Invite national and international experts to rate the samples, building a picture of expert vs novice rating behavior. This will enable feedback from leaders in the field of speech pathology education to provide feedback on the utility of the tool as they use it to rate speech and voice samples, and invite engagement with the project from educators.
Phase 3) : In this phase, we will concurrently a) increase capacity for use of the website nationally and internationally and b) collect and create exemplars of different voice and speech features as rated and greed upon by experts internationally. This would represent a unique and original innovation as this resource currently does not exist.
Phase 4) : Once we have agreed global partners in the project, we will beta test the tool in other courses nationally and internationally. At this time, we will also approach Speech Pathology organizations globally to consider use of the web-site as the first world-wide credentialing tool.