Abstract: Developing automatic systems for assessing spontaneous spoken utterances is important for second language learning, because it promotes and democratizes self-regulated learning and can serve as an auxiliary tool in language proficiency assessment and teacher training. While such systems are typically developed for languages with a large number of learners such as English, the languages with fewer learners such as Finnish and Swedish remain at a disadvantage due to the lack of training data. Nevertheless, due to recent advancements in self-supervised machine learning methods it is now possible to develop automatic speech recognition systems without a large amount of annotated training data. This means that it could also be now feasible to develop automatic speaking assessment systems also for under-resourced languages. In this talk I present our automatic speaking assessment demo system for spontaneous second language learners' speech in Finnish and Finland Swedish. Furthermore, I briefly describe the main components of the system and their evaluation results and the personalized feedback it provides to the language learners. Finally, the most important input features for the machine learning models are determined in order to increase the transparency and explainability of the system and to be able to return more accurate feedback for the students and their teachers. This automatic assessment system comprises several machine learning models: (1) an Automatic Speech Recogniser that converts the spoken utterances into written transcripts; (2) Lexico-grammatical Accuracy and Range Evaluators to the transcribed responses by leveraging textual features; (3) Pronunciation and Fluency Evaluators to the transcribed responses by leveraging acoustic and prosodic features; (4) a Task Accomplishment Evaluator to evaluate test-taker adherence to the task assignment; and (5) a separate Evaluator for the final CEFR-like score representing the holistic overall speaking proficiency.

Speakers:  Mikko Kurimo received his M.Sc., Lic.Tech and D.Sc.(Tech.) degrees from Helsinki University of Technology in 1992, 1994 and 1997. In his PhD thesis he developed neural networks based machine learning for automatic speech recognition (ASR). Since then he has been working as a research scientist at IDIAP, a Swiss research centre for artificial intelligence and visited as an international research fellow in a number of research groups specialized in machine learning and ASR including University of Colorado in Boulder, University of Edinburgh, SRI in Stanford, ICSI in Berkeley and NITech in Nagoya. At Aalto University Professor Kurimo has been the head of the automatic speech recognition group since his return from Switzerland in 2000. His work is internationally best known for unsupervised subword language modeling for morphologically complex languages such as Finnish, Estonian, Turkish and Arabic. His recent achievements include the winning of the 2017 multi-genre broadcast speech recognition challenge and success in Tekes Challenge Finland competition (348 competing projects) and in EC's H2020-ICT-2017 call (115 competing projects). His research interests include deep learning methods for automatic speech recognition and spoken language modeling.

Affiliation: Aalto University

Place of Seminar:  Otaniemi, T5 (in person) & Kumpula exactum C323 (streamed)