Abstract: We outline the role of forward and inverse modelling approaches in the design of systems to support human--computer interaction. Causal, forward models tend to be easier to specify and simulate, but the inverse problem is what typically needs to be solved in an HCI context. We illustrate the core issues in a case studies with capacitive sensing for finger pose inference and for using Google’s Soli radar to infer hand pose. We use machine learning to develop data-driven models to infer position, pose and sensor readings, based on data from electrostatic/radar simulators and human-generated data. A forward model is trained on this data using a Conditional Variational Autoencoder.
This emulation can accelerate the electrostatic simulation performance with a speedup factor of ca 2 million, and can be used to augment the training data when using a CVAE to train an inverse model (the VICI approach of Tonolini et al 2020). We compare forward and inverse model approaches to inference of finger pose. This combination of forward and inverse models improves the robustness of performance over previous inverse-only approaches.
(This approach to inverse problems was developed by my student Francesco Tonolini in his Ph.D. thesis, Variational learning for inverse problems, School of Computing Science, University of Glasgow, 2021.)
Related reading:
F. Tonolini, J. Radford, A. Turpin, D. Faccio, R. Murray-Smith, Variational Inference for Computational Imaging Inverse Problems, Journal of Machine Learning Research, Vol 21, No. 179, p1-46, 2020. pdf
Roderick Murray-Smith, John H Williamson, Andrew Ramsay, Francesco Tonolini, Simon Rogers, Antoine Loriette , Forward and Inverse models in HCI: Physical simulation and deep learning for inferring 3D finger pose, https://arxiv.org/abs/2109.03366
http://www.dcs.gla.ac.uk/~rod/Publications.htm
Speakers: Roderick Murray-Smith
Affiliation: University of Glasgow
Place of Seminar: Zoom
Meeting ID: 694 4226 1666
Passcode: 675107