AI that mirrors how humans behave can drive better designs for keyboards and charts
The human-like performance of these AI models is also transparent, paving the way for explainable AI and better human-computer interaction.
Comparing human eye movements to the Chartist model in three tasks: finding a certain value in a chart, filtering, and finding the biggest value. Image: Danqing Shi.
Two new research papers examine what happens when human behaviors are linked up with the capabilities of AI models. The Chartist and Typoist models interact with, respectively, data visualizations and touchscreen keyboards, in the same way a human user would, giving a clearer ‘glass-box’ view into how and why these models operate the way they do. The Chartist model can read and understand charts like a human would, which could provide instant feedback for creating more understandable designs. The human-like typing errors made by Typoist, created with the Google Gboard team, can help make text entry on mobiles more accurate. Both models are being presented in late April at CHI, the premier conference for research in human-computer interaction.
If you ask people to look at the same statistical chart and answer questions based on the data, their eye movements will vary a lot, as shown in the example image. “How people move their eyes around a chart and how they then use that to comprehend what the chart is about, these two things are interrelated,” says professor Antti Oulasvirta from the Finnish Center for Artificial Intelligence FCAI and Aalto University. Chartist can replicate the diversity of human eye movements when reading charts. Combined with the capability of a large language model, Chartist can then integrate what was viewed at each spot on a chart. “This connection has not been made before,” continues Oulasvirta. “We can ask the model to gather information from a chart to answer a particular question, and it can make sense of the chart as a whole, in ways that really simulate human behavior.”
The step-by-step reasoning for how the Chartist model reaches its conclusions is also transparent, explains postdoctoral researcher Danqing Shi, who is first author of both the Chartist and Typoist papers. This ‘glass-box’ approach is important for making AI models explainable, as opposed to black boxes where the decision-making process is hidden. Concretely, models like Chartist can help people make better data visualizations, by simulating how easy different chart designs are to read and understand. “Chartist can also potentially take any chart image and allow a person to ‘converse’ with it and get information out of the image, even if they can’t see it or read it,” says Oulasvirta.
Typoist is the latest state-of-the-art model of typing behavior that Oulasvirta and colleagues have been working on for a number of years. “Nearly everyone has a mobile device, but we know that for some, like aging adults, text entry can be very hard,” says Oulasvirta. “If we want a society where everyone can participate, we need to know why people make errors when they’re typing.” Errors might arise from being in a rush while texting to forgetfulness, and researchers classify them in three ways: slips of the finger; lapses or forgetting to give a command to fingers; and mistakes, where what is seen onscreen is wrong and a typo-in-progress is missed.
But how does having a computer model that makes these same errors help make typing easier for people? Simulating how people type can lead to the design of better keyboards, without the need to test them with lots of users. When a keyboard app like Gboard has a billion users, even a small change to it can have a huge societal impact, Shi points out, so it’s crucial to make sure it works optimally for users. “A simulated typist, like Typoist, that behaves like a human, can produce lots of data to guide the development of better text entry systems for real human users,” says Shi.
Both Chartist and Typoist, models that can accurately simulate how people read charts or type on touchscreen keyboards, offer routes to quicker and cheaper user research through synthetic data, the production of large volumes of human-like measurements. Moreover, the new way to create models using simulations and machine learning offers a path to increased transparency and explainability in AI systems for human-computer interaction. The Typoist paper has been recognized with an honorable mention, given to the top 5% of papers at the CHI conference.
Read more in the papers:
Simulating Errors in Touchscreen Typing
Chartist: Task-driven Eye Movement Control for Chart Reading