Research

Com­puter vis­ion iden­ti­fies ripe fruit and coun­ter­feit drugs

Researchers are developing an application based on AI algorithms that works in regular smartphones and brings extremely accurate hyperspectral imaging within anyone’s reach.

When would this avocado be suitably ripe for making guacamole? Is the drug I bought on my travels to far-off places the real thing or a fake? How big an apple crop are we getting this year?

Soon, answers to questions like these will be easily obtained, as computer vision technology developed by researchers will be made available to consumers at a reasonable price.

Mikko Toivonen and Chang Rajani, doctoral students in computer science, together with Assistant Professor Arto Klami from the University of Helsinki and FCAI, have designed computer vision algorithms that can convert photos taken with a phone into extremely accurate hyperspectral images.

To take hyperspectral images using their solution only requires a camera phone and a peripheral device attached to the phone. Photos are converted by computer vision algorithms into hyperspectral images in a cloud service.

Hy­per­spec­tral im­ages re­veal more

Hyperspectral images are different from regular photographs because they reveal things unseen to the naked eye in the object photographed. The technique is not based on transillumination; rather, hyperspectral images interpret the wavelengths of light more accurately than regular photos.

“Normal photos use three colour channels, such as red, blue and green. In hyperspectral imaging, the light wavelength resolution is finer, comprising a hundred colour channels,” Klami explains.

“A simple three-colour camera is unable to distinguish the spectrum of, for example, chlorophyll. In a hyperspectral image taken outdoors, it’s easier to identify the bits with chlorophyll, that is, the areas with vegetation,” Toivonen says.

Many uses, ex­pens­ive equip­ment

Hyperspectral cameras date back years, and there are many uses for such imaging. The technology is used, among other fields, in geographical remote sensing and estimating yield sizes in agriculture. Hyperspectral images can also be used to identify counterfeit art and pharmaceuticals.

“Fake drugs are a problem, especially in developing nations. By using a mobile spectral imaging device, pharmacies and consumers could take a photo of a drug and check whether the pill corresponds with a reference spectrum supplied by the drug manufacturer,” says Klami.

However, the devices currently available are specialist equipment, with prices starting from several thousand euros. The less expensive technology developed by the University of Helsinki researchers could bring the solution to regular consumers.

The high cost of previous devices is partly because they produce hyperspectral images independently from start to finish. In the researchers’ version, images are converted from regular photographs taken with a smartphone equipped with a peripheral device, and the heavy lifting, or developing the hyperspectral images, is carried out by computer vision algorithms in a cloud service.

“The phone peripheral is cheap and compatible with basically all smartphones. Consumers don't need to buy a separate device,” Klami says.

Above is an RGB photo of avocados created from a hyperspectral image. Below are the spectra of the avocados' surfaces. The colour-coded squares indicate the areas where the spectra have been shot. The avocado on the right is clearly greener than the others, which can be seen as a spike in the blue spectrum curve at 550 nanometres. The spike indicates that the avocado in question is likely to be less ripe than the others.

Above is an RGB photo of avocados created from a hyperspectral image. Below are the spectra of the avocados' surfaces. The colour-coded squares indicate the areas where the spectra have been shot. The avocado on the right is clearly greener than the others, which can be seen as a spike in the blue spectrum curve at 550 nanometres. The spike indicates that the avocado in question is likely to be less ripe than the others.

Aim­ing for the con­sumer mar­ket in 2021

The seed for the idea on consumer market spectral imaging was planted a couple of years ago when Toivonen and Rajani were developing computer vision algorithms for hyperspectral imaging as part of their doctoral studies.

“I noticed that we could also make the algorithms create the image, instead of leaving this cumbersome task to the device taking the pictures,” Toivonen describes.

His long-standing interest in amateur photography helped him to make this observation. Prototypes for the smartphone peripheral have also been created with Toivonen’s personal 3D printer.

The invention is patent pending and next year begins the search for investors. The team’s goal is to make Toivonen’s home printer obsolete by 2021, replacing it with a spin-out company running the development of the peripheral device and mobile application.

Arto Klami, Mikko Toivonen and Chang Rajani. Photo: Susan Heikkinen

Arto Klami, Mikko Toivonen and Chang Rajani. Photo: Susan Heikkinen

Find out more about hyperspectral imaging at the University of Helsinki / Helsinki Innovation Services (HIS) stand at Slush in Helsinki on 21-22 November 2019.

More information:

Arto Klami
Assistant Professor, FCAI & University of Helsinki
arto.klami@helsinki.fi
+358505823654

Mikko Toivonen
Doctoral student, University of Helsinki
mikko.e.toivonen@helsinki.fi
+358443303131

Chang Rajani
Doctoral student, University of Helsinki
chang.rajani@helsinki.fi
+358503115791

When physician and AI work together, the patient benefits

The machine learning method by Iiris Sundin and her colleagues taken into account uncertainties of the world. This is important in decision-making. Photo Matti Ahlgren / Aalto University

The machine learning method by Iiris Sundin and her colleagues taken into account uncertainties of the world. This is important in decision-making. Photo Matti Ahlgren / Aalto University

Doctoral student Iiris Sundin learned in her studies that a machine learning model could make use of a physician's silent knowledge which usually is never written down. This kind of model predicts best how a given patient will react to specific treatment.

Artificial Intelligence opens up new avenues into health care, for example, but its potential is as of now not fully put to use. There are many reasons, but the most important one became clear to Iiris Sundin as she was starting her doctoral studies on machine learning: machine and Man must learn to work together.

"When I acquainted myself with the research of my advisor, Professor Samuel Kaski, on user modeling where the machine tries to understand the human, I realized the huge potential. Cooperation can mean other things than the human operating Excel or passively staring at projections on the screen," says Sundin, a doctoral student at the Department of Computer Science at Aalto University.

Samuel Kaski is the director of FCAI. Sundin's research combines Medicine, technology and many other interests of hers. She has also always wanted to work at something that benefits other people. On the other hand, she was interested in mathematics and programming.

The role of AI and machine learning in health care is researched a lot. Sundin's point of view is unique in having the machine make use of the doctor's knowledge to define the best possible care for the patient.

 

There are uncertainties in the world that must factor in when decisions are taken

Physicians have vast quantities of knowledge that is never written down and that is impossible to feed directly into a learning algorithm. Sundin and her colleagues have found out just how the machine could make use of such knowledge in e.g. figuring out the efficacy of a given type of medication.

Sundin and her colleagues have at their disposal, for example, data on gene specimens collected from cancerous cells, courtesy of FIMM, Institute for Molecular Medicine Finland. Researchers took a look at the mutations in the specimens and tried to predict which cancer medicine would work best for each patient.

We don't just discuss the average person, we acknowledge there are differences, and take it into account when reaching decisions.

It is crucial to remember that there are always uncertainties in the world, and thus, also a physician's knowledge of the effect of different mutations can be uncertain. The machine learning model devised by the research group takes this into account. The results show including expert knowledge in machine learning models and emphasizing data improve prognoses on how a given patient will react to a particular treatment.

So the model developed by researchers depicts realistically how sure people are of the different properties of real matters. "I would that such thinking were more common: we don't just discuss the average person, we acknowledge there are differences, and take it into account when reaching decisions. That way, things can be modeled in a more useful manner."

 

The researcher gets to tell something new about the world

Even though Sundin was fond of mathematics already as a child, and it comes naturally to her to see the world via mathematical thinking, art held a big role in her life, especially when she was young.

As a schoolgirl, she played the piano, sang in a choir, read books and attended art school. Now, she is into yoga, air acrobatics and camping. A researcher can be interested in many things, and creativity, for example, helps at work, too. "When you e.g. make posters for a conference, it is very useful if you can do some of the visual elements yourself."

The antics of engineering students were something Sundin grew up with. Both her parents graduated from the Helsinki University of Technology, and the family always celebrated First of May with parents’ college pals. All the adults sported the traditional tassel caps of engineering students. So engineering studies came naturally. "The degree of an engineer is really well-rounded and offers a good start. You can choose to be a researcher, industrial work or pretty much anything."

I realized researching was such great fun that I rather do more of the same.

Sundin completed her Master’s in Automation and System Technology. Although she never fancied herself a researcher, her interest in it was kindled when she was working on her dissertation. Sundin was modeling the properties of a drop of fluid on different surfaces.

Research offered a way to bring received mathematical ways of thinking and skills into the real world, and to tell something about that world. "I realized it was such great fun that I rather do more of the same."

 

”You succeed at work if you are open”

Sundin says she sometimes imagined researchers worked alone in their chambers. That is no longer the case, anyway. "You succeed better at work is you are open, socially adept and like to travel to conferences and network with the people there."

Research work is social. "You succeed better at work if you are open, socially adept and like to travel to conferences and network with the people there,” says Iiris Sundin. Photo Matti Ahlgren / Aalto University

Research work is social. "You succeed better at work if you are open, socially adept and like to travel to conferences and network with the people there,” says Iiris Sundin. Photo Matti Ahlgren / Aalto University

It is crucial for a doctoral student to possess the wish and motivation to understand things. The point of further studies is to delve into one fairly narrow area of learning very profoundly, something which can be demanding. Therefore, it is important to have tenacity and the ability to stick to it. "It is needed, if you want to make it to the end."

Even though Sundin has hitherto worked in basic research, she feels the methods developed need to be taken to a practical level, once the foundation laid by basic research is robust enough.

In the future, she hopes to apply what she has learned and continue work for the benefit of people and also the environment. "I hope to work at some Finnish institute doing research that help Finland make smarter decisions when it comes to the environment."

Authored by Anu Haapala
English translation by Susanna Bell

Smartphone typing speeds catching up with keyboards

mobile_phone.png

A study of over 37,000 users shows that the ‘typing gap’, the difference typing speeds between mobile devices and physical keyboards is decreasing, and 10–19-year olds can type about ten words-per-minute faster than their parents' generation.

The largest experiment to date on mobile typing sheds new light on average performance of touchscreen typing and factors impacting the text input speed. Researchers from Finnish Centre for Artificial Intelligence (FCAI), University of Cambridge and ETH Zürich analysed the typing speed of tens of thousands of users on both phones and computers. Their main finding is that typing speeds on smartphones are now catching up with physical keyboards.

“We were amazed to see that users typing with two thumbs achieved 38 words per minute on average, which is only about 25% slower than the typing speeds we observed in a similar large-scale study of physical keyboards,” said Anna Feit, a researcher at ETH Zürich and one of the co-authors.

“While one can type much faster on a physical keyboard, up to 100 wpm, the proportion of people who actually reach that is decreasing. Most people achieve between 35-65 WPM.”

The authors call the difference between typing on a keyboard and a smartphone “the typing gap” and predict that as people get less skilled with physical keyboards, and smart methods for keyboards improve further (such as auto-correction and touch models), the gap may be closed at some point. The fastest speed the researchers saw on a touchscreen was a user who managed the remarkable speed of 85 words per minute.

Six hours a day phone time

The research team collected a dataset from over 37,000 volunteers in an online typing test, with the help of the typing speed test service TypingMaster.com. With the consent of the participants, they recorded the keystrokes they made while transcribing a set of given sentences to assess their typing speed, errors and other factors related to their typing behaviour on mobile devices. 

 The dataset is unique in its size and made publicly available. While the majority of volunteers were women in their early twenties and about half of the participants came from U.S., the dataset includes people from all ages and from over 160 countries. On average, the participants reported spending about 6 hours per day on their mobile device.

Anna Feit explains: “Such large amount of experience transfers to the development of typing skill and explains why young people, who spend more time with social media, communicating with each other, are picking up higher speeds.”

One finger, or two thumbs?

The best predictor of performance is whether you use one finger or two thumbs to type. Over 74% of people type with two thumbs, and the speed increase it offers is very large. The study also found that enabling the auto-correct of words offers a clear benefit, whereas word prediction, or manually choosing word suggestions, does not.

As Sunjun Kim, a researcher at Aalto University, explains, “The given understanding is that techniques like word completion help people, but what we found out is that the time spent thinking about the word suggestions often outweighs the time it would take you to type the letters, making you slower overall.” Most users used some type of intelligent support. Only 14% of people typed without auto-correction, word suggestions or gesture typing.

The study also exposed a strong generation effect. Young people, between 10 and 19 years of age are about 10 wpm faster than people in their 40s. Antti Oulasvirta, professor at Aalto University and researcher at FCAI: “We are seeing a young generation that has always used touchscreen devices, and the difference to older generations that may have used devices longer, but different types, is staggering.”

The authors found no benefit from formal training on the ten-finger typing system on physical keyboards. Oulasvirta continues: “This is a type of motor skill that people learn on their own with no formal training, which is very unlike typing on physical keyboards. It is an intriguing question what could be achieved with a careful training program for touchscreens.”

If you want to type faster on mobile, the researchers recommend using two thumbs and enabling auto-correction of words. 

The study will be presented at the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), in Taipei, Taiwan, 2 October 2019. 

More information on the study, including the paper and the dataset: https://userinterfaces.aalto.fi/typing37k/. If you want to try the typing-speed test yourself, you can have a go at http://typingtest.aalto.fi/.

Contact information

Associate Professor Antti Oulasvirta
Aalto University, Finnish Centre for Artificial Intelligence
antti.oulasvirta@aalto.fi 

Dr. Anna Feit
ETH Zurich
anna.feit@inf.ethz.ch

Ten research papers from FCAI accepted to the prestigious NeurIPS conference

Photo Matti Ahlgren / Aalto University

Photo Matti Ahlgren / Aalto University

Research conducted at the Finnish Centre for Artificial Intelligence (FCAI) is well presented at this year’s NeurIPS conference. In total, the prestigious conference has accepted ten submissions from either Aalto University or the University of Helsinki.

“Especially in European terms, we did well this year and the number of publications is within the top 10 of European academic institutions,” says Antti Honkela, Associate Professor of Computer Science at the University of Helsinki.

NeurIPS is the largest and most prestigious conference on machine learning, and it has become increasingly popular in the past years. In 2018, the main conference was sold out in under 12 minutes and therefore this year’s registration was based on a lottery.

“When I visited NeurIPS for the first time 15 years ago, there were 207 publications and less than a thousand participants. This year, the conference lasts for about the same time but there will be more than 1,400 presentations and possibly more than 12,000 participants,” says Honkela.

Over the past 32 years, the NeurIPS conference has been held at various locations around the world. This year’s conference will be held in Canada, at the Vancouver Convention Center.

Honkela and his research group wrote one of the accepted papers, which Honkela sees as a valuable acknowledgement for his group’s hard work. Researchers studied privacy in machine learning, and due to this work, they can assure that no one’s privacy will be violated by using a learning algorithm utilizing, for example, health data. Honkela and his colleagues developed a new version of Markov chain Monte Carlo, one of the most widely used Bayesian algorithms.

Their version of the algorithm assures privacy for a larger variety of models than any previously designed algorithm. “Therefore, this algorithm opens new, important possibilities for statistical inference that aims to secure privacy,” explains Honkela.

Accepted papers from FCAI (Aalto University and the University of Helsinki):

Regularizing Trajectory Optimization with Denoising Autoencoders
Rinu Boney (Aalto University) · Norman Di Palo (Italian Institute of Technology) · Mathias Berglund (Curious AI) · Alexander Ilin (Aalto University) · Juho Kannala (Aalto University) · Antti Rasmus (Curious AI) · Harri Valpola (Curious AI)

Improved Precision and Recall Metric for Assessing Generative Models
Tuomas Kynkäänniemi (Aalto University; NVIDIA) · Tero Karras (NVIDIA) · Samuli Laine (NVIDIA) · Jaakko Lehtinen (Aalto University; NVIDIA) · Timo Aila (NVIDIA)

Differentially Private Markov Chain Monte Carlo
Mikko Heikkilä (University of Helsinki) · Joonas Jälkö (Aalto University) · Onur Dikmen (Halmstad University) · Antti Honkela (University of Helsinki)

On Adversarial Mixup Resynthesis
Christopher Beckham (Mila, Polytechnique Montréal) · Sina Honari (Mila, Polytechnique Montréal) · Alex Lamb (Mila, University of Montreal) · Vikas Verma (Aalto University) · Farnoosh Ghadiri (Mila, Polytechnique Montréal) · R Devon Hjelm (Mila, University of Montreal; Microsoft Research) · Yoshua Bengio (Mila, University of Montreal) · Chris Pal (Mila, Element AI, Polytechnique Montréal)

High-Quality Self-Supervised Deep Image Denoising
Samuli Laine (NVIDIA) · Tero Karras (NVIDIA) · Jaakko Lehtinen (Aalto University; NVIDIA) · Timo Aila (NVIDIA)

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
Wenzheng Chen (University of Toronto) · Huan Ling (University of Toronto; NVIDIA) · Jun Gao (University of Toronto) · Edward Smith (McGill University) · Jaakko Lehtinen (Aalto University; NVIDIA) · Alec Jacobson (University of Toronto) · Sanja Fidler (University of Toronto)

Machine Teaching of Active Sequential Learners
Tomi Peltola (Aalto University) · Mustafa Mert Çelikok (Aalto University) · Pedram Daee (Aalto University) · Samuel Kaski (Aalto University)

ODE2VAE: Deep generative second order ODEs with Bayesian neural networks
Cagatay Yildiz (Aalto University) · Markus Heinonen (Aalto University) · Harri Lähdesmäki (Aalto University)

Identifying Causal Effects via Context-specific Independence Relations
Santtu Tikka (University of Jyväskylä) · Antti Hyttinen (University of Helsinki) · Juha Karvanen (University of Jyväskylä)

Variational Bayesian Decision-making for Continuous Utilities
Tomasz Kuśmierczyk (University of Helsinki) · Joseph Sakaya (University of Helsinki) · Arto Klami (University of Helsinki)

The full list of accepted papers is available on the NeurIPS website.

University of Helsinki announces AI-themed PhD positions – apply in September

University of Helsinki, one of the institutions behind FCAI, announces AI-themed doctoral candidate positions. The application period opens on Tuesday 3 September and ends on Tuesday 17 September.

Find more information about the positions, eligibility criteria and application process on the University of Helsinki website.