AI Day 2020

Session 1a: Language and explainability in AI (13:10-14:10)

13:10 - Multilingual Dynamic Topic Model - Elaine Zosa (University of Helsinki); Mark Granroth-Wilding (University of Helsinki) [click for abstract]

Dynamic topic models capture the evolution of topics and trends in textual time series data. Current DTMs are applicable only to monolingual datasets. In this paper we present the multilingual dynamic topic model (ML-DTM), a novel topic model that combines DTM with an existing multilingual topic modeling method to capture crosslingual topics that evolve across time. We present results of this model on a parallel German-English corpus of news articles and a comparable corpus of Finnish and Swedish news articles. We demonstrate the capability of ML-DTM to track significant events related to a topic and show that it finds distinct topics and performs as well as existing multilingual topic models in aligning cross-lingual topics.

[paper]

13:22 - Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation - Alessandro Raganato (University of Helsinki); Yves Scherrer (University of Helsinki); Jörg Tiedemann (University of Helsinki) [click for abstract]

Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input. However, recent works have shown that attention heads learn simple positional patterns which are often redundant. In this paper, we propose to replace all but one attention head of each encoder layer with fixed--non-learnable--attentive patterns that are solely based on position and do not require any external knowledge. Our experiments show that fixing the attention heads on the encoder side of the Transformer at training time does not impact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios. This work is part of the FoTran project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 771113).

[paper]

13:34 - BiographySampo: Artificial Intelligence Reading Biographies for the Semantic Web – Eero Hyvonen (Aalto University and University of Helsinki) [click for abstract]

This presentation discusses a shift of focus in research on Cultural Heritage semantic portals, based on Linked Data, and envisions and proposes new directions of research. Three generations of portals are identified in addition to publishing contents as printed texts: Ten years ago the research focus in semantic portal development was on data harmonization, aggregation, search, and browsing (“first generation systems”). At the moment, the rise of Digital Humanities research has started to shift the focus to providing the user with integrated tools for solving research problems in interactive ways (“second generation systems”). This presentation envisions and argues that the next step ahead to “third generation systems” is based on Artificial Intelligence: future portals not only provide tools for the human to solve problems but are used for finding research problems in the first place, for addressing them, and even for solving them automatically under the constraints set by the human researcher. Such systems should preferably be able to explain their reasoning, which is an important aspect in the source critical humanities research tradition. The second and third generation systems set new challenges for both computer scientists and humanities researchers.

As an example of this paradigm shift, the semantic portal "BiographySampo - Finnish biographies on the Semantic Web" is presented. The portal is in use and has had some 40 000 users. The system is explained in more detail with scientific publications about it on the homepages of the BiographySampo project: https://seco.cs.aalto.fi/projects/biografiasampo/

BiographySampo is a member "Sampo" portals that have had millions of users on the Web, based on the Sampo model: https://seco.cs.aalto.fi/applications/sampo/

[video] [link] [link]

13:46 - SRL-ESA-TextSum: A text summarization approach based on semantic role labeling and explicit semantic analysis – Mourad Oussalah (University of Oulu) [click for abstract]

Automatic text summarization attempts to provide an effective solution to today’s unprecedented growth of textual data. This paper proposes an innovative graph-based text summarization framework for generic single and multi document summarization. The summarizer benefits from two well-established text semantic representation techniques; Semantic Role Labelling (SRL) and Explicit Semantic Analysis (ESA) as well as the constantly evolving collective human knowledge in Wikipedia. The SRL is used to achieve sentence semantic parsing whose word tokens are represented as a vector of weighted Wikipedia concepts using ESA method. The essence of the developed framework is to construct a unique concept graph representation underpinned by semantic role-based multi-node (under sentence level) vertices for summarization. We have empirically evaluated the summarization system using the standard publicly available dataset from Document Understanding Conference 2002 (DUC 2002). Experimental results indicate that the proposed summarizer outperforms all state-of-the-art related comparators in the single document summarization based on the ROUGE-1 and ROUGE-2 measures, while also ranking second in the ROUGE-1 and ROUGE-SU4 scores for the multi-document summarization. On the other hand, the testing also demonstrates the scalability of the system, i.e., varying the evaluation data size is shown to have little impact on the summarizer performance, particularly for the single document summarization task. In a nutshell, the findings demonstrate the power of the role-based and vectorial semantic representation when combined with the crowd-sourced knowledge base in Wikipedia.

[paper]

13:58 - Personalized Explanations for Machine Learning – Alexander Jung (Aalto University) [click for abstract]

Automated decision making is used routinely throughout our everyday life. Recommender systems decide which jobs, movies, or other user profiles might be interesting to us. Spell checkers help us to make good use of language. Fraud detection systems decide if a credit card transactions should be verified more closely. Many of these decision making systems use machine learning methods that fit complex models to massive datasets. The successful deployment of machine learning (ML) methods to many (critical) application domains crucially depends on its explainability. Indeed, humans have a strong desire to get explanations that resolve the uncertainty about experienced phenomena like the predictions and decisions obtained from ML methods. Explainable ML is challenging since explanations must be tailored (personalized) to individual users with varying backgrounds. Some users might have received university-level education in ML, while other users might have no formal training in linear algebra. Linear regression with few features might be perfectly interpretable for the first group but might be considered a black-box by the latter. We propose a simple probabilistic model for the predictions and user knowledge. This model allows to study explainable ML using information theory. Explaining is here considered as the task of reducing the "surprise" incurred by a prediction. We quantify the effect of an explanation by the conditional mutual information between the explanation and prediction, given the user background.

[paper] [video]

Session 1b: Computer Vision (13:10-14:10)

13:10 - Encoding Temporal Information for Automatic Depression Recognition from Facial Analysis – Wheidima C Melo (University of Oulu); Eric Granger (ETS Montreal); Miguel Bordallo Lopez (VTT Technical Research Centre of Finland) [click for abstract]

Depression is a mental illness that may be harmful to an individual's health. Using deep learning models to recognize the facial expressions of individuals captured in videos has shown promising results for automatic depression detection. Typically, depression levels are recognized using 2D-Convolutional Neural Networks (CNNs) that are trained to extract static features from video frames, which impairs the capture of dynamic spatio-temporal relations. As an alternative, 3D-CNNs may be employed to extract spatiotemporal features from short video clips, although the risk of overfitting increases due to the limited availability of labeled depression video data. To address these issues, we propose a novel temporal pooling method to capture and encode the spatio-temporal dynamic of video clips into an image map. This approach allows fine-tuning a pre-trained 2D CNN to model facial variations, and thereby improving the training process and model accuracy. Our proposed method is based on two-stream model that performs late fusion of appearance and dynamic information. Extensive experiments on two benchmark AVEC datasets indicate that the proposed method is efficient and outperforms the state-of-the-art schemes.

[paper]

13:22 - Can You Trust Your Pose? Confidence Estimation in Visual Localization – Luca Ferranti (University of Vaasa); Xiaotian Li (Aalto University); Jani Boutellier (University of Vaasa); Juho Kannala (Aalto University, Finland) [click for abstract]

Camera pose estimation in large-scale environments is still an open question and, despite recent promising results, it may still fail in some situations. The research so far has focused on improving subcomponents of estimation pipelines, to achieve more accurate poses. However, there is no guarantee for the result to be correct, even though the correctness of pose estimation is critically important in several visual localization applications, such as in autonomous navigation. In this paper we bring to attention a novel research question, pose confidence estimation, where we aim at quantifying how reliable the visually estimated pose is. We develop a novel confidence measure to fulfil this task and show that it can be flexibly applied to different datasets, indoor or outdoor, and for various visual localization pipelines. We also show that the proposed techniques can be used to accomplish a secondary goal: improving the accuracy of existing pose estimation pipelines. Finally, the proposed approach is computationally light-weight and adds only a negligible increase to the computational effort of pose estimation.

[paper]

13:34 - Temporal Hierarchical Dictionary Guided Decoding for Online Gesture Segmentation and Recognition – Haoyu Chen (University of Oulu); Xin Liu (University of Oulu); Jingang Shi (University of Oulu); Guoying Zhao (University of Oulu) [click for abstract]

Online segmentation and recognition for skeleton-based human gestures are challenging. Compared with the offline case, the inference of the online setting can only rely on the current few frames and always completes before the whole temporal movements are performed. However, incompletely performed gestures are quite ambiguous and the early recognition of gestures is easy to fall into a local optimum. In this work, we address the problem with a temporal hierarchical dictionary to guide the hidden Markov model (HMM) decoding procedure. The intuition of this work is that, gestures are ambiguous with high uncertainty at the early performing phases, and they only become discriminate after certain phases being performed. This uncertainty naturally can be measured by entropy. Thus, we propose a measurement called ''relative entropy map'' (REM) to encode this temporal context into a dictionary to guide the HMM decoding. Furthermore, we introduce a progressive learning strategy with which deep neural networks could learn a robust recognition of the HMM states in an iterative manner. The performance of our method is intensively evaluated on three challenging databases and achieves state-of-the-art results. Our method shows the abilities of both extracting the discriminate connotations of gestures and reducing large redundancy in the HMM transition process. It is verified that, our proposed framework can be implemented to continuous gesture streams and achieve online gesture recognition even when they are halfway performed.

[paper]

13:46 - Snapshot Hyperspectral Imaging Using Wide Dilation Networks – Mikko Toivonen (University of Helsinki); Chan Rajani (University of Helsinki); Arto Klami (University of Helsinki) [click for abstract]

Hyperspectral (HS) cameras record the spectrum at multiple wavelengths for each pixel in an image, and are used, e.g., for quality control and agricultural remote sensing. We introduce a fast, cost efficient and mobile method of taking HS images using a regular digital camera equipped with a passive diffraction grating filter, using machine learning for constructing the HS image. The grating distorts the image by effectively mapping the spectral information into spatial dislocations, which we convert into a HS image by a convolutional neural network utilising novel wide dilation convolutions that accurately model optical properties of diffraction. We demonstrate high quality HS reconstruction using a model trained on only 271 pairs of diffraction grating and ground truth HS images.

13:58 - Exploring human-nature interactions in national parks using social media photographs and computer vision – Tuomas Väisänen (University of Helsinki); Vuokko Heikinheimo (University of Helsinki); Tuomo Hiippala (University of Helsinki); Tuuli Toivonen (University of Helsinki) [click for abstract]

Understanding the activities and preferences of visitors is crucial for managing protected areas and planning conservation strategies. Digital conservation and conservation culturomics promote the use of user-generated content in conservation science. Geotagged social media content is a unique source of in-situ information on human presence and activities in nature. Photographs posted on social media platforms are a promising source of information but analyzing large volumes of photographs manually remains laborious. In this article, we demonstrate the application of state-of-the-art computer vision methods to studying human-nature interactions. We use three methods, namely semantic clustering, scene classification and object detection, to automatically analyze photographs taken in Finnish national parks by domestic and international visitors. Our results show that (1) the application of computer vision methods to user-generated photographs provides meaningful information on human-nature interactions, and different methods complement each other; (2) geotagged photographs reveal distinct regional profiles for national parks; and (3) photographic content varies between domestic and international visitors, which indicates differences in activities and preferences. Information extracted automatically from photographs can help to identify preferences among diverse visitor groups, to create profiles of national parks for conservation marketing and to support conservation strategies that rely on public acceptance. We conclude that the application of computer vision methods to automatic content analysis of photographs should be explored further in digital conservation and conservation culturomics, particularly in combination with rich metadata available on social media platforms.

This research is part of the project Social Media Data for Conservation Science funded by the Kone Foundation.

SESSION 1C: MACHINE LEARNING AND APPLICATIONS (13:10-14:10)

13:10 - Crop loss identification at field parcel scale using satellite remote sensing and machine learning – Samantha Wittke (Aalto University); Santosh Hiremath (Aalto University); Taru Palosuo (Natural Resources Institute Finland); Jere Kaivosoja (Natural Resources Institute Finland); Fulu Tao (Natural Resources Institute Finland); Eetu Puttonen (Finnish Geospatial Research Institute); Maximilian Proll (Aalto University); Pirjo Peltonen-Sainio (Natural Resources Institute Finland); Pekka Marttinen (Aalto University); Hiroshi Mamitsuka (Aalto University) [click for abstract]

Identifying crop loss at field parcel scale using satellite images is challenging: first, crop loss is caused by many factors during the growing season; second, reliable reference data about crop loss are lacking; third, there are many ways to define crop loss. This study investigates the feasibility of using satellite images from Landsat 7 to train machine learning (ML) models to classify agricultural field parcels into those with and without crop loss. The reference data for this study was provided by Finnish Food Authority (FFA) containing crop loss information of approximately 1.4 million field parcels in Finland covering about 3.5 million ha from 2000 to 2015. This reference data was combined with Normalized Difference Vegetation Index (NDVI) derived from Landsat 7 images, in which more than 80% of the possible data are missing. Despite the hard problem with extremely noisy data, among the four ML models we tested, random forest (with mean imputation and missing value indicators) achieved the average AUC (area under the ROC curve) of 0.688 +- 0.059 over all 16 years with the range 0.602 - 0.795, in identifying new crop-loss fields based on reference fields of the same year. The linear regression of AUC results against the missing data ratio further demonstrated the possibility of having exceptionally high AUC of more than 0.9 if the noise amount was reduced to less than 30% in data. To our knowledge, this is the first large scale benchmark study of using machine learning for crop loss classification at field parcel scale. The classification setting and trained models have numerous potential applications, for example, allowing government agencies or insurance companies to verify crop-loss claims by farmers and realize efficient agricultural monitoring.

This work has been supported by the joint AI-CropPro project of Aalto University and the Natural Resources Institute Finland in the AIPSE programme of the Academy of Finland.

13:22 - Deep Learning Method for Mandibular Canal Segmentation in Dental Cone Beam Computed Tomography Volumes – Joel Jaskari (Aalto University); Jaakko Sahlsten (Aalto University); Jorma Järnstedt (Tampere University Hospital); Helena Mehtonen (Tampere University Hospital); Kalle Karhu (Planmeca Oy); Osku Sundqvist (Planmeca Oy); Ari Hietanen (Planmeca Oy); Vesa Varjonen (Planmeca Oy); Vesa Mattila (Planmeca Oy); Kimmo Kaski (Aalto University) [click for abstract]

Accurate localisation of mandibular canals in lower jaws is important in dental implantology, in which the implant position and dimensions are currently determined manually from 3D CT images by medical experts to avoid damaging the mandibular nerve inside the canal. Here we present a deep learning system for automatic localisation of the mandibular canals by applying a fully convolutional neural network segmentation on clinically diverse dataset of 637 cone beam CT volumes, with mandibular canals being coarsely annotated by radiologists, and using a dataset of 15 volumes with accurate voxel-level mandibular canal annotations for model evaluation. We show that our deep learning model, trained on the coarsely annotated volumes, localises mandibular canals of the voxel-level annotated set, highly accurately with the mean curve distance and average symmetric surface distance being 0.56 mm and 0.45 mm, respectively. These unparalleled accurate results highlight that deep learning integrated into dental implantology workflow could significantly reduce manual labour in mandibular canal annotations.

[paper]

13:34 - Federated Multi-view Matrix Factorization for Personalized Recommendations – Adrian Flanagan (Huawei Technologies Oy); Were Oyomno (Cloud Service Competence Center of Helsinki, Huawei Technologies Oy (Finland) Co. Ltd); Alexander Grigorievskiy (Cloud Service Competence Center of Helsinki, Huawei Technologies Oy (Finland) Co. Ltd ); Kuan Eeik Tan (Cloud Service Competence Center of Helsinki, Huawei Technologies Oy (Finland) Co. Ltd); Suleiman Ali Khan (Cloud Service Competence Center of Helsinki, Huawei Technologies Oy (Finland) Co. Ltd ); Muhammad Ammad-ud-din (Helsinki Research Center, Cloud Competence Center of Helsinki, Huawei) [click for abstract]

With an increasing focus on user privacy and legislation such as the GDPR, there exists a strong need to develop new approaches that allow training machine learning models on user devices (such as mobiles, tablets or laptops). Federated learning (FL) has emerged as a promising approach to addresses this issue of users' privacy. In FL, the model learning is distributed to the end clients (i.e. user's devices), and model updates are generated locally with the users' data and only the model updates are uploaded and aggregated in a central server ensuring the raw user's private data never leaves the client device.

We introduce the federated multi-view matrix factorization (FED-MVMF) method that learns a multi-view model without transferring the user’s personal data to a central server. The method extends the federated learning framework to matrix factorization with multiple data sources ("views" or "side-information sources"). FED-MVMF performs a federated factorization of the multi-view data (distributed across devices) jointly, to learn the low-dimensional latent factors. The joint federated factorization is formulated with a stochastic gradient descent based approach. The proposed method is presented for the particular case of personalized recommendations, though is applicable to other domains as well. As far as we are aware, this is the first federated model to provide recommendations using multi-view matrix factorization. In addition, it is the first method to provide federated cold-start recommendations. Using a scalable production equivalent FL client-server architecture, the model is rigorously evaluated on three datasets. Empirical validation confirms that federated multi-view matrix factorization outperforms simpler methods that do not take into account the multi-view structure of the data. In addition, we also demonstrate the usefulness of the proposed method for the challenging prediction task of cold-start federated recommendations. We also analyze the impact of federated multi-view model on the communication payloads.

[paper] [slides]

13:46 - Understanding WiFi Cross-technology Interference Detection in the Real World – Teemu Pulkkinen (Ekahau Oy); Jukka Nurminen (University of Helsinki); Petteri Nurmi (University of Helsinki) [click for abstract]

WiFi networks are increasingly subjected to cross-technology interference with emerging IoT and even mobile communication solutions all crowding the 2.4 GHz ISM band where WiFi networks conventionally operate. Due to the diversity of interference sources, maintaining a high level of network performance is becoming increasingly difficult. Recently, deep learning based interference detection has been proposed as a potentially powerful way to identify sources of interference and to provide feedback on how to mitigate their effects. The performance of such approaches has been shown to be impressive in controlled evaluations. However, little information exists on how they generalize to the complexity of everyday environments. In this paper, we contribute by conducting a comprehensive performance evaluation of deep learning based interference detection. In our evaluation, we consider five orthogonal but complementary metrics: correctness, overfitting, robustness, efficiency, and interpretability. Our results show that, while deep learning indeed has excellent correctness (i.e., detection accuracy), it can be prone to noise in measurements (e.g., struggle when transmission power is dynamically adjusted) and suffers from poor interpretability. Deep learning is also highly sensitive to the quality and quantity of training data, with performance decreasing rapidly when the training and testing measurements come from environments with different characteristics. To compensate for weaknesses of deep learning, as our second contribution we propose a novel signal modeling approach for interference detection and compare it against deep learning. Our results demonstrate that, in terms of errors, there are some differences across the two approaches, with signal modeling being better at identifying technologies that rely on frequency hopping or that have dynamic spectrum signatures but suffering in other cases. Based on our results, we draw guidelines for improving interference detection performance.

13:58 - Computing Tight Differential Privacy Guarantees Using FFT – Antti Koskela (University of Helsinki); Joonas Jälkö (Aalto University); Antti Honkela (University of Helsinki) [click for abstract]

Privacy accountants have emerged as an important tool for quantifying the privacy loss of differentially private (DP) machine learning algorithms. We present the Fourier Accountant (FA), which evaluates numerically the privacy loss for compositions of DP mechanisms. As an application we obtain accurate privacy loss bounds for DP algorithms such as the DP-SGD, a differentially private version of the stochastic gradient descent. We provide a rigorous error analysis for the method which gives theoretical accuracy guarantees for the obtained bounds.

FA is based on the privacy loss distribution formalism that has been introduced recently. Mathematically, in this formalism the compositions correspond to convolutions of the privacy loss distributions. These convolutions can be efficiently evaluated using the fast Fourier transform.

Empirically, FA yields significantly tighter privacy loss bounds for the Gaussian mechanism than, e.g., the moments accountant included in the TensorFlow Privacy library. Thus, to obtain a machine learning model with given privacy guarantees, one needs to add less noise in the learning process which further leads to a better utility of the model.

SESSION 2A: PROBABILISTIC AND GENERATIVE MODELS (14:20-15:20)

14:20 - Variational Bayesian Monte Carlo with Noisy Likelihoods – Luigi Acerbi (University of Helsinki) [click for abstract]

Variational Bayesian Monte Carlo (VBMC) is a recently introduced framework that uses Gaussian process surrogates to perform approximate Bayesian inference in models with black-box, non-cheap likelihoods. In this work, we extend VBMC to deal with noisy log-likelihood evaluations, such as those arising from simulation-based models. We introduce new 'global' acquisition functions, such as expected information gain (EIG) and variational interquantile range (VIQR), which are robust to noise and can be efficiently evaluated within the VBMC setting. In a novel, challenging, noisy-inference benchmark comprising of a variety of models with real datasets from computational and cognitive neuroscience, VBMC+VIQR achieves state-of-the-art performance in recovering the ground-truth posteriors and model evidence. In particular, our method vastly outperforms 'local' acquisition functions and other surrogate-based inference methods while keeping a small algorithmic cost. Our benchmark corroborates VBMC as a general-purpose technique for sample-efficient black-box Bayesian inference also with noisy models.

[paper] [code] [video]

14:32 - State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processes – William Wilkinson (Aalto University); Michael R Andersen (Aalto University); Paul Chang (Aalto University); Arno Solin (Aalto University) [click for abstract]

We formulate approximate Bayesian inference in non-conjugate temporal and spatio-temporal Gaussian process models as a simple parameter update rule applied during Kalman smoothing. This viewpoint encompasses most inference schemes, including expectation propagation (EP), the classical (Extended, Unscented, etc.) Kalman smoothers, and variational inference. We provide a unifying perspective on these algorithms, showing how replacing the power EP moment matching step with linearisation recovers the classical smoothers. EP provides some benefits over the traditional methods via introduction of the so-called cavity distribution, and we combine these benefits with the computational efficiency of linearisation, providing extensive empirical analysis demonstrating the efficacy of various algorithms under this unifying framework. We provide a fast implementation of all methods in JAX.

[paper]

14:44 - Deep State-Space Gaussian Processes – Zheng Zhao (Aalto University); Muhammad Emzir (Aalto University); Simo Särkkä (Aalto University) [click for abstract]

This presentation is concerned with a state-space approach to deep Gaussian process (DGP) regression. We construct the DGP by hierarchically putting transformed Gaussian process (GP) priors on the length scales and magnitudes of the next level of Gaussian processes in the hierarchy. The idea of the state-space approach is to represent the DGP as a non-linear hierarchical system of linear stochastic differential equations (SDEs), where each SDE corresponds to a conditional GP. The DGP regression problem then becomes a state estimation problem, and we can estimate the state efficiently with sequential methods by using the Markov property of the state-space DGP. The computational complexity scales linearly with respect to the number of measurements. Based on this, we formulate state-space MAP as well as Bayesian filtering and smoothing solutions to the DGP regression problem. We demonstrate the performance of the proposed models and methods on synthetic non-stationary signals and apply the state-space DGP to detection of the gravitational waves from LIGO measurements.

14:56 - Likelihood-Free Inference with Deep Gaussian Processes – Alexander Aushev (Aalto University); Henri Pesonen (University of Oslo); Markus Heinonen (Aalto University); Jukka Corander (University of Helsinki ); Samuel Kaski (Aalto University and University of Manchester) [click for abstract]

In recent years, surrogate models have been successfully used in likelihood-free inference to decrease the number of simulator evaluations. The current state-of-the-art performance for this task has been achieved by Bayesian Optimization with Gaussian Processes (GPs). While this combination works well for unimodal target distributions, it is restricting the flexibility and applicability of Bayesian Optimization for accelerating likelihood-free inference more generally. We address this problem by proposing a Deep Gaussian Process (DGP) surrogate model that can handle more irregularly behaved target distributions. Our experiments show how DGPs can outperform GPs on objective functions with multimodal distributions and maintain a comparable performance in unimodal cases. This confirms that DGPs as surrogate models can extend the applicability of Bayesian Optimization for likelihood-free inference (BOLFI), while adding computational overhead that remains negligible for computationally intensive simulators.

[paper]

15:08 - Deep Automodulators – Ari Heljakka (Aalto University); Yuxin Hou (Aalto University); Juho Kannala (Aalto University, Finland); Arno Solin (Aalto University) [click for abstract]

We introduce a new category of generative autoencoders called automodulators. These networks can faithfully reproduce individual real-world input images like regular autoencoders, but also generate a fused sample from an arbitrary combination of several such images, allowing instantaneous "style-mixing" and other new applications. An automodulator decouples the data flow of decoder operations from statistical properties thereof and uses the latent vector to modulate the former by the latter, with a principled approach for mutual disentanglement of decoder layers. Prior work has explored similar decoder architecture with GANs, but their focus has been on random sampling. A corresponding autoencoder could operate on real input images. For the first time, we show how to train such a general-purpose model with sharp outputs in high resolution, using novel training techniques, demonstrated on four image data sets. Besides style-mixing, we show state-of-the-art results in autoencoder comparison, and visual image quality nearly indistinguishable from state-of-the-art GANs. We expect the automodulator variants to become a useful building block for image applications and other data domains.

[paper] [video]

SESSION 2B: REASONING (14:20-15:20)

14:20 - Succinct Reductions of Partially Observable Planning to SAT – Saurabh Fadnis (Aalto University); Jussi Rintanen (Aalto University) [click for abstract]

Reduction to SAT is a powerful approach to solving deterministic (classical) planning problems, whereas non-determinism and partial observability had been thought to require far more powerful techniques with computational complexity far above NP. However, Geffner & Geffner (2018) have recently shown that SAT actually does have sufficient power for practically interesting non-deterministic planning and that SAT yields an effective method for solving succinctly represented fully observable (conditional) planning problems. This approach uses is in strong contrast with earlier works on conditional planning, which have required formalisms stronger than SAT and only enumerative non-succinct representations, in which every state is represented explicitly, had been solved with SAT before. Succinct (state-variable based) representations allow representing state spaces that have a size that is exponential in the size of the representation, whereas for enumerative representations the state-space size is linear, limiting the scalability of the search methods.

In our work, we develop a deeper understanding of Geffner & Geffner's ideas on conditional planning with full observability, and show that what we have identified as a core idea in their work can be generalized to still far more complex planning problems and more powerful transition system models such as partially observable planning problems and unobservable problems (known as conformant planning). We show how planning with a complex combination of features, non-determinism and partial observability, can be effectively reduced to the propositional logic, and solved with SAT solvers. With our approach, this type of planning with partial observability can be solved with single calls to a SAT solver that both, find a plan and determine its correctness, showing the significant potential in the ideas first presented by Geffner & Geffner (2018). Earlier works for partially observable planning use separate calls to SAT solvers for finding candidate plans and for determining their validity [Ferraris and Giunchiglia, 2000], or use constraint-satisfaction frameworks outside NP such as QBF [Rintanen, 2007]. Our experiments demonstrate potential of the proposed approach.

14:32 - Path Problems in Temporal Graphs: Algebraic Methods and Applications – Suhas Thejaswi (Aalto University); Aristides Gionis (KTH Royal Institute of Technology) [click for abstract]

The presentation will be based on two papers which are accepted/published and one paper which is currently under review.

In our first work, we study a family of pattern-detection problems in vertex-colored temporal graphs. In particular, given a vertex-colored temporal graph and a multiset of colors as a query, we search for temporal paths in the graph that contain the colors specified in the query. These types of problems have several applications, for example in recommending tours for tourists or detecting abnormal behavior in a network of financial transactions.

In our second work, we study a family of temporal reachability problems under waiting-time restrictions. In particular, given a temporal graph and a set of source vertices, we find the set of vertices that are reachable from a source via a time-respecting path, and such that the difference in timestamps between consecutive edges is at most a resting time. This kind of problems have several interesting applications in understanding the spread of a disease in a network, tracing contacts in epidemic outbreaks, and finding signaling pathways in the brain network.

For the family of problems we consider, we establish complexity results and design an algebraic-algorithmic framework based on constrained multilinear sieving. We demonstrate with an open source implementation that our solution scales to massive graphs with up to tens of millions of edges, despite the problems being NP-hard.

[paper] [paper]

14:44 - Finding Most Compatible Phylogenetic Trees over Multi-State Characters – Tuukka Korhonen (University of Helsinki); Matti Järvisalo (University of Helsinki) [click for abstract]

The reconstruction of the evolutionary tree of a set of species based on qualitative attributes is a central problem in phylogenetics. In the NP-hard perfect phylogeny problem the input is a set of taxa (species) and characters (attributes) on them, and the task is to find an evolutionary tree that describes the evolution of the taxa so that each character state evolves only once. However, in practical situations a perfect phylogeny rarely exists, motivating the maximum compatibility problem of finding the largest subset of characters admitting a perfect phylogeny. Various declarative approaches, based on applying integer programming (IP), answer set programming (ASP) and pseudo-Boolean optimization (PBO) solvers, have been proposed for maximum compatibility. In this work we develop a new hybrid approach to solving maximum compatibility for multi-state characters, making use of both declarative optimization techniques (specifically maximum satisfiability, MaxSAT) and an adaptation of the Bouchitte-Todinca approach to triangulation based graph optimization problems. Empirically our approach outperforms in scalability the earlier proposed approaches w.r.t. various parameters underlying the problem.

[paper] [code]

14:56 - On Robustness in Qualitative Constraint Networks – Michael Sioutis (Bamberg University); Zhiguo Long (Southwest Jiaotong University); Tomi Janhunen (University of Tampere) [click for abstract]

We introduce and study a notion of robustness in Qualitative Constraint Networks (QCNs), which are typically used to represent and reason about abstract spatial and temporal information. In particular, given a QCN, we are interested in obtaining a robust qualitative solution, or, a robust scenario of it, which is a satisfiable scenario that has a higher perturbation tolerance than any other, or, in other words, a satisfiable scenario that has more chances than any other to remain valid after it is altered. This challenging problem requires to consider the entire set of satisfiable scenarios of a QCN, whose size is usually exponential in the number of constraints of that QCN; however, we present a first algorithm that is able to compute a robust scenario of a QCN using linear space in the number of constraints. Preliminary results with a dataset from the job-shop scheduling domain, and a standard one, show the interest of our approach and highlight the fact that not all solutions are created equal.

[paper]

15:08 - Towards Scalable Bayesian Learning of Causal DAGs – Jussi Viinikka (University of Helsinki); Antti Hyttinen (University of Helsinki); Johan Pensar (University of Oslo); Mikko Koivisto (University of Helsinki) [click for abstract]

We give methods for Bayesian inference of directed acyclic graphs, DAGs, and the induced causal effects from passively observed complete data. Our methods build on a recent Markov chain Monte Carlo scheme for learning Bayesian networks, which enables efficient approximate sampling from the graph posterior, provided that each node is assigned a small number K of candidate parents. We present algorithmic tricks to significantly reduce the space and time requirements of the method, making it feasible to use substantially larger values of K. Furthermore, we investigate the problem of selecting the candidate parents per node so as to maximize the covered posterior mass. Finally, we combine our sampling method with a novel Bayesian approach for estimating causal effects in linear Gaussian DAG models. Numerical experiments demonstrate the performance of our methods in detecting ancestor–descendant relations, and in effect estimation our Bayesian method is shown to outperform existing approaches.

[paper]

SESSION 2C: GAMES AND HUMAN ASPECTS (14:20-15:20)

14:20 - Teaching to Learn: Sequential Teaching of Agents with Inner States – Samuel Kaski (Aalto University and University of Manchester); Pierre-Alexandre Murena (Aalto University); Mustafa Mert Celikok (Aalto University) [click for abstract]

The spread of technologies based on machine learning has positioned artificial intelligence at the core of our everyday life. Among other applications, AI assistants are tools that are expected to assist us in our daily tasks. To be more efficient, they should incorporate a model of how the user works and thinks.

As an illustration, in machine teaching, a teacher’s objective is to provide the optimal sequence of inputs to sequential learners in order to guide them towards the best model for their data. We extend this setting from current static one-data-set analyses to learners which change their learning algorithm or latent state to improve during learning, and to generalize to new datasets. We introduce a multi-agent formulation in which learners’ inner state may change with the teaching interaction, which affects the learning performance in future tasks. To teach such learners, we propose an optimal control approach that takes the future performance of the learner after teaching into account. This provides tools for modelling human learners having inner states, and machine teaching of meta-learning algorithms. We demonstrate that a teacher who would like to teach the best model but would not try to trigger transitions in the learner’s inner state might be condemned to a manipulative teaching, which can be done by effectively hiding data. We distinguish this manipulative teaching, which can be used for indoctrination, from more general education which aims to help the learner become better at generalization and learning in new datasets in the absence of a teacher.

We apply these ideas to the simple example of building linear regression models. We show that it is critical for users to understand the notion of collinearity between variables. By inferring this understanding, our teaching assistant displays tutorials when needed and can therefore educate the users to linear regression.

[paper]

14:32 - Building a game AI for Clash Royale – Jarno Seppänen (Supercell) [click for abstract]

Traditionally, commercial game AI bots are developed using rule systems which need lots of manual engineering effort. Whereas games have been widely used as a test domain for conducting machine learning research, relatively little has been said of using machine learning for commercial game development.

We describe how we built a real time game AI for the Clash Royale game, by using deep learning methods. Clash Royale is a mobile game where two players play against each other in real time across the world.

The game has been designed to be easy to learn but hard to master. On the surface, the game seems approachable for machine learning, with a 32 x 18 observation space and a discrete action space. But looking closer, it has rich complexity arising from real time interactions between dozens of game troops. Also the action space is time varying and may contain thousands of legal actions at any given time.

We applied convolutional neural nets in creating a game AI bot that learns to play Clash Royale end-to-end from real replay data. The bot is trained with imitation learning, which is motivated by the intended use for the bot as a human-like training opponent. The trunk of the bot network is straightforward convolutional neural net, but both observation representation and action decoding have been specialized for the Clash Royale game domain.

The machine learned bot was evaluated by game testing it against an earlier rule-based production bot, where it was found to be much better and more human-like. Consequently, the machine learned bot was implemented in C++, and currently it has been running in production for two years.

14:44 - Brainsourcing: Crowdsourcing Recognition Tasks via Collaborative Brain-Computer Interfacing – Keith M Davis (University of Helsinki); Lauri Kangassalo (FMI); Michiel Spape (University of Helsinki); Tuukka Ruotsalo (University of Helsinki) [click for abstract]

This paper introduces brainsourcing: utilizing brain responses of a group of human contributors each performing a recognition task to determine classes of stimuli. We investigate to what extent it is possible to infer reliable class labels using data collected utilizing electroencephalography (EEG) from participants given a set of common stimuli. An experiment (N=30) measuring EEG responses to visual features of faces (gender, hair color, age, smile) revealed an improved F1 score of 0.94 for a crowd of twelve participants compared to an F1 score of 0.67 derived from individual participants and a random chance of 0.50. Our results demonstrate the methodological and pragmatic feasibility of brainsourcing in labeling tasks and opens avenues for more general applications using brain-computer interfacing in a crowdsourced setting. This work is funded in part by the Academy of Finland AIPSE program under grant 313610. Additional funding was also provided by the Academy of Finland, grant numbers 322653 and 328875.

[paper] [video]

14:56 - Predicting Game Difficulty and Churn Without Players – Shaghayegh Roohi (Aalto University); Asko Relas (Rovio Entertainment); Jari Takatalo (Rovio Entertainment); Henri Heiskanen (Rovio); Perttu Hämäläinen (Aalto University) [click for abstract]

We propose a novel simulation model that is able to predict the per-level churn and pass rates of Angry Birds Dream Blast, a popular mobile free-to-play game. Our primary contribution is to combine AI gameplay using Deep Reinforcement Learning (DRL) with a simulation of how the player population evolves over the levels. The AI players predict level difficulty, which is used to drive a player population model with simulated skill, persistence, and boredom. This allows us to model, e.g., how less persistent and skilled players are more sensitive to high difficulty, and how such players churn early, which makes the player population and the relation between difficulty and churn evolve level by level. Our work demonstrates that player behavior predictions produced by DRL gameplay can be significantly improved by even a very simple population-level simulation of individual player differences, without requiring costly retraining of agents or collecting new DRL gameplay data for each simulated player.

[paper] [video]

15:08 - Learning Task-Agnostic Action Spaces for Movement Optimization – Amin Babadi (Aalto University); Michiel van de Panne (University of British Columbia); Karen Liu (Stanford); Perttu Hämäläinen (Aalto University) [click for abstract]

We propose a novel method for exploring the dynamics of physically based animated characters, and learning a task-agnostic action space that makes movement optimization easier. Like several previous papers, we parameterize actions as target states, and learn a short-horizon goal-conditioned low-level control policy that drives the agent’s state towards the targets. Our novel contribution is that with our exploration data, we are able to learn the low-level policy in a generic manner and without any reference movement data. Trained once for each agent or simulation environment, the policy improves the efficiency of optimizing both trajectories and high-level policies across multiple tasks and optimization algorithms. We also contribute novel visualizations that show how using target states as actions makes optimized trajectories more robust to disturbances; this manifests as wider optima that are easy to find. Due to its simplicity and generality, our proposed approach should provide a building block that can improve a large variety of movement optimization methods and applications.

[paper] [video]

Interactive Scientific Exhibition

Computer Vision

Movement Tracking by Optical Flow Assisted Inertial Navigation - Lassi Meronen (Aalto University); William Wilkinson (Aalto University); Arno Solin (Aalto University) [click for abstract]

Robust and accurate six degree-of-freedom tracking on portable devices remains a challenging problem, especially on small hand-held devices such as smartphones. For improved robustness and accuracy, complementary movement information from an IMU and a camera is often fused. Conventional visual-inertial methods fuse information from IMUs with a sparse cloud of feature points tracked by the device camera. We consider a visually dense approach, where the IMU data is fused with the dense optical flow field estimated from the camera data. Learning-based methods applied to the full image frames can leverage visual cues and global consistency of the flow field to improve the flow estimates. We show how a learning-based optical flow model can be combined with conventional inertial navigation, and how ideas from probabilistic deep learning can aid the robustness of the measurement updates. The practical applicability is demonstrated on real-world data acquired by an iPad in a challenging low-texture environment.

[paper] [paper] [video]

Automatic assessment of morphological changes in knee cartilage using deep learning - Egor Panfilov (University of Oulu); Aleksei Tiulpin (University of Oulu); Simo Saarakkala (University of Oulu, Finland); Miika T Nieminen (University of Oulu); Victor Casula (University of Oulu) [click for abstract]

Purpose: Morphological changes in knee cartilage sub-regions are valuable imaging-based biomarkers for understanding progression of osteoarthritis, and they are typically detected from MRI. So far, accurate segmentation of cartilage has been done manually. Deep learning approaches show high promise in automating the task, however, they lack clinically relevant evaluation. We introduce a fully automatic method for segmentation and sub-regional assessment of articular cartilage, and evaluate its predictive power in context of radiographic osteoarthritis progression.

Methods: Two datasets of 3D DESS MRI derived from the Osteoarthritis Initiative were used: first (n=88), second (n=600, 0-/24-month visits). Our method performed DL-based segmentation of knee cartilage tissues, their sub-regional division via multi-atlas-registration, and extraction of sub-regional volume and thickness. The segmentation model was developed and assessed on the first dataset. Subsequently, on the second dataset, the morphological measurements from our and the prior methods were analyzed in correlation and agreement, and, eventually, by their discriminative power of radiographic osteoarthritis progression over 24 months, retrospectively.

Results: The segmentation model showed very high correlation (r2>0.872) and agreement (mean difference<116mm3) data-preserve-html-node="true" in volumetric measurements with the reference segmentations. Comparison of our and state-of-the-art manual methods yielded r2=0.718-0.946 and mean differences=267-502mm3 for weight-bearing cartilage volume, and r2=0.588-0.926 and mean differences=0.512-1.137mm for sub-regional cartilage thickness.

Conclusions: Segmentation of thin cartilage structures is a challenging task, especially, considering degenerative changes caused by OA. In the scope of radiographic OA progression over 24 months, our method based on deep learning segmentation demonstrated comparable discriminative power as the state-of-the-art semi-automatic system, however, the observed ORs were generally lower. Importantly, the method achieved significance in all the sub-regions where the significant associations were discovered with the semi-automatic method. The results show high promise in developing and applying fully automatic methods for segmentation and morphological assessment of articular cartilage in OA progression studies.

Constraints, Planning and Reasoning

Preprocessing in Incomplete MaxSAT Solving - Jeremias Berg (University of Helsinki); Marcus Leivo (University of Helsinki); Matti Järvisalo (University of Helsinki) [click for abstract]

Motivated by the success of preprocessing in Boolean satisfiability (SAT) solving, development and analysis of preprocessing in maximum satisfiability (MaxSAT)---the optimization extension of SAT---has received noticeable attention recently. The correctness of preprocessing techniques for MaxSAT is standardly established by arguing that optimal solutions are maintained. However, the effects of preprocessing on the relative perceived costs of non-optimal solutions has not been considered, despite the fact that one of the most recent directions in MaxSAT research is developing incomplete solvers, i.e., solvers that are designed to provide good (but not necessarily optimal) solutions fast. In this paper, we bridge this gap by showing that employing central preprocessing techniques misleads MaxSAT solvers in terms of their interpretation of the costs of non-optimal solutions seen during search. This issue impacts both complete and incomplete solvers and the effects can be shown to be present also in practice with different types of MaxSAT solvers. Furthermore, we propose ideas for circumventing these negative effects in the context of stochastic local search algorithms for MaxSAT.

[paper] [video]

Planning as Satisfiability with C-step Plans - Masood Feyzbakhsh Rankooh (Aalto University); Jussi Rintanen (Aalto University) [click for abstract]

Planning as satisfiability (SAT) has been shown to be an efficient approach in AI planning. Typically, in planning as SAT, the plan to be produced, and analogously the propositional formula whose satisfiability is being checked, are assumed to be structured using a number of consecutive steps. There have been a number of semantics for steps introduced in previous SAT-based planners. Most notable examples are semantics for steps with parallel actions such as A-step and E-step, which allow more than one action to be included in each step. Almost all previous major works on SAT-based planning don't allow providing the precondition of an action by another action in the same step. In this work, we tackle the problem of lifting this restriction by introducing semantics for our C-step plans for STRIPS planning problems. We also show that when STRIPS planning problems are considered, any given semantic for steps can be combined with any A-step semantic to produce new kinds of semantics. A-step semantic allows a set of actions to be considered as a step only if they are executable in every possible order. We use this method to introduce C-step semantics for planning problems that allow negative preconditions and conditional effects. Our empirical study on numerous benchmark domains shows that C-step encoding often produces satisfiable formulas with considerably fewer steps, which in turn increases the efficiency of planning.

Smallest Explanations and Diagnoses of Rejection in Abstract Argumentation - Andreas Niskanen (University of Helsinki); Matti Järvisalo (University of Helsinki) [click for abstract]

Deciding acceptance of arguments is a central problem in the realm of abstract argumentation. Beyond mere acceptance status, when an argument is rejected it would be informative to analyze reasons for the rejection. Recently, two complementary notions---explanations and diagnoses---were proposed for capturing underlying reasons for rejection in terms of (small) subsets of arguments or attacks. We provide tight complexity results for deciding and computing argument-based explanations and diagnoses. Computationally, we identify that smallest explanations and diagnoses for argumentation frameworks can be computed as so-called smallest unsatisfiable subsets (SMUSes) and smallest correction sets of propositional formulas. Empirically, we show that SMUS extractors and maximum satisfiability solvers (computing smallest correction sets) offer effective ways of computing smallest explanations and diagnoses.

[code]

Boosting Answer Set Optimization with Weighted Comparator Networks - Tomi Janhunen (Tampere University); Jori Bomanson (Aalto University) [click for abstract]

Answer set programming (ASP) is a paradigm for modeling knowledge intensive domains and solving challenging reasoning problems. In ASP solving, a typical strategy is to preprocess problem instances by rewriting complex rules into simpler ones. Normalization is a rewriting process that removes extended rule types altogether in favor of normal rules. Recently, such techniques led to optimization rewriting in ASP, where the goal is to boost answer set optimization by refactoring the optimization criteria of interest. In this paper, we present a novel, general, and effective technique for optimization rewriting based on comparatornetworks, which are specific kinds of circuits for reordering the elements of vectors. The idea is to connect an ASP encoding of a comparator network to the literals being optimized and to redistribute the weights of these literals over the structure of the network. The encoding captures information about the weight of an answer set in auxiliary atoms in a structured way that is proven to yield exponential improvements during branch-and-bound optimization on an infinite family of example programs. The used comparator network can be tuned freely, e.g., to find the best size for a given benchmark class. Experiments show accelerated optimization performance on several benchmark problems.

[paper]

Bricklayer: Resource Composition on the Spot Market - Walter Wong (University of Helsinki); Lorenzo Corneo (Uppsala University); Aleksandr Zavodovski (University of Helsinki); Pengyuan Zhou (University of Helsinki); Nitinder Mohan (Technical University Munich); Jussi Kangasharju (University of Helsinki) [click for abstract]

AWS offers discounted transient virtual instances as a way to sell unused resources in their data-centers, and users can enjoy up to 90% discount as compared to the regular on-demand pricing. Despite the economic incentives to purchase these transient instances, they do not come with regular availability SLAs, meaning that they can be evicted at any moment. Hence, the user is responsible for managing the instance availability to meet the application requirements. In this paper, we present Bricklayer, a software tool that assists users to better use transient resources in the cloud, reducing costs for the same amount of resources, and increasing the overall instance availability. Bricklayer searches for possible combinations of smaller and cheaper instances to compose the requested amount of resources while deploying them into different spot markets to reduce the risk of eviction. We implemented and evaluated Bricklayer using 3 months of historical data from AWS and found out that it can reduce up 54% of the regular spot price and up to 95% compared to the standard on-demand pricing.

Work funded by Academy of Finland AIPSE program under grant 317086.

[paper]

Specification Languages with Complex Data Types - Mojtaba Elahi (Aalto University); Jussi Rintanen (Aalto University) [click for abstract]

Although with remarkable advances in artificial intelligence planning, the current state of the art domain-independent planners can solve complex problems, there is still a significant gap to deploy those planners in real-world applications. Planning in domains with complex data types for which simple data types such as Booleans and numeric variables are insufficient, is outside of current planning methods and modeling languages. In such domains, planning has been based on ad hoc instance generators, which are conventional software programs that produce planner-friendly input in a lower-level language directly understood by an automated planner.

The need for instance generators negates much of the idea of domain-independent planning; if an instance generator is needed, why not use ad hoc state representation, ad hoc search methods, and ad hoc heuristics to begin with.

To obviate the need for instance generators, we propose modeling language extensions for handling the required forms of complex data, which we need to describe problems with complex data types. More precisely, in our proposed modeling language, we support functions, arrays, tuples, user-defined data structures, existential quantifiers, universal quantifiers, and conditional statements, in addition to the Boolean and the numeric data types. With these features, we can describe problems with complex data types as easily as other conventional imperative programming languages can describe them; therefore, we do not need instance generators anymore.

After introducing those high-level features, we demonstrate the translation of them to logical formulas to show how we can reduce problems with complex data types to satisfiability problems and solve them by SAT solvers.

A modeling language supporting complex data types not only bridges the gap between research and real-world applications but also opens new doors to a more comprehensive understanding of the problem and the relationships between different elements of it, which consequently helps in solving it more efficiently.

HuboOPT: Automatic Query Optimization via Adversarial Training - Qingsong Guo (University of Helsinki); Jiaheng Lu (University of Helsinki) [click for abstract]

The future is autonomous - and this applies to data management as well. Recently, there emerged a great interest in building a self-learning architecture for database systems with deep learning technologies. With the self-learning feature, a database system can be autonomous - it can not only tune its configurations and optimize the query execution, but also synthesize data structures for layouts and indexes. Therefore, human experts, such as system designers and administrators, can be largely removed out from the loop of data management to keep up with the evolution of data and workloads. In this paper, we focus on developing a self-supervised training framework for query optimization.

Query optimization is one of the fundamental problems in data management. It aims at finding an efficient execution plan for a given query which is key to achieving good performance in database systems. There has been extensive work in relational query optimization since the early 70s. Most of them are cost-based approaches, which consists of three essential parts: 1) a search space of query execution plans; 2) a cost model to estimate the cost for each plan; 3) an enumeration algorithm to search for the optimal plan. For a relational database, the search space is determined by the set of algebraic equivalence and the physical operators supported in its optimizer. Cost estimates are heavily dependent upon the optimizer’s estimates for the number of rows (i.e., cardinality) that will result in each step of the plan for complex queries involving many predicates and operations. These estimations rely upon statistics (e.g., selectivity for each predicate) and modeling assumptions for a given database. The core of enumeration is join-ordering and the existing methods are normally based on dynamic programming or time-limited branch and bound search.

The existing approaches have two serious limitations: 1) The statistics and modeling assumptions for a given database may be invalid due to the evolution of data and workloads. In addition, some estimates such as query selectivity are too coarse-grained. For example, the predominant approach for selectivity estimation is to collect single-column summaries (e.g., histograms and sketches) and to combine these models assuming column independence. 2) The optimizer always fails to find the optimal execution plan because of the exponential search space. Despite the progress made over the past decades, query optimization remains one of the most challenging problems because it requires a great deal of hand-tuning for specific workloads and datasets.

The limitations make it a good candidate to be learned. Recently, several work has been done based on this idea. Neo is a learned optimizer that generates efficient query executions plans using deep neural networks. It maintains a tree convolution network as a value network and applies the learning-from-demonstration framework to efficiently learn the optimal query execution plan from the value network. Neo does not explicitly estimate cardinalities or cost but utilizes a value network to make estimations. Neo needs to bootstrap its optimization model from PostgreSQL and the experimental results show Neo can learn a model that offers similar performance to state-of-the-art commercial optimizers. In another learned optimizer Bao, it formalizes query optimization as a multi-armed bandit problem. Bao is trained with reinforcement learning algorithms by using query hints on a case-by-case basis. However, the learned optimizers (Neo and Bao) have shown limited gains due to substantive training overhead, inability to adapt to changes, and poor tail performance. In addition, these optimizers often yield sub-optimal query plans and are extremely hard to maintain and transfer to other databases.

In this paper, we introduce HuboOPT, a learned optimizer trained via a self-supervised adversarial neural network. HuboOPT overcomes the limitations of the existing learned optimizers by following designs: 1) We model query optimization as two-player min-max games between a query execution plan generator G and a discriminator D for determining whether a plan generated by G is optimal or not. 2) To precisely estimate the costs of execution plans, we use a deep likelihood model to learn accurate joint distributions for selectivity estimations instead of the single-column statistics. 3) In order to enhance the search capability, we implement the generator G as a cascaded series of improved policy guided by MCTS (Monte Carlo Tree Search). In addition, the existing optimization methods and off-the-shelf heuristics are applied to avoid training the optimizer from scratch. Therefore, we can bootstrap our optimization model from any existing optimizer (e.g., PostgreSQL’s optimizer) and thus tremendously save the training overhead. The initial experiment results show that HuboOPT has significant improvement on both the query performance and training overhead.

Human Aspects in AI

Trustworthy AI services in the public sector: what are citizens saying about it? - Karolina M Drobotowicz (Aalto University); Marjo Kauppinen (Aalto University) [click for abstract]

Artificial Intelligence (AI) brings many opportunities for public institutions, such as more flexible services. However, existing examples show that the opaque use of AI in the public sector can reduce citizens' trust. Multiple experts share guidelines on what trustworthy AI means. There is, however, little of citizens’ voice in this discussion. The aim of this study was to identify what kind of requirements citizens have for trustworthy AI services in the public sector. We conducted a qualitative study by interviewing 21 Finnish residents and by organising a design workshop of four public AI services. The main finding of this study was that all the participants wanted public AI services to be transparent. This transparency requirement covers a number of questions that trustworthy AI services should provide answers. The participants wanted to know what the purpose of the AI service is and what benefits it brings. They also had questions about the data used in the AI service. For example, the participants asked how the data is collected, from which sources it is collected and who has the access to the data. The transparency requirement also covers several questions about the AI process. The participants wanted AI services to explain how the results are created, what data is used in the AI process and what factors affect the final results. They also pointed out that the AI service must give explanations that citizens can understand easily. Moreover, the participants requested to know what tasks are done by AI and what by humans. They also showed a strong need to keep human staff involved in public AI services and wanted to know who is accountable for the service and its results. Based on these findings, we suggest that transparency is a particularly important requirement of public AI services.

AI for all: A Practical Discussion on AI and Accessibility - Tero Avellan (Tampere University); Biju Thankachan (Tampere University); Sumita Sharma (University of Oulu) [click for abstract]

Artificial Intelligence (AI) is the buzz word these days, and paeans have been written in academic literature and folklore portraying it as the “manna” that humankind had been waiting for. In practice, technologies such as predictive text, automatic language translation, visual and speech recognition are already showing the potential of AI for helping people with disabilities. In general, accessibility means that technology is designed to meet the needs of different users. Notwithstanding the tall claims and the supported benefits of AI-based systems, AI is found to be lacking in many aspects such as fairness, accountability and, transparency. The opaque nature of AI systems presents added challenges in aspects of accessibility. Users are often receiving the unintended consequences of an inadequate and erroneous AI model. There is also a valid concern over its diversity, inclusiveness, and accessibility in a social context. AI systems are driven by pattern recognition and classification, which generally lead to often marginalized social processes. Choices based on AI should be ethically sustainable and promote accessibility. The algorithms, or technology behind AI, should represent multiple perspectives on the same topics. The data used to train the algorithms should also represent different user groups. From the design point of AI, disability can be considered a discriminatory feature. The discussion should be more about adaptability and “abilities” rather than “disabilities”. For the individual, one of the biggest practical challenges is the legal environment related to security and privacy and how this affects accessibility. Over time, technologies and the global context of privacy have changed. Accessibility options need to be available, but their use needs to be kept private, which is a challenge for both design and use. Ultimately, people should be able to trust AI. This presentation attempts to raise some discussions on AI in the context of accessibility.

[paper]

Designing AI that Respects Human Autonomy and Sustainability - Kaisa Väänänen (Tampere University); Saara Ala-Luopa (Tampere University); Anu Lehtiö (Tampere University); Thomas Olsson (Tampere University); Maria Hartikainen (Tampere University); Otto Sahlgren (Tampere University) [click for abstract]

While technical advancement is central to the current third wave of AI, focus on human-centered AI (HCAI) is needed to make AI applications acceptable to users and other stakeholders. Human autonomy, defined as “an individual’s capacity for self-determination or self-governance”, is a central human value. While AI’s autonomic nature is a technological advantage, it may also lead to violation of human autonomy. The far-reaching influences of AI systems call for new sensitivities and perspectives in the practice of HCI design. In addition to people’s individual needs, sustainability goals should be considered in all development efforts. Environmental and social sustainability should be in the core of developers’ responsibility.

We propose to approach respecting human autonomy and sustainability on two levels: First, the overall AI design approach and second, on the more detailed application design level. For HCAI, the basic methods of HCD need to be adapted, for example by extensive data-based testing to catch algorithmic biases, or by evaluate the long-term effects of the evolving AI functionality. Value-sensitive design builds goes deep into understanding people’s values to guide design. Sustainable design, speculative design, socially responsible design and planet-centered design can complement the conventional HCD approach, in order to achieve solutions that are sustainable.

On the more detailed application design level, the central target of human-centered design is to design applications that are effective and satisfying to use. Since AI makes actions on behalf of people, user control is a very central aspect to consider in HCAI. Further central considerations are appropriate task allocation to support purposeful human-AI collaboration and respecting privacy of various stakeholders. To achieve explainability, central design targets are informativeness, interactivity, interpretability and comprehensibility.

We argue that future AI developers need competences that allow them to holistically reflect on the values underlying proposed solutions, to elicit discussion with the help of early prototypes, and to conceive artefacts that are capable of communicating ideas with psychological and ideological weight. This will enable developers to envision alternative socio-technical futures where AI technology can impact different behavioral, cultural and societal dynamics, and being mindful of how we design things.

Human-Computer Interaction

Brain Relevance Feedback for Interactive Image Generation - Carlos de la Torre-Ortiz (University of Helsinki); Michiel Spape (University of Helsinki); Lauri Kangassalo (FMI); Tuukka Ruotsalo (University of Helsinki) [click for abstract]

Brain-computer interfaces (BCIs) are increasingly used to perform simple operations such as a moving a cursor, but have remained of limited use for more complex tasks. In our new approach to BCI, we use brain relevance feedback to control a generative adversarial network (GAN). We obtained EEG data from 31 participants who viewed face images while concentrating on particular facial features. Following, an EEG relevance classifier was trained and propagated as feedback on the latent image representation provided by the GAN. Estimates for individual vectors matching the relevant criteria were iteratively updated to optimize an image generation process towards mental targets. A double-blind evaluation showed high performance (86.26% accuracy) against random feedback (18.71%), and not significantly lower than explicit feedback (93.30%). Furthermore, we show the feasibility of the method with simultaneous task targets demonstrating BCI operation beyond individual task constraints. Thus, brain relevance feedback can validly control a generative model, overcoming a critical limitation of current BCI approaches.

[paper]

Mid-Air Gesture Recognition from Point Clouds - Dariush Salami (Aalto University); Sameera Palipana (Aalto University); Luis Leiva (Aalto University); Manila Kodali (Aalto University); Stephan Sigg (Aalto University) [click for abstract]

Motion gesture recognition is the heart of the human-computer interaction world. Human-robot interactions, human-vehicle interactions, smart homes, and touchless screens in shopping malls and restaurants are a few applications in which mid-air gestures play a crucial role. There are several approaches in the literature, including RGB cameras, depth sensors, WiFi, Leap Motion, and Google Soli. Vision-based systems such as Kinect provide 2D color frames, full-body 3D skeleton, and 3D point clouds. However, they suffer from occlusion and darkness issues, and often they raise privacy concerns.

We proposed two systems in this direction: Pantomime and PointGest. Pantomime is a system based on an FMCW radar designed for gesture recognition from highly sparse moving point clouds. However, we introduced PointGest for gesture recognition from highly dense point clouds generated from either depth videos or sensors like Kinect. The details of each of these two systems are discussed in what follows.

In the first work submitted to the IMWUT, we introduce an FMCW radar-based gesture recognition system called Pantomime. Pantomime is positioned in a unique region of the RF technology landscape as a miniaturized medium-resolution medium-range high-frequency RF sensing approach, which is privacy-aware and robust to occlusion, weather conditions, and visibility issues. High operating RF frequencies allow for smaller radar sensors so that they can be placed everywhere. Unlike Soli, which operates in short ranges, Pantomime is ideal for sensing full-body motion gestures. Further, we argue that a medium-resolution device provides the right balance for device performance: If the carrier frequency is too low, it is impossible to recognize fine-grained gestures. However, if it is too high, the signal attenuation increases, and occlusion problems may arise.

We configure a commercial FMCW radar device to promote spatial information over the temporal resolution utilizing sparse 3D point clouds and contribute a deep learning architecture that directly consumes the point cloud, enabling real-time performance with low computational demands. Pantomime achieves 95% accuracy and 99% AUC in a challenging dataset outperforming four state-of-the-art 3D point cloud recognizers.

To evaluate the performance of the system, we recorded more than 10,000 samples from 41 participants in two environments with different speeds, angles toward the radar, and distances. We analyze the effect of environment, articulation speed, angle, and distance to the sensor. We conclude that Pantomime is resilient to various input conditions and may enable novel applications in industrial, vehicular, and smart home scenarios.

In the second work published in MLSP 2020, we proposed a neural network architecture called PointGest for gesture recognition from dense moving point clouds. We demonstrate the feasibility of our approach with point cloud datasets of hand gestures. The architecture, PointGest, directly feeds on un-processed timelines of point cloud data without any need for voxelization or projection. The model is resilient to noise in the input point cloud through abstraction to lower-density representations, especially for high-density regions. We evaluate the architecture on a benchmark dataset with ten gestures. PointGest achieves an accuracy of 98.8%, outperforming five state-of-the-art point cloud classification models.

To evaluate the performance of PointGets, we utilized an existing open-source dataset named UBPG collected using an MS Kinect device. The point cloud is highly dense, with more than 2000 points in each frame. The dataset is collected from 6 participants, and it has nine classes of gesture plus a class named 'no-gesture', which includes recordings of an idle person and random gestures.

We managed to outperform five state-of-the-art models in the literature in five different settings of the UBPG dataset, including two subject-independent settings. In these subject-independent settings, we split the users into two groups for training and testing the model. The aim was to evaluate the proposed method's generalizability on new participants that were unseen to the model. Based on the results, the model is able to classify the gestures from unseen subjects with reasonable accuracy.

Finally, we also showed that PointGest is highly robust to noisy and missing points. We performed an extensive evaluation of PointGest with different numbers of frames and points in each frame. In an extreme case, we showed that by dropping 97% points from the input frames, the accuracy drops only 16%. We also visualized the critical point set of PointGest to show that although we fed frames with 1024 points to the model, it used only a small fraction of those points (200-250 points) for extracting the Spatio-Temporal features.

We appreciate partial funding in the frame of the ChistEra project RadioSense and the MSCA-ITN-ETN Windmill for both of the works.

[slides] [video]

Machine Learning

Recurrent Neural Network Based Simulation of a Single Shaft Gas Turbine - Hamid Asgari (VTT); Emmanuel Ory (VTT); jari lappalainen (Semantum Oy) [click for abstract]

A model of a single shaft gas turbine (GT) is developed by using artificial intelligence (AI). A recurrent neural network (RNN) is employed to train the datasets of the GT variables in Python programming environment by using Pyrenn Toolbox. The resulting model is validated against the Test datasets. Thirteen significant variables of the gas turbine are considered for the modelling process. The results show that the RNN model developed in this study is capable of performance prediction of the system with a high reliability and accuracy. This methodology provides a simple and effective approach in dynamic simulation of gas turbines, especially when real datasets are only available over a limited operational range and using simulated datasets for modelling and simulation purposes is unavoidable.

Federated Learning in Big Data over Networks - Alexander Jung (Aalto University) [click for abstract]

Federated learning is a recent machine learning paradigm for applications involving decentralized collections of local datasets. Federated learning methods train machine learning models collaboratively without exchanging local raw data. These methods are,therefore, attractive for sensitive applications involving confidential data. One main application domain for federated learning is healthcare, where different service providers (hospitals) jointly use their individual confidential patient data to learn improved models. In contrast to existing work on federated learning we develop methods that exploit an intrinsic network structure relating the local datasets. To this end, we introduce networked exponential families as a new probabilistic model for big data over networks. The local datasets are interpreted as realizations of random variables with distributions belonging to some exponential family. The parameters of the distributions can be different for different local datasets. The distributions of the local datasets are coupled by requiring the exponential family parameters of datasets belonging to a well-connected subset (cluster) to be similar. This model allows us to jointly leverage the information contained in the network structure between and statistical properties of local datasets.

[paper] [video]

Stacked Boosters Network Architecture for Short-Term Load Forecasting in Buildings - Jussi Kiljander (VTT); Daniel Pakkala (VTT); Tuukka Salmi (Talenom) [click for abstract]

This paper presents a novel deep learning architecture for short-term load forecasting of building energy loads. The architecture is based on a simple base learner and multiple boosting systems that are modelled as a single deep neural network. The architecture transforms the original multivariate time series into multiple cascading univariate time series. Together with sparse interactions, parameter sharing and equivariant representations, this approach makes it possible to combat against overfitting while still achieving good presentation power with a deep network architecture. The architecture is evaluated in several short-term load forecasting tasks with energy data from an office building in Finland. The proposed architecture outperforms state-of-the-art load forecasting model in all the tasks.

[paper]

Monte Carlo Simulations of Au38(SCH3)24 Nanocluster Using Distance-Based Machine Learning Methods - Antti Pihlajamäki (University of Jyväskylä) [click for abstract]

In this study we present an implementation of distance-based machine learning methods to create atomistic interaction potentials for Au38(SCH3)24 monolayer protected cluster (MPC). MPCs are versatile nanostructures consisting of three distinguishable part: metallic core, protecting ligand layer and interface structure binding ligands to the particle. They have several applications in the fields of biological imaging, catalysis and medicine. Experimentally there have been found two stable structures of Au38(SR)24, which both have been studied extensively with density functional theory level molecular dynamics. We used this data to train our algorithm so that it can predict the potential energy of the cluster solely from structural information. This algorithm can be plugged into a Monte Carlo method to produce dynamic simulations of the cluster using only a fraction of computational resources required by density functional theory. The advantage of the distance-based machine learning methods is not only their computational speed but also the low number of parameters and the fact that they rarely overfit. The study shows that even simple machine learning methods can be used to study complex nanostructures like MPCs.

This work has been supported by the projects 315549 and 315550 in the AIPSE programme of the Academy of Finland.

[paper]

Cyberbullying Detection with Fairness Constraints - Oguzhan Gencoglu (Tampere University) [click for abstract]

Cyberbullying is a widespread adverse phenomenon among online social interactions in today's digital society. While numerous computational studies focus on enhancing the cyberbullying detection performance of machine learning algorithms, proposed models tend to carry and reinforce unintended social biases. In this study, we try to answer the research question of "Can we mitigate the unintended bias of cyberbullying detection models by guiding the model training with fairness constraints?". For this purpose, we propose a model training scheme that can employ fairness constraints and validate our approach with different datasets. We demonstrate that various types of unintended biases can be successfully mitigated without impairing the model quality. We believe our work contributes to the pursuit of unbiased, transparent, and ethical machine learning solutions for cyber-social health.

[paper] [paper]

Differentially private cross-silo federated learning - Mikko A Heikkilä (University of Helsinki); Antti Koskela (University of Helsinki); Kana Shimizu (Waseda University, National Institute of Advanced Industrial Science and Technology); Samuel Kaski (Aalto University and University of Manchester); Antti Honkela (University of Helsinki) [click for abstract]

Strict privacy is of paramount importance in distributed machine learning. Federated learning, with the main idea of communicating only what is needed for learning, has been recently introduced as a general approach for distributed learning to enhance learning and improve security. However, federated learning by itself does not guarantee any privacy for data subjects. To quantify and control how much privacy is compromised in the worst-case, we can use differential privacy.

In this paper we combine additively homomorphic secure summation protocols with differential privacy in the so-called cross-silo federated learning setting. The goal is to learn complex models like neural networks while guaranteeing strict privacy for the individual data subjects. We demonstrate that our proposed solutions give prediction accuracy that is comparable to the non-distributed setting, and are fast enough to enable learning models with millions of parameters in a reasonable time. To enable learning under strict privacy guarantees that need privacy amplification by subsampling, we present a general algorithm for oblivious distributed subsampling. However, we also argue that when malicious parties are present, a simple approach using distributed Poisson subsampling gives better privacy.

Finally, we show that by leveraging random projections we can further scale-up our approach to larger models while suffering only a modest performance loss.

[paper]

Automated Tip Functionalization and Image interpretation with Machine Learning in Atomic Force Microscopy - Benjamin Alldritt (Aalto university); Chen Xu (Aalto university); Prokop Hapala (Czech Academy of Sciences); Ondrej Krejci (Aalto university); Niko Oinonen (Aalto university); Fedor Urtev (Aalto university); Filippo Federici Canova (Aalto university); Juho Kannala (Aalto University, Finland); Peter Liljeroth (Aalto university); Adam Foster (Aalto university) [click for abstract]

Atomic force microscopy (AFM) is an ubiquitous nanoscale characterization technique, where a probe is scanned across a surface of a sample to measure a 3D map of surface roughness at atomic resolutions [1]. It is a powerful tool not only for imaging at atomic scales, but also for manipulation of molecules and assembly of atomic structures at the nanoscale. AFM data interpretation and quantitative analysis for complex mixtures of molecules and bulky 3D molecules can be difficult [2], due to the complex nature of contrast in AFM images, and need significant acceleration and automation to make AFM technique available to a wide range of laboratories and clinics. AFM can achieve atomic resolution with microscope’s tip apex functionalization by a CO molecule. Here, we introduce a machine learning (ML) approach both for the preparation of AFM experiments and for data interpretation in AFM. For the first objective our method involves a convolutional neural network (CNN) that has been trained to analyze the quality of a CO-terminated tip from other CO molecules on a copper surface. For the interpretation of AFM images, we introduce ML image descriptors characterizing the molecular configuration, allowing us to predict the molecular structure directly [3].

[1] L. Gross, F. Mohn, N. Moll, P. Liljeroth, and G. Meyer, “The Chemical Structure of a Molecule Resolved by Atomic Force Microscopy,” Science, vol. 325, no. 5944, pp. 1110–1114, Aug. 2009, doi: 10.1126/science.1176210

[2] O. M. Gordon and P. J. Moriarty, “Machine learning at the (sub)atomic scale: next generation scanning probe microscopy,” Mach. Learn. Sci. Technol., vol. 1, no. 2, p. 023001, May 2020, doi: 10.1088/2632-2153/ab7d2f

[3] B. Alldritt et al., “Automated structure discovery in atomic force microscopy,” Sci. Adv., vol. 6, no. 9, p. eaay6913, Feb. 2020, doi: 10.1126/sciadv.aay6913

[paper]

Software Quality for AI - Valentina Lenarduzzi (LUT University); Francesco Lomio (Tampere University); Sergio Moreschini (Tampere University); Davide Taibi (Tampere University); Damian Tamburri (Jheronimus Academy of Data Science) [click for abstract]

As any software system, AI systems require attention attaining quality assurance, and in particular to their code quality. Conversely, current development processes, and in particular agile models, enable companies to decide the technologies to adopt in their system in a later stage and it becomes hard to anticipate if a system, or if a data pipeline is used for Machine-Learning (ML) produces high-quality models.

The need for considering the quality of AI-enabled systems was highlighted already even more than 30 years ago. For the time being, different approaches have been proposed to evaluate the quality of the AI-models, defining metrics to evaluate their accuracy (e.g. precision, recall, AUC, …).

Conversely, the overall quality of the AI-enabled systems, and in particular the ML code has never been investigated systematically so far if not anecdotally. For example, a report from Informatics Europe (https://www.informatics-europe.org) and the ACM Europe Council (https://europe.acm.org), highlighted the importance of assessing the quality of AI-related code. The EU has also proposed a whitepaper discussing a high-level approach to the regulatory compliance of AI but did not focus on code quality issues at all.

In this presentation, we aim at reporting the results published in our recent paper “Software Quality for AI: Where we are now?” accepted in the next Software Quality Days 2021 conference, and on the further results obtained in the last months in our four research groups

The goal of this presentation is three-folded:

• highlight the quality-related issues of AI software, as well as possible solutions that can be adopted to solve them.
• present a taxonomy of misleading terms adopted both in AI and in software engineering, that contributes even more to increase the communication issues between AI developers and other developers with a background in software engineering.
• share our research road map on software quality for AI with local companies and researchers.

The identification of such quality issues is based on the experience of our four research groups: (1) the Software Engineering group of the LUT University, (2) the Machine Learning and (3) Software Engineering groups of the Tampere University, and the (4) Jheronimus Academy Data and Engineering (JADE) lab of the Jheronimus Academy of Data Science.

The insights in this work will enable researchers to understand possible problems on the quality of AI-enabled systems opening new research topics and allows companies to understand how to better address quality issues in their systems and how to reduce the communication burden between AI developers and software engineers.

The paper will be present to the workshop "Workshop on Quality Assurance for AI (QAAI)" hosted at the Software Quality Days in Vienna (Austria) on January 2021.

Image-to-graph translation of atomic force microscopy images using graph neural networks - Niko Oinonen (Aalto university); Fedor Urtev (Aalto university); Alexander Ilin (Aalto University); Juho Kannala (Aalto University, Finland); Adam Foster (Aalto university) [click for abstract]

The atomic force microscope (AFM) is an important tool in nanoscale science for imaging surfaces and molecules on surfaces. The AFM device measures the force interaction between an extremely sharp probe tip and a sample, such that a scan over the sample produces an image of the sample structure. State-of-the-art AFM setups operating in vacuum at low temperatures are able to resolve features on the scale of individual atoms in molecules. However, the process of interpreting the resulting AFM images often requires a high level of expertise. Furthermore, in the case of more complicated sample structures, the interpretation task can be very challenging or even impossible even for highly trained experts in the field. We are working towards greater interpretability and greater automation of the processing of AFM images using machine learning methods. Initially, we formulated this task as an image-to-image translation problem where the target is a 2D image descriptor that characterises the molecular structure in an easily interpretable image, and we tackled this problem using convolutional neural networks (CNN) [1]. We are currently exploring the possibility of directly predicting the atomic structure of the sample as a graph using graph neural networks (GNN) [2], which shifts the problem to one of image-to-graph translation.

GNNs are a class of deep learning models that operate on graph-structured data. This makes them a good choice for working with molecular structures, which are quite naturally presented as graphs, with atoms as the vertices and bonds as the edges. Previous studies have demonstrated the capability of GNNs in, for example, predicting the results of quantum chemistry calculations for molecules with high accuracy [3]. Our approach builds on the work of Li et al. [4] who demonstrated an iterative GNN model capable of generating realistic graphs by learning the distribution of an existing database of graphs. We propose a similar GNN model, which is additionally conditioned on an AFM image processed into a vector representation by a CNN, in order to iteratively predict the molecular structure in the AFM image. This is still a work-in-progress, but our initial results are showing promise.

[1] B. Alldritt et al. “Automated structure discovery in atomic force microscopy”. Sci. Adv. 6(9), eaay6913, 2020. DOI: 10.1126/sciadv.aay6913, URL: https://advances.sciencemag.org/content/6/9/eaay6913

[2] P. W. Battaglia et al. “Relational inductive biases, deep learning, and graph networks”. arXiv:1806.01261. URL: http://arxiv.org/abs/1806.01261.

[3] J. Gilmer et al. “Neural Message Passing for Quantum Chemistry”. arXiv:1704.01212. URL: http://arxiv.org/abs/1704.01212.

[4] Y. Li et al. “Learning Deep Generative Models of Graphs”. arXiv:1803.03324. URL: http://arxiv.org/abs/1803.03324.

Adversarial sampling in box-constrained settings - Bruno Ordozgoiti (Aalto University); Aristides Gionis (KTH Royal Institute of Technology); Ananth Mahadevan (University of Helsinki) [click for abstract]

In recent years, deep learning methods have proved effective in fields such as computer vision and speech recognition. As a result, commercial applications such as automated image tagging are now possible. However, these methods have been shown to be vulnerable to certain types of attack. For instance, by making a small set of carefully chosen modifications, imperceptible to the human eye, one can ensure that a computer vision system will lose its ability to recognize an object in an image. The resulting image is often referred to as an adversarial example. A recent line of work has shown that this can be accomplished even without access to the classifier's parameters, only by making queries to a black-box model.

In tasks related to perception, such as image classification, an example is considered to be adversarial if it is misclassified but remains perceptually similar to the original one. As this is a subjective assessment, the similarity is usually enforced by putting constraints or penalties on the modification procedure, such as bounds on the l-p norm between the original example and the result.

Here we consider a different scenario, where the validity of the adversarial is not determined by its perceptual qualities, but by a set of box constraints. Suppose we want to attack a network server, which identifies malicious requests by means of a classifier. We rely on feedback from the server, which reveals whether an attack is successful. In order to produce an adversarial example, we must preserve certain qualities of the request so that the attack remains effective. However, some of its characteristics, independent of the malevolent effect, can be freely modified, and thus l-p norm penalized adversarial crafting methods can be ineffective.

In this work we explore sampling techniques for adversarial attacks in this scenario, making certain regularity assumptions about the classifier, with the goal of minimizing the number of queries before crafting a successful attack.

Artificial Intelligence Method for Identifying Adsorbate Structures in Microscopy Images - Jari Järvi (Aalto University); Milica Todorović (Aalto University); Patrick Rinke (Aalto University) [click for abstract]

Atomic force microscopy (AFM) has considerable resolution for imaging and characterization of adsorbed surface nanostructures. However, interpreting images of complex 3-dimensional adsorbates can be difficult and atomistic simulations are often required to provide insight. The most stable simulated structures correspond to the minima of the computed potential energy surface (PES). In the case of complex adsorbates, thorough exploration of the PES with quantum mechanical methods such as density-functional theory (DFT) is prohibitively expensive.

We combine DFT with artificial intelligence (AI) for global atomistic structure search of the most stable adsorbates. Bayesian Optimization Structure Search (BOSS) [1, 2] is a new AI tool, which accelerates the structure search via a strategic sampling of the PES. BOSS computes the complete PES with minimal number of expensive DFT simulations. This allows a clear identification of the most stable minimum energy structures and the barriers between them.

We apply BOSS to study the adsorption of a camphor molecule on the Cu(111) surface as a function of molecular orientation and translations [3]. We identify 8 unique types of stable adsorbates, in which camphor chemisorbs via an oxygen bond (global minimum) or physisorbs via hydrocarbon interactions to the Cu(111) surface. By matching the stable structures to AFM images [4], we conclude that the experiments feature 3 different structures of chemisorbed camphor molecules. This study demonstrates the power of new cross-disciplinary tools in detecting complex interface structures.

Work funded by the Academy of Finland's AIPSE program via the Artificial Intelligence for Microscopic Structure Search (AIMSS) project No. 316601, and by the Emil Aaltonen Foundation.

[1] M. Todorović et al., npj Comput. Mater. 5, 35 (2019). https://doi.org/10.1038/s41524-019-0175-2

[2] https://gitlab.com/cest-group/boss

[3] J. Järvi et al., Beilstein J. Nanotechnol. 11, 1577-1589 (2020). https://doi.org/10.3762/bjnano.11.140

[4] J. Järvi et al., in preparation. https://doi.org/10.21203/rs.3.rs-50783/v1.

[paper] [code]

Sensors and AI Techniques for Situational Awareness in Autonomous Ships: A Review - Zheng Zhao (Aalto University); Simo Särkkä (Aalto University) [click for abstract]

Autonomous ships are expected to improve the level of safety and efficiency in future maritime navigation. Such vessels need perception for two purposes: to perform autonomous situational awareness and to monitor the integrity of the sensor system itself. In order to meet these needs, the perception system must fuse data from novel and traditional perception sensors using Artificial Intelligence (AI) techniques. This presentation overviews the recognized operational requirements that are imposed on regular and autonomous seafaring vessels, and then proceeds to consider suitable sensors and relevant AI techniques for an operational sensor system. The integration of four sensors families is considered: sensors for precise absolute positioning (Global Navigation Satellite System (GNSS) receivers and Inertial Measurement Unit (IMU)), visual sensors (monocular and stereo cameras), audio sensors (microphones), and sensors for remote-sensing (RADAR and LiDAR). Additionally, sources of auxiliary data, such as Automatic Identification System (AIS) and external data archives are discussed. The perception tasks are related to well-defined problems, such as situational abnormality detection, vessel classification, and localization, that are solvable using AI techniques. Machine learning methods, such as deep learning and Gaussian processes, are identified to be especially relevant for these problems. The different sensors and AI techniques are characterized keeping in view the operational requirements, and some example state-of-the-art options are compared based on accuracy, complexity, required resources, compatibility and adaptability to maritime environment, and especially towards practical realization of autonomous systems.

[paper]

Enhancing Industrial X-ray Tomography by Data-Centric Statistical Methods - Jarkko Suuronen (LUT University); Lassi Roininen (LUT University); Simo Särkkä (Aalto University); Sari Lasanen (University of Oulu); Muhammad Emzir (Aalto University) [click for abstract]

We talk about Bayesian methods for the X-ray tomography reconstruction. X-ray tomography has applications in various industrial fields such as sawmill industry, oil and gas industry, as well as chemical, biomedical and geotechnical engineering. In Bayesian methods, the inverse problem of tomographic reconstruction is solved with help of a statistical prior distribution which encodes the possible internal structures by assigning probabilities for smoothness and edge distribution of the object. We compare Gaussian random field priors, that favour smoothness, to non-Gaussian total variation, Besov, and Cauchy priors which promote sharp edges and high-contrast and low-contrast areas in the object. Structures with sharp edges and steep contrast-difference areas are common in industrial tomography, and we show with synthetic examples what kind of artefacts they might induce in reconstruction. We also present computational schemes for solving the resulting high-dimensional Bayesian inverse problem with 100,000-1,000,000 unknowns. That is, we study the applicability of a no-U-turn variant of Hamiltonian Monte Carlo methods and of a more classical adaptive Metropolis-within-Gibbs algorithm to enable full uncertainty quantification of the reconstructions.

We also demonstrate the maximum a posteriori estimates with limited-memory BFGS optimisation algorithm. We show that Cauchy priors produce smaller number of artefacts than other choices, especially with sparse high-noise measurements, and choosing Hamiltonian Monte Carlo enables systematic uncertainty quantification, provided that the posterior is not pathologically multimodal or heavy-tailed. Finally, we briefly present ideas for future work with the subject. Uncertainty quantification could be improved by using an advanced MCMC sampling method like Pseudo-Extended MCMC. Likewise, employing an approximation or parametrization of a general stable distribution as a prior seems noteworthy. The random field priors could be also upgraded with a Bayesian mixture of experts model that divides the area of interest into clusters with their own priors, while simultaneously enabling the uncertainty quantification.

Work has been funded by Academy of Finland (project numbers 326240, 326341, 314474, 321900,313708) and by European Regional Development Fund (ARKS project A74305).

[paper]

Machine Learning Methods for Neonatal Mortality and Morbidity Classification - Joel Jaskari (Aalto University); Janne Myllärinen (Aalto University); Markus Leskinen (Helsinki University Hospital); Ali Bahrami Rad (Aalto University); Jaakko Hollmén (Aalto University); Sture Andersson (Helsinki University Hospital); Simo Särkkä (Aalto University) [click for abstract]

Preterm birth is the leading cause of mortality in children under the age of five. In particular, low birth weight and low gestational age are associated with an increased risk of mortality. Preterm birth also increases the risks of several complications, which can increase the risk of death, or cause long-term morbidities with both individual and societal impacts. In this work, we use machine learning for prediction of neonatal mortality as well as neonatal morbidities of bronchopulmonary dysplasia, necrotizing enterocolitis, and retinopathy of prematurity, among very low birth weight infants. Our predictors include time series data and clinical variables collected at the neonatal intensive care unit of Children's Hospital, Helsinki University Hospital. We examine 9 different classifiers and present our main results in AUROC, similar to our previous studies, and in F1-score, which we propose for classifier selection in this study. We also investigate how the predictive performance of the classifiers evolves as the length of time series is increased, and examine the relative importance of different features using the random forest classifier, which we found to generally perform the best in all tasks. Our systematic study also involves different data preprocessing methods which can be used to improve classifier sensitivities. Our best classifier AUROC is 0.922 in the prediction of mortality, 0.899 in the prediction of bronchopulmonary dysplasia, 0.806 in the prediction of necrotizing enterocolitis, and 0.846 in the prediction of retinopathy of prematurity. Our best classifier F1-score is 0.493 in the prediction of mortality, 0.704 in the prediction of bronchopulmonary dysplasia, 0.215 in the prediction of necrotizing enterocolitis, and 0.368 in the prediction of retinopathy of prematurity.

[paper]

Surrogate modeling of a permanent magnet synchronous machine finite element models based on artificial neural networks - Mikko Tahkola (VTT Oy); Janne Keränen (VTT Oy); Denis Sedov (Aalto University); Mehrnaz Farzam Far (VTT Oy); Juha Kortelainen (VTT Oy) [click for abstract]

Electrical machines (EMs), with their applications ranging from domestic appliances to power plants, are an intrinsic part of our life. Mathematical models are used in, for example, engineering design, control applications, and condition monitoring of EMs. In addition, the evolving concept of digital twins highlights the utilization of these models. Physics-based mathematical simulation, such as finite element method (FEM), lay the foundation for the analysis of EMs. However, the nature of physics-based methods cause them to be either accurate but slow, or inaccurate but fast. For example, the FEM-based analyses of EMs are typically accurate, but tend to be computationally expensive.

Surrogate modeling offers a way to avoid the trade-off between the efficiency and accuracy in simulation. In this study, we present the surrogate modeling approaches used in the EM domain with the focus on ANN-based models. The experimental part focuses on surrogate modeling of torque behavior of a permanent magnet synchronous machine. We have compared the accuracy and efficiency of ANNs and gradient boosting decision trees (GBDTs), finding ANN to be more accurate and faster than GBDT. We also evaluated the effect of different data sampling approaches on the accuracy of the ANN surrogate, where a combination of Latin hypercube sampling and grid sampling provided better results than either of them used alone. We also present a hybrid approach to enhance the accuracy of a data-driven surrogate model. The accuracy of the hybrid-ANN surrogate model was close to the accuracy of FEM. In the efficiency evaluation, the ANNs were several thousand times faster than FEM. The results show the potential of the ANN-based surrogate modeling approach in producing fast but accurate models for applications.

Acknowledgment: The work has been done in the Arrowhead Tools project, funded by European Commission and Business Finland, under grant 826452.

Twinify: A software package for differentially private data release - Joonas Jälkö (Aalto University); Lukas Prediger (Aalto University); Antti Honkela (University of Helsinki); Samuel Kaski (Aalto University and University of Manchester) [click for abstract]

Differential privacy allows quantifying privacy loss from computations on sensitive personal data. This loss grows with the number of accesses to the data, making it hard to open the use of such data while respecting privacy. To avoid this limitation, we propose privacy-preserving release of a synthetic twin of a data set, which can be used for an unlimited number of analyses with any methods without affecting the privacy guarantees. The synthetic data generation is based on differentially private learning of a generative probabilistic model which can capture the probability distribution of the original data. We demonstrate empirically that we can reliably reproduce statistical discoveries from the synthetic data. We expect the method to have broad use in sharing anonymised versions of key data sets for research.

We have implemented the data sharing pipeline into an easy to use software package called twinify (https://github.com/DPBayes/twinify/). Twinify allows a user to build highly expressive probabilistic models tailored for the particular data sharing task in hand. To make it accessible for a broad audience not versed in probabilistic modelling, twinify additionally features an automatic model building capability requiring only minimal guidance from the user. Twinify realises fast inference using CPU/GPU accelerated kernels via the JAX framework.

[code]

Stationary Activations for Uncertainty Calibration in Deep Learning - Lassi Meronen (Aalto University); Christabella Irwanto (Aalto University); Arno Solin (Aalto University) [click for abstract]

We introduce a new family of non-linear neural network activation functions that mimic the properties induced by the widely-used Matérn family of kernels in Gaussian process (GP) models. This class spans a range of locally stationary models of various degrees of mean-square differentiability. We show an explicit link to the corresponding GP models in the case that the network consists of one infinitely wide hidden layer. In the limit of infinite smoothness the Matérn family results in the RBF kernel, and in this case we recover RBF activations. Matérn activation functions result in similar appealing properties to their counterparts in GP models, and we demonstrate that the local stationarity property together with limited mean-square differentiability shows both good performance and uncertainty calibration in Bayesian deep learning tasks. In particular, local stationarity helps calibrate out-of-distribution (OOD) uncertainty. We demonstrate these properties on classification and regression benchmarks and a radar emitter classification task.

[paper]

Privacy-preserving Data Sharing on Vertically Partitioned Data - Razane Tajeddine (University of Helsinki); Antti Honkela (University of Helsinki); Joonas Jälkö (Aalto University); Samuel Kaski (Aalto University and University of Manchester) [click for abstract]

With the increasing use of predictive machine learning, the concern for user data privacy is increasing. Differential privacy (DP) assumes that privacy is preserved if the outcome of an algorithm is barely affected whether a certain individual record is in the dataset or not. DP is widely used for privacy-preserving machine learning because it provides a strong mathematical guarantee and is satisfied by a computationally rich class of algorithms. However, in many cases, the datasets are partitioned between parties. Vertically partitioned data refers to the case where each party holds different features for the same set of individuals. Being able to explore such data without privacy concerns would allow addressing many new interesting questions, e.g. by combining large-scale health and behavioural data. We present a method for differentially private data sharing by training a mixture model on vertically partitioned data.

In this method, we use secure multiparty computation (MPC) to combine the contribution of the data from the parties to train the model. After that, we add noise to guarantee differential privacy. We apply the differentially private variational inference (DPVI) for learning the model. Assuming the mixture components contain no dependencies across different parties, the objective function can be factorized into a sum of products of individual components of each party. Therefore, each party can calculate its shares on its own without the use of MPC. Then MPC is only needed to get the product between the different shares and add the noise. Applying the method to demographic data from the US Census, we obtain comparable accuracy to the non-partitioned case with approximately 20-fold increase in computing time.

[paper]

Model-free prediction of optical supercontinuum generation dynamics - Lauri Salmela (Tampere University); Mathilde Hary (Tampere University); John M. Dudley (Institut FEMTO-ST); Goëry Genty (Tampere University) [click for abstract]

Ultrabroadband supercontinuum (SC) lasers are light sources with unique characteristics which have had a major impact in many research fields ranging from precision metrology, to high resolution imaging [1]. The generation of a supercontinuum is a highly complex process resulting from the propagation of intense and short pulses of light in a nonlinear medium, such as an optical fibre. As a result of a very rich landscape of nonlinear dynamics, narrowband input pulses injected into the fibre experience massive spectral broadening, spanning from the visible to infrared region of the electromagnetic spectrum [1].

The dynamics of SC generation can be accurately modelled by the generalized nonlinear Schrödinger equation (GNLSE) [1]. In the GNLSE model, the propagation of light is represented as a sequence of electric field amplitude distributions along the propagation. Direct integration of the GNLSE, however, can be extremely time-consuming due to a large number of steps. This creates a bottleneck, in particular for the analysis and optimization of optical experiments in ‘real-time’. To overcome these computational limitations, we introduce a “model-free” forecasting method [2] based on a recurrent (LSTM) neural network that can predict the nonlinear propagation dynamics using only the input pulse intensity profiles [3]. The speed of our approach exceeds the conventional methods by a few orders of magnitude. In addition, we show how feed-forward neural networks can be used for predicting the full-field (intensity and phase) propagation dynamics of SC.

Our work is the first to introduce the use of machine learning techniques for predicting highly nonlinear dynamics in optics, and we anticipate that our results will stimulate similar studies in all areas of physics where GNLSE-like dynamics play a governing role. Work funded by Academy of Finland AIPSE program under grant 318082 (SMALL).

[1] J. M. Dudley, G. Genty, and S. Coen, Rev. Mod. Phys. 78, 1135 (2006).

[2] P. Vlachas, et al., Neural Netw. 126, 191–217 (2020).

[3] L. Salmela, et al., arXiv preprint arXiv:2004.14126 (2020).

[paper]

Multi-scale Cloud Detection in Remote Sensing Images Using a Dual Convolutional Neural Network - Markku Luotamo (University of Helsinki); Sari Metsämäki (Finnish Environment Institute); Arto Klami (University of Helsinki) [click for abstract]

Semantic segmentation by convolutional neural networks (CNN) has advanced the state of the art in pixel-level classification of remote sensing images. However, processing large images typically requires analyzing the image in small patches, and hence features that have large spatial extent still cause challenges in tasks such as masking of atmospheric clouds from optical satellite images. To support a wider scale of spatial features while simultaneously reducing computational requirements for large satellite images, we propose an architecture of two cascaded CNN model components successively processing undersampled and full resolution images. The first component distinguishes between patches in the inner cloud area from patches at the cloud's boundary region. For the cloud-ambiguous edge patches requiring further segmentation, the framework then delegates computation to a fine-grained model component. We apply the architecture to a cloud detection dataset of complete multispectral images from the ESA Sentinel-2 satellite, approximately annotated for minimal false negatives in a land use application. On this specific task and data, we achieve a 16% relative improvement in pixel accuracy over a CNN baseline based on patching.

[paper]

Fast Variational Learning in State-Space Gaussian Process Models - Paul Chang (Aalto University); William Wilkinson (Aalto University); Mohammad Emtiyaz Khan (RIKEN); Arno Solin (Aalto University) [click for abstract]

Gaussian process (GP) regression with spatio-temporal inputs can often be performed in linear time via a stochastic differential equation formulation. However, for non-Gaussian likelihoods, this requires application of approximate inference methods which can make the implementation difficult. In this paper, we propose a new method that removes such difficulties. Building upon an existing method called conjugate-computation variational inference, our approach enables linear-time inference via Kalman recursions. We provide an efficient JAX implementation which exploits just-in-time compilation and allows for fast automatic differentiation through large for-loops. Overall, our approach leads to fast and stable variational inference in state-space GP models that can be scaled to time series with millions of data points.

[paper] [video]

Multidisciplinary Topics and Applications

Towards Collective Artificial Intelligence - Fabrice Saffre (VTT); Caj Södergård (VTT) [click for abstract]

The definition of intelligence is highly problematic. According to the Cambridge Dictionary, intelligence is “the ability to learn, understand and make judgements or have opinions that are based on reason”. The emphasis is on the combination of “learning”, “understanding” and “reason” and makes it a cognition-centric notion. Neuromorphic Artificial Intelligence is seemingly rooted in that same philosophical tradition: it was recently described as “focus(ing) on developing techniques to perform tasks that we normally associate with the application of intelligence” and “(…) mimicking tasks such as learning, reasoning, planning and interacting with other intelligent agents”. By way of contrast, Merriam-Webster proposes a more inclusive definition and considers intelligence to be “the ability to learn or understand or deal with new or trying situations”. According to this other school of thought, even biological evolution could be considered a form of intelligence, since its “ability to deal with new or trying situations” is its key attribute.

Consequently, we propose to expand the definition of AI to include other forms of intelligence or, rather, to “reinstate” them since it is arguably one of the most profound scientific discoveries of the second half of the last century that complex adaptive behaviour does not require advanced cognitive functions. We do this with the explicit objective to promote alternative models for machine intelligence that may provide more economical, more sustainable, more transparent or just simpler and more robust solutions to certain real-world problems than neuromorphic AI and Deep Learning. In order to demonstrate this approach, we choose drone swarms, a particular case of distributed robotics, as a proof-of-concept. From a technical standpoint, autonomous drones have become very feasible (thanks in part to advances in artificial neural networks). Furthermore, from surveillance to precision agriculture, and from disaster recovery to small parcel delivery, there is no shortage of applications. What is lacking though is a well thought-through organisational paradigm capable of orchestrating the operation of drone swarms beyond “formation flying”, over extended time-scales and without requiring human intervention or causing “information overload” (whether on the processing or on the communication side).

Indeed, such a paradigm does exist in Nature and has been extensively studied, but is still to be systematically applied to distributed robotics: it consists in the kind of collective intelligence exhibited by a variety of social (or merely gregarious) organisms. The primary modus operandi of collective (or swarm) intelligence is stigmergetic self-organisation, a process whereby individual actions recorded in the form of a modification of the environment add up to an adaptive response at group level. The challenge is to transpose the mechanisms presiding to this phenomenon in biology to the realm of technology, in which problems and objectives are often fundamentally similar (e.g. how to redeploy resources in response to a changing situation) but with different constraints and information channels (e.g. insects communicate mostly through chemicals, computers through electromagnetic signals).

In this paper, we will show how methods that were successfully used by theoretical biologists to decipher collective intelligence in the natural world can be “reverse-engineered” to formulate distributed algorithms and identify optimal parameter values allowing a drone swarm tasked with the continuous observation (or surveillance) of a geographical area of arbitrary size and shape to operate fully autonomously. We will also demonstrate how the concept of “Digital Twin” can be leveraged to support forms of interaction that cannot easily be implemented in the physical world but are known to foster emergent coordination (e.g. stigmergetic positive and negative feedback loops that modulate recruitment).

Artificial Intelligence Platforms: New Research Agenda for Digital Platform Economy - Tomasz Mucha (Aalto University); Timo Seppälä (Department of Industrial Engineering and Management, Aalto University; Research Institute of the Finnish Economy, ETLA) [click for abstract]

This paper expands the research agenda for digital platform economy scholars by identifying multiple areas for future inquiry in artificial intelligence (AI) platforms. The need for a revision of the research agenda is driven by rapid developments in the industry. Many digital platform companies were already using machine learning algorithms in their internal business processes or as part of customer offering long before battles for mobile platform domination were fought. After a series of breakthroughs in modern AI and performance improvements of deep neural networks, these technologies became omnipresent and platform companies are the key providers. Together with the increasing capabilities and performance of AI technologies, their application areas and the role they play for digital platforms have evolved. Starting from automating specific tasks in internal business processes and offering capabilities to deliver new services that previously required human involvement, these technologies have enriched the repertoire of things that digital platforms can do to improve their business incrementally. However, proliferation of AI technologies has also enabled platformization of this AI middleware layer in digital platform technology stack, thus effectively creating a platform inside a platform.

There is overwhelming empirical evidence of AI technologies being central to running a digital platform business. However, the current research agenda for digital platform economy is not directing researchers to study AI technologies in this context. We remedy this by developing a broad research agenda identifying micro- and macro-level questions related to AI platforms. We have divided the proposed AI platforms research agenda as follows: The first set of questions we propose relates to an overall conceptualization of AI platforms. Thereafter, we recognize specific aspects of AI platforms, which need to be investigated in detail to gain understanding that is more complete. The second set of questions we propose relates to understanding the dynamics between AI platforms and the broader socio-economic context. This topic might be particularly relevant to economies of countries without indigenous AI platforms. Our paper builds on the proposition that AI is a general-purpose technology, which by itself carries properties of a digital platform.

[link]

Quantum computing for big data analytics - Ville Kotovirta (VTT); Hannu Reittu (VTT) [click for abstract]

Quantum computing (QC) has a promise to speed up computation of specific tasks, such as search and optimisation. This could lead to new hybrid methods e.g. in machine learning, where computationally hard tasks are implemented as quantum programs executed on quantum devices, while the more data intensive routines are run on classical computers. Similarly, quantum computing can speed up big data analysis, not by processing large amounts of data, but by solving hard problems induced by e.g. big data graphs. Representing big data as graphs or networks takes into account not just data points but also their relations, which has much potential for future applications. Such large-scale graphs are computationally extremely hard to analyse and most problems are usually beyond the capacity of even the most powerful supercomputers. Graph analysis problems are thus one area where keenly looked-after ‘killer applications’ for QC can be searched. Our work is aimed at finding possibilities in applying QC to large graph analysis, in order to understand the data and underlying phenomena and to pre-process data for machine learning and AI applications. A main avenue for applying QC is so called quantum adiabatic computing and quantum annealing in particular which is suitable for almost any optimization task, and another is universal quantum gate computing that could be used for many problems, but there each efficient algorithm is an non-trivial milestone in itself.

We present a new quantum algorithm called Community panning for graph partitioning and, in particular, graph community detection. A starting point is Szemeredi’s Regularity Lemma (SRL), a cornerstone of extremal graph theory, which justifies a kind of stochastic block model structure of bounded complexity or all large graphs. SRL has had a great impact in the theoretical study of large graphs and that is why it can have a decisive role in future big data induced large graph analysis as well. We developed one version of Community panning using quantum annealing and evaluated it on a real quantum annealer. In addition, under construction is another version for quantum gate computing using Grover’s algorithm and the quantum phase estimation algorithm. For the quantum annealing algorithm we defined regularity check as an optimization problem and wrote the task in a form of quadratic binary optimization problems (qubo) which is suitable for quantum annealers. Regularity check has simple formulation and is extremely hard problem (co-NPcomplete), and that is why it is fruitful for evaluating QC. The same optimization task used in regularity check can be used to find communities of an arbitrary graph. We tested the algorithm using D-wave’s 2000-qubit annealer and compared the results with classical algorithms. Both classical and quantum computing found similar solutions for moderate size problems, but based on the performance analysis of the quantum computer in relation to problem size, we conjecture that future quantum annealers can produce high quality solutions for large scale problems outperforming classical computers.

Next, we will evaluate Community panning using the fresh version of D-wave’s quantum annealer with 5000 qubits. We continue developing the gate-based version to be evaluated on quantum gate simulators and real quantum gate computers, demonstrating principles of circuit implementations of so called quantum oracles. We want to study real applications for quantum computing and we aim at applying the algorithm to a biotech problem of finding new microbe species in field samples. 16S-metagenomics data can be represented as graphs and the detection of novel species can be defined as finding communities in the graph. Bioinformatics approaches exist, but they are CPU-intensive, and the sequencing data might be of poor quality. New algorithms are needed in the field for speed-up and to tolerate the errors in the source data. Medical and biological applications of QC are considered as one of the main direction of potential quantum breakthroughs in real-life applications.

[paper]

AI in Software Engineering - David Hastbacka (Tampere University); Hannu-Matti Järvinen (Tampere University); Terhi Kilamo (Tampere University); Outi SIevi-Kerte (Tampere University); Kari Systä (Tampere University); Davide Taibi (Tampere University); Zheying Zhang (Tampere University) [click for abstract]

In this presentation we claim that AI should be used more in software engineering since there is both a need and opportunity. The role of software in our lives is increasing, and it is obvious that the demand cannot be fulfilled by educating new professionals only. We need a major leap in productivity and AI is an obvious tool for that. Luckily, the AI methods and tools are maturing and there is an increasing amount of machine readable data for AI methods to run on. In this presentation we introduce a collection of examples from the research of our group:

• AI for software quality: use of genetic algorithms for architectural design, project coordination and management of technical debts, visualization and analysis framework for SWENG data, test case selection
• AI for Quality of Service: anomaly detection in cloud systems
• AI for business value creation in software development: automatic analysis of user feedback for software development and evolution

Moreover, we will propose some calls for action for researchers, industry and their collaboration considering

• interoperability of data
• need for a data & analysis tools specific to SW engineering
• need for business-specific aspects that can be easily analyzed with AI

Natural Language Processing

Visual Interpretation of DNN-based Acoustic Models using Deep Autoencoders - Tamas Grosz (Aalto University); Mikko Kurimo (Aalto University) [click for abstract]

Using Deep Neural Networks (DNN) is the standard approach in automatic speech recognition since they can achieve the best accuracy. Unfortunately, their inner workings are still a mystery, and it is difficult to translate their function into an understandable format for humans. Interpretation is also essential in verifying that a highly accurate DNN has actually learned to use a proper problem representation and not just exploited some artifacts of the data.

We focus on visually explaining how DNN-based acoustic model functions. Specifically, we wanted to see which phonetic categories were recognized by it. The visual interpretation task is quite simple; given some test data, the hidden representations of the network are inspected. Naturally, the hidden activation vectors of the DNN need to be transformed into a low dimensional space so humans can visually inspect them.

Currently, several solutions exist for this like T-Distributed Stochastic Neighbor Embedding (t-SNE), Uniform Manifold Approximation and Projection (UMAP). The main drawback of these is that they rely on an optimization step to determine the best low-dimensional layout. A problematic consequence is that after modifying the data might produce a very different transformation. Another problem is that both t-SNE and UMAP optimize the layout of the data globally, severely limiting the number of vectors we can inspect. To solve these issues, we investigated the use of deep Autoencoders (AE) for transforming the data. Our AE-based method is compared with two standard algorithms: t-SNE and UMAP.

To ensure that we evaluate the approaches appropriately, we employ three metrics; Procrustes Distance (PD), Mutual Information (MI) and Distance Correlation (DC). Our results show that AEs are applicable to visual interpretation. Furthermore, the learned transformation (the encoder part) can be reused to visualize new data without any optimization, which is the main advantage of our system.

This work was supported by the Kone Foundation.

[paper]

Controlling the Imprint of Passivization and Negation in Contextualized Representations - Hande Celikkanat (University of Helsinki); Sami Virpioja (University of Helsinki); Jörg Tiedemann (University of Helsinki); Marianna Apidianaki () [click for abstract]

Contextualized word representations encode rich information about syntax and semantics, alongside specificities of each context of use. While contextual variation does not always reflect actual meaning shifts, it can still reduce the similarity of embeddings for word instances having the same meaning. We explore the imprint of two specific linguistic alternations, namely passivization and negation, on the representations generated by neural models trained with two different objectives: masked language modeling and translation. Our exploration methodology is inspired by an approach previously proposed for removing societal biases from word vectors. We show that passivization and negation leave their traces on the representations, and that neutralizing this information leads to more similar embeddings for words that should preserve their meaning in the transformation. We also find clear differences in how the respective features generalize across datasets.

Developing Open Tools and Data Sets for Low Resource and Multilingual Machine Translation -- The Case of the Tatoeba Translation Challenge - Jörg Tiedemann (University of Helsinki) [click for abstract]

This paper describes the development of a new benchmark for machine translation that provides training and test data for thousands of language pairs covering over 500 languages and tools for creating state-of-the-art translation models from that collection. The main goal is to trigger the development of open translation tools and models with a much broader coverage of the World's languages. Using the package it is possible to work on realistic low-resource scenarios avoiding artificially reduced setups that are common when demonstrating zero-shot or few-shot learning. For the first time, this package provides a comprehensive collection of diverse data sets in hundreds of languages with systematic annotation and data splits to extend the narrow coverage of existing benchmarks. Together with the data release, we also provide a growing number of pre-trained baseline models for individual language pairs and selected language groups.

[code]

Paraphrase Generation and Evaluation on Colloquial-Style Sentences - Eetu Sjöblom (University of Helsinki); Mathias J.P. Creutz (University of Helsinki); Yves Scherrer (University of Helsinki) [click for abstract]

Paraphrases are a set of sentences or phrases that have the same meaning. The study of paraphrases has both theoretical and practical implications: On the one hand, it is possible to explore semantic representations that go deeper than surface-level features. Two expressions may carry the same meaning, although they may not contain the same words or their syntactic structures may be completely different. These two expressions could have an identical or similar underlying semantic representation or there could be a mapping that transforms one surface form to another.

On the other hand, there are practical applications of paraphrase models. Such models can be useful in information retrieval or data mining for discovering expressions with the intended meaning but with totally different surface realization than the original query. Paraphrasing has also been used in abstractive summarization as part of the summarization models, as well as for evaluation. Paraphrases can also be used for proofing or grammar checking, producing suggested corrections. Similarly, someone perfecting their skills in a second language, or someone looking for alternate, possibly more idiomatic, expressions may benefit from paraphrase models. For instance, to pick one word, to corroborate, in a few contexts, we can find the following paraphrase pairs: “She’ll corroborate my story.” → “She’ll back me up.”, “Can you corroborate that?” → “I need proofs.”, “Will people corroborate your account?” → “Is there anybody who can vouch for that?”

In this work, we focus on paraphrase generation using neural machine translation methods. In paraphrase generation we are interested in models that take in an arbitrary input sentence and generate an output with the same meaning but different surface form. We apply traditional recurrent encoder-decoder networks with attention (Luong et al., 2015) as well as Transformer based models (Vaswani et al., 2017), which are the state of the art of modern machine translation.

We perform systematic evaluation and analysis of paraphrase generation in six European languages: German, English, Finnish, French, Russian, and Swedish. To assess the adequacy of the generated paraphrases, we compute scores from manual annotations, which we compare to three automatically computed metrics: (1) BLEU (Papineni et al., 2002) is the standard evaluation metric for machine translation based on n-gram overlap; unfortunately BLEU can penalize interesting paraphrases that are completely different on the surface despite the same or similar meaning. (2) The recently proposed BERTScore (Zhang et al., 2019) attempts to remedy the shortcomings of BLEU and is based on deep contextualized embeddings. (3) We quantify the novelty of the phrases using PINC scores (Chen and Dolan, 2011), which unlike BLEU and BERTScore measures how dissimilar the candidate is from the source.

Our experiments show that our RNN model (recurrent encoder-decoder with attention trained on bidirectional data) produces the most accurate paraphrase predictions in most cases. We also show that it is possible to produce paraphrases that are different from their source sentences, which is a valuable feature for many downstream applications. The two metrics for measuring semantic adequacy, BLEU and BERTScore, show good correlations with human assessment especially in corpus-level evaluation. BERTScore, in particular, can be a valuable replacement or complement to labor-intensive manual annotation efforts.

References

Chen, D. L. and Dolan, W. B. (2011). Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190–200. Association for Computational Linguistics.

Luong, M.-T., Pham, H., and Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.

Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019). BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675.

[paper]

Assessing Grammatical Correctness in the Context of Language Learning: The Case of Russian - Anisia Katinskaia (University of Helsinki, Department of Computer Science); Roman Yangarber (University of Helsinki) [click for abstract]

We present our experiments with assessing grammatical correctness of answers in an online language-learning system Revita for learners beyond the beginner level. Revita automatically generates exercises based on a given text, collects the learners' answers, and returns automatically created feedback. Automatic exercise generation is based on a text which was randomly chosen by the learner and depends on her current language competence level. The main type of exercises is fill-in-the-blank, where the learner receives a text with some words removed and replaced by base forms as hints. The task is to produce a correct grammatical form of the missing words, given the context. Since Revita checks answers automatically, it currently expects only one type of answers as correct ones—the forms which were in the original text. Most of existing language learning systems have pre-set number of possible correct answers for pre-made exercises which are the same for all learners. However, considering that the exercise generation in Revita is personalised, this approach is not viable.

An intelligent learning system should be able to accept more than one grammatically valid alternative (alternative-correct answers) in the given context for any automatically generated exercise. This problem is also relevant to the exercise context that requires a free-form answer, such as an answer to a question or an essay: the learner can produce “unexpected” but valid grammatical forms. The distribution of errors depends on the language skills of learners: beginners and advanced learners make different errors, and advanced learners make fewer errors overall. Users with a high level of language competence insert 2.6 more alternative-correct answers than on average, which makes this problem crucial for Revita. We release a manually annotated learner dataset for Russian, which we used for testing. In this paper, we explore the ability of BERT to detect if learners’ answers are grammatically correct. Unlike in our previous experiments with LSTM models, we extend the context from sentences to paragraphs, because in the practice mode learners receive texts by paragraphs with all generated exercises. We evaluate all exercises in a paragraph jointly since correctness depends on the joint fills in all exercises.

First, we started with a pretrained BERT as masked language model (MLM) and performed a test similar to described by Goldberg at (2019) but with masking all inserted answers in a paragraph at once. To evaluate performance on alternative-correct answers, we consider the MLM prediction as correct if the alternative answer’s score is the same as or higher than the expected answer’s score. This approach showed that BERT as a MLM for Russian, unlike for English, is not able to distinguish between correct and incorrect forms in the context. It is based on a comparison of absolute scores returned by a LM that makes us choose some ad hoc thresholds. We also cannot know which difference between scores means a confident prediction, which is essential in language learning—we cannot return incorrect feedback because it is harmful to the learning process.

Next, we fine-tuned BERT using synthetic data. For this, we designed a new method for data generation, particularly for the task of detecting alternative-correct answers. One difficulty we had to address stems from the sparseness of errors, which causes unbalance in the datasets: only 10% of all words has grammatical errors in the simulated training and the real test data. We have to consider it during training and evaluation, overwise we cannot trust results for alternative-correct answers—models will learn to predict that most of the words are always correct. Because of that, we have to evaluate how good are predictions for actual errors: weak models will show poor results on the smaller class. The evaluation only on inserted answers shows that weighted cross-entropy gives better results for detecting errors. However, all models show worse results for assessing alternative-correct answers. Our previous experiments with LSTM-based models were showing worse results for alternative-correct answers rather than grammatical errors.

Our method of generating data is promising—models trained on it can distinguish between correct/incorrect answers and locate error positions. However, assessing grammatical correctness of alternative-correct answers is more difficult for pre-trained BERT than detecting actual errors. We plan to continue investigating this problem. It is the first paper which investigates the problem of assessing grammatical correctness in the context of language learning and establishes new baselines for Russian.

[link]

Robotics

Real-time robotic fabrication and characterization of artificial fiber - Houari BETTAHAR (Aalto university); Quan Zhou (Aalto university) [click for abstract]

Recently, gel-like matter is used extensively in a wide range of application fields including the industrial applications such as, manufacture and assembly of garment and footwear products, the packaging industry and aircraft manufacturing, as well as, soft macro/micro-robotics, water collection, medical diagnostics, and drug delivery. However, gel-like matter manipulation is very complicated and challenging, due to the nonnewtonian mechanical properties, fast solidification, high deformability, and high viscosity. Moreover, gel-like matters are pliable combinations of liquids, gases, and solids with multiple interfaces, exhibiting nonlinear responses, and are strongly affected by thermal fluctuations. Manipulating gel-like matter requires a deep understanding and detailed characterization of their properties. On the other hand, many nature species manipulate gel-like material by exploiting their body structures. Archetypal examples include silk spinning in silkworms and spiders. They can perform a sophisticated pultrusion process based on natural real time sensing and manipulation control, in which protein dope can be continuously transformed into fibres with extraordinary mechanical properties. It is among the strongest and toughest biological materials. In this work, we control in real-time the mechanical properties of gel-like matter by manipulation, aiming to mimic the pultrusion process of natural species such as spiders and worms. We developed a new robotic manipulation strategy based on force tracking impedance control to control in real-time the mechanical properties of gel-like substance. The obtained results showed that the proposed robotic approach allows fabrication of artificial fibres in real-time with high performances, i.e. great trade-off between toughness and stiffness. The system will be further developed to mimic the natural pultrusion process using reinforcement learning.

This work has been supported by the project 317018 AI spider silk threading (ASSET) in the AIPSE program of the Academy of Finland.

Intuitive adaptive robotics - Petri T Tikka (VTT ); Joonas Linnosmaa (VTT) [click for abstract]

Human way of independent learning is often trial and error. If in the future we want industrial robots/manipulators to become independent and to be able to quickly adapt to scenarios unforeseen by their creators, it is essential that they learn similar skills. Safe and fast way of doing this is with the help of simulation. We let the robots simulate their scenario, environment and actions to learn a way to accomplish the task, without external guidance from their designers or operators. We wanted to do it in a way that is intuitive (understandable) and adaptive (scalable) in nature. We studied the possibilities of Reinforcement Learning (RL) with robot manipulator. RL is a machine learning paradigm, which allowed the robot to learn its action independently using trial and error. In RL the agent (in this case, a robot) tries to accomplish a given task by using a reward function to guide its actions and learn the most rewarding way to do it. To find an optimal way to accomplish a given task requires agent to try different approaches (exploring). The amount of tries varies according to given task, but in any real-life case, it will anyway be a big number. Physically doing this can be time consuming and even dangerous as we cannot let the robot to totally explore on its own without a completely closed environment. To speed up the learning process (which is often around 100k trials), we used simulation-based learning, where we first built a simulation model of the robot and the physical environment to allow faster than real life speed. We then transfer this knowledge to a real industrial manipulator via Robot Operating System (ROS) and combining it with machine vision to allow also autonomous real-time feedback from physical world. Robotization, flexible automation and Artificial Intelligence (AI) offer opportunities for enhanced production. Bringing more agile development and operation of industrial robots will benefit the Finnish and European production industry. Self-learning systems can help, for instance, logistics industry in piling packages on a pallet in optimized order autonomously.

Safe, Explainable and Trustworthy AI

Explainable AI Without Interpretable Model - Kary Å Framling (Umeå University) [click for abstract]

Explainability has been a challenge in AI for as long as AI has existed. With the recently increased use of AI in society, it has become more important than ever that AI systems would be able to explain the reasoning behind their results also to end-users in situations such as being eliminated from a recruitment process or having a bank loan application refused by an AI system. Especially if the AI system has been trained using Machine Learning, it tends to contain too many parameters for them to be analysed and understood, which has caused them to be called ‘black-box’ systems. Most Explainable AI (XAI) methods are based on extracting an interpretable model that can be used for producing explanations. However, the interpretable model does not necessarily map accurately to the original black-box model. Furthermore, the understandability of interpretable models for an end-user remains questionable. The notions of Contextual Importance and Utility (CIU) presented in this paper make it possible to produce human-like explanations of black-box outcomes directly, without creating an interpretable model. Therefore, CIU explanations map accurately to the black-box model itself. CIU is completely model-agnostic and can be used with any black-box system. In addition to feature importance, the utility concept that is well-known in Decision Theory provides a new dimension to explanations compared to most existing XAI methods. Finally, CIU can produce explanations at any level of abstraction and using different vocabularies and other means of interaction, which makes it possible to adjust explanations and interaction according to the context and to the target users.

The work is partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

[paper] [code]

Methodologies for the validation of AI systems - Lalli Myllyaho (University of Helsinki); Mikko Raatikainen (University of Helsinki); Tomi Männistö (University of Helsinki); Tommi Mikkonen (Metropolia University, University of Helsinki); Jukka Nurminen (University of Helsinki) [click for abstract]

Context: Artificial intelligence (AI) has made its way into everyday activities. Powerful new techniques and tools especially in the realm of machine learning (ML) have increased the visibility AI. However, these new tools have made the techniques easily implementable even to those with very little knowledge of the techniques themselves. This, combined with the difficulty of testing AI systems with traditional methods, has made the questions of software quality even more pressing than before. Objective: This paper studies the methods used to validate systems utilizing AI in the research literature. The goal is to gain a view to the methods that are considered suitable to ensure the software quality in AI systems. Method: The study is conducted as a systematic literature review (SLR). Automated search and three stages of paper selection resulted in 90 papers. Systems presented in the papers were analysed based on domain, task and complexity, and which methods were used to validate them. Results: The systems performed tasks in 18 categories in 14 domains. Main methods for validation were described: trial, simulation, statistical proof, and expert opinion. Failure monitors, safety channels, redundancy, voting, and input and output restrictions were used to continuously validate the systems after deployment. Only a handful of papers addressed more than one validation method. Conclusions: The wide variety of domains and performed tasks suggest that AI is a flexible and adaptable tool for solving a number of problems. A taxonomy of validation methods emerged from the included papers. A taxonomy for continuous validation methods was also synthesised but were rarely found in the papers and often described in less detail. In the future, the presented taxonomy should be assessed and refined, and more emphasis should be put on continuous validation methods when describing systems.

Differentially Private Bayesian Inference For GLMs - Tejas Kulkarni (Aalto University); Joonas Jälkö (Aalto University); Antti Koskela (University of Helsinki); Antti Honkela (University of Helsinki); Samuel Kaski (Aalto University and University of Manchester) [click for abstract]

The framework of differential privacy (DP) upper bounds the information disclosure risk involved in using sensitive datasets for statistical analysis. A DP mechanism typically operates by adding carefully calibrated noise to the data release procedure. The generalized linear models (GLMs) are among the most widely used arms in data analyst's repertoire. In this work, with logistic regression as a running example, we propose a generic noise aware Bayesian framework to quantify the parameter uncertainty for a GLM at hand, given noisy sufficient statistics. The running time of our method is independent of the number of observations. We experimentally demonstrate that the posteriors obtained from our model are similar to the non-private ones with a strong privacy guarantee.

[paper]

Semantic Technologies

Integrating Historical Person Registers as Linked Open Data in the WarSampo Knowledge Graph - Mikko Koho (University of Helsinki / HELDIG); Petri Leskinen (Aalto University / SeCo); Eero Hyvönen (University of Helsinki & Aalto University) [click for abstract]

Semantic data integration from heterogeneous, distributed data silos enables Digital Humanities research and application development employing a larger, mutually enriched and interlinked knowledge graph. However, data integration is challenging, involving aligning the data models and reconciling the concepts and named entities, such as persons and places. This paper concerns the entity reconciliation of person entities in military historical person registers for semantic data integration. A probabilistic record linkage process is presented to reconcile person references in different person registers with structured metadata into a single knowledge graph. The process was applied to reconcile three person registers of the popular semantic portal "WarSampo - Finnish World War 2 on the Semantic Web". The registers contain detailed information about some 100,000 people and are individually maintained by domain experts. This sets demands on the integration process to be automated and adaptable to changes in the registers. An evaluation of the record linkage results is promising, and provides some insight into military person register reconciliation in general.

[link] [paper] [video] [link]

Linked Open Data Service about Historical Finnish Academic People in 1640–1899 - Petri Leskinen (Aalto University / SeCo); Eero Hyvonen (Aalto University and University of Helsinki) [click for abstract]

The Finnish registries ‘Ylioppilasmatrikkeli’' 1640–1852 and 1853–1899 contain detailed biographical data about virtually every academic person in Finland during the respective time periods. This paper presents first results on transforming these registries into a Linked Open Data service using the FAIR principles. The data is based on the student registries of the University of Helsinki, formerly the Royal Academy of Turku, that have been digitized, transliterated, and enriched with additional data about the people from various other registries. Our goal is to transform this largely textual data into Linked Open Data using named entity recognition and linking techniques, and to enrich the data further based on links to internal and external data sources and by reasoning new associations in the data. The data will be published as a Linked Open Data service on top of which a semantic portal AcademySampo'' with tools for searching, browsing, and analyzing the data in biographical and prosopographical research are provided.

[paper] [video]