Expert knowledge elicitation for interactive improvement of machine learning models

Helsinki Institute for Information Technology HIIT is a joint

research institute of Aalto University and the University of Helsinki

for basic and applied research on information technology.

Helsinki Institute for Information Technology HIIT is a joint research institute of Aalto University and

the University of Helsinki for basic and applied research on information technology.

Case study: Predicting citations of scientific articles

Evaluation and User Study

• Predict the relative number of citations a scientific article will get in

Artificial Intelligence domain, based on its title, abstract and keywords

• An expert user indicates whether the words in the article (features)

are relevant for the prediction task

Solving elicitation on a budget – User model

• Balance between querying additional input on either the most

promising relevant features (exploitation), or on the most uncertain

ones (exploration)

References[1] Soare, M., Ammad-ud-din, M., Kaski, S. (2016). Regression with n 1 by Expert Knowledge Elicitation. In the 15th IEEE International Conference on Machine Learning and Applications, pp. 734 – 739.[2] Afrabandpey, H, Peltola, T., Kaski, S. (2016). Regression Analysis in Small-n-Large-p Using Interactive Prior Elicitation of Pairwise Similarities. In FILM 2016, NIPS Workshop on Future of Interactive Learning Machines.[3] Micallef, L.*, Sundin, I.*, Marttinen, P.*, Ammad-ud-din, M., Peltola, T., Soare, M., Jacucci, G., Kaski, S. (2017). Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 547 – 552. (*equal contribution).[4] Daee, P., Peltola, T., Soare, M., Kaski, S. (2017). Knowledge Elicitation via Sequential Probabilistic Inference for High-Dimensional Prediction. In arXiv preprint arXiv:1612.03328.

• Reinforcement learning using linear bandit: Features with the highest

Upper Confidence Bounds are shown to the expert user for input

• At each iteration, the user model updates estimated relevance of

features based on expert’s previous inputs

Prediction model

• A standard linear regression model, expert

knowledge brought in by adjusting the prior

distributions of the model’s parameters

• Interaction improves predictions compared to the model that doesn’t

use any expert user input

• Our user model method provides significantly better elicitation than a

naive method with random selection

Iiris Sundin1, Pedram Daee1, Homayun Afrabandpey1, Tomi Peltola1, Marta Soare1,

Muhammad Ammad-ud-din1, Luana Micallef1, Pekka Marttinen1, Giulio Jacucci2, Samuel Kaski1

1 Helsinki Institute for Information Technology HIIT, Department of Computer Science, Aalto University

2 Helsinki Institute for Information Technology HIIT, Department of Computer Science, University of Helsinki

firstname.lastname@hiit.fii

Interactive expert knowledge elicitation in small n, large p problems

Results

• User study with 23

participants

• Users indicated their

domain knowledge with

an interactive interface

• User input for 200

keywords

• Modeling and making predictions are challenging

when there are only few samples (n), but many

predictive features (p)

• Applications: Genomic medicine and other cases

where collecting more data is not feasible, or even

not possible

Interactive system brings an expert to the loop