thesis proposal maximizing long-term roi for active learning systems
TRANSCRIPT
Thesis Proposal
Maximizing long-term ROI for Active Learning Systems
Interactive ClassificationGoal: Optimize life-time Return On Investment
Large volume (in millions) of transactions coming in
Majority transactions automatically cleared
Minority transactions flagged for auditing
Transactions processed
successfully
Domain specific transaction processing
Machine Learning model
Defining Characteristics • Expensive domain experts• Skewed class distribution (minority events)• Concept/Feature drift• Biased sampling of labeled historical data• Lots of unlabeled data
Lower false positive rates
based on learning model
Learning Model to Flag Transactions for Manual
Intervention
Interactive Classification Applications
• Fraud detection (Credit Card, Healthcare)• Network Intrusion detection• Video Surveillance• Information Filtering / Recommender Systems• Error prediction/Quality Control– Health Insurance Claims Rework
Health Insurance Claim Process - Rework
Underpayments Overpayments
Why is solving Claims Rework important?
• Inefficiencies in the healthcare process result in large monetary losses affecting corporations and consumers– For large (10 million+) insurance plan, estimated $1 billion in loss of revenue
– $91 billion over-spent in US every year on Health Administration and Insurance (McKinsey study’ Nov 2008)
– 131 percent increase in insurance premiums over past 10 years
• Claim payment errors drive a significant portion of these inefficiencies– Increased administrative costs and service issues of health plans
– Overpayment of Claims - direct loss
– Underpayment of Claims – loss in interest payment for insurer, loss in revenue for provider
•Classifier trained from labeled data•Human (user/expert) in the loop using the results but also providing feedback at a cost
•Goal: Maximize long-term Return on Investment (equivalent to the productivity of the entire system)
Interactive Classification Setting – Machine Learning Setup
Unlabeled + Labeled Data
Trained Classifier
Ranked List scored by classifier
Factorization of the problem
Cost (Time of human expert)
Exploration (Future classifier
performance)
Exploitation (Relevancy to the expert)
Exploration-Exploitation Tradeoffs
Cost-Sensitive Active Learning
Standard Ranking / Relevance Feedback Active Learning
Cost-
Sens
itive
Expl
oita
tion
Factorization of the problem – characterization of the models
• Uniform– Each instance has same value
• Variable– Each instance has different value
which is dependent on the properties of the instance
• Markovian– Each instance has dynamically
changing value depending on the (ordered) history of instances already observed, in addition to the factors for Variable model
Cost (Time of human expert)
Exploration (Future classifier
performance)
Exploitation (Relevancy to the expert)
Example Cases for Factorization of Cost Model
• Uniform: Speculative versus definitive language usage distinction for biomedical abstracts– [Settles et al., 2008]
• Variable: Part Of Speech tagging – Annotation time dependent on the sentence length with
longer documents taking more time to label [Ringger et al., 2008]
• Markovian: Claims Rework Error Prediction– If similar claims are shown to the auditors in sequence
reducing the cognitive switching costs, the time taken to label reduces [Ghani and Kumar, 2011]
Example Cases for Factorization of Exploitation Model
• Uniform: Claims Rework Error Prediction– If we only account for the administrative overhead
of fixing a claim [Kumar et al., 2010]• Variable: Claims Rework Error Prediction– If we take into account the savings based on the
adjustment amount of the claim [Kumar et al., 2010]
• Markovian: Claims Rework Error Prediction– Root cause detection [Kumar et al., 2010]
Example Cases for Factorization of Exploration Model
• Uniform: Extracting contact details from email signature lines– Random strategy gives results comparable to other
strategies [Settles et al., 2008]• Variable: KDD Cup 1999, Network Intrusion detection
– Sparsity based strategy gives good performance [Ferdowsi et al., 2011]
– Dependent on the properties of the examples (or population) which can be pre-determined.
• Markovian: Uncertainty based active sampling strategy– Most commonly used strategy
Problem Statement
How can we maximize long term ROI of active learning systems for interactive classification problems?
Proposed Hypothesis
Jointly managing the cost, exploitation and exploration factors will lead to increased long term ROI compared to managing them independently
Proposed Contributions
• A framework to jointly manage cost, exploitation and exploration
• Extensions of Active Learning along the following dimensions– Differential utility of a labeled example– Dynamic cost of labeling an example– Tackling concept drift
Proposed Framework
• Choice of Cost model• Choice of Exploitation model• Choice of Exploration model• Utility metric• Algorithms to optimize the utility metric
Choice of Models
Cost Model
Exploitation Model
Exploration Model
Uniform
Variable
Markovian
Unifo
rm
Varia
ble
Mar
kovia
n Uniform
Variable
Markovian
Utility Metric• Domain dependent• May or may not have a simple instantiation in the domain• Possible instantiations for Claims Rework domain
– Return on Investment (Haertal et al, 2008)• Corresponds to the business goal of the deployed systems• Return: Cumulative dollar value of claims adjusted• Investment: Cumulative time (equivalent dollar amount) for auditing the claims• Does not take into account the classifier improvement/degradation
– Amortized Return on Investment• Amortized return: Calculate the net present value of the returns based on the
expected future classifier improvement• Return: Cumulative dollar value of claims adjusted + net present value of the
increased returns due to future classifier improvement• Investment: Cumulative time (equivalent dollar amount) for auditing the claims• Takes into account exploration and exploitation
Algorithm to optimize the utility metric
• Optimization straightforward if a well defined utility metric exists for the domain– Computational approximations may still be required for practical
feasibility• Cases where a utility metric is not well defined based on the
constituent cost/exploration/exploitation models, approaches to explore– Rank fusion based approach
• Each model provides a ranking which are combined to get a final ranking
– Explore relevant approaches from reinforcement learning• Upper Confidence Bounds for Trees (Kocsis and Szepesvári, 2006)• Multi-armed bandit with dependent arms (Pandey et al, 2007)
Labeled Data (1,…,t-1)
Trained Classifier (1,…,t-1)
Ranked List
Cost (Time of human expert)
Exploration (Future classifier
performance)
Exploitation (Relevancy to the expert)
Labeled Data (t)
Unlabeled Data (t)
Interactive Classification Framework-Experimental Setup
Performance evaluation done on the set of labeled instances obtained at each iteration
Evaluation
Compare various approaches with multiple baselines• Random• Pure Exploitation
– Exploitation=Var/Mar; Exploration=Uniform; Cost=Uniform• Pure Exploration
– Exploration=Var/Mar; Exploitation=Uniform; Cost=Uniform• Pure Cost sensitive
– Cost=Var/Mar; Exploitation=Uniform; Exploration=Uniform
Preliminary results
• Graph with results from framework
Generalizing Active Learning for Handling Temporal Drift
• What is temporal drift?– Changing data distribution– Changing nature of classification problem– Adversarial actions
• Related Work– Traditional active learning assumes static unlabeled pool– Stream-based active learning (Chu et al., 2011) assumes no memory
to store the instances and makes online decisions to request labels• Not completely realistic as labeling requires human effort and is usually not
real-time
– Learning approaches from data streams with concept drift predominantly use ensembles over different time period (Kolter and Maloof, 2007)
Proposed Setup for Temporal Active Learning
• Periodically changing unlabeled pool, corresponding to the experimental setup for interactive framework– Cumulative streaming pool– Recent streaming pool– Novel setup
• Three components for handling temporal drift– Instance selection strategy– Type of model: Ensemble or Single– Instance or model weighing scheme
Proposed Instance Selection Strategies
• Model Weight Drift Strategy• Feature Weight Drift Strategy• Feature Distribution Drift Strategy
Detecting Drift – Change in Models over Time
• Claims rework domain• 15 models built over 15 time periods• Similarity between the models based on cosine measure
Preliminary results
• Evaluation metric: Precision at 5 percentile• Represented in graph as percentage of the best strategy at a given
iteration to give a sense that the mentioned strategies are not the best strategies at all iterations
• Uncertainty begins to perform poorly at later iterations and feature drift based strategy starts performing better
Proposed Work
• More experiments and analysis for claims rework data with data from different clients
• More experiments based on synthetic dataset with longer observation sequence to analyze the performance of sampling strategies
• Generation of synthetic data based on Gaussian Mixture models to mimic real data
Cost-Sensitive Exploitation
Cost (Time of human expert)
Exploration (Future classifier
performance)
Exploitation (Relevancy to the expert)
Cost-
Sens
itive
Expl
oita
tion
More Like This strategy
Labeled DataRanked List scored
by classifier
Select Top m% claims
Ran k
Online Strategy
Cluster
Online “More-like-this” AlgorithmRequire a labeled set L and an unlabeled set U
1. Train classifier C on L2. Label U using C3. Select top m% scored unlabeled examples UT
4. Cluster the examples UT U L into k clusters
5. Rank the k clusters using a exploitation metric6. For each cluster ki in k
1. Rank examples in ki
2. For each example in ki
1. Query expert for label of 2. If precision of cluster ki is < Pmin and number of labels > Nmin, Next
Offline Comparison – MLT vs Baseline
• 9% relative improvement over baseline for Precision at 2nd percentile metric
Precision@1 Precision@2 Precision@50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
MoreLikeThis
Baseline
Live System Deployment
Baseline - batch classifier
More-Like-This0
0.1
0.2
0.3
0.4
0.5
0.6
0
50
100
150
200
250
Precision
Time Taken per audit
Prec
isio
n
Tim
e(se
cond
s)
• Number of claims audited:– Baseline system: 200– More-Like-This: 307
• 90% relative improvement over baseline
• 27% reduction in audit time over baseline
~$10 Million savings /year for a typical insurance company
SummaryProblem Statement
How to maximize long term ROI of active learning systems for interactive classification problems
SummaryThesis Contributions
• Characterization of the interactive classification problem– Defining the cost/exploration/exploitation models
• Uniform• Variable• Markovian
• Generalization (Extensions?) of Active Learning along the following dimensions– Differential utility of a labeled example– Dynamic cost of labeling an example– Tackling concept drift– A framework to jointly manage these considerations
SummaryEvaluation
• Empirical Evaluation of the proposed framework– Using evaluation metric motivated by real business tasks– Datasets
• Real world dataset: Health Insurance Claims Rework• Synthetic dataset
– Comparison with multiple baselines based on underlying cost/exploitation/exploration models
• Methodological contribution – Novel experimental setup– Intend to make the synthetic dataset and its generators
public
SummaryProposed Work: Temporal Active Learning
• Creation of synthetic datasets• Evaluation and analysis of proposed strategies
on synthetic and claims rework dataset
SummaryProposed Work: Framework for interactive classification
• Evaluate multiple utility metrics/optimization algorithm for Claims Rework domain
• Augment temporal drift synthetic data for evaluating framework
• Evaluate multiple utility metrics/optimization algorithm for synthetic dataset
Cost Model
Exploitation Model
Exploration Model
Uniform
Variable
Markovian
Unifo
rm
Varia
ble
Mar
kovia
n
Uniform
Variable
Markovian
Thanks
• Problem Description– High level factorization of the problem
• Related Work– Triangle
• Our proposed approach – framework– Broad categorization of the models
• Choice of models
– Choice of utility metric– Choice of optimization– Proposed work (various aproaches)
• Temporal active learning– Some initial results
• Cost sensitive exploitation• Summary
– Problem statemnt– Contributions– Evaluation
Thesis Contributions• Problem Statement: How to generalize active learning to incorporate differential utility of a
labeled example(dynamic/variable exploitation), dynamic cost of labeling an example, concept drift in a unified framework that makes the deployment of such learning systems practical
• Contributions– Characterization of the interactive learning problem– Generalization of Active Learning along the following dimensions
• Differential utility of a labeled example• Dynamic cost of labeling an example• Tackling concept drift• Cost-Sensitive Exploitation• A unified framework to solve these considerations jointly
– First solution: Optimizing joint utility function based on cost, exploration utility and exploitation utility– Second solution: Using Upper Confidence Bound approach with contextual multi-armed bandit setup to incorporate the
different factors
– Empirical Evaluation of the proposed framework• Using evaluation metric motivated by real business tasks• Datasets
– Synthetic dataset– Real world dataset: Health Insurance Claims Rework
• Comparison with multiple baselines based on underlying factors
Situating the thesis work wrt related work
Active Learning
Cost-sensitiveProactiveLearning• Unreliable Oracle• Oracle variation
PrActiveLearning• Differential Utility• Dynamic cost• Concept Drift
Efficiency & Representation• Feature level feedback• Feature acquisition• Batch active learning
Problem Statement
How to generalize active learning to incorporate differential utility of a labeled example(dynamic/variable exploitation), dynamic cost of labeling an example, concept drift in a unified framework that makes the deployment of such learning systems practical