Possible Futures: Complexity in Sequential Decision-Making Dan Lizotte Assistant Professor, David R. Cheriton School of Computer Science University of Waterloo
Waterloo Institute for Complexity and Innovation 22 Jan 2013
Agents, Rationality, and Sequential Decision-Making • Sequential decision-making from the AI tradition • Supporting sequential clinical decision-making • The assumption of rationality: • mitigates complexity ☺ • reduces expressiveness ☹ • Alternatives to “simple rationality” • Please interrupt at any time
“An agent is just something that perceives and acts.� -Russell & Norvig
Graphic: Leslie Pack Kaelbling, MIT
Agent state
st
reward
action
rt
at
rt+1 st+1
Environment
The “Reward Hypothesis” “That all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward).” — Rich Sutton A.K.A. “Rationality.” (“reward” ≡ “utility”)
Agent Status state st
Outcome reward
Treatment action
rt
at
rt+1 st+1
Environment
Autonomous Agent Paradigm
Agent
•
st
Good
• • • •
state
reward
action
rt
at
rt+1 st+1
Environment
Goal is to maximize long term reward Makes context-dependent, individualized decisions Handles uncertain environments naturally
Bad
• • •
Need to prepare for every eventuality Relies on the reward hypothesis Relies on the assumption of rational* behaviour *Rational in the non-judgemental, mathematical sense.
Decision Support Agent Place a human in-between
Agent state
st
reward
action
rt
at
rt+1 st+1
Environment
Recommend rather than perform actions, still leverage autonomous agent idea
Rational Decision Making Should I Move or Stay?
Assume we have: Observation of state of the world Rewards associated with each state Model of how the world changes
P = 0.1, R = 0
Move
P = 0.1, R = 100 Q(
,Move) = 5
P = 0.7, R = 0
P = 0.1, R = -50
P = 0.1, R = -25
Stay
P = 0.1, R = 10 Q(
,Stay) = -1.5
P = 0.7, R = 0
P = 0.1, R = 0
How do we decide what to do? • Often, “we” assume we want to maximize expected reward
•
For each potential action
•
Compute expected reward over all possible future states
• Choose the action with the highest expected reward
• “Q-function” Q(s,a) = E[R|s,a]
P = 0.1, R = 0
Move
P = 0.1, R = 100 Q(
,Move) = 5
P = 0.7, R = 0
P = 0.1, R = -50
P = 0.1, R = -25 Q( Stay
,Move) = 5 P= = -1.5 0.1, R = 10 ,Stay)
Q( π(
V(
) = Move Q(
,Stay) = -1.5
)=5 P = 0.7, R = 0
P = 0.1, R = 0
Sequential Decision-Making •
For each potential action
•
Compute expected reward over all possible future states
• Feasible if reward is known for each state. • What if reward is influenced by subsequent decision-making?
Mosey P = 0.1, R = 10
P = 1.0, R = 10 Q(
,Mosey) = 10
Q( Scramble!
Q( π(
,Mosey) = 10 ,Scramble) = 90 P = 0.9, R = 100 ) = Scramble Q( ,Scramble) = 90 V( ) = 90 P = 0.1, R = 0
P = 0.1, R = -25
Q(
Stay
,Move) = 5 P = 0.1, V R == 90 10 Q( ,Stay) = -1.5 6.5 ,Stay) = -1.5 6.5 MoveQ( π( ) = Stay V( ) = 6.5 5 P = 0.7, R = 0
P = 0.1, R = 0
Sequential Decision-Making • Optimal decisions are influenced by future plans
• Not a problem, if we know what we will do when we get to any future state.
• Replace R (short-term) with V (long-term) by the law of total expectation
• Expectation is now over sequences of states
Policy Evaluation
Policy Evaluation
Policy Evaluation
Policy Evaluation
Policy Evaluation
Time to evaluate: O(S2⋅T) Number of policies: O(AS⋅T) Exhaustive search: O(AS⋅T ⋅S2⋅T)
Policy Learning
Policy Learning
Policy Learning
Policy Learning
Policy Learning
Policy Learning
Policy Learning
Policy Learning
Time to evaluate: O(A⋅S2⋅T) Exhaustive search: O(AS⋅T ⋅S2⋅T)
Policy Learning • Enormous complexity savings:
From O(AS⋅T ⋅S2⋅T) to O(A⋅S2⋅T)
•
What assumptions are required?
• • •
Markov property Rationality (linearity of expectation)
What if we want to avoid the rationality assumption?
Beyond Simple Rationality • maximization of the expected value of the cumulative sum of a received scalar signal
• What is the “right” scalar signal? • Symptom relief? • Side-effects? • Cost? • Quality of life? • Different for every decision-maker, every patient.
Multiple Reward Signals state
st
•
How can we reason with multiple reward signals?
Agent
reward
action
rt
at
rt+1 st+1
Environment
• Make use of the human in the loop! • Preference Revealing • Multi-outcome Screening • Each adds computational complexity (DL,Bowling,Murphy)
(DL,Ferguson,Laber)
Preference Elicitation Suppose D different rewards are important for decision-making, r[1], r[2], ..., r[D] Assume each person has a function f that takes these and gives utility, that person’s happiness given any configuration of the r[i] expressed as a scalar value. We could use this as our new reward! Preference Elicitation attempts to figure out an individual’s f.
Preference Elicitation 1. Determine preferences of the decision-maker 2. Construct reward function from “basis rewards” (different outcomes) 3. Compute the recommended treatment, by policy learning
•
Cost: O(A⋅S2⋅T) per elicitation
Agent state
st
reward
action
rt
at
rt+1 st+1
Environment
Preference Elicitation One way: Assume f has a nice form: f(r[1], r[2], ..., r[D]) = δ[1]r[1] + δ[2]r[2] + ... + δ[D]r[D] Then Preference Elicitation figures out the δ, or weights, an individual attaches to different rewards. How? The values δ[i] and δ[j] defines an exchange rate between r[i] and r[j].
Preference Elicitation f(r[1], r[2], ..., r[D]) = δ[1]r[1] + δ[2]r[2] + ... + δ[D]r[D] “If I lost δ[j] units of r[i], but I gained δ[i] units of r[j], I would be equally happy.” Preference elicitation asks questions like: “If I took away 5 units of r[i], how many units of r[j] would you want?” Once f is known, standard single-outcome methods can be applied.
Preference Elicitation • Are the questions based in reality? • Even if they are, can the decision-maker answer them?
• How will the decision-maker respond to “I know what you want.” ?
Agent state
st
reward
action
rt
at
rt+1 st+1
Environment
Preference Revealing (DL, S Murphy, M Bowling)
1. Determine the preferences of the decision-maker 2. Compute the recommended treatment for all possible preferences (δ) 3. Show, for each action, what preferences are consistent with that action being recommended
Positive And Negative Syndrome Scale vs. Body Mass Index vs. Heinrichs Quality of Life Scale
Phase 1
Data:
Reward: BMI δ = (0, 1, 0)
Clinical Antipsychotic Trials of Intervention Effectiveness (NIMH)
Reward: PANSS δ = (1, 0, 0)
! Ziprasidone — Olanzapine
Reward: HQLS δ = (0, 0, 1)
Preference Revealing Benefits No reliance on preference elicitation Facilitates deliberation rather than imposing a single recommended treatment Information still individualized through patient state Treatments that are not suggested for any preference are implicitly screened
Costs O(AT⋅ST) worst case cf. O(A⋅S2⋅T) for single reward cf. 2ℵ0 for “enumeration” Feasible for short horizons Value function/policy now a function of state and preference Value functions not convex in preference, thus related methods for POMDPs do not apply
Multi-outcome Screening 1. Elicit “clinically meaningful difference” for each outcome 2. Screen out treatments that are “definitely bad” 3. Recommend the set of remaining treatments
Multi-outcome Screening (DL, B Ferguson, E Laber)
Suppose two* different rewards are important for decision making: r[1], r[2] Screen out a treatment if it is much worse than another treatment for one reward and not any better for the other reward. Do not screen if 1) treatments are not much different or 2) one treatment is much worse for one reward but much better for the other Output: Set containing one or both treatments, possibly with a reason if both are included.
[D]t
ˆ
t+1
t (·, ·, p), ..., Q[D]t (·, ·, p))
g Praction
Multi-outcome Screening Eliminate using Strong Practical Domination
a3
6
a2
QT [2] (sT , a)
a1
4
a)
in Qt
a1 a4
4
a5 2
6
0
0
2
4 QT [1] (sT , a) (b)
6
Multi-outcome Screening Benefits
Costs
No model of preference required
Must consider all policies the user might follow in future
Suggests a set rather than imposing a single recommended treatment Information still individualized through patient state Treatments with bad evidence are explicitly screened Screening criterion is intuitive
Possible Futures 6
Positive And Negative Syndrome Scale and Body Mass Index Daniel J. Lizotte, Eric B. Laber
Fig. 2 Plot of a vector-valued Q-function (QT 1[1] (sT 1 , aT 1 ; pT ), QT 1[2] (sT 1 , aT 1 ; pT )) for a fixed state sT 1 . Here, aT 1 2 A = {+, 4, #, 竍･, 竚マ, and pT 2 P, a collection of 20 possible future policies. Thus each
Multi-outcome Screening Benefits
Costs
No notion of preference required
Must consider all policies the user might follow in future
Suggests a set rather than imposing a single recommended treatment Information still individualized through patient state Treatments with bad evidence are explicitly screened Screening criterion is intuitive
Enumeration: O(AS⋅T ⋅S2⋅T) Restriction to policies that 1) follow recommendations and 2) are “not too complex” makes computation feasible CATIE: 10124 possible futures reduced to 1213 possible futures
Multi-Outcome Screening
Non-deterministic multiple-reward fitted-Q for decision support
17
Fig. 7 CATIE NDP for Phase 1 made using P 9 ; “warning” actions that would have been eliminated by Practical Domination but not by Strong Practical Domination have been removed.
Figure 7 shows the NDP learned for Phase 1 using our algorithm with Strong Practical Domination (D1 = D2 = 2.5) and P 9 , and actions that receive a “warning” according to Practical Domination have been removed. Here, choice is further increased by requiring an action to be practically better than another action in order to dominate it. Furthermore, although we have removed actions that were warned to have a bad tradeoff—those that were slightly better for one reward but practically worse for another—we still achieved increased choice over using the Pareto frontier alone. In this NDP, the mean number of choices per
Summary • Autonomous Agent model is for sequential
decision making but we want decision support.
• The Reward Hypothesis is useful
computationally, but too restrictive for us
• Preference Revealing:
Compute for all possible scalar rewards
• Multi-outcome Screening:
Compute for all “reasonable” future policies
• In both cases, increase in complexity is tolerable
Thank you! • Dan Lizotte dlizotte@uwaterloo.ca Acknowledgements: My colleagues Susan Murphy, Michael Bowling, Eric Laber Natural Sciences and Engineering Research Council of Canada (NSERC) Data were obtained from the limited access datasets distributed from the NIHsupported “Clinical Antipsychotic Trials of Intervention Effectiveness in Schizophrenia” (CATIE-Sz). The study was supported by NIMH Contract N01MH90001 to the University of North Carolina at Chapel Hill. The ClinicalTrials.gov identifier is NCT00014001. This talk reflects the view of the author and may not reflect the opinions or views of the CATIE-Sz Study Investigators or the NIH.