Implementing custom rollout strategies for JuliaPOMDP

[loslapleo]

Implementing custom rollout strategies for JuliaPOMDP

Hello all,

I've been trying to wrap my head around the JuliaPOMDP packages for a while now and I need some help and/or advice. Please do keep in mind that I've only just started using Julia and this is likely the cause of most, if not all, of my frustrations.

Goal

At the moment I only want to create a rollout strategy for BasicPOMCP where multiple random rollout are applied and the average reward of all the rollouts are returned.

Findings

I first looked at the main file. Here I found that you can set the estimation value of the POMCPSolver using the estimate_value argument which is initially set to RolloutEstimator(RandomSolver(rng)) where the RolloutEstimator calls the RandomSolver to evaluate the leaf nodes. This is where things start getting fuzzy to me.

I'm guessing the RolloutEstimator gets translated into a SolvedRolloutEstimator and then estimate_value is called when a rollout needs to be simulated with RolloutSimulator.

This brings me to the simulate functions for RolloutSimulator. My initial thought was to override this simulate function. However, there are a few concerns I have:

• I'm using a HistoryRecorder since I need extra information such as number of states evaluated by solver.
  • How do we go from RolloutSimulator to HistoryRecorder?
  • How to use the edited simulate function for the rollout and HistoryRecorder?

If anyone has any advice on how to do this or can tell me if I got anything wrong, please let me know. I know this is messy but it's the best I could do with my current understanding.

[zsunberg]

Thanks for asking!

Most languages these days are very object oriented. Julia allows for object-oriented programming, but it is often easier to do things with a more functional style.

The easiest way to accomplish your initial task of making a Monte Carlo estimate is by creating a function:

function mc_estimate(pomdp, s, h, steps)
    sim = RolloutSimulator(max_steps=steps)
    policy = RandomPolicy(pomdp)
    return mean(simulate(sim, pomdp, policy) for i in 1:10)
end

solver = POMCPSolver(
    estimate_value = mc_estimate
    #...
)

Now, you have complete control of anything that happens within mc_estimate with a lot less boilerplate code than an object-oriented approach. I am not sure exactly what you're trying to do with the HistoryRecorder, but this general approach is quite flexible, so I am sure that we can do it. Can you say a bit more about what you're trying to do?

(All of your reasoning about RolloutEstimator is correct, but I wouldn't recommend emulating that design pattern - it is from our early days)

[loslapleo]

Thank you so much for your reply.

This is my first time using a functional programming language, so I really do appreciate the help.

My current implementation the HistoryRecorder (recorder) so I can get the return with discounted_reward(history) and the amount of belief nodes with length(belied_hist(history)) where history = simulate(recorder, pomdp, policy). I'm doing this to emulate the way that results were displayed by Ross et al. in [1] (on page 28 for example). Please do let me know if this is the correct way of going about this.

I also wanted to ask what would be necessary to use this approach for other policies such as the ValuePolicy or any other custom policy. Below you can find how I'm currently trying to use other policies.

function xact_estimate(pomdp, s, h, steps)
    sim = RolloutSimulator(max_steps=steps)
    policy = ValuePolicy(pomdp)
    return simulate(sim, pomdp, policy)
end

I'm aware that this was naive on my part and I'm still actively seeking the correct way to use other policies. I'm confused since this looks similar to how ValuePolicy is used policy tests for POMDP.jl.

I'm using RockSample as my POMDP as follows.

recorder = HistoryRecorder(max_steps=1000)
pomdp = RockSamplePOMDP(
  map_size=(7,7),
  rocks_positions=[(1,2), (2,2), (2,5), (2,7), (4,7), (5,4), (7,2), (7,6)],
  init_pos=(4,1),
  sensor_efficiency=20.0,
  discount_factor=0.95,
  good_rock_reward = 20.0
)

This is the current error I am getting:

ERROR: LoadError: MethodError: no method matching updater(::ValuePolicy{RockSamplePOMDP{8}, Matrix{Float64}, Int64})

[zsunberg]

Hi @loslapleo ,

Sorry for the delayed reply - I am just coming up for air after a very busy semester.

I'm doing this to emulate the way that results were displayed by Ross et al. in [1] (on page 28 for example). Please do let me know if this is the correct way of going about this.

If my understanding is correct, what Ross et al. are referring to as "belief nodes" are nodes in the search tree, which would not include beliefs encountered on the rollout. To count these, you would do something like:

using POMDPs
using POMDPModels
using BasicPOMCP
using POMDPTools: action_info

m = TigerPOMDP()

planner = solve(POMCPSolver(tree_in_info=true), m)
a, info = action_info(planner, initialstate(m))
@show length(info[:tree].o_labels) # this is the number of belief nodes

(the POMCP tree structure does not have documentation, but you can see it in the code here)

Regarding using other policies, first note that the ValuePolicy documentation is not very up-to-date, so I am not surprised you didn't figure out how to use it. First, it is useful to note that with POMCP, it is usually most useful to do a fully observable rollout since each leaf node contains only one state particle anyways (The reason you were getting a no method matching updater error is that ValuePolicy is not correctly configured for a POMDP, so it does not know how to do belief updates.). To perform a fully observable rollout, use UnderlyingMDP as below.

Also, you constructed the ValuePolicy without a value table, so it will just assume that all actions have the same value, which is not very useful. The most useful policy for these rollouts is probably FunctionPolicy, which takes in a function of the state or belief. The example below just always returns the first action in the action space, but you can make it do whatever you want.

function first_action_rollout_estimate(pomdp, s, h, steps)
    sim = RolloutSimulator(max_steps=steps)
    function select_first_action(s)
        return first(actions(pomdp, s))
    end
    policy = FunctionPolicy(select_first_action)
    mdp = UnderlyingMDP(pomdp)
    return simulate(sim, mdp, policy)
end

Hope that helps!