Thiago D. Simão

selected publications

1. AAAI

   Safe Policy Improvement for POMDPs via Finite-State Controllers

   Simão, Thiago D., Suilen, Marnix, and Jansen, Nils

   In Proceedings of the AAAI Conference on Artificial Intelligence 2023

   Abs arXiv Bib HTML PDF Code Details

   We study safe policy improvement (SPI) for partially observable Markov decision processes (POMDPs). SPI is an offline reinforcement learning (RL) problem that assumes access to (1) historical data about an environment, and (2) the so-called behavior policy that previously generated this data by interacting with the environment. SPI methods neither require access to a model nor the environment itself, and aim to reliably improve the behavior policy in an offline manner. Existing methods make the strong assumption that the environment is fully observable. In our novel approach to the SPI problem for POMDPs, we assume that a finite-state controller (FSC) represents the behavior policy and that finite memory is sufficient to derive optimal policies. This assumption allows us to map the POMDP to a finite-state fully observable MDP, the history MDP. We estimate this MDP by combining the historical data and the memory of the FSC, and compute an improved policy using an off-the-shelf SPI algorithm. The underlying SPI method constrains the policy-space according to the available data, such that the newly computed policy only differs from the behavior policy when sufficient data was available. We show that this new policy, converted into a new FSC for the (unknown) POMDP, outperforms the behavior policy with high probability. Experimental results on several well-established benchmarks show the applicability of the approach, even in cases where finite memory is not sufficient.

   @inproceedings{Simao2023safe,
     title = {Safe Policy Improvement for {POMDP}s via Finite-State Controllers},
     author = {Sim{\~a}o, Thiago D. and Suilen, Marnix and Jansen, Nils},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
     year = {2023},
     publisher = {{AAAI} Press},
     pages = {15109--15117}
   }

2. AAMAS

   AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training

   Simão, Thiago D., Jansen, Nils, and Spaan, Matthijs T. J.

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2021

   Abs Bib HTML PDF Supp Code Details

   Deploying reinforcement learning (RL) involves major concerns around safety. Engineering a reward signal that allows the agent to maximize its performance while remaining safe is not trivial. Safe RL studies how to mitigate such problems. For instance, we can decouple safety from reward using constrained Markov decision processes (CMDPs), where an independent signal models the safety aspects. In this setting, an RL agent can autonomously find tradeoffs between performance and safety. Unfortunately, most RL agents designed for CMDPs only guarantee safety after the learning phase, which might prevent their direct deployment. In this work, we investigate settings where a concise abstract model of the safety aspects is given, a reasonable assumption since a thorough understanding of safety-related matters is a prerequisite for deploying RL in typical applications. Factored CMDPs provide such compact models when a small subset of features describe the dynamics relevant for the safety constraints. We propose an RL algorithm that uses this abstract model to learn policies for CMDPs safely, that is without violating the constraints. During the training process, this algorithm can seamlessly switch from a conservative policy to a greedy policy without violating the safety constraints. We prove that this algorithm is safe under the given assumptions. Empirically, we show that even if safety and reward signals are contradictory, this algorithm always operates safely and, when they are aligned, this approach also improves the agent’s performance.

   @inproceedings{Simao2021alwayssafe,
     author = {Sim{\~a}o, Thiago D. and Jansen, Nils and Spaan, Matthijs T. J.},
     title = {AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training},
     year = {2021},
     booktitle = {Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)},
     publisher = {IFAAMAS},
     location = {Online},
     pages = {1226--1235}
   }

3. AAAI

   Safe Policy Improvement with Baseline Bootstrapping in Factored Environments

   Simão, Thiago D., and Spaan, Matthijs T. J.

   In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence 2019

   Abs Bib HTML PDF Details

   We present a novel safe reinforcement learning algorithm that exploits the factored dynamics of the environment to become less conservative. We focus on problem settings in which a policy is already running and the interaction with the environment is limited. In order to safely deploy an updated policy, it is necessary to provide a confidence level regarding its expected performance. However, algorithms for safe policy improvement might require a large number of past experiences to become confident enough to change the agent’s behavior. It can achieve a better sample complexity by exploiting independence between features of the environment, but it lacks a confidence level. We study how to improve the sample efficiency of the safe policy improvement with baseline bootstrapping algorithm by exploiting the factored structure of the environment. Our main result is a theoretical bound that is linear in the number of parameters of the factored representation instead of the number of states. The empirical analysis shows that our method can improve the policy using a number of samples potentially one order of magnitude smaller than the flat algorithm.

   @inproceedings{Simao2019safe,
     author = {Sim{\~a}o, Thiago D. and Spaan, Matthijs T. J.},
     title = {{Safe Policy Improvement with Baseline Bootstrapping in Factored Environments}},
     booktitle = {Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence},
     pages = {4967--4974},
     publisher = {{AAAI} Press},
     year = {2019}
   }