Thiago D. Simão

selected publications

1. AAAI

   Safe Policy Improvement for POMDPs via Finite-State Controllers

   Simão, Thiago D., Suilen, Marnix, and Jansen, Nils

   In Proceedings of the AAAI Conference on Artificial Intelligence 2023

   Abs arXiv Bib Code Details

   We study safe policy improvement (SPI) for partially observable Markov decision processes (POMDPs). SPI is an offline reinforcement learning (RL) problem that assumes access to (1) historical data about an environment, and (2) the so-called behavior policy that previously generated this data by interacting with the environment. SPI methods neither require access to a model nor the environment itself, and aim to reliably improve the behavior policy in an offline manner. Existing methods make the strong assumption that the environment is fully observable. In our novel approach to the SPI problem for POMDPs, we assume that a finite-state controller (FSC) represents the behavior policy and that finite memory is sufficient to derive optimal policies. This assumption allows us to map the POMDP to a finite-state fully observable MDP, the history MDP. We estimate this MDP by combining the historical data and the memory of the FSC, and compute an improved policy using an off-the-shelf SPI algorithm. The underlying SPI method constrains the policy-space according to the available data, such that the newly computed policy only differs from the behavior policy when sufficient data was available. We show that this new policy, converted into a new FSC for the (unknown) POMDP, outperforms the behavior policy with high probability. Experimental results on several well-established benchmarks show the applicability of the approach, even in cases where finite memory is not sufficient.

   @inproceedings{Simao2023safe,
     title = {Safe Policy Improvement for POMDPs via Finite-State Controllers},
     author = {Sim{\~a}o, Thiago D. and Suilen, Marnix and Jansen, Nils},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
     year = {2023},
     publisher = {{AAAI} Press},
     arxiv = {2301.04939}
   }

2. NeurIPS

   Robust Anytime Learning of Markov Decision Processes

   Suilen, Marnix,  Simão, Thiago D., Parker, David, and Jansen, Nils

   In Advances in Neural Information Processing Systems 2022

   Abs arXiv Bib Code Details

   Markov decision processes (MDPs) are formal models commonly used in sequential decision-making. MDPs capture the stochasticity that may arise, for instance, from imprecise actuators via probabilities in the transition function. However, in data-driven applications, deriving precise probabilities from (limited) data introduces statistical errors that may lead to unexpected or undesirable outcomes. Uncertain MDPs (uMDPs) do not require precise probabilities but instead use so-called uncertainty sets in the transitions, accounting for such limited data. Tools from the formal verification community efficiently compute robust policies that provably adhere to formal specifications, like safety constraints, under the worst-case instance in the uncertainty set. We continuously learn the transition probabilities of an MDP in a robust anytime-learning approach that combines a dedicated Bayesian inference scheme with the computation of robust policies. In particular, our method (1) approximates probabilities as intervals, (2) adapts to new data that may be inconsistent with an intermediate model, and (3) may be stopped at any time to compute a robust policy on the uMDP that faithfully captures the data so far. We show the effectiveness of our approach and compare it to robust policies computed on uMDPs learned by the UCRL2 reinforcement learning algorithm in an experimental evaluation on several benchmarks.

   @inproceedings{Suilen2022robust-anytime,
     title = {Robust Anytime Learning of Markov Decision Processes},
     author = {Suilen, Marnix and Sim{\~a}o, Thiago D. and Parker, David and Jansen, Nils},
     booktitle = {Advances in Neural Information Processing Systems},
     year = {2022},
     publisher = {Curran Associates, Inc.},
     volume = {35},
     arxiv = {2205.15827}
   }

3. ML

   Safety-constrained reinforcement learning with a distributional safety critic

   Yang, Qisong,  Simão, Thiago D., Tindemans, Simon H., and Spaan, Matthijs T. J.

   Machine Learning 2022

   Abs Bib HTML PDF Code Details

   Safety is critical to broadening the real-world use of reinforcement learning. Modeling the safety aspects using a safety-cost signal separate from the reward and bounding the expected safety-cost is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, it can be risky to set constraints only on the expectation neglecting the tail of the distribution, which might have prohibitively large values. In this paper, we propose a method called Worst-Case Soft Actor Critic for safe RL that approximates the distribution of accumulated safety-costs to achieve risk control. More specifically, a certain level of conditional Value-at-Risk from the distribution is regarded as a safety constraint, which guides the change of adaptive safety weights to achieve a trade-off between reward and safety. As a result, we can compute policies whose worst-case performance satisfies the constraints. We investigate two ways to estimate the safety-cost distribution, namely a Gaussian approximation and a quantile regression algorithm. On the one hand, the Gaussian approximation is simple and easy to implement, but may underestimate the safety cost, on the other hand, the quantile regression leads to a more conservative behavior. The empirical analysis shows that the quantile regression method achieves excellent results in complex safety-constrained environments, showing good risk control.

   @article{Yang2022safety-constrained,
     title = {Safety-constrained reinforcement learning with a distributional safety critic},
     author = {Yang, Qisong and Sim{\~a}o, Thiago D. and Tindemans, Simon H. and Spaan, Matthijs T. J.},
     journal = {Machine Learning},
     pages = {1--29},
     year = {2022},
     publisher = {Springer}
   }

4. AAMAS

   AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training

   Simão, Thiago D., Jansen, Nils, and Spaan, Matthijs T. J.

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2021

   Abs Bib HTML PDF Supp Code Details

   Deploying reinforcement learning (RL) involves major concerns around safety. Engineering a reward signal that allows the agent to maximize its performance while remaining safe is not trivial. Safe RL studies how to mitigate such problems. For instance, we can decouple safety from reward using constrained Markov decision processes (CMDPs), where an independent signal models the safety aspects. In this setting, an RL agent can autonomously find tradeoffs between performance and safety. Unfortunately, most RL agents designed for CMDPs only guarantee safety after the learning phase, which might prevent their direct deployment. In this work, we investigate settings where a concise abstract model of the safety aspects is given, a reasonable assumption since a thorough understanding of safety-related matters is a prerequisite for deploying RL in typical applications. Factored CMDPs provide such compact models when a small subset of features describe the dynamics relevant for the safety constraints. We propose an RL algorithm that uses this abstract model to learn policies for CMDPs safely, that is without violating the constraints. During the training process, this algorithm can seamlessly switch from a conservative policy to a greedy policy without violating the safety constraints. We prove that this algorithm is safe under the given assumptions. Empirically, we show that even if safety and reward signals are contradictory, this algorithm always operates safely and, when they are aligned, this approach also improves the agent’s performance.

   @inproceedings{Simao2021alwayssafe,
     author = {Sim{\~a}o, Thiago D. and Jansen, Nils and Spaan, Matthijs T. J.},
     title = {AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training},
     year = {2021},
     booktitle = {Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)},
     publisher = {IFAAMAS},
     location = {Online},
     pages = {1226-1235}
   }

5. AAMAS

   Safe Policy Improvement with an Estimated Baseline Policy

   Simão, Thiago D., Laroche, Romain, and Tachet des Combes, Rémi

   In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2020

   Abs Bib HTML PDF Code Details

   Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.

   @inproceedings{Simao2020safe,
     author = {Sim{\~a}o, Thiago D. and Laroche, Romain and {Tachet des Combes}, Rémi},
     title = {{Safe Policy Improvement with an Estimated Baseline Policy}},
     year = {2020},
     booktitle = {Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)},
     publisher = {IFAAMAS},
     pages = {1269–1277}
   }

6. AAAI

   Safe Policy Improvement with Baseline Bootstrapping in Factored Environments

   Simão, Thiago D., and Spaan, Matthijs T. J.

   In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence 2019

   Abs Bib HTML PDF Details

   We present a novel safe reinforcement learning algorithm that exploits the factored dynamics of the environment to become less conservative. We focus on problem settings in which a policy is already running and the interaction with the environment is limited. In order to safely deploy an updated policy, it is necessary to provide a confidence level regarding its expected performance. However, algorithms for safe policy improvement might require a large number of past experiences to become confident enough to change the agent’s behavior. It can achieve a better sample complexity by exploiting independence between features of the environment, but it lacks a confidence level. We study how to improve the sample efficiency of the safe policy improvement with baseline bootstrapping algorithm by exploiting the factored structure of the environment. Our main result is a theoretical bound that is linear in the number of parameters of the factored representation instead of the number of states. The empirical analysis shows that our method can improve the policy using a number of samples potentially one order of magnitude smaller than the flat algorithm.

   @inproceedings{Simao2019safe,
     author = {Sim{\~a}o, Thiago D. and Spaan, Matthijs T. J.},
     title = {{Safe Policy Improvement with Baseline Bootstrapping in Factored Environments}},
     booktitle = {Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence},
     pages = {4967--4974},
     publisher = {{AAAI} Press},
     year = {2019}
   }