Thiago D. Simão

selected publications

1. ML

   Safety-constrained reinforcement learning with a distributional safety critic

   Yang, Qisong,  Simão, Thiago D., Tindemans, Simon H., and Spaan, Matthijs T. J.

   Machine Learning 2022

   Abs Bib HTML PDF Code Details

   Safety is critical to broadening the real-world use of reinforcement learning. Modeling the safety aspects using a safety-cost signal separate from the reward and bounding the expected safety-cost is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, it can be risky to set constraints only on the expectation neglecting the tail of the distribution, which might have prohibitively large values. In this paper, we propose a method called Worst-Case Soft Actor Critic for safe RL that approximates the distribution of accumulated safety-costs to achieve risk control. More specifically, a certain level of conditional Value-at-Risk from the distribution is regarded as a safety constraint, which guides the change of adaptive safety weights to achieve a trade-off between reward and safety. As a result, we can compute policies whose worst-case performance satisfies the constraints. We investigate two ways to estimate the safety-cost distribution, namely a Gaussian approximation and a quantile regression algorithm. On the one hand, the Gaussian approximation is simple and easy to implement, but may underestimate the safety cost, on the other hand, the quantile regression leads to a more conservative behavior. The empirical analysis shows that the quantile regression method achieves excellent results in complex safety-constrained environments, showing good risk control.

   @article{Yang2022safety-constrained,
     title = {Safety-constrained reinforcement learning with a distributional safety critic},
     author = {Yang, Qisong and Sim{\~a}o, Thiago D. and Tindemans, Simon H. and Spaan, Matthijs T. J.},
     journal = {Machine Learning},
     pages = {1--29},
     year = {2022},
     publisher = {Springer}
   }

2. AAMAS

   AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training

   Simão, Thiago D., Jansen, Nils, and Spaan, Matthijs T. J.

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2021

   Abs Bib HTML PDF Supp Code Details

   Deploying reinforcement learning (RL) involves major concerns around safety. Engineering a reward signal that allows the agent to maximize its performance while remaining safe is not trivial. Safe RL studies how to mitigate such problems. For instance, we can decouple safety from reward using constrained Markov decision processes (CMDPs), where an independent signal models the safety aspects. In this setting, an RL agent can autonomously find tradeoffs between performance and safety. Unfortunately, most RL agents designed for CMDPs only guarantee safety after the learning phase, which might prevent their direct deployment. In this work, we investigate settings where a concise abstract model of the safety aspects is given, a reasonable assumption since a thorough understanding of safety-related matters is a prerequisite for deploying RL in typical applications. Factored CMDPs provide such compact models when a small subset of features describe the dynamics relevant for the safety constraints. We propose an RL algorithm that uses this abstract model to learn policies for CMDPs safely, that is without violating the constraints. During the training process, this algorithm can seamlessly switch from a conservative policy to a greedy policy without violating the safety constraints. We prove that this algorithm is safe under the given assumptions. Empirically, we show that even if safety and reward signals are contradictory, this algorithm always operates safely and, when they are aligned, this approach also improves the agent’s performance.

   @inproceedings{Simao2021alwayssafe,
     author = {Sim{\~a}o, Thiago D. and Jansen, Nils and Spaan, Matthijs T. J.},
     title = {AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training},
     year = {2021},
     booktitle = {Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)},
     publisher = {IFAAMAS},
     location = {Online},
     pages = {1226-1235}
   }

3. AAMAS

   Safe Policy Improvement with an Estimated Baseline Policy

   Simão, Thiago D., Laroche, Romain, and Tachet des Combes, Rémi

   In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2020

   Abs Bib HTML PDF Code Details

   Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.

   @inproceedings{Simao2020safe,
     author = {Sim{\~a}o, Thiago D. and Laroche, Romain and {Tachet des Combes}, Rémi},
     title = {{Safe Policy Improvement with an Estimated Baseline Policy}},
     year = {2020},
     booktitle = {Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)},
     publisher = {IFAAMAS},
     pages = {1269–1277},
     numpages = {9},
     location = {Auckland, New Zealand}
   }

4. AAAI

   Safe Policy Improvement with Baseline Bootstrapping in Factored Environment

   Simão, Thiago D., and Spaan, Matthijs T. J.

   In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence 2019

   Abs Bib HTML PDF Details

   We present a novel safe reinforcement learning algorithm that exploits the factored dynamics of the environment to become less conservative. We focus on problem settings in which a policy is already running and the interaction with the environment is limited. In order to safely deploy an updated policy, it is necessary to provide a confidence level regarding its expected performance. However, algorithms for safe policy improvement might require a large number of past experiences to become confident enough to change the agent’s behavior. It can achieve a better sample complexity by exploiting independence between features of the environment, but it lacks a confidence level. We study how to improve the sample efficiency of the safe policy improvement with baseline bootstrapping algorithm by exploiting the factored structure of the environment. Our main result is a theoretical bound that is linear in the number of parameters of the factored representation instead of the number of states. The empirical analysis shows that our method can improve the policy using a number of samples potentially one order of magnitude smaller than the flat algorithm.

   @inproceedings{Simao2019safe,
     author = {Sim{\~a}o, Thiago D. and Spaan, Matthijs T. J.},
     title = {{Safe Policy Improvement with Baseline Bootstrapping in Factored Environment}},
     booktitle = {Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence},
     pages = {4967--4974},
     publisher = {{AAAI} Press},
     year = {2019}
   }