Thiago D. Simão

selected publications

1. AAMAS

   AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training

   Simão, Thiago D., Jansen, Nils, and Spaan, Matthijs T. J.

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2021

   Abs HTML PDF Supp Code Bibtex Details

   Deploying reinforcement learning (RL) involves major concerns around safety. Engineering a reward signal that allows the agent to maximize its performance while remaining safe is not trivial. Safe RL studies how to mitigate such problems. For instance, we can decouple safety from reward using constrained Markov decision processes (CMDPs), where an independent signal models the safety aspects. In this setting, an RL agent can autonomously find tradeoffs between performance and safety. Unfortunately, most RL agents designed for CMDPs only guarantee safety after the learning phase, which might prevent their direct deployment. In this work, we investigate settings where a concise abstract model of the safety aspects is given, a reasonable assumption since a thorough understanding of safety-related matters is a prerequisite for deploying RL in typical applications. Factored CMDPs provide such compact models when a small subset of features describe the dynamics relevant for the safety constraints. We propose an RL algorithm that uses this abstract model to learn policies for CMDPs safely, that is without violating the constraints. During the training process, this algorithm can seamlessly switch from a conservative policy to a greedy policy without violating the safety constraints. We prove that this algorithm is safe under the given assumptions. Empirically, we show that even if safety and reward signals are contradictory, this algorithm always operates safely and, when they are aligned, this approach also improves the agent’s performance.

   @inproceedings{Simao2021alwayssafe,
     author = {Sim{\~a}o, Thiago D. and Jansen, Nils and Spaan, Matthijs T. J.},
     title = {AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training},
     year = {2021},
     booktitle = {Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)},
     publisher = {IFAAMAS},
     location = {Online},
     pages = {1226-1235}
   }

2. AAMAS

   Safe Policy Improvement with an Estimated Baseline Policy

   Simão, Thiago D., Laroche, Romain, and Tachet des Combes, Rémi

   In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2020

   Abs HTML PDF Code Bibtex Details

   Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.

   @inproceedings{Simao2020safe,
     author = {Sim{\~a}o, Thiago D. and Laroche, Romain and {Tachet des Combes}, Rémi},
     title = {{Safe Policy Improvement with an Estimated Baseline Policy}},
     year = {2020},
     booktitle = {Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)},
     publisher = {IFAAMAS},
     pages = {1269–1277},
     numpages = {9},
     location = {Auckland, New Zealand}
   }

3. AAAI

   Safe Policy Improvement with Baseline Bootstrapping in Factored Environment

   Simão, Thiago D., and Spaan, Matthijs T. J.

   In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence 2019

   Abs HTML PDF Bibtex Details

   We present a novel safe reinforcement learning algorithm that exploits the factored dynamics of the environment to become less conservative. We focus on problem settings in which a policy is already running and the interaction with the environment is limited. In order to safely deploy an updated policy, it is necessary to provide a confidence level regarding its expected performance. However, algorithms for safe policy improvement might require a large number of past experiences to become confident enough to change the agent’s behavior. It can achieve a better sample complexity by exploiting independence between features of the environment, but it lacks a confidence level. We study how to improve the sample efficiency of the safe policy improvement with baseline bootstrapping algorithm by exploiting the factored structure of the environment. Our main result is a theoretical bound that is linear in the number of parameters of the factored representation instead of the number of states. The empirical analysis shows that our method can improve the policy using a number of samples potentially one order of magnitude smaller than the flat algorithm.

   @inproceedings{Simao2019safe,
     author = {Sim{\~a}o, Thiago D. and Spaan, Matthijs T. J.},
     title = {{Safe Policy Improvement with Baseline Bootstrapping in Factored Environment}},
     booktitle = {Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence},
     pages = {4967--4974},
     publisher = {{AAAI} Press},
     year = {2019}
   }