Thiago D. Simão

selected publications

1. ICLR

   Safety-Prioritizing Curricula for Constrained Reinforcement Learning

   Koprulu, Cevahir,  Simão, Thiago D., Jansen, Nils, and Topcu, Ufuk

   In ICLR 2025

   Abs Bib HTML PDF Code Details

   Curriculum learning aims to accelerate reinforcement learning (RL) by generating curricula, i.e., sequences of tasks of increasing difficulty. Although existing curriculum generation approaches provide benefits in sample efficiency, they overlook safety-critical settings where an RL agent must adhere to safety constraints. Thus, these approaches may generate tasks that cause RL agents to violate safety constraints during training and behave suboptimally after. We develop a safe curriculum generation approach (SCG) that aligns the objectives of constrained RL and curriculum learning: improving safety during training and boosting sample efficiency. SCG generates sequences of tasks where the RL agent can be safe and performant by initially generating tasks with minimum safety violations over high-reward ones. We empirically show that compared to the state-of-the-art curriculum learning approaches and their naively modified safe versions, SCG achieves optimal performance and the lowest amount of constraint violations during training.

   @inproceedings{Koprulu2025safetyPrioritizing,
     title = {Safety-Prioritizing Curricula for Constrained Reinforcement Learning},
     author = {Koprulu, Cevahir and Sim{\~a}o, Thiago D. and Jansen, Nils and Topcu, Ufuk},
     booktitle = {ICLR},
     year = {2025}
   }

2. ECAI

   Reinforcement Learning by Guided Safe Exploration

   Yang, Qisong,  Simão, Thiago D., Jansen, Nils, Tindemans, Simon H., and Spaan, Matthijs T. J.

   In ECAI 2023

   Abs arXiv Bib HTML PDF Code Details

   Safety is critical to broadening the application of reinforcement learning (RL). Often, we train RL agents in a controlled environment, such as a laboratory, before deploying them in the real world. However, the real-world target task might be unknown prior to deployment. Reward-free RL trains an agent without the reward to adapt quickly once the reward is revealed. We consider the constrained reward-free setting, where an agent (the guide) learns to explore safely without the reward signal. This agent is trained in a controlled environment, which allows unsafe interactions and still provides the safety signal. After the target task is revealed, safety violations are not allowed anymore. Thus, the guide is leveraged to compose a safe behavior policy. Drawing from transfer learning, we also regularize a target policy (the student) towards the guide while the student is unreliable and gradually eliminate the influence from the guide as training progresses. The empirical analysis shows that this method can achieve safe transfer learning and helps the student solve the target task faster.

   @inproceedings{Yang2023reinforcement,
     title = {Reinforcement Learning by Guided Safe Exploration},
     author = {Yang, Qisong and Sim{\~a}o, Thiago D. and Jansen, Nils and Tindemans, Simon H. and Spaan, Matthijs T. J.},
     booktitle = {ECAI},
     year = {2023},
     pages = {2858--2865}
   }

3. AAAI

   Safe Policy Improvement for POMDPs via Finite-State Controllers

   Simão, Thiago D., Suilen, Marnix, and Jansen, Nils

   In AAAI 2023

   Abs arXiv Bib HTML PDF Code Details

   We study safe policy improvement (SPI) for partially observable Markov decision processes (POMDPs). SPI is an offline reinforcement learning (RL) problem that assumes access to (1) historical data about an environment, and (2) the so-called behavior policy that previously generated this data by interacting with the environment. SPI methods neither require access to a model nor the environment itself, and aim to reliably improve the behavior policy in an offline manner. Existing methods make the strong assumption that the environment is fully observable. In our novel approach to the SPI problem for POMDPs, we assume that a finite-state controller (FSC) represents the behavior policy and that finite memory is sufficient to derive optimal policies. This assumption allows us to map the POMDP to a finite-state fully observable MDP, the history MDP. We estimate this MDP by combining the historical data and the memory of the FSC, and compute an improved policy using an off-the-shelf SPI algorithm. The underlying SPI method constrains the policy-space according to the available data, such that the newly computed policy only differs from the behavior policy when sufficient data was available. We show that this new policy, converted into a new FSC for the (unknown) POMDP, outperforms the behavior policy with high probability. Experimental results on several well-established benchmarks show the applicability of the approach, even in cases where finite memory is not sufficient.

   @inproceedings{Simao2023safe,
     title = {Safe Policy Improvement for {POMDP}s via Finite-State Controllers},
     author = {Sim{\~a}o, Thiago D. and Suilen, Marnix and Jansen, Nils},
     booktitle = {AAAI},
     year = {2023},
     publisher = {{AAAI} Press},
     pages = {15109--15117}
   }