Evaluating Metaheuristic Optimization Methods for Quantum Reinforcement Learning

Evaluating Metaheuristic Optimization Methods for Quantum Reinforcement Learning

Abstract:

Quantum Reinforcement Learning offers the potential for advantages over classical Reinforcement Learning, such as a more compact representation of the state space through quantum states. Furthermore, theoretical studies suggest that Quantum Reinforcement Learning can exhibit faster convergence than classical approaches in certain scenarios. However, further research is needed to validate the actual benefits of Quantum Reinforcement Learning in practical applications. This technology also faces challenges such as a flat solution landscape, characterized by missing or low gradients, which makes the application of traditional, gradient-based optimization methods inefficient. In this context, it is necessary to examine gradient-free algorithms as an alternative. The present work focuses on the integration of metaheuristic optimization algorithms such as Particle Swarm Optimization, Ant Colony Optimization, Tabu Search, Simulated Annealing, and Harmony Search into Quantum Reinforcement Learning. These algorithms offer flexibility and efficiency in parameter optimization, as they utilize specialized search strategies and adaptability. The approaches are evaluated within two Reinforcement Learning environments and compared to random action selection. The results show that in the MiniGrid environment, all algorithms lead to acceptable or even very good results, with Simulated Annealing and Particle Swarm Optimization achieving the best performance. In the Cart Pole environment, Simulated Annealing and Particle Swarm Optimization achieve optimal results, while Ant Colony Optimization, Tabu Search, and Harmony Search perform only slightly better than an algorithm with random action selection. These results demonstrate the potential of metaheuristic optimization methods such as Particle Swarm Optimization and Simulated Annealing for efficient learning in Quantum Reinforcement Learning systems, but also highlight the need for careful selection and adaptation of the algorithm to the specific problem.

Author:

Daniel Seidl

Advisors:

Michael Kölle, Maximilian Zorn, Claudia Linnhoff-Popien

Student Thesis | Published May 2024 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors