Architectural Influence on Variational Quantum Circuits in Multi-Agent Reinforcement Learning: Evolutionary Strategies for Optimization

Architectural Influence on Variational Quantum Circuits in Multi-Agent Reinforcement Learning: Evolutionary Strategies for Optimization

Abstract:

The field of Multi-Agent Reinforcement Learning (MARL) is becoming increasingly relevant in domains that involve the interaction of multiple agents, such as autonomous driving and robotics. One challenge in MARL is the exponential growth of dimensions in the state and action spaces. Quantum properties o!er a solution by enabling compact data processing and reducing trainable parameters. One drawback of gradient-based optimization methods in Quantum MARL is the possibility of Barren Plateaus impeding effective parameter updating, thereby hindering convergence. Evolutionary Algorithms, however, bypass this issue as they do not rely on gradient information. Building on research that demonstrates the potential of Evolutionary Algorithms in optimizing Variational Quantum Circuits for MARL tasks, we examine how introducing architectural changes into the evolutionary process affects optimization. We explore three different architecture concepts for Variational Quantum Circuits — Layer-Based, Gate-Based, and Prototype-Based — by applying two evolutionary strategies: one involving both recombination and mutation (ReMu), and the other using mutation only (Mu). To evaluate the efficacy of these approaches, we tested them in the Coin Game, comparing them to a baseline without architectural modifications. The mutation-only strategy with the Gate- Based approach yielded the best results, achieving the highest scores, number of coins collected, and own coin rates while using the fewest parameters. Furthermore, a variant of the Gate-Based approach with results comparable to those of the baseline required significantly fewer gates, resulting in an acceleration of the runtime by 90.1%.

Author:

Karola Schneider

Advisors:

Michael Kölle, Leo Sünkel, Claudia Linnhoff-Popien

Student Thesis | Published November 2024 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors