Student-Abstracts-EN

Offline Quantum Reinforcement Learning using Metaheuristic Optimization Strategies

Abstract:

This thesis investigates offline quantum reinforcement learning (QRL) with variational quantum circuits (VQCs) and metaheuristic optimization. O!ine reinforcement learning (RL) provides a realistic training paradigm in which agents learn entirely from fixed datasets instead of online interaction, making it particularly suited for reproducible studies and controlled comparisons. For the offline training, we created a dataset for the CartPole-v1 environment by combining random, medium, and expert policies, resulting in 525,000 transitions with diverse state–action coverage. On this dataset, we evaluated the effectiveness of four gradient-free metaheuristic optimizers: Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Simulated Annealing (SA), and Tabu Search (TS). We trained a DQN agent with a 4-qubit, 2-layer VQC. Their performance is compared to a gradient-based gradient descent (GD) baseline with Adam optimizer. Each optimizer undergoes per-factor hyperparameter tuning, followed by an optimizer comparison on a single dataset pass.

Results show that all metaheuristics substantially outperform our GD baseline, with SA achieving the highest final performance, followed by TS, GA, and PSO. These findings demonstrate that gradient-free optimization offers clear advantages over gradient descent for VQCs, especially when learning from offline datasets, where optimization must proceed under limited data access and without environment interaction. By decoupling training from online interaction, the offline setting enables a rigorous comparison of optimizers and provides a practical path toward scaling QRL experiments under realistic resource constraints. This is particularly important in domains where online interactions are costly or safety-critical. Therefore, this study establishes offline QRL with metaheuristic optimization strategies as a promising research direction, while also highlighting limitations such as distribution shift and restricted convergence when training on a single dataset pass.

Author:

Frederik Bickel

Advisors:

Michael Kölle, Julian Hager, Claudia Linnhoff-Popien

Student Thesis | Published September 2025 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors

Assessing Noise Robustness of Variational Quantum Circuits in Reinforcement Learning Environments

Abstract:

This bachelor thesis investigates the robustness of variational quantum circuits (VQCs) in reinforcement learning (RL) compared to classical neural networks under the influence of observation noise. Observation noise describes the uncertainty that arises when the states perceived by an RL agent deviate from the actual states of the environment, for example due to sensor noise, environmental influences or targeted adversarial attacks. A deterministic REINFORCE algorithm is used, which always selects the action with the highest probability prediction instead of the usual stochastic sampling. This methodological decision enables a targeted analysis of the direct influence of observation noise on the agent’s policy, independent of random exploration effects. Robustness is investigated using the deterministic variant of the well-known reinforcement learning environment Frozen-Lake, which is extended by an observation noise model with a self designed hot zone logic. Within these hot zones, the agent receives deliberately incorrect observations orthogonal to its original direction of movement. A classical neural network in form of a multi-layer perceptron (MLP) is compared with a VQC. Although the MLP often converges faster, it exhibits volatile and non-monotonic performance under increasing noise influence. In contrast, the VQC demonstrates superior stability with a predictable performance degradation, especially at higher noise levels. The results suggest that the structural properties of VQCs may enable better generalisation and robustness against structured observation noise.

Author:

Justin Dominik Marinus Klein

Advisors:

Julian Hager, Michael Kölle, Thomas Gabor, Claudia Linnhoff-Popien

Student Thesis | Published July 2025 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors

Quantum Architecture Search for Solving Quantum Machine Learning Tasks

Abstract:

Quantum computing is a computing paradigm based on the principles of quantum mechanics. This makes it fundamentally different from classical computing. For selected problem domains, quantum computers are expected to offer a performance advantage, the so-called quantum advantage, which manifests itself in exponentially faster computation times or lower resource requirements. In the current Noisy Intermediate Scale Quantum era, quantum hardware is still limited in performance and highly error-prone. Variational Quantum Circuits represent an approach that is comparatively robust to these limitations. The performance of these quantum circuits is highly dependent on the underlying architecture of the parameterized quantum circuit. The development of powerful, hardware-compatible circuit architectures is therefore an important task, also known as Quantum Architecture Search. Developing good architectures manually is an inefficient and error-prone process. First attempts have been made to automate this process. In addition to Evolutionary Algorithms, Differentiable Architecture Search, and Monte Carlo Tree Search, Reinforcement Learning is another potentially suitable approach for finding good architectures, but it has been relatively little studied. In particular, little is known about its suitability as a search strategy for Machine Learning problems. The goal of this work is to investigate Reinforcement Learning as a suitable search strategy for quantum circuits in the context of Machine Learning problems. For this purpose, the RL-QAS framework is presented, which enables the automated search for circuit architectures using a Reinforcement Learning Agent. The RL-QAS framework is evaluated on the Iris and binary MNIST classification problems. RL-QAS enabled the discovery of architectures that achieve high test accuracy in the classification of the aforementioned datasets while exhibiting low complexity. RL-QAS demonstrated that Reinforcement Learning is indeed suitable for architecture discovery. However, in order for RL QAS to be applied to more complex problems, further development of the approach is necessary.

Author:

Simon Salfer

Advisors:

Michael Kölle, Philipp Altmann, Claudia Linnhoff-Popien

Student Thesis | Published June 2025 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors

An Empirical Evaluation of Quantum Annealing-Based Image Classification Using Discriminative Quantum Boltzmann Machines

Abstract:

The Boltzmann machine has been highly influential in the development of artificial intelligence, serving as a foundational framework for energy-based models and neural network research. However, its direct applications in modern deep learning have been severely limited due to computational constraints. Classical sampling methods have consistently proven to be inefficient, rendering the processing of high-dimensional inputs practically infeasible. Therefore, alternatives like the Restricted Boltzmann Machine (RBM) have been introduced, sacrificing expressiveness for faster computations. In contrast, Quantum Boltzmann Machines can efficiently sample from approximate Boltzmann distributions when implemented using quantum algorithms such as quantum annealing. Empirical results suggest that this approach can yield a more efficient sampling process than classical methods, enabling more effective exploration of energy landscapes while reducing computational overhead. Additionally, this also makes full connectivity possible, preserving the expressiveness of the original BM. Nonetheless, to the best knowledge of the author, only a sparse amount of other studies have explored the capability of QBMs for supervised learning. This is particularly true for an application-driven context using real quantum hardware. Thus, the primary goal of this work is the evaluation of the practical effectiveness of QBMs utilizing discriminative learning for the classification of real-world image data using a novel embedding approach to save expensive Quantum Processing Unit time. This is done by employing discriminative QBMs, which always clamp the input units to a data point regardless of the current phase. The model can therefore learn the conditional distribution of a label given a data point. The results demonstrate competitive performance compared to discriminative BMs trained with simulated annealing and discriminative RBMs, while also indicating a slight reduction in the number of training epochs required. Additionally, the embedding approach proposed in this work significantly accelerated sampling, with an average speedup of 69.65% over the conventional embedding.

Author:

Mark Vorapong Seebode

Advisors:

Jonas Stein, Daniëlle Schuman, Claudia Linnhoff-Popien

Student Thesis | Published June 2025 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors

Quantum Transformers: Leveraging Variational Quantum Circuits for Natural Language Processing

Abstract:

Recent advances in large language models (LLMs) have established transformer architectures as the dominant paradigm in natural language processing (NLP). While these models achieve state of the art performance, their exponential growth in parameter count and computational demands raises concerns regarding scalability, energy consumption and environmental impact. Simultaneously, quantum machine learning (QML) has emerged as a promising field that explores whether quantum computation can offer more efficient learning mechanisms, particularly using variational quantum circuits (VQCs), which have shown competitive performance with fewer parameters. This thesis investigates whether a quantum transformer model can be designed to structurally mirror the classical transformer while remaining feasible for execution on Noisy Intermediate-Scale Quantum (NISQ) hardware. To this end, we propose a modular, NISQ-compatible quantum transformer architecture that reproduces key classical components embedding, multi-head attention and encoder-decoder structure, using VQCs. Each component is implemented using shallow, strongly entangling circuits designed to minimize circuit depth and parameter count. The model is evaluated on synthetic language modeling tasks, comparing quantum and classical variants under matched conditions, including identical token vocabularies and equivalent parameter budgets. Results show that the quantum model is capable of learning simple formal languages, converging rapidly with fewer parameters and in some configurations achieving perfect reconstruction of deterministic token sequences. However, its performance degrades on more complex tasks requiring generalization, where classical models remain superior. These findings demonstrate the feasibility of the proposed quantum transformer architecture on near-term hardware and situate the model as a proof of concept for the architectural potential of encoder-decoder quantum transformers models in NLP.

Author:

Julian Hager

Advisors:

Michael Kölle, Gerhard Stenzel, Thomas Gabor, Claudia Linnhoff-Popien

Student Thesis | Published June 2025 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors

Emergent Cooperation in Quantum Multi-Agent Reinforcement Learning Using Communication

Abstract:

Emergent cooperation in classical Multi-Agent Reinforcement Learning has gained significant attention, particularly in the context of Sequential Social Dilemmas. While classical reinforcement learning approaches have demonstrated to be capable of emergent cooperation, research on extending these methods to the emerging field of Quantum Multi-Agent Reinforcement Learning remains limited, particularly through the usage of communication. In this work, we apply the two-phase communication protocol Mutual Acknowledgment Token Exchange (MATE), its extension Mutually Endorsed Distributed Incentive Acknowledgment Token Exchange (MEDIATE), the peer rewarding mechanism Gifting and Reinforced Inter-Agent Learning (RIAL), an approach to learn a discrete communication protocol, to quantum Q-Learning. We evaluate the resulting eight approaches in terms of their impact on emergent cooperation in three Sequential Social Dilemmas, namely the Iterated Prisoner’s Dilemma, the Iterated Stag Hunt and the Iterated Game of Chicken. Our experimental results show that the approaches MATETD, AutoMATE, MEDIATE-I and MEDIATE-S achieved high levels of cooperation across all three Sequential Social Dilemmas, demonstrating that communication is a viable method to achieve emergent cooperation in Quantum Multi-Agent Reinforcement Learning.

Author:

Christian Reff

Advisors:

Michael Kölle, Leo Sünkel, Claudia Linnhoff-Popien

Student Thesis | Published May 2025 | Copyright © QAR-Lab
Direct Inquiries to this work to the Advisors