Tobias Wasner, M.Sc.

At the Applied Education Technologies (AET) research group of the Technical University of Munich (TUM), both the applications it builds and the developers in its courses need access to large language models (LLMs). Logos provides this access, but represents users as a flat list of equally privileged Application Programming Interface (API) keys. Instructors have to create every user and key manually, cannot attribute requests to users or teams, and cannot cap capacity through budgets and rate limits. Existing gateways offer capacity control, but do not provision course teams from university-managed groups.

Institutional deployments of large language models (LLMs) face a fundamental tension between growing inference demand and constrained local Graphics Processing Unit (GPU) hardware. Without a dedicated orchestration layer, multiple applications sharing on-premise inference capacity must independently manage model selection and memory allocation, making coordinated hardware utilization impossible.

This guided research evaluates whether Logos, an open-source LLM orchestration layer developed at the Applied Education Technologies (AET) research group at TUM, provides measurable throughput benefits over direct access to LLM serving frameworks such as vLLM. The study adapts existing benchmarks for such frameworks to a multi-application, multi-model scenario and compares three conditions: Logos-orchestrated vLLM, direct vLLM access, and Ollama, across two request datasets and two inference nodes (dedicated GPU hosts). The results provide empirical evidence for the design choices in Logos and establish a reusable benchmark methodology for shared LLM infrastructure.

The Logos platform serves as a central gateway between users and AI models within a shared GPU infrastructure, hosting and serving models through vLLM. Currently, deploying new models requires manual intervention by the administrator, who must verify compatibility, resolve errors, and accept licensing agreements on behalf of all users. Users do not have visibility into the status of a model request, leaving them unable to make informed decisions about whether to wait, switch to an available model, or request an alternative.