Hybrid deployments of large language models (LLMs) that combine local and cloud-based providers can balance cost, privacy, and deployment autonomy, but constrained GPU memory and compute capacity make efficient local serving challenging. Existing orchestration approaches focus on model selection, while cold starts, low throughput under concurrent load, and backend-specific runtime behavior are rarely integrated into routing decisions.
This thesis extends the Logos orchestration platform to address both efficient local serving and runtime-aware scheduling. It introduces a Scheduling Data Interface (SDI) that provides a unified view of provider state, extends Logos with improved handling of local serving capacity and cold-start behavior, and adds lightweight operational monitoring to support scheduling decisions.
The evaluation uses deterministic workload replay and measures time to first token, total request latency, and tail latency percentiles. The results quantify the gap between static and runtime-aware routing and show how more efficient local serving reduces the hardware headroom required under realistic multi-model load.