Every time a developer sends a 2 + 2 or a JSON formatting task to a proprietary 400-billion-parameter model, somewhere in the world an H100 GPU weeps and a CFO gets an over-budget notification.<\/p>\n\n\n\n
At the monthly meetup hosted by ENG (Engineering) thanks to Andrea Raponi, Daniele Scasciafratte and Eugenio Petull\u00e0 took apart, piece by piece, the rhetoric of the \u201cone model to rule them all.\u201d The message is clear: to scale AI-based products without bleeding money on compute or being crushed by European compliance requirements, we need to stop treating AI as a black box and start doing what we do best: software and systems engineering.<\/p>\n\n\n\n
When we call an LLM, the tokenizer breaks the text into minimal units. These units are converted into vector embeddings and processed by the GPU. This is where the first systemic bottleneck appears: the KV Cache (Key-Value Cache).<\/p>\n\n\n\n
To avoid recomputing attention over all previous tokens for every single word generated in streaming, the inference engine stores the key and value tensors in the graphics card\u2019s VRAM. The longer the context grows (up to the million tokens promised by the market), the more video memory is cannibalized by the cache, at the expense of the throughput of concurrent batches:<\/p>\n\n\n\n
Dev request (HTTPS)\n \u2500\u2500> Tokenizer (splitting into syllables\/tokens)\n \u2500\u2500> Vector embedding generation\n \u2500\u2500> GPU Core (Attention processing)\n \u2500\u2500> KV Cache (VRAM progressively saturated to keep the context)<\/code><\/span><\/pre>\n\n\nvLLM vs SGLang: Two Philosophies Compared<\/h3>\n\n\n\n
The middle layer that governs all this is the inference engine. Today the standard is played out on two platforms:<\/p>\n\n\n\n
\n- vLLM:<\/strong> The historic giant, written mainly in Python and backed by hardware vendors like Nvidia and AMD. It introduced PagedAttention, an architecture that models video memory like the paged virtual memory of operating systems. It is robust, but its iteration and release speed is held back by a monolithic codebase.<\/li>\n\n\n\n
- SGLang:<\/strong> A modern engine written in Rust, based on RadixAttention. Instead of managing the cache as a set of isolated pages, it structures shared memory as a tree\/graph. In multi-turn architectures or agent flows with identical system prompts, it reuses huge portions of computation already done.<\/li>\n<\/ul>\n\n\n\n
Real Compliance and Zero Data Retention: The AI Act Gives No Discounts<\/h3>\n\n\n\n
An enterprise application doesn\u2019t fail only because of latency; it fails when the Legal department blocks the deployment.<\/p>\n\n\n\n
Between the GDPR and the gradual rollout of the EU AI Act, relying on foreign endpoints whose Terms of Service reserve the right to retain logs for fine-tuning exposes companies to sanctions and trade secret violations. The AI Act draws a sharp distinction between those who train foundation models from scratch and those who simply serve inference on already validated open-weights models.<\/p>\n\n\n\n
To guarantee full compliance, the inference infrastructure must respect a few iron pillars:<\/p>\n\n\n\n
\n- Data Sovereignty:<\/strong> Servers and data centers located on national and European territory (such as the Italian nodes between Frosinone and Lombardy of Regolo\/Seeweb).<\/li>\n\n\n\n
- Zero Data Retention (ZDR):<\/strong> No persistence of prompts or completions on the infrastructure\u2019s disks. The only traceable data is consumption metadata (tokens consumed, model called, time), needed for billing.<\/li>\n\n\n\n
- Encrypted Prompt Caching:<\/strong> If persistent prefix caching is enabled for performance reasons, it must be encrypted with per-tenant unique keys so as not to break ZDR.<\/li>\n\n\n\n
- ESG Impact & Green Energy:<\/strong> Inference isn\u2019t free for the environment either; using infrastructure powered 100% by renewable sources, with visible metrics on grams of CO2 and milliwatts consumed per prompt, becomes a concrete KPI for sustainability reports.<\/li>\n<\/ul>\n\n\n\n
From Cathedral to Construction Site: The Sub-Agent Architecture<\/h3>\n\n\n\n
Sending an entire code review or architectural analysis task to a giant \u201cgeneralist\u201d model generates unsustainable bills and watered-down answers.<\/p>\n\n\n\n
The most efficient engineering approach is to split roles across specialized models coordinated asynchronously:<\/p>\n\n\n\n
\u250c\u2500\u2500> [Sub-Agent 1: Mistral] \u2500\u2500> Critical analysis & bug finding\n[Orchestrator: GLM] \u2500\u253c\u2500\u2500> [Sub-Agent 2: Gemma] \u2500\u2500> UI\/UX & frontend code\n \u2514\u2500\u2500> [Sub-Agent 3: Qwen] \u2500\u2500> Specific task execution \/ core logic<\/code><\/pre>\n\n\n\n\n- GLM (e.g. GLM-5) as Orchestrator:<\/strong> Larger models handle a wide context without losing focus, plan the activities, query the sub-agents and synthesize the results.<\/li>\n\n\n\n
- Mistral (Large) as Adversarial Reviewer:<\/strong> A European model with strong analytical and multilingual skills, particularly inclined to spot logical flaws, edge cases and architectural violations.<\/li>\n\n\n\n
- Qwen (family) as Task Executor:<\/strong> Chinese models with an excellent cost\/token ratio in pure code execution. Given a bounded perimeter (\u201cdo this refactoring\u201d), they execute it cleanly. They shouldn\u2019t be used as open-ended orchestrators, though, or you risk structural hallucinations.<\/li>\n\n\n\n
- Gemma (31B) as Frontend & UI\/UX Specialist:<\/strong> An agile and effective model when it comes to laying out components, polishing interfaces (for example with Qt or HTML) and understanding visual logic.<\/li>\n<\/ul>\n\n\n\n
Context Engineering: The \u201cLinus Torvalds\u201d Case Study on Antirez\u2019s smallchat<\/h3>\n\n\n\n
An LLM released a year ago is a model that doesn\u2019t know the latest patches, the new framework releases or your repository\u2019s internal practices. How do you avoid the prohibitive cost of continually fine-tuning on hardware? With modular Skills injected into the context as specialized Markdown files.<\/p>\n\n\n\n
An empirical experiment by Daniele Scasciafratte illustrates how it works in practice:<\/p>\n\n\n\n
\n- A pipeline was built to process 38,000 emails written by Linus Torvalds on the Linux Kernel Mailing List over twenty years.<\/li>\n\n\n\n
- From this corpus, Torvalds\u2019 code review heuristics were extracted and synthesized into a reusable Markdown \u201cSkill.\u201d<\/li>\n\n\n\n
- The skill was then applied to the code review of smallchat, a C++ project by Salvatore Sanfilippo (Antirez) made up of about ten files.<\/li>\n<\/ul>\n\n\n\n
The benchmark results speak for themselves:<\/p>\n\n\n\n
\n- Without the Skill:<\/strong> The models analyze the codebase with standard heuristics, finding superficial anomalies.<\/li>\n\n\n\n
- With the Skill injected:<\/strong> Models like Mistral highlighted unique and precise vulnerabilities in memory management and race conditions, surpassing in depth of analysis even larger models that lacked the specific context.<\/li>\n\n\n\n
- The monolithic model paradox:<\/strong> Huge models like GLM recorded fewer anomalies when constrained by the skill compared to more aggressive intermediate runtimes like Mistral, showing that the biggest model is not automatically the best for vertical tasks.<\/li>\n<\/ul>\n\n\n\n
Developer ROI: How to Implement It From Monday<\/h3>\n\n\n\n
To put this strategy into practice without rewriting the entire application codebase, the starting point is decoupling:<\/p>\n\n\n\n