Anatomy of an enterprise AI assistant: the 7 layers that separate a demo from production

When a company says "we want to implement artificial intelligence", it is almost always picturing the top layer: an assistant that answers by text or by voice. What stays out of view is that the assistant is the tip of a seven-layer stack, and that 80% of the cost, the risk and the implementation time lives in the six layers underneath.

At SUMāTO we have spent 9 years integrating technology across Latin America, and the pattern repeats: AI projects do not fail at the model — they fail at the infrastructure, the data or the integration. So here we run the full architecture exercise: if we had to build an enterprise AI assistant with voice from scratch today, which layers would we need, how does each one work under the hood, and which official services from which vendors would we use?

This is the map. Each layer includes its technical definition, its internal components, the metrics it is measured by, and references to the vendors' official documentation.

The full stack, bottom to top

Layer 7 — Presentation     TTS, voice and text channels
Layer 6 — Interaction      STT/ASR, endpointing, VAD
Layer 5 — Orchestration    Agents, RAG, function calling, guardrails
Layer 4 — Model            LLM: inference, context, fine-tuning
Layer 3 — Data             Ingestion, chunking, embeddings, vector DB
Layer 2 — Platform         Kubernetes, APIs, LLMOps, observability
Layer 1 — Infrastructure   GPU, network, storage
─────────────────────────────────────────────────────
Cross-cutting              Security, governance, compliance

Each layer solves a different problem, is contracted with different vendors, is measured with different metrics and fails in different ways. Let's take them one at a time.

Layer 1 — Infrastructure: where everything runs

Definition. This is the compute, network and storage foundation the rest of the stack runs on. What sets it apart from traditional IT infrastructure comes down to two words: accelerated compute. An LLM at inference time executes billions of matrix multiplications for every token it generates; that workload is massively parallel and conventional CPUs cannot serve it at usable latency. Hence the reference hardware: data center GPUs (NVIDIA H100/H200/B200 and the Blackwell family) or cloud-proprietary accelerators (AWS Trainium/Inferentia, Google TPU).

The technical concepts that define this layer:

• Training vs. inference. Training a foundation model takes clusters of thousands of GPUs running for weeks — that is AI-lab territory, not enterprise territory. The enterprise consumes inference: serving an already-trained model. Every enterprise architecture decision is designed around inference.
• GPU memory (VRAM) as the dominant constraint. A 70-billion-parameter model at FP16 precision needs roughly 140 GB just to load the weights, plus the KV-cache memory that grows with context length. This determines how many GPUs you need, and it explains techniques like quantization (reducing weights to 8 or 4 bits, INT8/FP8/INT4) that cut memory and cost with minimal quality loss.
• Network latency. In a voice assistant, the total latency budget per turn is around 800 ms for the conversation to feel natural. Every network hop between the telephony channel, the STT engine, the LLM and the TTS eats into that budget; that is why the cloud region is chosen by proximity to the user, not by price.
• Object storage (S3, Azure Blob) for raw data, artifacts and model weights, which run from tens to hundreds of GB per version.

Official services we would use:

The consulting call: almost no company in LATAM should buy its own GPUs to get started. The right model is to consume inference as a service (layer 4) and reserve dedicated infrastructure only when volume justifies it — typically above several million interactions per month, or when regulation requires that data never leave a defined perimeter.

Layer 2 — Platform: the project's operating system

Definition. This is the engineering layer that turns raw infrastructure into an operable, auditable environment: containers, APIs, deployment pipelines and the discipline of LLMOps — MLOps extended to language models, where the artifact you version is no longer just the model but also the prompts, the RAG configurations and the evaluation sets.

Internal components:

• Container orchestration. Kubernetes is the de facto standard; in the cloud it is consumed managed as Amazon EKS or Azure AKS. The assistant's microservices (gateway, orchestrator, connectors, evaluators) are deployed, scaled and updated here.
• Inference server. If the model is self-hosted, you do not serve it by hand: you use an optimized engine like vLLM, NVIDIA Triton Inference Server or TensorRT-LLM, which implement techniques such as continuous batching and paged attention to multiply throughput per GPU by 5x to 20x over a naive implementation.
• API gateway and traffic management. A single entry point that authenticates, applies rate limiting, routes between model versions and enables canary deployments: sending 5% of traffic to the new version before promoting it.
• Classic observability + LLM observability. On top of the three traditional pillars (logs, metrics, traces) sits the semantic trace: which prompt came in, what context RAG retrieved, what the model answered, what it cost in tokens and what score it earned. Reference tooling: LangSmith, MLflow, Arize Phoenix.
• Continuous evaluation (evals). Test-case suites that run on every prompt or model change, with automated judges (LLM-as-judge) and human sampling. Without this, every "improvement" is a bet.

Official services we would use: Amazon SageMaker AI or Azure Machine Learning for the full lifecycle; Datadog or Grafana for platform observability.

The consulting call: this layer is invisible to the business and therefore chronically underestimated. Without LLMOps, there is no way to answer with data the question the executive committee will ask in month three: "is the assistant answering well, and better than last month?"

Layer 3 — Data: the raw material of knowledge

Definition. This is the layer that turns company knowledge (documents, policies, catalogs, system histories) into a format the model can query. The central mechanism is the embedding: a function that transforms text into a vector of hundreds or thousands of dimensions (typically 384 to 3,072) whose position in space captures meaning. Two texts that say the same thing with different words land close together; search stops being keyword-based and becomes semantic search.

The full pipeline, step by step:

1. Ingestion. Connectors to the source systems: ERP, CRM, SharePoint, document repositories. For scanned documents or complex PDFs, extraction runs through OCR and document intelligence: Amazon Textract or Azure AI Document Intelligence.
2. Chunking. Documents are split into fragments of 200 to 1,000 tokens. Strategy matters: segmenting by semantic structure (sections, overlapping paragraphs) retrieves better than cutting at a fixed size. It is one of the highest-impact decisions on final quality, and one of the least discussed.
3. Embedding generation. Each fragment passes through an embedding model — a different model from the conversational LLM. Official options: Amazon Titan Text Embeddings on Bedrock, OpenAI's text-embedding-3, Cohere Embed, with strong multilingual performance for Spanish.
4. Vector database indexing. Vectors are stored in an index optimized for approximate nearest neighbor search (ANN, typically using the HNSW algorithm), which responds in milliseconds across millions of vectors. Options: Pinecone, Amazon OpenSearch Service with k-NN, Azure AI Search, pgvector on PostgreSQL for moderate volumes.
5. Hybrid retrieval and reranking. Production systems combine vector search with traditional lexical search (BM25) and apply a reranker — a second model that reorders the initial 50 candidates and hands the top 5 to the LLM. Cohere Rerank and the Azure AI Search rerankers are the managed references.
6. Synchronization. The pipeline re-runs whenever the knowledge changes. An assistant working off six-month-old data is a liability, not an asset.

Metrics for this layer: recall@k (is the right fragment among the k retrieved?), retrieval precision, and index freshness.

The consulting call: this is where the assistant's real quality is decided. A state-of-the-art LLM with badly chunked data answers worse than an average model with a well-tuned retrieval pipeline. It is the highest-return investment per dollar in the entire stack, and it is why an orderly data analytics foundation is the real prerequisite for any AI project.

Layer 4 — Model: the language engine (LLM)

Definition. The LLM (Large Language Model) is a language model built on the Transformer architecture, pretrained on trillions of text tokens for a deceptively simple task — predicting the next token — from which capabilities in comprehension, reasoning and generation emerge. Three concepts define its behavior in production:

• Tokens. The unit of processing: fragments of roughly 4 characters on average. Everything is billed, measured and capped in tokens. A typical customer service conversation consumes between 2,000 and 10,000 tokens per turn once you count the context.
• Context window. How much text the model can "hold in mind" in a single request — today between 128 thousand and over a million tokens depending on the model. It defines how much conversation history and how much retrieved knowledge fits into each turn.
• Temperature and sampling. Parameters that control the balance between deterministic answers (for regulated processes) and creative ones.

The three mechanisms for adapting a model to the business, in increasing order of cost:

1. Prompt engineering. System instructions, examples and output format on every request. Low cost, iteration in hours.
2. RAG (Retrieval-Augmented Generation). The model queries layer 3 before answering and generates on the basis of cited fragments of company knowledge. It reduces hallucinations — plausible but false answers — because it anchors generation in verifiable sources, without retraining anything.
3. Fine-tuning. Partial retraining with your own data (techniques like LoRA adjust a tiny fraction of the parameters). It is justified only for highly repetitive tasks, strict output formats or extremely specialized vocabulary — and only after exhausting 1 and 2.

Official services we would use:

The core trade-off: managed API = top quality and zero operations, paying per token and accepting that data transits through the provider (with the contractual and technical controls that demands). Self-hosted open model = full control and data residency, in exchange for operating layers 1 and 2 end to end. Multi-model platforms (Bedrock, Foundry) are the pragmatic middle ground: one contract, several models, centralized governance.

The consulting call: the right question is not "which is the best model?" but "which is the best model for this use case, at this cost per interaction, with these data requirements?". That is why the correct design is model-agnostic: an abstraction layer that lets you switch providers without rewriting the system. LLM pricing and capabilities change every quarter; the architecture should not.

Layer 5 — Orchestration: from model to agent

Definition. An LLM only generates text. The orchestration layer turns it into an agent: a system that reasons in steps, queries sources, executes actions in external systems and holds the thread of a conversation from start to finish. This is the layer where AI stops answering and starts operating.

Internal components:

• Function calling / tool use. The mechanism by which the model, instead of returning text, emits a structured call to a function defined by the integrator — look up an order in the ERP, book an appointment, open a ticket, execute a payment. The model decides which tool to invoke and with what parameters; the orchestrator executes it and returns the result to the model. It is documented as a native capability in Anthropic, OpenAI and Bedrock.
• Context and memory management. What the agent knows about the current conversation (short-term memory, inside the context window) and about the customer over time (long-term memory, persisted and retrieved via layer 3).
• Multi-step flows and planning. Decomposing "I want to change my plan and have the invoice sent to my new email" into the four operations it actually involves, with error handling and intermediate confirmations.
• Guardrails. Declarative rules that bound what the agent can and cannot do or say: off-limits topics, maximum amounts, actions that require human confirmation, sensitive-data filtering. Managed implementations: Amazon Bedrock Guardrails and Azure AI Content Safety.
• Human-in-the-loop. The explicit design of when the agent escalates to a person — on low confidence, by policy, or at the customer's request. Autonomy without escalation is not maturity: it is risk.

Official frameworks and services: LangChain / LangGraph as the open-source orchestration framework; Amazon Bedrock Agents and the agent services in Microsoft Foundry as the managed option; Model Context Protocol (MCP), the open standard driven by Anthropic for connecting agents to tools and data sources interoperably.

An important architecture decision surfaces here: there are packaged cognitive agent (AI Agent) platforms that solve this layer — and much of layers 3 and 4 — as a product, with the full autonomous cycle: automate, decide, improve. As agnostic integrators, at SUMāTO we assess with each client when it makes sense to build custom orchestration on open frameworks and when it is better to deploy a platform that already ships that cycle ready to connect to the operation.

Layer 6 — Interaction: understanding the human voice (STT)

Definition. When the channel is voice, a critical step comes before the LLM: STT (Speech-to-Text), also called ASR (Automatic Speech Recognition) — the technology that converts the audio signal into text. Modern systems are end-to-end neural models trained on hundreds of thousands of hours of audio, and their quality is measured in WER (Word Error Rate): the percentage of words inserted, deleted or substituted relative to the correct transcript.

Internal components of a production STT:

• Streaming recognition. Transcribing while the person is speaking, emitting partial hypotheses that get corrected on the fly, with latencies of 100–300 ms per fragment. It is an absolute requirement for natural conversation; batch transcription is reserved for recordings.
• VAD (Voice Activity Detection). Detecting when there is human speech in the signal and when there is silence or noise, so you do not process (or pay for) empty audio.
• Endpointing / end-of-turn detection. Deciding when the person has finished speaking — the hardest problem in conversational voice. A short threshold interrupts; a long one produces awkward silences. The most advanced systems use semantic signals, not just acoustic ones: models like Deepgram Flux build turn detection into the recognition model itself, with Spanish support.
• Vocabulary adaptation (keyword boosting). Raising the probability of product names, internal acronyms and regional terms the generic model has never seen.
• Diarization. Distinguishing who is speaking in audio with multiple voices — relevant for recorded calls and conversation analytics.
• Telephony audio handling. Telephony delivers 8 kHz audio (narrowband), with lossy codecs and line noise: conditions very different from a smartphone microphone. The STT model must be evaluated specifically in that domain.

Official services we would use:

The consulting call: STT is where voice projects in LATAM fail quietly. A 15% WER on the client's accent and domain means one word in seven reaches the LLM corrupted — and no model, however good, reasons well over corrupted input. The SUMāTO rule: STT is chosen with your own benchmark on real audio from the client's operation (real calls, real accent, real noise), never with the WER on the vendor's website, which was measured under laboratory conditions.

Layer 7 — Presentation: answering in text and in voice (TTS)

Definition. This is the layer the user ultimately perceives. In text, it is delivery through the channel (web, WhatsApp, application). In voice, it is TTS (Text-to-Speech): the neural synthesis that turns the LLM's answer into speech with natural prosody, pauses and intonation. Today's models generate the waveform directly with generative neural architectures, and the gap versus the "robot voice" of a decade ago is categorical.

Internal components and metrics:

• Audio TTFB (Time To First Byte). How long it takes for the answer to start playing. With streaming synthesis — generating and playing back while the LLM is still writing — the production target is < 300 ms. Models optimized for real time such as ElevenLabs Flash v2.5 operate at around 75 ms of model latency.
• Perceived quality (MOS). Mean Opinion Score, the standard naturalness metric rated by human listeners on a 1–5 scale. Leading neural TTS engines score above 4.5.
• Brand voice. Choosing a catalog voice, designing a synthetic voice, or voice cloning from samples — with the non-negotiable requirement of documented consent from the voice talent.
• Regional Spanish support. Mexican, Colombian and Rioplatense intonation are not the same product. Today's multilingual models handle Spanish at high quality, but you validate it with local ears.
• SSML and prosody control. Standard markup for controlling pauses, emphasis, numbers and spelling — critical for reading out amounts, reference numbers and addresses without ambiguity.
• Channels. Telephony (SIP/PSTN), WhatsApp Business, web chat, mobile apps. Each channel has its own integration, audio formats and business rules.

Official services we would use:

And this is where the stack closes: the channel layer — telephony above all — is what an omnichannel AI contact center platform solves: routing, telephony, WhatsApp and handoff to a human agent when the conversation calls for it. The complete architecture — a cognitive agent as the brain and a channel platform as the contact system — is the concrete example of how these seven layers materialize on top of the client's real operation.

The 2026 architecture debate: cascading pipeline vs. native speech-to-speech

Everything so far describes the cascading architecture: STT → LLM → TTS, three models chained together. It is the dominant architecture in production because each link is chosen, measured and replaced independently — the agnostic design we recommend.

But a second architecture is emerging: native speech-to-speech models, which take audio in and generate audio out directly, with no intermediate text. OpenAI's Realtime API is the commercial reference: natively multimodal models that listen, reason and speak in a single session over WebRTC or WebSocket, with built-in turn detection and function calling mid-conversation. The promise: lower end-to-end latency and preservation of signals that text discards — tone, emotion, hesitation.

Our consulting read:

For regulated operations — banking, healthcare, insurance, government — the cascade remains the recommendation, on auditability and control. Native speech-to-speech is worth a pilot in premium-experience cases where latency is everything. It is exactly the kind of architecture decision we assess case by case.

Cross-cutting layer — Security, governance and compliance

None of the seven layers above goes to production without this one. Definition: the set of technical and organizational controls that guarantee the assistant handles personal data in line with applicable regulation (in Mexico, the LFPDPPP; across the region, equivalent frameworks; for clients with European operations, the AI Act as the reference for risk classification), that conversations are encrypted and audited, and that every automated decision is traceable.

Minimum components:

• Encryption in transit (TLS 1.2+) and at rest, with customer-managed keys (AWS KMS, Azure Key Vault).
• IAM and least privilege: the agent accesses only the functions and data its role requires — the agent's credentials are a first-class security asset.
• PII masking before data ever reaches the LLM when the use case does not require it, plus contractual no-retention and no-training policies with the model providers (Bedrock and Foundry document their enterprise data commitments explicitly).
• Defense against LLM-specific threats: prompt injection, system-instruction leakage, jailbreaks. The public reference is the OWASP Top 10 for LLM Applications.
• Perimeter and network: next-generation firewalls and segmentation — Fortinet territory in our cybersecurity ecosystem — plus native cloud controls (AWS GuardDuty, Microsoft Defender, Microsoft Purview for data governance).
• AI governance as a practice: a use-case registry, per-case risk assessment, a review committee and periodic audits of agent behavior. The public reference framework is the NIST AI Risk Management Framework.

The executive summary of the stack

Conclusion: the technology is new, the discipline to adopt it is not

Every piece of this stack exists as a service, is publicly documented by its vendor, and can be bought with a credit card. What you cannot buy with a credit card is the judgment to choose them, connect them and govern them.

And here it is worth being blunt: there is currently an oversupply of "AI experts" working on a technology where every one of us is still learning — labs, vendors, integrators and clients alike. Models change every quarter, best practices are being written in real time, and a good share of the architectures being proposed today come out of trial and error. That is not a market defect: it is the nature of a technology mid-way up its maturity curve. The defect is dressing up an experiment as certainty.

The good news is that, from a consulting standpoint, the procedure for adopting technology in a corporate environment has not changed in any essential way in recent years — and it is precisely what protects the company when the technology does change every quarter:

1. Understand the business before the technology. The process starts at the problem, not at the model: which process hurts, what that pain costs, what measurable outcome would fix it. A use case without a quantified baseline is not a use case; it is a demo waiting for a budget.

2. Rigorous vendor analysis. Verifiable success stories — with clients you can actually call — vendor financial health, a public roadmap, official documentation (which is why this article cites it layer by layer) and a support model. A vendor that cannot show references comparable to your industry and your scale is still experimenting; that can be a valid partner for a pilot, not for the core of the operation.

3. Demos and proofs of concept with controlled scope. Seeing the technology work on a scenario of your own, with exit criteria defined before you start: what accuracy, what latency, what cost per interaction turn the pilot into a project — and what results kill it. A pilot without cancellation criteria is not a pilot; it is a purchase in disguise.

4. Quality validation before implementation. Benchmarking with real operational data (WER on the real accent, recall on the real documents), security testing, regulatory compliance validation and experience evaluation with real users. In AI this stage carries more weight than ever, because the models are probabilistic: it is not enough that it works in the demo; you have to measure how often and under what conditions it stops working.

5. Gradual implementation with governance from day one. Phased rollout, with a human in the loop, operational metrics and a committee that reviews system behavior — not at the end of the project, but throughout its life.

And two rules that admit no exception, because mistakes on these points are not fixed by a new model release:

• Protect your company's data. Before connecting any AI system, the mandatory question is: where does my data travel, who retains it, is it used to train third-party models, and what does the contract say about it? Serious providers publicly document their no-retention and no-training commitments for enterprise data; if a provider cannot answer this in writing, the conversation ends there. Company information — customers, pricing, contracts, operations — is a strategic asset, not test data.
• Do not integrate with homegrown solutions or dubious implementations. A wrapper thrown together over a weekend on top of a model API can look identical to an enterprise platform in a 30-minute demo. The difference shows up in production: no serious encryption, no audit trail, no support, no roadmap and — the biggest risk — your company's data flowing through infrastructure nobody certified. The cost of undoing a bad integration always exceeds the cost of having evaluated it properly.

That is the job of an agnostic integrator: applying a proven methodology to a technology that is still maturing, without a conflict of interest pushing any one component. At SUMāTO we do not manufacture any of these pieces, and precisely for that reason we can pick the best one for each layer and each client — the right cloud, the right model, the STT that actually understands your operation's accent, and the platform — whether custom-built on open frameworks or packaged — that turns the stack into operational results. Our method — Consulting → Design → Processes → Governance — is exactly the sequence above, systematized.

Want to adopt AI with method instead of hype? The starting point is our 360° Diagnostic: in 90 days we understand your business, map your current architecture against this stack, evaluate vendors with objective criteria and deliver an implementation roadmap prioritized by value and feasibility — with your company's data protected at every step.

Schedule a 360° Diagnostic

Frequently asked questions

How many layers does an enterprise AI assistant really have?
Seven functional layers plus a cross-cutting security and governance layer: infrastructure, platform, data, model (LLM), orchestration, interaction (STT) and presentation (TTS). The assistant the user perceives is only the last one; 80% of the cost, the risk and the implementation time lives in the layers below.

What is the difference between RAG and fine-tuning?
RAG has the model consult company knowledge before answering, without retraining anything, and it reduces hallucinations because it anchors generation in citable sources. Fine-tuning partially retrains the model with your own data and is justified only for highly repetitive tasks, strict output formats or very specialized vocabulary — and only after exhausting prompt engineering and RAG.

Cascading architecture (STT→LLM→TTS) or native speech-to-speech?
For regulated operations — banking, healthcare, insurance, government — the cascade remains the recommendation: each component is chosen, audited and replaced independently, and the text transcript serves as evidence. Native speech-to-speech offers lower latency and is worth a pilot in premium experiences, but it implies more lock-in and traceability that is still being built.

How do you choose an STT engine for Spanish in Latin America?
With your own benchmark on real audio from the operation: real calls, real accent, real noise and 8 kHz telephony audio. Never with the vendor-published WER, measured in the lab. A 15% WER means one word in seven reaches the LLM corrupted, and no model reasons well over corrupted input.