LEAF Advisory · Confidential AI
AI on infrastructure you control.
Some workloads cannot send sensitive data to shared third-party AI services. LEAF works with your IT and security teams to define the boundary, evaluate model and infrastructure options, and design private AI that fits your existing identity, network and operating environment.
The objective is not isolation for its own sake. It is to keep the right workloads inside the right boundary while making the trade-offs in capability, hardware, operations and governance explicit.
What Confidential AI Is
Private inference is an architecture decision.
Confidential AI is private AI architecture for organizations that need tighter control over where models run and where data is processed. Data, prompts, outputs, embeddings, caches and logs should stay inside boundaries deliberately defined by the organization.
It is an Advisory capability: LEAF works through the decision with your IT and security teams, with LEAF Labs engineering support where the architecture needs implementation, integration or performance validation.
Data movement is defined explicitly per workload and architecture.
The decision touches
- Infrastructure
- Identity
- Networking
- Storage
- Model lifecycle
- Logging
- Governance
It is not
- A software product
- An appliance
- A security certification
- A compliance solution
- An LLM installed on a server
Start with the Workload
Not every AI workload needs the same boundary.
The deployment model follows the constraints of each workload, not a preference for one kind of infrastructure.
Sensitivity
What happens if prompts, documents or outputs are exposed.
Data classification
Which classes of data the workload reads, creates and stores.
Latency
How quickly an answer or action is needed in the real workflow.
Connectivity
Whether the environment is connected, restricted or isolated.
Regulatory requirements
Residency, retention, access or sector obligations that apply to the data.
Model capability
How much quality the task actually needs, measured on real examples.
Number of users
Who uses it, how many at the same time and how often.
Integration
Which identity provider, data sources and systems it has to work with.
Operational ownership
Who will run, update and support it after the pilot.
The order of decisions
- Workload
- Data classification
- Trust boundary
- Model choice
- Deployment model
- Hardware sizing
Hardware comes last: it is sized from the decisions before it.
Trust Boundary Architecture
Define the boundary first. Design the AI inside it.
The architecture has to show what crosses the boundary and what does not: identity, the gateway, the models, retrieval, tools, logs and the few connections that are allowed out.
- Flow inside the boundary
- Crosses only when policy permits
- Unapproved flow, stopped at the boundary
Can cross, when permitted
- Non-sensitive requests to approved external services
- Approved models, dependencies and updates, through a controlled import
- Operational alerts defined by policy
Designed to stay inside
- Confidential prompts, documents and outputs
- Embeddings, indexes and caches
- Logs and audit trails
- Credentials and tool permissions
Deployment Models
Four ways to draw the boundary.
Each model moves responsibilities, not just servers. None is safer by default: the controls around it decide that.
On-premises
Models and supporting services run on infrastructure operated within the organization's own environment and integrated with its network, identity and operational processes.
Not automatically safer: hardware, patching and operations move to your teams or a partner.
Private cloud
Dedicated cloud infrastructure can keep workloads within an environment and region controlled by the organization while retaining cloud operational models.
A cloud provider still operates the underlying infrastructure; the boundary is set by tenancy, region, network and contract.
Isolated / air-gapped
Environments with no external connectivity, where models and everything they depend on enter through controlled transfer procedures.
Not simply more secure: isolation adds requirements around
- Model import
- Dependency management
- Updates
- Security patches
- Evaluation datasets
- Monitoring
- Transfer procedures
Hybrid with explicit boundaries
Confidential workloads use private models, while permitted non-sensitive workloads can use external services, with explicit rules on which data may cross.
Often the most realistic architecture, and the one that needs the clearest rules.
Hybrid, Made Explicit
Privacy does not require treating every workload the same.
A hybrid architecture routes each request by rule. Confidential workloads stay on private models inside the boundary; non-sensitive workloads can use an external service where its terms are acceptable for that use.
The routing rules are part of the architecture: written down, reviewed with your security team and logged when they are applied. This is often more realistic than an all-or-nothing choice.
Boundary
Unclassified data can default to the stricter path.
Deployment Comparison
Qualitative trade-offs, not a score.
An orientation for the first conversation. The answer for a specific workload comes from the assessment.
| Dimension | On-premises | Private cloud | Air-gapped | Hybrid |
|---|---|---|---|---|
| External connectivity | Set by your network policy | Provider network, restricted by your configuration | None; controlled transfer only | Permitted per workload by routing rules |
| Operational burden | High: your team or a partner runs hardware and stack | Medium: the provider runs hardware, you run the stack | Highest: every change moves by procedure | Medium to high: two paths to run and govern |
| Model flexibility | Hostable models, limited by your hardware | Hostable models, hardware can scale with the provider | Models imported and approved in advance | Broadest: private models plus permitted external services |
| Data boundary | Your own environment | Your tenancy and region, on provider infrastructure | Isolated environment | Differs per workload, set by explicit rules |
| Update complexity | Moderate | Moderate | High | Moderate, on two paths |
| Typical fit | Existing data-center operations, sensitive internal data | Cloud operating model with residency or tenancy needs | Environments that must stay disconnected | Portfolios mixing sensitive and non-sensitive workloads |
Qualitative labels only. No deployment model ranks above the others in general.
The Model Follows the Task
Private does not mean choosing a model by parameter count.
Candidate models are assessed on what the workload needs, then tested on it.
Models available for private hosting may differ in capability from the largest proprietary cloud models.
Candidate models should be evaluated on the organization's own representative tasks before hardware or architecture is finalized.
Assessed on
- Task quality
- Language
- Context length
- Latency
- Concurrency
- Memory
- License
- Quantization
- Serving stack
- Hardware fit
More than language models
- Language
- Embedding
- Reranking
- Vision
- Speech
- Task-specific models
Confidential AI is not a synonym for a private chatbot: search, classification, document processing and vision can all run inside the boundary.
Hardware Sizing
Measure the workload before buying the infrastructure.
Sizing combines variables that are only known once the workload and the model are clear. Generic users-per-GPU figures hide all of them.
- Model
- Context
- Concurrency
- Latency target
- Memory
- Sizing
Measured in a pilot on representative load, not estimated from a datasheet.
Hardware LEAF has worked with in engagements
Evidence of range, not recommended Confidential AI hardware. Every engagement is sized on its own workload.
Compact and edge
Raspberry Pi 5 + Hailo accelerator
On-device camera and video inference, keeping visual data on site.
Team and department
NVIDIA DGX Spark (GB10)
Private inference, retrieval and model adaptation for a team.
Data center
NVIDIA servers with dual H200 GPUs
Larger models, more concurrent users and heavier document workloads.
Security & Governance
Private infrastructure still needs identity.
Running models privately changes where the controls live, not whether they are needed. Six areas are worked through with your security and IT teams.
Identity & access
Who can use which model, for which use case and on which data sources, through your existing identity provider.
Auditability
Requests, administrative actions and model changes logged and retained according to your policy.
Data residency
An inventory of where data is stored and processed, not only where the model runs.
Network & serving security
Segmentation, encryption, secrets management and hardening of the serving endpoints.
Model supply chain
Verified sources, license review, version tracking and a controlled path for new models and updates.
Enterprise governance
An inventory of AI use cases, owners and approvals that fits existing risk and change processes.
Identity covers
- Your existing identity provider
- Roles and groups
- Service identities
- Permissions by use case
- Permissions by data source
- Administrative privileges
Permissions are mapped and tested source by source. Automatic propagation across every system is not assumed.
Where data lives
- Prompts
- Outputs
- Documents
- Embeddings
- Vector indexes
- Caches
- Logs
- Backups
- Temporary files
- Model weights
Where the model runs is only part of data residency.
What is observable
- Requests
- Administrative actions
- Tool calls where relevant
- Model and version
- Deployment changes
- Errors
- Usage metadata
Logs record observable events, not a model's hidden reasoning. Retention follows your policy.
Compliance stays with the organization
Running AI on your infrastructure does not make the organization compliant.
LEAF does not present Confidential AI as holding security or compliance certifications.
Architectures can be designed to support requirements arising from data protection, security, residency or sector-specific obligations. Compliance is assessed by the organization and its auditors across processes, people and controls.
Design for the Threats That Matter
“Private” is not a threat model.
Private deployment removes some risks and leaves others in place. The relevant ones are assessed for each architecture.
Examples of risks to assess, not a complete security assessment checklist.
Unauthorized access
Users or services reaching models and data they should not reach.
Data exfiltration
Sensitive content leaving through outputs, logs, tools or integrations.
Compromised credentials
Stolen accounts or keys used against the gateway or the data sources.
Model and dependency supply chain
Tampered weights, images or libraries entering the environment.
Exposed serving endpoints
Inference APIs reachable from networks that should not reach them.
Inappropriate tool permissions
Models or agents allowed to act beyond what the use case needs.
Unapproved data flows
Connections to external services that nobody decided to allow.
Model Supply Chain
A model that runs locally still has to be trusted.
Open weights can be inspected and hosted, but running locally does not make a model, its dependencies or its serving image trustworthy by default.
- Model source
- License
- Weights
- Dependencies
- Serving image
- Version
- Checksums / signatures where appropriate
- Approval
- Update procedure
- Rollback
Every new model or update follows the same path, air-gapped environments included.
How We Work
Scout. Connect. Unlock.
The LEAF method, applied with your IT, security and compliance teams.
01/ Scout
Map the boundary
Map workloads, data classification, requirements, threat model and existing infrastructure.
02/ Connect
Co-design and validate
Co-design the architecture and validate model and hardware choices through a bounded pilot.
03/ Unlock
Operate and adopt
Integrate operating procedures, monitoring and adoption into the organization's environment.
Confidential AI is an Advisory capability. LEAF Labs provides engineering support where the architecture needs implementation, integration or performance validation.
Where it helps, LEAF Academy supports operators, administrators, users and governance teams. Not every Confidential AI project includes training.
Ownership is agreed per project
Every area has an owner before the pilot ends. There is no universal assignment.
Areas
- Architecture
- Model lifecycle
- Infrastructure
- Identity
- Security operations
- Updates
- Monitoring
- Application support
Each one sits with
- Customer
- LEAF
- Customer partner
- Shared
Validate Before Standardizing
A private model still has to perform the task.
Each workload is measured on representative tasks and load before the architecture becomes a standard.
Answer and task quality
Correct, useful results on the organization's own examples.
Retrieval quality
Where retrieval is used, the right passages found and ranked.
Latency
Response time under the conditions of the real workflow.
Throughput
Requests or documents processed in a given period.
Memory
Model, context and cache footprint on the target hardware.
Concurrency
Behavior with the expected number of simultaneous users.
Refusal and failure behavior
What happens with out-of-scope, unanswerable or malformed requests.
Integration
Identity, data sources and tools working end to end.
Operational stability
Restarts, updates and monitoring behaving as planned.
Trade-offs made visible
- Quality
- Latency
- Hardware cost
- Operational complexity
- Model choice
- Update cadence
Private architecture is not unambiguously better than a public API. It trades some of these for control, and the trade has to be worth it.
Cost depends on workload volume, hardware utilization, model choice, operations and the external services being compared.
Where It Fits
Where a private boundary is often justified.
Sensitive enterprise knowledge
Internal documents, contracts, research or designs that should not be processed by shared services.
Regulated or residency-constrained data
Workloads where data protection, residency or sector obligations shape where processing can happen.
Isolated operational environments
Plants, laboratories or networks that are restricted or disconnected by design.
Private RAG and enterprise assistants
Retrieval and assistants over internal knowledge, running inside the same boundary.
Workloads requiring tighter infrastructure control
Cases where the organization needs to decide models, versions, logs and update timing itself.
Illustrative patterns, not a list of completed projects.
When It May Not Fit
Private deployment is not always the better architecture.
Part of the assessment is saying so when a private boundary is not justified.
Low-volume experimentation
Early exploration rarely justifies dedicated infrastructure.
Already acceptable under provider terms
If an external provider's terms meet the requirements for that data, a private stack can add cost without adding control that matters.
Frontier proprietary capabilities needed
Some tasks need capabilities that privately hostable models do not match yet.
Insufficient operational capacity
Without people to run, patch and monitor it, a private stack becomes a risk of its own.
Hardware investment not justified
Usage may be too low or too uncertain to justify the purchase.
Explicit Limits
What private AI does not solve on its own.
The objective is controlled architecture, not maximum isolation.
- Private deployment does not guarantee compliance.
- Private deployment does not automatically make a system secure.
- Self-hosted models may underperform leading proprietary alternatives.
- The organization takes on more lifecycle and infrastructure responsibility.
- Air-gapped environments complicate updates and monitoring.
- Hardware sizing based on estimates alone is unreliable.
- Local hosting does not fix poor application permissions or governance.
Start with the Boundary
Which data and workloads need to stay under your control?
Show us the workload, the data it uses, the systems around it and the constraints your teams have to respect. We can assess whether private AI is justified and what architecture should be validated first.