Running a private LLM in-house: infrastructure requirements
Nanda Teknoloji Engineering Team · · 8 min read
In brief
A private LLM means the model and the data run in an environment under the organization's control. This requires a GPU compute layer, model serving software, a vector database and document pipeline, access control and logging, and a team to operate them.
Running an in-house (private) LLM means running a large language model, and the data it processes, in an environment the organization controls. That environment can be your own data center, rented dedicated infrastructure, or a private cloud; the common point is that prompts, documents, and responses do not flow uncontrolled to a third party’s service. Five layers are needed for it to work: GPU compute, model serving software, the data layer that connects enterprise knowledge (RAG), identity and access control, and monitoring and logging. This article explains what is needed at each layer before you make the decision, where surprises are encountered, and when a private approach is unnecessary.
When does a private LLM make sense?
Not every organization needs to run its own model. If several of the following situations apply to you, a private approach should be seriously evaluated:
- It is unacceptable for prompts containing personal data, trade secrets, or regulated data to go to external services.
- Data residency and auditability are mandatory by contract or legislation.
- There is high-volume, continuous usage connected to internal systems (document archive, ticketing system, code repository), and you do not want dependence on external services.
- You want to manage the model’s version, behavior, and update timing yourself.
On the other hand, if usage is infrequent, data sensitivity is low, and the team is not ready to operate infrastructure, a controlled external service and a strong governance framework may be more practical. The management approach in the What is Shadow AI article applies in both cases, to prevent employees from turning to unapproved tools.
Architecture layers: what does a private LLM consist of?
| Layer | Role | Decision points |
|---|---|---|
| Compute (GPU) | Runs model inference | Memory capacity, GPU count, interconnect, rented/owned |
| Model serving | Exposes the model as an API, batches requests | Inference engine, scaling, version management |
| Knowledge layer (RAG) | Adds enterprise documents to the context | Vector database, chunking, permission awareness |
| Control layer | Identity, quota, logging, content inspection | Gateway, identity provider integration, DLP |
| Operations | Monitoring, backup, updates | Observability, incident management, backup |
Each row is a separate work package. Projects usually start from the first row (hardware) and then try to “add” the other four afterward; this is the most common cause of delays and budget overruns.
GPU and compute: how is sizing done?
The model choice determines the hardware. The weights must fit in memory, room must be left for the KV cache, and the number of concurrent users must be calculated. For the detailed method, see our GPU selection for LLMs article. Three practical principles stand out here:
- Start small, measure. Pilot first with a smaller model; if quality is sufficient, the hardware requirement falls noticeably.
- Measure concurrency. A pilot with ten people and a deployment for thousands of employees do not require the same architecture.
- Plan for redundancy. A single GPU server is a single point of failure. For critical use, at least a second node and a failover plan are needed.
Renting dedicated capacity instead of buying hardware provides serious flexibility in the early stage; for this option, see the scope of the GPU as a Service solution.
Knowledge layer: connecting enterprise data with RAG
In most enterprise use cases, you do not need to retrain the model. In the RAG approach, documents are split into chunks, indexed as vectors, and when a question arrives the relevant chunks are given to the model as context. What is decisive here is not the model but the quality of the data pipeline:
- Document quality: Outdated, contradictory, and ownerless documents are the first source of wrong answers.
- Chunking strategy: Chunks that are too small lose context, and ones that are too large add noise.
- Permission awareness: Documents the user cannot already access must not leak into answers. Permission information should be stored when entering the index, and a filter should be applied at query time.
- Update flow: When a source document changes, the index must be updated too.
The permission filter is the most frequently skipped point. A search index opened to everyone’s access can accidentally make human resources or finance documents visible.
Control layer: identity, quota, and data inspection
The model running in your own environment does not solve the security problem; it only moves the boundary. This layer requires:
- Single sign-on and role-based access with the existing identity provider.
- Quotas and rate limits per user, team, and application.
- Logging for prompts and responses; a clear rule on which data is kept and for how long.
- Sensitive data detection and masking; an inspection layer such as AI DLP can be used for this.
- All model calls passing through a single point; the AI Gateway architecture provides this, and you can find the details in the What is an AI Gateway article.
Risks at the application layer (for example prompt injection, over-privileged tools) should be addressed separately. The OWASP Top 10 for LLM Applications list offers a good checklist for these risks.
Personal data and the KVKK dimension
When data stays inside the organization, the risk of external transfer is reduced, but the obligations do not disappear. The processing purpose, access permissions, retention period, personal data in logs, and the application of deletion requests to the index as well must be handled. For the detailed framework, see the AI and KVKK article; for official information and decisions, the KVKK Authority’s website is the primary source. Carry out the legal assessment together with your organization’s legal and data protection officers.
Operations: keeping the model alive in production
A private LLM is not a product installed once but a service that is operated. The following work is continuous:
- Security patches and version updates for the model and the inference engine.
- Monitoring GPU utilization, queue length, latency, and error rates.
- Quality evaluation: regression testing with a fixed question set when versions change.
- Backing up the index, configuration, and logs.
- Incident management: a process for misuse, suspected data leakage, and erroneous model responses.
If this load exceeds the in-house team, it is possible to hand off part of the operation under a managed services model.
Example scenario: an internal document assistant
A legal and compliance unit wants an assistant that can search across hundreds of contracts and internal policies. The documents are confidential, and sending them to an external service is not wanted. The team first tries a mid-size open-weight model on a single GPU server and prepares an evaluation set with around a hundred real questions. Once the results prove sufficient, the document pipeline is built: documents are classified, their owners are labeled, and they enter the index together with permission information.
Two problems appear in the first pilot. The first is that older versions of contracts get mixed with newer versions and produce contradictory answers; the fix is to reflect the document lifecycle in the index. The second is that users do not trust the answers because the source document is not shown; adoption rises when source links are added to answers. For the move to production, the identity provider is integrated, a retention period is written for logs, and all requests are routed through a single gateway. In this scenario the hard part was not the GPU but document quality and permission management.
Deployment model options
Where the private LLM will run is also a decision. There are three basic options:
- In your own data center: Provides full control; power, cooling, hardware procurement, and refresh responsibility are yours.
- Dedicated rented infrastructure: The hardware runs allocated to you at the provider; it reduces the physical operations load, and control is still high.
- Hybrid: The sensitive workload runs in the environment under organizational control, while trials and peak capacity run on rented infrastructure.
Data class, usage profile, in-house team capability, and budget predictability are decisive in the choice. For most organizations, starting with rented capacity and deciding as usage becomes clear is safer than locking into hardware early.
Common mistakes
- Starting with hardware: Buying GPUs before the use case and data sources are clear.
- Skipping permissions in RAG: Content users have no access to appearing in answers.
- Making the logs themselves a risk: Personal data accumulates in prompt logs; if access and retention period are not defined, a new data store is born.
- Not measuring quality: Going live on a “looks good” feeling.
- Leaving employees out: An unusable internal tool sends employees back to external services.
- Unclear ownership: No one assigned as responsible for the model, the index, and the policies.
Decision summary
- Put use cases and data sensitivity in writing; prove the need for private.
- Measure model quality and concurrency needs with a small pilot.
- Plan the five layers (compute, serving, knowledge, control, operations) together.
- Design document quality and the permission filter in RAG from the first version.
- Clarify logging, retention period, and KVKK compliance from the start.
- Identify operational responsibility and the skills gap early.
Moving forward with Nanda Teknoloji
Nanda Teknoloji carries out the discovery, design, migration, and operation steps in projects like these: use case and data classification, sizing GPU capacity, designing the control layer, and continuity. To clarify your environment and data control requirements together, you can explore the AI security and GPU as a Service solution pages.