How to fine-tune an open model for your business

Three different things people keep confusing

When someone says train the model for us, they usually mean one of three quite different jobs. The cost and timeline differ by multiples, so it is worth settling which one before the project starts.

The first is the prompt, the system instruction. You tell the model who it is, how to answer, what to avoid and what format to produce. It changes instantly, costs little, and a surprisingly large share of tasks is solved right here.

The second is search over your own documents, commonly called RAG. The model does not memorize your materials; it receives the relevant fragments from your knowledge base at the moment of answering: policies, price lists, instructions, customer history. The model stays ordinary, only what is handed to it changes.

The third is fine-tuning proper, where you take examples of correct answers and change the model itself. It starts answering in the required tone and format without lengthy instructions, at the price of a training cycle, a separate model version and the duty to maintain all of it.

Why a knowledge base usually comes first

The main reason is simple: fine-tuning is a poor fit for knowledge that changes. If prices, delivery terms or internal policies are updated monthly, baking them into the weights means retraining every time. Document search is updated by replacing a file.

The second reason is verifiability. When answers come from a knowledge base, you can see which document they came from. You can show the source to the user, investigate an error and find exactly which passage misled the model. A fine-tuned model answers from the inside, and tracing an error is considerably harder.

The third reason is the cost of entry. Setting up document search is usually faster than assembling a good training set. It also reveals which materials the company does not have and which are written so badly that even a human cannot follow them.

A sensible order looks like this: instruction first, knowledge base second, fine-tuning only for whatever the first two did not cover. What remains is usually tone, format, narrow terminology and specific behavioral rules.

LoRA and QLoRA in plain language

Full fine-tuning means changing every number inside the model, and there are billions of them. That is slow and expensive. LoRA solves the problem differently: the model itself stays untouched while a small add-on is trained alongside it, a thin layer of corrections attached to the model at run time.

That add-on takes vastly less space than the model and trains many times faster. It can be stored separately, switched on and off, kept in several variants for different tasks and swapped without replacing the base model. For a business the appeal is that rolling back to the previous behavior takes nothing complicated.

QLoRA is the same idea plus compression of the base model during training, so it fits into less GPU memory. Quality drops slightly compared with uncompressed training, but training becomes feasible on a single card instead of a cluster.

The overwhelming majority of business projects take this route. It is cheaper, more reversible and easier to maintain, and on typical tasks such as tone and format the quality gap is rarely decisive.

Full fine-tuning: when it is justified

Full fine-tuning makes sense when the task is not about tone and format but effectively requires a different model. For instance, a narrow domain with its own language where a general model struggles even to parse the question. Or a rare language or dialect barely represented in the base model. Or a shift in modality, where the model is taught to work with a data type it was not built for.

The second case is a very large volume of proprietary data. Once you are dealing with millions of quality examples, a lightweight add-on stops holding everything you want to convey.

The price tag is different: a GPU cluster or rented capacity, people who have run such training before, and budget for several failed attempts, because it almost never works on the first pass.

For most companies the honest answer is that full fine-tuning is not needed. If it feels necessary, first confirm that the instruction, the knowledge base and the lightweight add-on are genuinely exhausted.

How much data you actually need

The usual expectation is that fine-tuning requires tens of thousands of examples. In practice, for typical tasks such as answer style, format and rule compliance, several hundred well-chosen examples are often enough.

The reason is that the model already knows how to speak, reason and understand questions. You are not teaching it language; you are showing how answers should look in your specific situation. That calls for precision rather than volume.

Hence the main rule: a thousand bad examples are worse than a hundred good ones. A bad example is an answer that is substantively wrong, an answer in the wrong format, an answer your company considers incorrect, or simply an awkward phrasing that slipped in from real correspondence. The model cannot tell good from bad; it reproduces what it sees and repeats your mistakes with confidence.

Data collection is therefore editorial work, not technical work. Examples should be selected by someone who knows which answer is right and can explain why. Usually that is a department head or your strongest specialist, not a contractor.

What the data looks like in practice

The format of a training set is simpler than people expect. At its core are pairs: an input and the correct answer. The input is a customer question, a task, a document or some combination. The answer is what you consider exemplary.

The usual sources for such pairs:

• Real customer correspondence, with the good answers selected and the poor ones removed.
• An existing library of standard questions and approved answers, if the team already kept one.
• Annotated documents, when the task is field extraction: a contract plus a table of what must be pulled out of it.
• Examples written specifically for rare but important situations that barely appear in correspondence.
• Examples of how not to answer, if the model is being taught to decline and hand the conversation to a human.

Spend separate effort on variety. If nine examples out of ten cover the same topic, the model will conclude that this is the job and start steering every conversation in that direction.

One more thing: personal information should be stripped from the data before training, not afterwards.

Hardware and time

As a rough guide, a lightweight LoRA fine-tune for a model of roughly up to eight billion parameters is genuinely doable on a single GPU with enough memory. That fits a short rental or a machine you already own.

Mid-sized models require a stronger card or QLoRA-style compression. Large models, heavy even to run, require several cards or a rented cluster, and that is a different budget conversation.

Training time on a small set is usually measured in hours, not weeks. Most of the project goes into preparing data and evaluating results, not into training. Plan accordingly, because the idea of training over a weekend breaks precisely on example collection.

Running a finished model has its own, generally lighter requirements. A compressed model of the right size runs comfortably on a single card, and that is what makes on-premise deployment realistic for an ordinary company.

How to tell whether it actually improved

The most common project mistake sounds like this: we looked at a dozen answers, liked them, and called it a success. The person who built the model sees in the answers what they hoped to see, and that applies to everyone without exception.

An honest minimum check has three parts. The first is a held-out set: examples the model never saw during training, set aside in advance and used only for evaluation. If the evaluation data took part in training, the result means nothing.

The second is blind comparison. Answers from the old and new versions are shown to an expert shuffled together, with no labels. The person picks the better answer, and only then is the source revealed. This removes the expectation effect and often produces unpleasant but useful results.

The third is review by the person responsible for substance rather than for the technology. A metric can rise while the answers become formally correct and useless to the customer.

Separately, check that nothing that used to work is now broken. Keep a small fixed set of ordinary questions and run it on every update.

Common mistakes

What shows up most often in projects, regardless of industry.

• Fine-tuning instead of instructing. A task solvable with one paragraph in the system prompt is attacked with a training cycle. Test the prompt before collecting data.
• Knowledge baked into the weights. Prices and policies change while the model keeps confidently quoting last year's terms. Changing facts belong in the knowledge base, not in the model.
• Personal data in the training set. Phone numbers, addresses and amounts travel from correspondence into training. Removing them from a trained model is far harder than cleaning them beforehand.
• Forgetting general skills. A model trained only on short templated replies often stops handling ordinary questions. The cure is a varied data set and careful training settings.
• No versioning. Nobody remembers which version is live, what data it was trained on or how to roll back. Every version needs a number, a description of its data and a way back.
• Training without a goal. If nobody stated in advance what should change, the result cannot be assessed and the project ends in an argument about taste.

The legal side in general terms

A training set is company data, and the same rules apply to it as to any other data. If the examples include customer correspondence, they almost certainly contain personal data, and whether it may be used for training is not a decision for the technical team.

Broadly, the picture is this: anonymization removes part of the problem but not all of it, and each company has its own wording on processing purposes and consent. Regulation in Russia and in the UAE differs, and a project operating in both usually needs two separate answers.

A practical approach that saves time: before collecting data, write down where the examples come from, what fields they contain and what exactly gets stripped out, then show that description to a lawyer and to whoever owns company policy. It is a short document, but it is the one that answers the awkward questions later.

The rights to the base model belong here too: a fine-tuned version generally stays bound by the original license, including distribution limits. That is covered in the separate guide on licenses.

What this ends up looking like

The working setup most projects arrive at is not a single fine-tuned model but a small system of several parts.

The model runs on your own server or on a server rented by the company, so data never leaves for external services. Next to it sits a knowledge base updated without retraining: drop in a new policy document and the answers change. Above that lies the system instruction that defines behavior and boundaries. And, if it turned out to be necessary, a lightweight fine-tuned add-on handles tone, format and narrow terminology.

All of it comes with a written routine: who owns the content of the knowledge base, how often answers are reviewed, what happens when an error is found, who approves a model update and how to roll back. Without that part the system degrades invisibly: materials go stale, answers drift, and one day a customer notices first.

This arrangement is duller than the idea of training a neural network just for us, but the company can maintain it in house and it survives a change of contractor, which is usually the real goal of the project.