LLM Engineering · 19 Nov 2026 · 6 min
Understanding LLM Tool Calling
How an application asks a model for a structured tool call, checks the arguments, runs the tool itself, and sends the result back to the model.
RAG · 19 Nov 2026 · 6 min read
How to ingest business documents, retrieve the right passages, and keep the written answer tied to those passages instead of model memory alone.
Retrieval-augmented generation is a search step in front of a model. You split documents into passages, store vectors for those passages, retrieve a handful for the question in front of you, and ask the model to answer from that handful. The model is the last stage. If you treat it as the first stage, you have a chatbot with a search engine you forget to call.
Business documents are a worse fit for a tutorial corpus than people expect. They repeat themselves, they go stale, they use the same heading in six departments, and they often contain data you are not allowed to drop into a prompt log. The design work is metadata, filters, and a rule for what the model may say when retrieval is empty. It is not a leaderboard.
Nothing here is a measured accuracy. A percentage without a test set, a judge, and a date is decoration.
People ask questions that sound like they have one answer and actually have a section, a footnote, and a newer file that replaced both. A model with no retrieval will still answer. That is the failure mode RAG is supposed to reduce, and it comes back the moment retrieval returns the wrong section or no section and the prompt says "be helpful."
A second problem is tenancy. An insurance or operations team is not one pile of PDFs. Department A should not retrieve Department B's procedures because the embedding happened to be nearby. If the index has no department, no source id, and no updated date, you cannot enforce that, and you cannot explain an answer a week later.
A third problem is evaluation theatre. Cosine similarity is not correctness. A high score means the vectors are near each other. It does not mean the passage contains the policy rule the person asked about.
Ingestion is a pipeline with files as the input and passage records as the output. A passage record needs the text, the embedding, and metadata: source id, title, department or other tenant key, and a timestamp you trust. Optional and useful: a heading path, so a chunk knows it came from "Claims / Documents / Accepted files" and not from a floating paragraph.
Chunking is where most quiet bugs start. A fixed window of a few hundred tokens is a reasonable default when the source is prose with no structure. When the source has headings, split on the headings first and only then window a section that is still too long. A chunk that begins mid-sentence will retrieve, and the model will treat the fragment as a rule.
Overlapping windows reduce the chance of cutting a sentence in half. They also duplicate text, so your citations start repeating. Prefer a small overlap, and dedupe exact chunks from the same source before you embed them.
Embeddings have a model id. Store it. If you change embedding models you have to rebuild the index. Querying a new model against an old index does not fail loudly. It fails as slightly worse search, which is the hardest bug to notice.
Retrieval is a filter plus a nearest-neighbor query. Apply the tenant filter first. Then take a small top-k. Then, if you have a second cheap signal such as keyword match on an identifier, use it. Do not start with top-k of 30 and hope the model ignores the noise. The model will use the noise if it makes a smoother paragraph.
The prompt should contain the passages, each with a citation id you minted, and an instruction that is specific: answer only from these passages, quote the citation id, and say you do not know when they do not contain the answer. "Be helpful" fights that instruction. Pick one.
Diagram
An admin ingestion screen is just a way to start that left-hand path and to show which source landed in which namespace. It does not need to be clever. It needs to show failure: a file that did not parse should not look ingested.
At query time the application, not the model, decides the department filter from the signed-in user. Never let the question text choose the tenant. A user who types another department's name is not a reason to drop the filter.
Build ingestion until you can point at a passage and see its source. Only then call a model. If you cannot retrieve the right paragraph with a search box, a paragraph of prompt engineering will not fix the index.
Use one namespace or metadata filter per department, team, or corpus. Pinecone-style namespaces and a local FAISS index with a metadata map are both workable. The hosted service buys you less code. The local index buys you a machine you control. Either way the filter is the feature.
Return citations in the product, not only in the prompt. A citation the user cannot see is a claim you cannot check. The smallest UI is the answer plus the source title and the heading path. A page number or URL belongs there when the parser actually has one. Do not invent a page number the parser did not emit.
When retrieval returns nothing, skip the model or call it with an explicit empty set and a required refusal. Calling it with no passages and a friendly system prompt is how the general-knowledge answer sneaks back in.
For a first evaluation, write twenty questions you know the answer to, from documents you ingested, including five the corpus cannot answer. Run them after every chunking change. A person reads the answers. That is slower than an automatic score and it is the evaluation you can defend. Add an automatic check later for cheap invariants: empty retrieval must not produce a confident answer, every citation id in the reply must be one you supplied.
Re-embed when the source changes. Store a hash of the source and the embedding model id. A nightly job that rewrites the world is worse than an incremental one, but an incremental one that forgets deletes will serve a procedure you removed. Deletes are part of ingestion.
Keep document text out of debug logs at the same level as application logs. Retrieval systems become a second copy of the corpus. Treat the index with the same access rules as the files.
This Python is a sketch of the record and the query, not a client for a specific vector database. The embed and search functions are yours.
from dataclasses import dataclass
@dataclass
class Passage:
id: str
text: str
source_id: str
department: str
heading: str
def build_prompt(question: str, passages: list[Passage]) -> str:
if not passages:
return (
"The library has no matching passages. "
"Say that you do not know. Do not answer from general knowledge.\n\n"
f"Question: {question}"
)
blocks = []
for passage in passages:
blocks.append(
f"[{passage.id}] {passage.heading}\n{passage.text}"
)
joined = "\n\n".join(blocks)
return (
"Answer only from the passages. Cite ids in brackets. "
"If they do not contain the answer, say you do not know.\n\n"
f"{joined}\n\nQuestion: {question}"
)
def answer(question: str, department: str) -> str:
passages = search(embed(question), department=department, limit=5)
return complete(build_prompt(question, passages))search must apply department in the index, not as a substring check after you have already fetched a global top-k. Filtering after the fact hides the passages that were merely ranked sixth inside the right department.
A TypeScript service has the same two functions. The type is the point: a passage without a department cannot be inserted.
Most RAG bugs are ingestion bugs. The model is an easy thing to blame because its mistakes are sentences. When the heading was dropped on the way into the index, the sentence was never going to be right.
Metadata you do not filter on is a comment. If department matters, the query must require it. If time matters, the query must know which timestamp wins. Write those rules down next to the index, not in a slide.
Refuse cleanly. Users trust a system that says the document does not say, more than they trust a system that is usually eloquent and sometimes invents a clause.
A RAG application is a maintained index plus a strict prompt. Parse, chunk on structure where you have it, embed with a model id you store, filter by tenant before you rank, and show citations. Evaluate with questions a person can judge, including questions the corpus cannot answer. The model is allowed to write the reply. It is not allowed to supply the facts.
Tool calling is the same idea with a function instead of a passage: the application fetches the fact, the model phrases it. Fallback matters when the embedding or the completion call fails. Neither one repairs a missing department filter.
LLM Engineering · 19 Nov 2026 · 6 min
How an application asks a model for a structured tool call, checks the arguments, runs the tool itself, and sends the result back to the model.
LLM Engineering · 19 Nov 2026 · 6 min
A practical way to call a second model when the first one times out or returns an unusable response, without repeating the same side effect.