The system's limits
Lesson 9 of 10
9The system's limits
The big pictureA good system knows when to say "I do not know". A safe system never retrieves what the user may not see in the first place. These are two different decisions: the first is a threshold you choose by measuring, the second a condition in the database query, not a sentence in the prompt hoping the model obeys.
What the picture shows: As you raise the threshold the lower line falls, so there is less invention out of scope, but the upper line also drops after 0.55 and the system declines legitimate questions. No point makes both errors zero, so you choose which error harms your users less.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A good receptionist says "I do not know, ask that department" instead of inventing an answer. A smart system is the same: if it finds no source, it declines politely. And if the receptionist may speak about one department only, that is enforced in the files they can open, not by a spoken reminder.
Refusal rests on a threshold: if the top similarity is below a limit, or the judge finds no support in the passages, the system says it does not know. Scope is enforced in the query itself with a WHERE filter on metadata (department, permission), not by instructions in the system message.
Before You Start
🧲Vector search
You know every result has a similarity score, and that pgvector allows a WHERE condition in the same query.
📐Evaluation first
You have in-scope and out-of-scope questions, which are exactly what the threshold is chosen from.
The Core Idea
A good system knows when to say "I do not know", and a safe one never retrieves what the user may not see. In this lesson you choose the refusal threshold, put permissions in the query, write them in pgvector, then word a refusal that leaks nothing.
1The refusal threshold
The system declines when the top retrieval score is below a threshold. The threshold is a number you choose from your data: questions you know are in scope and others out of scope, and the top score for each.
# top retrieval score for questions we labelled by hand
in_scope = [0.81, 0.77, 0.74, 0.69, 0.66, 0.72, 0.58, 0.79, 0.63, 0.71, 0.55, 0.68]
out_scope = [0.41, 0.52, 0.38, 0.47, 0.60, 0.35, 0.44, 0.50, 0.29, 0.46]
print("threshold answers-in-scope wrongly-answers-out-of-scope")
for t in [0.45, 0.50, 0.55, 0.60, 0.65]:
kept = sum(s >= t for s in in_scope)
leaked = sum(s >= t for s in out_scope)
print(f" {t:.2f} {kept:2}/{len(in_scope)} {leaked}/{len(out_scope)}")
At 0.45 the system answers every valid question, and also half the out-of-scope ones, meaning it invents. At 0.55 it answers all valid ones and errs on one outside question. At 0.65 it never errs outside but declines three legitimate questions. There is no perfect threshold: you choose which error is less harmful to your users, and write down why.
2Permissions as a query condition
Treat anything that enters the context window as visible to the user. No sentence in the prompt reliably stops the model from mentioning what it saw.
Look at the picture: permissions are applied before ranking, so only what the user may see reaches the model.
An engineer asks "what is my salary and leave balance?". The salary scale document is the closest match, but it is for HR only.
docs = [
{"id": 1, "text": "Salary scale for grade 7", "access": {"hr"}},
{"id": 2, "text": "Annual leave is 21 working days", "access": {"all"}},
{"id": 3, "text": "Disciplinary case of employee 4471", "access": {"hr", "legal"}},
{"id": 4, "text": "Travel allowance paid in 10 days", "access": {"all"}},
]
scores = {1: 0.83, 2: 0.80, 3: 0.78, 4: 0.40} # similarity to "what is my salary and leave?"
def retrieve_prompt_only(k=2):
return [d["id"] for d in sorted(docs, key=lambda d: -scores[d["id"]])][:k]
def retrieve_scoped(user_groups, k=2):
allowed = [d for d in docs if d["access"] & (user_groups | {"all"})]
return [d["id"] for d in sorted(allowed, key=lambda d: -scores[d["id"]])][:k]
print("filter in the prompt (hope the model obeys):", retrieve_prompt_only())
print("filter in the query, for an engineer: ", retrieve_scoped({"engineering"}))
print("filter in the query, for HR: ", retrieve_scoped({"hr"}))
With filtering in the prompt, the salary document (1) reached the engineer's context, and now everything hangs on the model obeying "do not mention restricted documents". With filtering in the query, the model never saw the document, and the HR employee sees it as they should. Note the disciplinary case file (3) did not appear for either here because it was not in the top two, but without the predicate it would have reached the engineer with a different question.
3The predicate in pgvector
In a real database the condition becomes a line in WHERE next to the vector search. It can carry more than permissions, such as a policy's expiry date.
SELECT doc_id, body, 1 - (embedding <=> $1) AS similarity FROM chunks WHERE access && $2::text[] -- $2: the user's groups plus 'all', from the session, never from the prompt AND (valid_until IS NULL OR valid_until > now()) ORDER BY embedding <=> $1 LIMIT 5;
The user's groups come from their authenticated session on the server, never from anything they typed in the question. The second condition stops expired policies from being retrieved, another kind of "scope". Write an automated test: a user without permission asks a question matching a restricted document, and the results must not contain it.
4Leakage and the refusal message
Leakage is information from a restricted document reaching the user through the answer, even if the document itself is never shown. The refusal message itself can leak if it says the document exists.
Look at the picture: the restricted document is never retrieved, so the top score drops, and the user sees the same ordinary refusal.
How It Works, Step by Step: how a question passes the system's limits
Take the user's groups from their authenticated session on the server, not from the text of their question.
Retrieve with a WHERE condition on permissions and expiry before ranking, so nothing restricted ever reaches the model.
If the top score is below the threshold you chose, decline before calling the model.
The refusal message is the same for absence and for missing permission, suggests where to ask, and never reveals that a document exists.
Worked example
When should the system decline? When it finds no document close enough. But "close enough" is a number, chosen from your data: take questions you know are in scope and others out of scope, and look at the top similarity score for each.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A line saying "do not reveal salaries" in the prompt protects the data. | It is a request, not a guarantee. What enters the context can come out in the answer. The only guaranteed protection is never retrieving the document for this user. |
| There is one refusal threshold that suits every system. | The threshold is chosen from your data and from which error is worse for you: for medical questions declining beats a confident wrong answer. |
| "You lack permission" is an honest, helpful refusal. | It is itself a leak: it told the user the document exists. Refusal for absence and for missing permission should look identical. |
Try it yourself
A 0.45 threshold answers all valid questions and half the outside ones. 0.65 never errs outside and declines 3 of 12 valid ones. Which do you pick for a system answering medical questions?
The threshold is a decision about which error is less harmful. In sensitive domains, declining a legitimate question and pointing the user to a specialist is far better than a confidently wrong answer.
Where do the user's permission groups in the retrieval query come from?
Anything the user types or the model decides can be manipulated. Permissions come from the same source of truth the rest of your application uses.
Quick review cards
WHERE), so only what the user may see reaches the model. They are not applied with a sentence in the prompt.Check Your Understanding
Why is "do not reveal salary information to anyone outside HR" in the system message not enough?
Because it is a request, not a guarantee. The model can be fooled by clever phrasing, summarise the whole context when asked, or mention a figure inside an answer about something else. And if the model changes, so does how well it obeys. A query predicate works the same with any model and can be tested automatically.
Why does the pgvector query add a valid_until condition beside permissions?
So expired policies are never retrieved. It is another kind of scope: the document exists and is allowed, but is no longer true, so it must not reach the model.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.