Calling a model

Two ways into the same idea

Before You Start

The Core Idea

Calling a model is one line, but four decisions sit behind it. In this lesson you build the messages and price them, send a real request, set the temperature, then ask for data your code can check.

1Messages and cost

A conversation is a list of messages, each with a role: system, then user and assistant in turns. The model has no memory, so every request resends the whole list. Look at the picture: the new question travels with everything before it.

Cost is counted in tokens: input tokens at their price, output tokens at theirs, and output is usually three to five times dearer.

messages = [
    {"role": "system", "content": "You answer questions about the leave policy. Be brief."},
    {"role": "user", "content": "How many days of annual leave do I get?"},
    {"role": "assistant", "content": "21 working days per year."},
    {"role": "user", "content": "And after five years?"},
]

def rough_tokens(text):
    return max(1, round(len(text) / 4))      # about 4 characters per English token

prompt_tokens = sum(rough_tokens(m["content"]) + 4 for m in messages)
answer_tokens = 60
price_in, price_out = 0.50, 1.50              # dollars per million tokens, an example price

cost = prompt_tokens * price_in / 1e6 + answer_tokens * price_out / 1e6
print("turns sent:", len(messages))
print("prompt tokens ~", prompt_tokens)
print(f"cost of this call: ${cost:.6f}")
print(f"cost of 100,000 such calls: ${cost * 100_000:.2f}")

The fourth request sends all four messages. The estimate here is rough (four characters per English token, far fewer in Arabic as you saw in the tokenisation lesson), but multiplying by the number of requests is what matters: fractions of a cent become hundreds of dollars. Work this out before you choose a model, not after the bill arrives.

2A real call

Now you actually send the messages. Three things matter in every request: where the key comes from, the ceiling on the reply, and how many tokens you were billed for.

Most providers accept the same OpenAI-compatible shape. This needs an API key, so it is here for reading.

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.deepseek.com", api_key=os.environ["DEEPSEEK_API_KEY"])
reply = client.chat.completions.create(
    model="deepseek-chat",
    messages=[{"role": "system", "content": "أجب عن أسئلة سياسة الإجازات باختصار."},
              {"role": "user", "content": "كم يوم إجازة سنوية لي؟"}],
    temperature=0.2,
    max_tokens=200,
)
print(reply.choices[0].message.content)
print(reply.usage)   # prompt_tokens, completion_tokens: what you are billed for

The key comes from an environment variable, not the code; max_tokens is a ceiling that protects you from a needlessly long reply; and usage is the real number you are billed for. Log it with every request from day one.

3Temperature

The model gives every possible next word a score, then turns the scores into probabilities with softmax. Temperature divides the scores first: near zero the strongest takes everything, and above 1 the probabilities flatten.

Look at the picture: as the temperature rises, approved falls and rejected climbs, and the two lines draw closer.

The same scores for five possible next words, at four temperatures.

import numpy as np

words = ["approved", "accepted", "pending", "rejected", "banana"]
logits = np.array([3.0, 2.6, 1.5, 1.0, -2.0])

def probs(t):
    z = logits / t
    e = np.exp(z - z.max())
    return e / e.sum()

for t in [0.2, 0.7, 1.0, 1.5]:
    p = probs(t)
    row = "  ".join(f"{w}={x:.2f}" for w, x in zip(words, p))
    print(f"T={t}: {row}")

At 0.2, approved takes 88% and the rest nearly vanish. At 1.5, rejected reaches 11%: in a system answering about a request's status, that means roughly one reply in nine could say "rejected" for an accepted request. And notice banana never quite disappears at high temperature. That is why systems answering from documents start with a low temperature.

The working rule: 0 to 0.3 for answers from documents, and higher only for creative writing.

4Structured output

Code does not read prose. Ask for JSON with a defined schema, check every field and its type, and if validation fails, ask again with the reason.

Look at the picture: three attempts, each rejection goes back to the model with its reason, and the third passes.

Three possible replies from a model asked for JSON. The code validates each, rejecting and asking again with the reason.

import json

replies = [
    'Sure! Here is the data: {"name": "Sara", "days": 3}',
    '{"name": "Sara", "days": "three"}',
    '{"name": "Sara", "days": 3, "type": "sick"}',
]
schema = {"name": str, "days": int, "type": str}

def parse(reply):
    try:
        data = json.loads(reply)
    except json.JSONDecodeError as e:
        return None, f"not JSON: {e.msg}"
    for key, kind in schema.items():
        if key not in data:
            return None, f"missing field {key!r}"
        if not isinstance(data[key], kind):
            return None, f"{key!r} should be {kind.__name__}"
    return data, None

for attempt, reply in enumerate(replies, 1):
    data, error = parse(reply)
    if data:
        print(f"attempt {attempt}: ok -> {data}")
        break
    print(f"attempt {attempt}: rejected ({error}), asking again with the error")

The first reply has a polite preamble before the JSON, the second wrote the number in words. Both look "fine" to a person and break the code. Providers now offer a "JSON mode" or "structured outputs" that constrains generation to the schema, so use it where available, and keep validation in code anyway: the schema guarantees the shape, not the meaning.

How It Works, Step by Step: how one request to a model goes

Worked example

The model remembers nothing between requests. Every request carries the whole conversation from the start: the system message that sets the role, then user and assistant messages taking turns. Which is why every old message you keep, you pay for again in every new request.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding