From a notebook to a service

Two ways into the same idea

Before You Start

The Core Idea

A service is an endpoint any program can send a request to and get a reply. Here you build loading the model once, then the request handler, then status codes, then the readiness check with FastAPI.

1Load the model once

Loading a model takes seconds to minutes and uses a lot of memory. If it happened on every request, the service would be unusable.

Look at the picture: four requests. Loading inside the request means four loads, and loading at startup means one, shared by everyone.

2The handler: validate, then predict

The endpoint is an address like POST /predict that takes JSON and returns JSON in a fixed, documented shape. Before anything reaches the model we check type, length and required fields.

Before FastAPI, let us see what any framework does: receive text, validate it, call the preloaded model, and return a status code and a JSON reply.

import json

class SentimentModel:
    loads = 0
    def __init__(self):
        SentimentModel.loads += 1          # in real life: seconds of reading weights from disk
        self.version = "sentiment-2026-05-01"
    def predict(self, text):
        return "positive" if any(w in text for w in ["ممتاز", "شكرا", "رائع"]) else "negative"

MODEL = SentimentModel()                   # loaded once, when the service starts

def handle(body):
    try:
        data = json.loads(body)
    except json.JSONDecodeError:
        return 400, {"error": "body must be JSON"}
    text = data.get("text")
    if not isinstance(text, str) or not text.strip():
        return 422, {"error": "field 'text' must be a non-empty string"}
    if len(text) > 2000:
        return 422, {"error": "text is longer than 2000 characters"}
    return 200, {"label": MODEL.predict(text), "model_version": MODEL.version}

def health():
    return 200, {"status": "ok", "model_version": MODEL.version}

for body in ['{"text": "خدمة ممتاز وشكرا"}', '{"text": ""}', 'not json', '{"text": "تأخر الطلب"}']:
    print(handle(body))
print(health())
print("model loads after 4 requests:", SentimentModel.loads)

Four requests and the model was loaded once. And the codes tell the client what happened: 200 success, 400 the request itself is broken, 422 the request is well-formed but its content is not acceptable. Every successful reply carries the model version, so when someone complains about an answer you know which model gave it.

3Status codes

The status code tells the client what happened before it reads the reply. Look at the picture: three codes cover most cases.

400 means the request itself is broken, like invalid JSON. 422 means the request is well formed but its content is not acceptable, like empty or overlong text.

4FastAPI and the readiness check

The service is alive as soon as the server runs, but not ready until the model finishes loading. So we split two checks: /health/live and /health/ready.

Look at the picture: the deployment platform sends requests only after the readiness check answers 200.

This is what you will actually write. It needs pip install fastapi uvicorn, so it is here for reading.

from contextlib import asynccontextmanager
from fastapi import FastAPI
from pydantic import BaseModel, Field

state = {}

@asynccontextmanager
async def lifespan(app):
    state["model"] = load_model("/models/sentiment-v3")   # once, before the first request
    yield
    state.clear()

app = FastAPI(lifespan=lifespan)

class PredictIn(BaseModel):
    text: str = Field(min_length=1, max_length=2000)

class PredictOut(BaseModel):
    label: str
    model_version: str

@app.post("/predict", response_model=PredictOut)
def predict(body: PredictIn):
    model = state["model"]
    return PredictOut(label=model.predict(body.text), model_version=model.version)

@app.get("/health/ready")
def ready():
    return {"status": "ok", "model_version": state["model"].version}

lifespan loads the model before the first request, Pydantic validates input and returns 422 with error details automatically, and interactive docs appear at /docs with no extra line. Run it with uvicorn app:app and try it from the browser.

How It Works, Step by Step: how the service handles a request

Going deeper

For large language models, do not write the service from scratch. Dedicated servers like vLLM handle loading, smart request batching and context memory management, and expose an OpenAI-compatible API directly. FastAPI suits your small and medium models, and the business-logic layer around a language model running in a dedicated server.

Worked example

The first mistake in nearly every model service: loading the model inside the request function. Loading a model takes seconds to minutes, so if it happens on every request the service is unusable. The right way is to load it once when the service starts and keep it in memory.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding