Fixing AI-generated code
Lesson 12 of 12
12Fixing AI-generated code
The big pictureGenerated code looks confident and clean, which is what makes its bugs dangerous. It works for the happy example and falls over on an empty list, odd input, or a library whose name was invented.
What the picture shows: The fix loop: the test exposes the bug and proves the fix, and the fix is the smallest change that makes it pass.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Like a recipe a friend wrote in a hurry: it looks right, but they forgot "preheat the oven" and listed an ingredient no shop near you sells. Before cooking for guests, try a small batch and fix what is missing.
Common patterns: API hallucination (a function or argument that does not exist in the installed version), package hallucination (a non-existent library name an attacker may register: slopsquatting), edge cases (empty, zero, None, Unicode, time zones), security (string-built SQL, secrets in code, eval, missing validation), shared state (mutable defaults, globals), and swallowed errors (except: pass). The method: a test that reproduces the bug, then the smallest fix, then run everything. Check packages on PyPI/npm before installing, and pin versions.
Before You Start
🧪Tests
You write simple asserts or pytest.
🔎Assisted review
The previous lesson in this track.
The Core Idea
1Generated code with three bugs
This is a function that "averages a product's ratings" as an assistant generated it. It looks fine. The tests say otherwise:
def average_rating(ratings, history=[]):
history.extend(ratings)
return round(sum(history) / len(history), 1)
def check(name, fn):
try:
print(name, "->", fn())
except Exception as e:
print(name, "-> CRASH", type(e).__name__)
check("first product", lambda: average_rating([4, 5]))
check("second product", lambda: average_rating([1, 1]))
check("no ratings", lambda: average_rating([]))The second product's average is 1, but it came out 2.8 because history is shared across every call. And with no ratings the answer should be clear, not a random crash.
2The smallest fix
Do not rewrite everything. Remove the shared state, decide the empty-list behaviour explicitly, then rerun the same tests:
def average_rating(ratings):
if not ratings:
return None
return round(sum(ratings) / len(ratings), 1)
assert average_rating([4, 5]) == 4.5
assert average_rating([1, 1]) == 1.0
assert average_rating([]) is None
print("all tests pass")3Invented functions and packages
Before pip install or npm install of a name the assistant suggested: look it up on PyPI or npm, and check the repository, download count and last release date. Before calling an unfamiliar function, open the docs for the installed version or try help().
import importlib.util
import json
suggested = ["json", "requests_magic_retry", "pathlib"]
for name in suggested:
found = importlib.util.find_spec(name) is not None
print(f"{name:22} {'installed' if found else 'NOT FOUND, check PyPI first'}")
print("dumps has 'indent':", "indent" in json.dumps.__code__.co_varnames)requirements.txt or package-lock.json; the assistant may write code for a newer or older version than yours.Worked example
Problem: a generated function splits a 100-riyal bill among 3 with round(total / n, 2) each. Test: sum(split(100, 3)) == 100 fails: 33.33 × 3 = 99.99. Smallest fix: work in halalas (integers), giving the remainder to the first person: [3334, 3333, 3333] halalas. The test passes and no money vanishes.
Myth vs Reality
| Myth | Reality |
|---|---|
| Code that works for the example is correct. | The happy example is the easiest case. The bugs live at the edges. |
| If the assistant suggests a library, it exists. | It may invent a name, and that name may have been registered by an attacker. |
Test your understanding
A function with a history=[] default gives results that pile up across calls. Why?
Defaults are evaluated when the function is defined, not on each call.
The assistant suggested a package you have never heard of. First step?
The assistant may confirm what it invented. The real source is the registry.
What is a good fix?
A small change is easier to review and less risky.
Quick review cards
Check Your Understanding
Why write the test before the fix?
To prove the bug first, then prove the fix solved it and broke nothing else.
What edge cases do you always try?
Empty, zero, negative, None, Arabic text, a very large value.
How do you handle money in code?
With integers in the smallest unit (halalas) or Decimal, not float.