Log Parsing
Lesson 4 of 11
4Log Parsing
Your first time reading an endless stream instead of a file: line by line, skipping the malformed, printing metrics every ten lines and on interruption.
What the picture shows: Each line is split and checked, then adds to the total size and to the counter of its status, and the stats print every ten lines.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Picture an endless strip of receipts coming out of a machine. You cannot wait for it to end; you read each receipt as it arrives, ignore torn ones, and every ten receipts write a summary: how much you sold and how often each code appeared.
You read sys.stdin line by line, match each line with a regex, skip non-matching ones and update counters in a dict. You print stats every ten lines and on KeyboardInterrupt, in constant memory however long the stream runs.
Required fundamentals
Standard input, sys.stdin
An object you read like a file, except its source is whatever another program or the user writes. for line in sys.stdin reads line by line without loading everything into memory.
A dict as a counter
counts.get(k, 0) + 1 is the safe form: it returns zero for a new key instead of raising KeyError.
Nested try/except
One for the failed conversion inside a line (ValueError), and a bigger one for the interruption (KeyboardInterrupt). Each error in its own place.
The __main__ guard
The brief explicitly requires that the code not run on import. Without the guard, a plain import starts reading input.
Core concepts
The task: read lines from standard input, and after every ten lines (or on Ctrl+C) print the total size and the number of lines per status code. Lines that do not match the format are skipped.
1A stream, not a file
This is the first fundamental departure from everything before. The program does not know when the input ends, it may be a ten-line file, or a generator that writes forever. So:
- We do not use
sys.stdin.read(), which tries to load everything and hangs on an endless stream. - We use
for line in sys.stdin, which reads one line, handles it, then asks for the next. - Memory stays constant however large the input is.
2Anatomy of a line: why index from the end?
1.2.3.4, the IP address-, a fixed separator[2026-10-22 12:00:00.000000], the date (two fields, it has a space!)"GET /projects/260 HTTP/1.1", the request (three fields)200, the status code → parts[-2]128, the file size → parts[-1]The date and the request both contain spaces, so the field count from the start is not fixed. But the two values we need are always the last and the one before it. parts[-1] and parts[-2] work no matter what precedes them, and that is not a trick, it is reading the format correctly.
split and not a regex?A regex works, but it is longer, harder to maintain, and slower per line. The brief says "a non-matching line must be skipped", and split() with try/except ValueError implements exactly that in two lines. Pick the simplest tool that meets the requirement.3The accumulator pattern
| Variable | Type | Role |
|---|---|---|
total_size | int | Running total of every valid size |
counts | dict | status code → how many times it appeared |
lines | int | Count of valid lines only, this is what decides when to print |
Note the precision in the third: malformed lines are not counted, so the report comes after every ten accepted lines, not every ten lines read.
4Status codes: ordering and skipping
Two rules in the brief that are easy to forget:
- A code not in the list gets nothing printed for it, which is why we check
status in CODESand only then add to the dict. - Codes are printed in ascending order, and
sorted(counts)takes care of that. Notice in the real output that 200 comes before 301 before 404, even though 301 was inserted first.
And note something subtle: the file size is always added, even when the code is not in the list, because the brief separates "total size" from "lines by status code".
5The interruption: except, not finally
On Ctrl+C Python raises KeyboardInterrupt. We need to print the metrics and then let the exception continue (raise) so the program exits with the right code.
finally instead of except KeyboardInterrupt. finally runs in every case, so on a normal end it prints once from inside itself and once from the end-of-input check, and your metrics come out duplicated. finally is for cleanup that always happens; except is for one specific case.How it works
Three pictures: the pipeline between the two programs, then the single-line decision, then the difference between a normal end and an interruption.
Diagram 9, The pipeline: the generator writes while the parser reads line by line, both run together, neither waits for the other to finish.
Diagram 10, The single-line decision: a skip does not raise the counter, a code outside the list still adds its size without being counted, and printing is gated on multiples of ten.
Diagram 11, The two endings: a normal end prints once more only if the last print was not on the tenth line; an interruption prints and then re-raises.
Worked example, step by step
We write the solution, run it on a fixed input whose answer we know, then on a random generator, then break it on purpose. Every output below is real.
Step 1, the solution
#!/usr/bin/python3
"""0-stats"""
import sys
CODES = (200, 301, 400, 401, 403, 404, 405, 500)
def report(total_size, counts):
"""Print the metrics gathered so far."""
print("File size: {:d}".format(total_size))
for code in sorted(counts):
print("{:d}: {:d}".format(code, counts[code]))
def main():
total_size, counts, lines = 0, {}, 0
try:
for line in sys.stdin:
parts = line.split()
if len(parts) < 2:
continue
try:
size = int(parts[-1])
status = int(parts[-2])
except ValueError:
continue
total_size += size
if status in CODES:
counts[status] = counts.get(status, 0) + 1
lines += 1
if lines % 10 == 0:
report(total_size, counts)
except KeyboardInterrupt:
report(total_size, counts)
raise
if lines % 10 != 0:
report(total_size, counts)
if __name__ == "__main__":
main()Step 2, a fixed input we can check by hand
$ cat fixed.txt 1.2.3.4 - [2026-10-22 12:00:00.000000] "GET /projects/260 HTTP/1.1" 200 128 5.6.7.8 - [2026-10-22 12:00:01.000000] "GET /projects/260 HTTP/1.1" 404 512 9.9.9.9 - [2026-10-22 12:00:02.000000] "GET /projects/260 HTTP/1.1" 200 64 oops this line is garbage 1.1.1.1 - [2026-10-22 12:00:03.000000] "GET /projects/260 HTTP/1.1" 301 32 2.2.2.2 - [2026-10-22 12:00:04.000000] "GET /projects/260 HTTP/1.1" 500 1024 3.3.3.3 - [2026-10-22 12:00:05.000000] "GET /projects/260 HTTP/1.1" 200 16 $ ./0-stats.py < fixed.txt File size: 1776 200: 3 301: 1 404: 1 500: 1
Manual check: 128+512+64+32+1024+16 = 1776 ✓ and the malformed line counted toward neither the total nor the line count.
Step 3, ten lines from a generator (printing on the tenth)
$ ./0-generator.py | ./0-stats.py File size: 4039 301: 4 200: 1 404: 1 405: 1 500: 3
Here lines = 10 exactly, so the print came from inside the loop and the end-of-input check lines % 10 != 0 did not apply, no duplication.
Step 4, cases that print nothing
$ printf '' | ./0-stats.py # empty input: nothing to report $ printf 'hello\nworld\n' | ./0-stats.py # all lines malformed: nothing to report $ echo 'x - [d] "GET /" 200 abc' | ./0-stats.py # size not a number: line skipped
Step 5, interrupting with Ctrl+C
^C
File size: 300
200: 3
Traceback (most recent call last):
File "./0-stats.py", line 41, in <module>
main()
File "./0-stats.py", line 18, in main
for line in sys.stdin:
KeyboardInterruptThe metrics go to stdout and the traceback to stderr, and the program exits with code 130, which is what a shell expects from an interrupted program.
Step 6, walk the fixed input line by line
Tracing six lines, one of them malformed
Line 1. … "GET /projects/260 HTTP/1.1" 200 128. parts[-2:] = ['200','128'], so we accept it. total_size = 128 and counts = {200: 1}. lines = 1, so nothing is printed.
Line 2. Code 404, size 512. total_size = 640 and counts = {200:1, 404:1}.
Line 3. 200 and 64. total_size = 704 and counts = {200:2, 404:1}. lines = 3.
Line 4: oops this line is garbage. parts = ['oops','this','line','is','garbage'], so its length is 5 ≥ 2 and it passes the first check. But int('garbage') raises ValueError, so we skip the line, and lines stays at 3.
Lines 5 to 7. 301/32, 500/1024 and 200/16. The final total is 1776 with counts = {200:3, 301:1, 404:1, 500:1}. lines = 6, and 6 % 10 != 0, so we print one last time.
The print. sorted(counts) yields [200, 301, 404, 500], ascending, not insertion order. 301 appeared before 404 in the input and still prints before it, because numerically 200 < 301 < 404 < 500.
Myth vs. reality
| The myth | The reality |
|---|---|
| “I'll read the whole input and then parse it.” | sys.stdin.read() hangs on an endless stream and consumes memory proportional to the input. A stream is read line by line. |
“finally is cleaner than except for cleanup.” | finally is for cleanup that must always happen. Here the print is not cleanup but behaviour for one specific case, and finally causes a double print on a normal end. |
| “The counter goes up for every line read.” | It goes up only for every valid line. Malformed lines do not count, so the report lands every ten accepted lines. |
| “An unknown code should be printed as zero.” | Never print anything for it. The brief is explicit: "if a status code doesn't appear or is not an integer, don't print anything for this status code". |
| “The size of a line with an unknown code is not added.” | It is added. The total covers every well-formed line; classifying codes is a separate matter. |
| “Python dicts are unordered, so the order is random.” | Dicts have preserved insertion order since Python 3.7, but what is required is ascending numeric order, so sorted() is needed either way. |
Where it is used in practice
Server monitoring
Every monitoring tool does exactly this at a larger scale: read streaming logs, aggregate status codes, and alert when the 5xx rate climbs.
Command-line tools
Your program is now a link in a pipeline: cat log | ./0-stats.py. That is the whole Unix model, small programs talking through stdin/stdout.
Real-time analytics
Counting events over a moving window (every ten lines here) is the heart of stream processing, and the same logic runs in Kafka or Flink at a far larger scale.
Security auditing
A sudden rise in 401 or 403 in auth logs is the first signal of an intrusion attempt, and that is a counter just like the one you wrote.
Try it yourself
You swap except KeyboardInterrupt for finally. What happens at the end of an input with 7 valid lines?
finally runs whenever the try block is left, including a normal end. So the metrics print from inside it, and then if lines % 10 != 0 runs afterwards and prints them again. except KeyboardInterrupt runs only on an actual interruption.
Quick review flashcards
for line in sys.stdin and not sys.stdin.read()?parts[-1] and parts[-2]?continue and does not count.except or finally for the interruption?except KeyboardInterrupt. finally prints one extra time on a normal end.__main__ guard here?import starts reading input immediately.Check your understanding
1. Why do we check len(parts) < 2 before converting?
Because parts[-2] on a one-element list raises IndexError, which except ValueError does not catch. The check protects the conversion before it happens.
2. An input with 10 valid lines. How many times do the metrics print?
Exactly once, from inside the loop at the tenth line. The end check lines % 10 != 0 does not apply because 10 % 10 == 0. If yours prints twice, the condition is wrong.
3. (Spot the bug) Your program prints File size: 0 for a completely empty input. Acceptable?
It depends on your reading: the brief says "print after every 10 lines", so with zero lines there is nothing to print, and the solution above prints nothing. What matters is being deliberate about the decision and documenting it, not leaving it to chance.
4. Someone used collections.Counter instead of a dict. Better or worse?
Genuinely cleaner: counts[status] += 1 is enough. The brief only forbids non-standard libraries, and collections is standard, so it is allowed. A plain dict stays clearer for a learner, because it exposes the logic instead of hiding it.
5. (Decision) You want a moving window: "metrics for the last 100 lines", not "from the beginning". What changes?
You have to keep the lines themselves, not just the totals, because a running total cannot be undone. The usual answer is a collections.deque of length 100: add at the front and subtract what falls off the back. This exposes an important difference: running totals are cheap, while moving windows cost memory proportional to the window.
Resources for this part
Sorted by kind: start with the official docs when you want the precise reference, or with an article or video when you want the explanation first.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs3
Done with this lesson?
Sign in to save your progress.