Error Handling, Debugging, Logging & Testing with pytest

Part 1: Python Exception System Internals

Every Python program you write will eventually fail. The question is not if an error occurs but how gracefully you handle it. Python's exception system is one of the most expressive in any language — deeply object-oriented, fully inspectable, and designed to be extended. Understanding it at the internals level transforms error handling from defensive boilerplate into a deliberate design tool.

The Exception Hierarchy Tree

Python's exceptions form a strict inheritance hierarchy rooted at BaseException. Every exception class you can catch or raise is a node in this tree:

BaseException
├── SystemExit                 ← raised by sys.exit()
├── KeyboardInterrupt          ← Ctrl+C
├── GeneratorExit              ← generator.close() / garbage collection
└── Exception                  ← all "normal" exceptions
    ├── ArithmeticError
    │   ├── ZeroDivisionError
    │   ├── OverflowError
    │   └── FloatingPointError
    ├── LookupError
    │   ├── IndexError
    │   └── KeyError
    ├── ValueError
    ├── TypeError
    ├── AttributeError
    ├── NameError
    │   └── UnboundLocalError
    ├── ImportError
    │   └── ModuleNotFoundError
    ├── OSError  (also IOError, EnvironmentError — aliases)
    │   ├── FileNotFoundError
    │   ├── PermissionError
    │   ├── IsADirectoryError
    │   ├── FileExistsError
    │   ├── TimeoutError
    │   └── ConnectionError
    │       ├── ConnectionResetError
    │       └── ConnectionRefusedError
    ├── RuntimeError
    │   └── RecursionError
    ├── StopIteration
    ├── StopAsyncIteration
    ├── MemoryError
    ├── BufferError
    ├── EOFError
    ├── SyntaxError
    │   └── IndentationError
    │       └── TabError
    ├── UnicodeError
    │   ├── UnicodeDecodeError
    │   ├── UnicodeEncodeError
    │   └── UnicodeTranslateError
    ├── Warning
    │   ├── DeprecationWarning
    │   ├── UserWarning
    │   └── RuntimeWarning
    └── ExceptionGroup (Python 3.11+)

The critical insight: except Exception catches everything except SystemExit, KeyboardInterrupt, and GeneratorExit. Those three inherit directly from BaseException because they represent program lifecycle events, not errors in your logic. Accidentally catching KeyboardInterrupt would make your program unresponsive to Ctrl+C.

How Exceptions Propagate: Call Stack Unwinding

When an exception is raised, Python does call stack unwinding: it travels up the call stack frame by frame, looking for a matching except handler. If none is found at the outermost frame, the program prints a traceback and terminates.

The traceback shows the full path — level_1 called level_2 called level_3, which is where the exception originated. Each frame has its own local variables (accessible via traceback.extract_tb).

try / except / else / finally — Every Clause

Each block has a specific semantic purpose. Using them correctly communicates intent to readers.

The else clause is underused. It was designed exactly for "code that should only run when the try block succeeded." Using else instead of putting success code inside try avoids accidentally catching exceptions raised by the success code itself.

The finally clause always runs. Even if you return inside try or except, Python executes finally before actually returning. This makes it the right place for resource cleanup.

Multiple except Clauses and Exception Tuples

Python evaluates except clauses top to bottom. Subclasses must come before parent classes.

Accessing Exception Attributes

Re-raising Exceptions: bare raise, raise e, raise NewExc from e

The from e syntax sets __suppress_context__ = True, which makes the traceback cleaner — Python only shows the chained exception, not the original context. Without from, Python prints both, with "During handling of the above exception, another exception occurred."

Suppressing Exceptions with contextlib.suppress

Part 2: All Built-in Exceptions — Thorough Reference

Understanding when Python raises each exception — and what data it carries — lets you write precise except clauses instead of catching Exception everywhere.

Part 3: Custom Exceptions — Professional Patterns

Why Custom Exceptions

Using Python's built-in exceptions for domain errors is a code smell. When a library raises ValueError, callers cannot distinguish between "bad argument to int()" and "business rule violation." Custom exceptions create a precise, testable contract.

Exception Groups (Python 3.11+)

Python 3.11 introduced ExceptionGroup for concurrent code where multiple exceptions can occur simultaneously (e.g., asyncio.gather with multiple failures).

Part 4: Context Managers — Complete Mastery

The with Statement Protocol

A context manager is any object with __enter__ and __exit__ methods. The with statement calls __enter__ on entry and __exit__ on exit — even if an exception is raised.

contextlib.contextmanager — Generator-Based Context Managers

The @contextmanager decorator lets you write context managers as generators, using yield to split setup from teardown.

Part 5: Logging — Production Patterns

Why logging beats print()

print() is for exploratory debugging. logging is for production. Here is why:

• Levels: DEBUG/INFO/WARNING/ERROR/CRITICAL — filter verbosity without code changes
• Destinations: Console, file, network, syslog — without changing call sites
• Format: Timestamps, module names, line numbers — free with formatters
• Off switch: Set level to WARNING in production — all DEBUG/INFO disappear with no performance hit
• Thread-safe: The logging module is thread-safe; print() is not guaranteed

Logger Architecture: Logger, Handler, Formatter, Filter

Part 6: Testing with pytest — Complete Guide

Why Testing?

Tests are executable documentation. They verify your code works, prevent regressions, and — most importantly — force you to design code that can actually be tested (which usually means well-structured code).

Part 7: Real Project — Robust File Processing Library

Exercises

Exercise 1 (Easy): Write a function safe_divide(a, b) that returns a/b for valid inputs, raises ZeroDivisionError with the message "Cannot divide {a} by zero" when b is zero, and raises TypeError when either argument is not numeric. Write 4 assert statements that test all paths.

Exercise 2 (Easy): Create a custom exception NetworkTimeoutError that extends ConnectionError (itself an OSError), with attributes host, port, and timeout_seconds. Raise it and verify isinstance(e, OSError) is True.

Exercise 3 (Easy): Implement a retry context manager using @contextmanager that runs the body up to n times, catching a specific exception type, and raising on the final failure.

Exercise 4 (Medium): Implement a full exception hierarchy for a payment processing system: PaymentError → CardError (with card_last4) → DeclinedError, ExpiredCardError; PaymentError → FraudError (with risk_score). Write a process_payment function that raises the appropriate exception for mock scenarios.

Exercise 5 (Medium): Build a logging system with two handlers: one StreamHandler at DEBUG level with a human-readable format, and one that captures JSON-formatted records into a list. Log 10 events at various levels and verify the JSON handler captured only ERROR and above.

Exercise 6 (Medium): Write a ManagedPool class that implements __enter__ and __exit__, managing a pool of 3 simulated resources. Track checked-out resources, and on __exit__, return them to the pool. If an exception occurs, mark the resource as "dirty" instead of returning it.

Exercise 7 (Hard): Implement pytest-style parametrized testing without pytest. Create a @parametrize(cases) decorator that runs the decorated test function with each set of arguments and prints a summary table: test name, parameters, PASS/FAIL, and error message.

Exercise 8 (Hard): Build a complete ConfigLoader class with: custom exceptions (ConfigNotFoundError, ConfigParseError, ConfigValidationError), a @contextmanager that handles loading and automatic saving on clean exit, structured logging for every operation, and a full unittest.TestCase with at least 8 test methods including mocking open() with unittest.mock.mock_open.