Error Handling, Logging & Testing with pytest

Why Error Handling Matters

Amateur code crashes. Professional code fails gracefully.

Every program encounters unexpected conditions: a file doesn't exist, a network times out, a user passes bad data. How your code responds to those conditions is the difference between software people trust and software people avoid.

Python's exception system is one of the most expressive in any language. Understanding it deeply — not just the mechanics, but the design — will change how you write every function.

The Exception Hierarchy

Python's exceptions form an inheritance tree rooted at BaseException. Understanding the hierarchy explains which except clauses catch what:

BaseException
├── SystemExit              ← raised by sys.exit()
├── KeyboardInterrupt       ← raised by Ctrl+C
├── GeneratorExit           ← raised when a generator is closed
└── Exception               ← base for all "normal" exceptions
    ├── ArithmeticError
    │   ├── ZeroDivisionError
    │   ├── OverflowError
    │   └── FloatingPointError
    ├── LookupError
    │   ├── IndexError
    │   └── KeyError
    ├── ValueError
    ├── TypeError
    ├── AttributeError
    ├── ImportError
    │   └── ModuleNotFoundError
    ├── OSError
    │   ├── FileNotFoundError
    │   └── PermissionError
    ├── RuntimeError
    ├── StopIteration
    └── ...hundreds more

The key insight: except Exception catches everything except SystemExit, KeyboardInterrupt, and GeneratorExit. Those three inherit directly from BaseException because they represent program lifecycle events, not errors in your logic. Catching them accidentally is a bug — your program would ignore Ctrl+C, for example.

try / except / else / finally

Each block has a specific purpose — using them correctly communicates intent:

try:
    # Code that might raise an exception
    result = risky_operation()
except SomeError as e:
    # Handle the specific error
    handle_error(e)
else:
    # Runs ONLY if no exception was raised in try
    # This is where you put code that should run on success
    # but that you don't want inside the try block
    use_result(result)
finally:
    # ALWAYS runs — even if an exception propagated
    # Use for cleanup: closing files, releasing locks
    cleanup()

The else block is underused but important. Code in else runs only when try succeeded, but is outside the try block — so if else itself raises an exception, it won't be caught by the except clauses above. This makes the intent clear: the except handlers are for risky_operation() specifically, not for everything that follows.

Catching Specific vs Broad Exceptions

The cardinal rule: catch the most specific exception you can handle.

# BAD: catches everything, including bugs in your own code
try:
    process_user_input(data)
except Exception as e:
    log.error(f"Something went wrong: {e}")
    # You've now silently swallowed a NameError, AttributeError,
    # or any bug you introduced — it looks like "user input was bad"

# GOOD: catch only what you can actually handle
try:
    process_user_input(data)
except ValueError as e:
    # We know exactly what this means and how to respond
    return {"error": f"Invalid input: {e}"}
except KeyError as e:
    return {"error": f"Missing required field: {e}"}
# Everything else propagates — that's intentional

When is except Exception acceptable? At the outermost layer of a system — a web server's request handler, a background job runner — where you truly want to catch and log any crash without killing the process. Even then, log the full traceback.

raise, raise from, and Re-raising

raise creates a new exception. raise from chains exceptions, preserving the original context. bare raise re-raises the current exception.

# raise with a message
raise ValueError("expected positive number, got -5")

# raise from — tells Python (and the user) that this new exception
# was CAUSED BY the original one. Shows both tracebacks.
try:
    data = json.loads(raw_input)
except json.JSONDecodeError as e:
    raise ConfigurationError("Invalid config file") from e

# raise from None — suppress the original exception chain
# Use when the cause is an implementation detail not useful to callers
try:
    result = cache[key]
except KeyError:
    raise MissingResourceError(key) from None

# Bare re-raise — re-raises the current exception unchanged
# Use when you want to log but still propagate
try:
    risky()
except Exception:
    log.exception("Unexpected error in risky()")
    raise  # propagates the original exception

Custom Exception Classes

Custom exceptions are first-class objects. Give them structured data, not just strings:

Context Managers

Context managers guarantee cleanup — they run __exit__ even if an exception occurs inside the with block. This is more reliable than try/finally because it packages the cleanup with the resource.

The logging Module

print() is for development. logging is for production. The difference: log records have levels, timestamps, caller info, and can be routed to multiple destinations (console, file, external service) simultaneously.

DEBUG    → Detailed diagnostic info — only for troubleshooting
INFO     → Confirmation that things are working as expected
WARNING  → Something unexpected happened, but the program continues
ERROR    → A serious problem that prevented an operation
CRITICAL → A serious error that may crash the program

pytest: Testing Your Code

pytest is Python's most popular testing framework. Its philosophy: test functions are just functions, assertions use Python's built-in assert statement, and failures give you rich context.

The core patterns:

# Test function — must start with test_
def test_add():
    assert add(2, 3) == 5

# Testing exceptions
def test_divide_by_zero():
    with pytest.raises(ZeroDivisionError):
        divide(10, 0)

# Testing exception message
def test_bad_input():
    with pytest.raises(ValueError, match="must be positive"):
        process(-1)

# Fixtures — reusable setup/teardown
@pytest.fixture
def sample_data():
    return {"name": "Alice", "age": 30}

def test_with_fixture(sample_data):
    assert sample_data["name"] == "Alice"

# Parametrize — run one test with many inputs
@pytest.mark.parametrize("input,expected", [
    (2, 4),
    (3, 9),
    (-4, 16),
    (0, 0),
])
def test_square(input, expected):
    assert square(input) == expected

PROJECT: Validated Data Pipeline

We'll build a complete data processing pipeline with:

• Three custom exception classes with structured context
• A DataValidator that enforces schema, types, and value ranges
• A DataProcessor that reads CSV data, validates each row, transforms, and reports errors
• A full test suite demonstrating every failure mode

Now let's see the full test suite:

Key Takeaways

• Exception hierarchy matters: catch Exception only at system boundaries; prefer specific exceptions that communicate intent
• except Exception hides bugs: a broad catch that logs "something went wrong" obscures AttributeError, NameError, and other programmer mistakes
• else clarifies intent: code that runs only on success belongs in else, not inside try — it won't be caught by your except clauses
• raise from preserves cause: use raise NewError("...") from original_error so callers see the full chain; use raise ... from None to suppress implementation details
• Custom exceptions carry data: give exceptions structured fields (code, field, row_num) instead of cramming everything into the message string
• Context managers guarantee cleanup: __exit__ is called even when exceptions occur — more reliable than try/finally for resource management
• Logging beats printing: use named loggers (logging.getLogger(__name__)), appropriate levels, and extra= for structured context; print is for REPL exploration
• pytest is just functions and assert: test functions start with test_, use plain assert, and pytest.raises for exception testing — parametrize eliminates copy-paste test boilerplate