Testing strategy, fixtures & flakiness

Explain the testing pyramid — and the criticism of it.

Many fast unit tests, fewer integration tests, very few end-to-end tests. The reasoning: cost and runtime rise, and reliability falls, as you go up.

The criticism worth voicing: a codebase with 100% unit coverage and no integration tests can be entirely broken — units pass in isolation while the wiring is wrong, and heavily-mocked unit tests mostly assert that your mocks match your mocks. The modern framing is the testing trophy: heaviest investment in integration tests (a real HTTP request against a real database, external services stubbed), with unit tests for genuine logic and a thin end-to-end layer over the two or three flows that make money.

The question to answer for each test: "if this test passes and the feature is broken, what did I get?"

What makes a good test?

• Tests behaviour, not implementation. It should survive a refactor. If renaming a private method breaks twenty tests, those tests were coupled to the wrong thing.
• One reason to fail, with a name that says what broke: test_refund_fails_when_order_already_refunded.
• Arrange / Act / Assert, visibly.
• Deterministic — no real clock, no network, no random seed, no reliance on test order.
• Fast enough to run constantly. A suite people skip protects nothing.
• Fails informatively — the assertion message should be enough to diagnose without a debugger.

When do you mock, and when is mocking a mistake?

Mock at the edges you do not own: third-party HTTP APIs, payment providers, email, the system clock, randomness. Do not mock what you own — your own database, your own ORM, your own service layer — because those mocks encode assumptions that drift silently from reality.

# Bad: this test passes forever, even when the query is wrong.
mock_db.query.return_value = [FakeUser(id=1)]

# Better: a real database in a transaction that rolls back.
def test_active_users(session):
    session.add_all([User(active=True), User(active=False)])
    session.flush()
    assert len(list_active_users(session)) == 1

Terminology, since it gets asked: a stub returns canned data; a mock asserts on interactions; a fake is a working lightweight implementation (an in-memory repository); a spy wraps a real object and records calls. Prefer fakes to mocks — they are far less brittle. And for HTTP, record/replay (VCR, responses, MSW) beats hand-written mocks because the fixtures came from the real API.

How do you test code that depends on time, randomness or external services?

Inject the dependency. A function that calls datetime.now() internally is untestable; one that takes now: datetime or a clock is trivially testable.

def test_expiry(freezer):                # pytest: freezegun / time-machine
    freezer.move_to("2025-01-01")
    token = issue_token(ttl=timedelta(hours=1))
    freezer.move_to("2025-01-01 02:00")
    assert token.is_expired()

For randomness, seed it or inject the generator. For external services: stub at the HTTP layer (responses, respx, MSW) rather than mocking your own client class, so the test also covers your serialisation. Keep a small suite of contract tests that hit the real sandbox API on a schedule — that is what catches the provider changing a field.

Write good pytest fixtures.

@pytest.fixture(scope="session")
def db_engine():                                   # expensive: build once per session
    with PostgresContainer("postgres:16") as pg:
        engine = create_engine(pg.get_connection_url())
        Base.metadata.create_all(engine)
        yield engine

@pytest.fixture
def session(db_engine):                            # cheap + isolated: per test
    conn = db_engine.connect(); trans = conn.begin()
    s = Session(bind=conn)
    yield s
    s.close(); trans.rollback(); conn.close()      # nothing persists between tests

@pytest.fixture
def user(session):
    u = UserFactory()                              # factory_boy: readable, defaults filled in
    session.add(u); session.flush()
    return u

@pytest.mark.parametrize("email,valid", [
    ("a@b.com", True), ("no-at-sign", False), ("", False), ("a@b", False),
])
def test_email_validation(email, valid):
    assert is_valid_email(email) is valid

Points to make: choose fixture scope deliberately (session-scoped state that tests mutate is the top cause of order-dependent failures); use factories rather than fixtures-per-scenario so tests state only what they care about; and parametrize instead of loops so each case reports separately.

What causes flaky tests, and how do you fix them?

Ranked by how often it is actually the cause:

1. Shared state between tests — a module-level cache, a session-scoped database row, a singleton. Fix: reset in a fixture; run with pytest -p no:randomly --forked or pytest-randomly to expose order dependence deliberately.
2. Time — datetime.now() at a boundary, sleeps racing with real work, timezone/DST. Fix: freeze the clock; never sleep in a test, wait for a condition.
3. Async/concurrency races — asserting before the write lands. Fix: await the actual signal, or poll with a timeout.
4. Test order and leftover data — the fix is transactional rollback per test.
5. External network — a rate-limited or slow third party. Fix: stub it.
6. UI timing — fixed waitForTimeout. Fix: web-first assertions that retry (expect(locator).toBeVisible() in Playwright).

Policy matters as much as technique: quarantine flaky tests immediately (mark and route them out of the required check) and fix them on a deadline. A suite that is red 10% of the time trains the team to ignore red, which is far more expensive than the flaky test itself.

What should you actually test in a web application?

• Unit: pure business rules, calculations, validators, permission logic, state machines. Cheap, fast, high value.
• Integration: an HTTP request through routing, auth, validation, the service layer, and a real database, asserting the status code, response body and the resulting database state. This is where most of your value is.
• Contract: your API's schema against consumers (Pact, or OpenAPI schema assertions in CI) — prevents breaking a mobile client you cannot deploy.
• End-to-end: signup → login → core action → payment. Two to five flows, running in CI, in a real browser.
• Non-functional: a load test for the endpoint you know is hot, a query-count assertion so an N+1 fails the build, and a security scan (dependency audit, SAST).

Is code coverage a useful metric?

It is useful as a detector of untested areas and useless as a target. 100% coverage proves every line ran, not that any behaviour was verified — a test suite with no assertions can hit 100%. Goodhart applies immediately: mandate 90% and people write tests for getters.

The useful version: track coverage on changed lines in a pull request (so new code arrives tested), look at which branches are uncovered rather than the headline number, and pay attention to whether the critical paths — payments, auth, permissions — are covered at all. Mutation testing (mutmut, Stryker) is the honest measure of assertion quality if you want one.

How do you structure CI so it stays fast and trustworthy?

Stages, failing fast: lint and type-check (seconds) → unit tests (a minute) → integration tests with service containers (a few minutes) → build → e2e on a deployed preview → deploy. Run test files in parallel (pytest -n auto), cache dependencies, and only run the slow e2e suite on the main branch and on pull requests that touch relevant paths.

Trust rules: the required check must be green to merge, no one merges through a red build, flaky tests are quarantined not retried blindly (an automatic retry-on-failure hides real races), and the suite must be runnable locally with one command — if it only works in CI, people stop running it.

What is TDD, and do you use it?

Red → green → refactor: write a failing test, write the minimum code to pass, then clean up with the test as a safety net. Be honest in the answer, because "always, religiously" is not credible: it is excellent for well-specified logic (parsers, pricing rules, state machines, bug fixes — write the failing test that reproduces the bug first), and awkward for exploratory UI work or when you do not yet know the design. The defensible position is "test-first where the behaviour is clear, test-after where I was exploring, and always before the code merges."