Researching Errors and Organizing Analytical Work

A rides table has a start time and an end time, and the report asks how long each ride took. Most rows look right. Then one rider starts at 23:00 and ends at 06:00, and the report says minus seventeen hours. Nothing crashed, the total is wrong, and the question is what to do next.

A useful debugging method is to reproduce the failure, inspect the evidence, reduce the example, search for relevant guidance, test a candidate change, and record the result. This article follows that method through the ride calculation. Some problems also need domain knowledge or help from colleagues; an unresolved assumption is a reason to ask, not to keep changing formulas.

Each code block continues the same session, so run them in order in one notebook or Python prompt. Outputs were recorded with Python 3.12; error wording can differ between versions, which is itself worth knowing when you search.

Reproduce the failure before you explain it

The first move is not a theory about what went wrong. It is a small piece of code that fails on demand, because a failure you can trigger is a failure you can test a fix against. Start from the two rows that matter: one ordinary ride and the one that misbehaves.

from datetime import date, datetime, time, timedelta

rides = [
    {"rider": "A-11", "start": time(10, 0), "end": time(12, 0)},
    {"rider": "A-12", "start": time(23, 0), "end": time(6, 0)},
]
try:
    rides[0]["end"] - rides[0]["start"]
except TypeError as error:
    print("TypeError:", error)
# TypeError: unsupported operand type(s) for -: 'datetime.time' and 'datetime.time'

The first attempt does not even reach the wrong number: a clock reading is not a quantity you can subtract, because 10:00 says where you are in a day, not how much time has passed. This is a useful kind of failure. It says the type is wrong before any arithmetic can produce a plausible-looking lie.

Make the wrong answer visible

Attaching a date turns the clock readings into points in time, which can be subtracted. Now the arithmetic runs and produces the reported nonsense, so the bug is reproduced rather than described.

day = date(2024, 9, 24)
start = datetime.combine(day, rides[1]["start"])
end = datetime.combine(day, rides[1]["end"])
print(end - start)
# -1 day, 7:00:00
print((end - start).total_seconds() / 3600)
# -17.0

Read the first line before the second. Python is not confused: it is saying minus one day plus seven hours, which is exactly right for two points on the same calendar day. The seven hours you wanted is already in the output. The error is in the question — both readings were placed on one day, and the ride crossed into the next.

The same bug in a spreadsheet

This is worth a detour because the same report often starts life in a spreadsheet, where times are stored as a fraction of a day. Subtracting them gives a negative fraction for exactly the same reason, and the classic fix is a MOD formula. The Python equivalent shows what that formula is doing.

print(23 / 24, 6 / 24)
# 0.9583333333333334 0.25
print(6 / 24 - 23 / 24)
# -0.7083333333333334
print((6 / 24 - 23 / 24) % 1)
# 0.29166666666666663
print(((6 / 24 - 23 / 24) % 1) * 24)
# 6.999999999999999

With the under-24-hour, fixed-clock assumption, taking a remainder of one day recovers the intended fraction. Some day fractions, such as 1/4, are exactly representable in binary; 23/24 is not, which explains the small error here. Round for display. For approximate results, use a justified tolerance such as math.isclose; integer-minute cases can instead use exact integer arithmetic.

A rule that handles both cases

The same remainder idea works directly on durations, and it is worth naming what it is: modular arithmetic, counting on a circle that wraps at one day. Written this way, one expression covers the ordinary ride and the overnight ride, with no condition to get wrong.

print((end - start) % timedelta(days=1))
# 7:00:00
same_day = datetime.combine(day, time(12, 0)) - datetime.combine(day, time(10, 0))
print((same_day % timedelta(days=1)).total_seconds() / 3600)
# 2.0

For clock-only inputs, adding a day when the end is earlier and taking modulo one day agree under the same assumptions: a nonnegative duration under 24 hours, in one unchanged clock system. Equal readings map to zero. Neither method can verify those assumptions from the two readings. When full dates are available, subtract them directly; an IF that leaves positive dated differences alone does not lose the extra days the way modulo does.

The case the rule cannot see

Wrapping at one day assumes every ride is shorter than a day. A bike kept for two nights breaks that assumption, and the failure is silent: the rule returns a number that looks like a normal ride.

long_start = datetime(2024, 9, 24, 23, 0)
long_end = datetime(2024, 9, 26, 6, 0)
print((long_end - long_start).total_seconds() / 3600)
# 31.0
print(((long_end - long_start) % timedelta(days=1)).total_seconds() / 3600)
# 7.0

Thirty-one hours became seven. This is the edge case that decides the design: if the data only ever has clock readings, no rule can recover the missing day, and the honest fix is to carry real dates through the pipeline rather than to choose a cleverer formula. Deciding what a duration means in your data is the same question as elapsed versus active handling time: the arithmetic is easy once the definition is settled.

Read the error message

Not every failure is silent. When Python does complain, the message usually names the thing it could not do and the value it could not do it to, and reading it closely is faster than any search. Three common ones, produced on purpose:

try:
    datetime.strptime("11:00 PM", "%H:%M")
except ValueError as error:
    print("ValueError:", error)
# ValueError: unconverted data remains:  PM
row = {"rider": "A-12", "start": "23:00"}
try:
    row["ride_end"]
except KeyError as error:
    print("KeyError:", error)
# KeyError: 'ride_end'
try:
    int("06:00")
except ValueError as error:
    print("ValueError:", error)
# ValueError: invalid literal for int() with base 10: '06:00'

Separate reusable error wording from input-specific details. Here PM, 06:00, and ride_end help identify what to inspect locally. For a search, retain diagnostic type names and format directives while removing private values. Reading an error message this way helps narrow the problem; the message alone does not establish its root cause.

Read the traceback from the bottom

For a simple exception, a traceback lists calls from outermost to innermost and ends with the exception type and message. Start with that message, then inspect the innermost relevant frame in your code and follow its callers and inputs outward. The failing line may contain the bug or merely expose it; neither the outermost nor the innermost frame is automatically the root cause.

import traceback

def elapsed_hours(row):
    return (row["end"] - row["start"]).total_seconds() / 3600

try:
    elapsed_hours({"start": start, "end": "06:00"})
except TypeError as error:
    frames = traceback.extract_tb(error.__traceback__)
    print(type(error).__name__ + ":", error)
    print("raised in:", frames[-1].name)
# TypeError: unsupported operand type(s) for -: 'str' and 'datetime.datetime'
# raised in: elapsed_hours

The message names both operand types, and that pair is the diagnosis: a string arrived where a point in time was expected. Here the caller supplies text, but the function contract must also say whether parsing and validation belong to the caller or the function. Tracing a wrong value back to where it entered is the same discipline as root cause analysis on a business problem, and the five whys questions can help identify hypotheses, which still need evidence.

Cut it down to a minimal example

To prepare a question, remove unrelated columns and formatting while checking that the same failure remains. Keep file loading if it contributes to the bug. A minimal reproducible example should include the needed imports and input, the actual result, and the expected result. This standalone example reproduces the negative duration before we repair the calculation.

from datetime import date, datetime, time

on_day = date(2024, 9, 24)
a_clock = time(23, 0)
b_clock = time(6, 0)
actual = (datetime.combine(on_day, b_clock) - datetime.combine(on_day, a_clock)).total_seconds() / 3600
print(actual)
# -17.0
print("expected with next-day arrival:", 7.0)
# expected with next-day arrival: 7.0

def minutes_between(start_clock, end_clock, on_day=date(2024, 1, 1)):
    """Fixed-clock minutes in [0, 1440); caller must verify under-24h trips."""
    if not isinstance(start_clock, time) or not isinstance(end_clock, time):
        raise TypeError("clock reading expected")
    if start_clock.tzinfo is not None or end_clock.tzinfo is not None:
        raise ValueError("timezone-bearing clocks need dated instants")
    a = datetime.combine(on_day, start_clock)
    b = datetime.combine(on_day, end_clock)
    return ((b - a) % timedelta(days=1)).total_seconds() / 60

print(minutes_between(time(23, 0), time(6, 0)))
# 420.0
print(minutes_between(time(10, 0), time(12, 0)))
# 120.0

A reduced example must still show the original failure. The standalone example above records the actual -17-hour result and the expected 7 hours under the stated next-day assumption; the following function is a candidate correction. Removing a component only shows it is unnecessary for this particular reproduction, not that it is innocent in every failure. If the symptom disappears, restore the removed part and test a smaller change. Keep the docstring with the function so its limits travel with it.

Turn the message into a search query

A useful query combines the tool and version, the failing operation, and relevant error wording. Remove private identifiers, but keep diagnostic details such as operand types, function names, and format directives. The regex below only illustrates replacing simple single-quoted substrings. It is not a redaction tool: it can remove useful details and miss secrets outside those quotes. Review the complete query or shared example before sending it.

import re

def search_phrase(message):
    return re.sub(r"'[^']*'", "<value>", message)

print(search_phrase("KeyError: 'ride_end'"))
# KeyError: <value>
print(search_phrase("ValueError: invalid literal for int() with base 10: '06:00'"))
# ValueError: invalid literal for int() with base 10: <value>

When there is no error message to quote, name the operation instead of describing the appearance. “Four characters in a column” finds nothing; “SQL extract left 4 characters from string” finds the function. A technical search query improves along three axes: the tool it runs in, the operation you want, and the condition that makes your case awkward — here, times that cross midnight. Each tool has its own vocabulary, and using it is most of the skill.

Verify before you adopt

Whatever comes back from a search, a colleague, or an assistant is a claim, not an answer. It was written for someone else’s data, possibly another version of the tool. Check it against cases where you already know the result, including the boundaries — the moment either side of midnight, and the zero-length case.

cases = [
    (time(10, 0), time(12, 0), 120.0),
    (time(23, 0), time(6, 0), 420.0),
    (time(0, 0), time(0, 0), 0.0),
    (time(23, 59), time(0, 1), 2.0),
]
for start_clock, end_clock, expected in cases:
    got = minutes_between(start_clock, end_clock)
    assert got == expected, (start_clock, end_clock, got, expected)
print(len(cases), "cases passed")
# 4 cases passed

The third case assumes identical readings mean zero because the trip is known to last less than 24 hours. Clock-only data cannot distinguish that from a full-day trip. If the business permits such trips, retain dates rather than choosing a duration from identical clock readings. Keep these assumptions beside the tests so their scope stays visible.

Know what your fix assumes

The clock-only rule measures wall-clock difference on a fixed 24-hour cycle. Local UTC offsets can change, so real elapsed time needs dated, timezone-aware instants. In New York on 2024-11-03, midnight to noon spans 12 wall-clock hours and 13 elapsed hours. Python subtracts datetimes with the same tzinfo object using wall-clock fields; converting both to UTC exposes the elapsed difference.

from zoneinfo import ZoneInfo

tz = ZoneInfo("America/New_York")
a = datetime(2024, 11, 3, 0, 0, tzinfo=tz)
b = datetime(2024, 11, 3, 12, 0, tzinfo=tz)
print((b - a).total_seconds() / 3600)
# 12.0
print((b.astimezone(ZoneInfo("UTC")) - a.astimezone(ZoneInfo("UTC"))).total_seconds() / 3600)
# 13.0

Choose wall-clock or elapsed duration explicitly. A repeated local time during a backward clock change needs an offset or a fold choice; a nonexistent local time during a forward change needs a validation policy. Merely attaching ZoneInfo does not validate such input. The example uses unambiguous endpoints. If ZoneInfoNotFoundError occurs, the environment needs an IANA timezone database, supplied by the system or the tzdata package.

Keep a work log

A work log helps avoid repeating attempts and lets another person continue the investigation. Record the input and environment, what changed, the observed result, and the reason to keep or reject it. For this example, adoption is conditional on a verified under-24-hour, unchanged-clock data contract; a successful 7-hour example alone is not enough.

log = [
    {"tried": "end - start on time objects", "result": "TypeError", "kept": False},
    {"tried": "datetime.combine then subtract", "result": "-17.0 hours", "kept": False},
    {"tried": "modulo one day", "result": "7.0 hours", "kept": True},
]
for entry in log:
    print(f"{entry['tried']:32} {entry['result']:14} {'kept' if entry['kept'] else 'dropped'}")
# end - start on time objects      TypeError      dropped
# datetime.combine then subtract   -17.0 hours    dropped
# modulo one day                   7.0 hours      kept

Read as a record, those three lines also answer the question a reviewer asks later: why is there a modulo in the duration calculation? The rejected attempts explain the search, but do not establish correctness. This compact log must be read with the under-24-hour, fixed-clock assumptions and boundary tests recorded above. The wider habits around a change — tests, review, and documentation — are covered in software engineering habits for data work.

Putting it together

The report below validates the fields needed by this example and records invalid rows as skipped. The calculation rejects time-zone-bearing clock readings, since it cannot infer their dates and offset changes. It still cannot detect a multi-day trip or reversed input from clock readings alone. Use it only after establishing the fixed-clock, under-24-hour contract externally. Skipped rows must remain visible when judging report completeness.

def report(rows):
    out = []
    for row in rows:
        if not isinstance(row, dict) or not all(key in row for key in ("rider", "start", "end")):
            out.append(("unknown", "skipped: rider/start/end required"))
            continue
        try:
            minutes = minutes_between(row["start"], row["end"])
        except (TypeError, ValueError) as error:
            out.append((row["rider"], f"skipped: {error}"))
            continue
        out.append((row["rider"], f"{minutes:.0f} min"))
    return out

for rider, result in report(rides + [{"rider": "A-13", "start": "23:00", "end": time(6, 0)}]):
    print(rider, result)
# A-11 120 min
# A-12 420 min
# A-13 skipped: clock reading expected

The method transfers. Reproduce it, read what the computer said, cut it down, search with the words that match, check against cases you know, and leave a record. The ride bug took one session; the same six steps handle the ones that take three days.

Exercises

1. A ride runs from 22:30 to 01:15. Compute its length in minutes with the verified function, and say which assumption you are relying on.

print(minutes_between(time(22, 30), time(1, 15)))
# 165.0

2. Parsing "2024-09-24 23:00" with the format "%Y-%m-%d %H:%M:%S" fails. Show the message, then write the phrase you would search for.

try:
    datetime.strptime("2024-09-24 23:00", "%Y-%m-%d %H:%M:%S")
except ValueError as error:
    print("ValueError:", error)
# ValueError: time data '2024-09-24 23:00' does not match format '%Y-%m-%d %H:%M:%S'
print(search_phrase(str(ValueError("time data '2024-09-24 23:00' does not match format '%Y-%m-%d %H:%M:%S'"))))
# time data <value> does not match format <value>

3. Run the rule on a ride from 23:00 to 06:00 and on one from 06:00 to 23:00, and explain why both results are right.

overnight = [(time(23, 0), time(6, 0)), (time(6, 0), time(23, 0))]
print([minutes_between(a, b) / 60 for a, b in overnight])
# [7.0, 17.0]

References: Python datetime, Python traceback, Python zoneinfo, how to create a minimal reproducible example.