How Computers Store Data

In plain terms

A computer stores names, dates, and amounts as bit patterns. A format tells a program how to interpret each pattern: the same byte can represent a number or part of a letter, depending on that agreement. If a reader uses the wrong agreement, it may reject the data or accept it with a different meaning.

A name decoded with the wrong encoding can arrive as 김민수. Binary floating-point arithmetic can change a rounding result, and a file layout can restrict parallel reads. These are different problems at different layers of storage; recognizing the layer helps you choose the right fix.

The Python examples reproduce representation errors rather than a production incident or a runtime prediction. Basic Python reading is enough to follow them. They use Python 3.9 or later, NumPy, pandas, and IANA time-zone data for the zoneinfo example.

Bits and bytes

A bit is one on/off switch, 0 or 1. Eight bits make a byte, which can hold 2⁸ = 256 distinct patterns, usually read as the numbers 0 to 255. A file is a long row of bytes; so is a network message; so is a database page.

Because 256 possibilities is not many, larger values are built by combining bytes and agreeing on how to read the combination. Two questions follow from that:

• How many bytes is one value? (Where does this value end and the next begin?)
• How should those bytes be interpreted? (As a number, as text, as a moment in time?)

A file format is a written answer to those two questions. So is a database column type, and so is an API contract.

For multi-byte numbers, byte order is another part of the agreement. The two bytes 01 00 represent 256 as an unsigned big-endian integer and 1 as little-endian. A format must define width, signedness, and byte order; the byte pattern alone does not choose them.

Text: how names break

A character encoding specifies how text is represented as bytes. ASCII covers 128 characters, including English letters, digits, punctuation, and control characters, and uses one byte per character. It cannot represent 한글, 漢字, é, or emoji.

Different regions solved that differently and incompatibly. Korea used EUC-KR, Japan Shift-JIS, Western Europe ISO-8859-1 and its Windows cousin Windows-1252. A byte sequence could mean different characters under different encodings, so the reader needed the writer’s encoding. That information should come from the contract or metadata; guessing can fail even when decoding succeeds.

Unicode assigns code points to characters; UTF-8 encodes Unicode scalar values in one to four bytes. ASCII text remains one byte per character, and precomposed Korean syllables use three bytes each. A displayed emoji can combine several code points and therefore need more than four bytes. Python’s Unicode HOWTO explains the distinction. str.encode produces bytes; len counts code points for a string and bytes for a byte sequence:

for text in ("hello", "김민수", "café"):
    print(text, len(text), len(text.encode("utf-8")))
# hello 5 5
# 김민수 3 9
# café 4 5

The second number counts code points, which match visible characters in these examples; the third counts bytes. Combining marks can make a visible character span several code points. A byte limit and a character limit therefore answer different questions: always check the database type and configuration before interpreting a declared length.

Reading garbled text

Reading bytes with the wrong encoding can produce garbled text, often called mojibake, or raise a decoding error. The examples below reproduce three specific failure paths. Their appearance is a clue, not proof of a unique cause:

name = "김민수"
raw = name.encode("utf-8")
print(raw.decode("cp1252"))
# 김민수
print(raw.decode("cp1252").encode("cp1252") == raw)
# True
print(name.encode("ascii", errors="replace").decode("ascii"))
# ???
print(raw[:4].decode("utf-8", errors="replace"))
# 김�

• 김민수: UTF-8 bytes read through the Windows-1252 table. Each Korean syllable became three nonsense characters. The second line shows the important part: encoding the garbage back with the same wrong table returns the original bytes exactly. In this example the wrong decode was reversible. Recovery elsewhere depends on retaining the bytes without replacement or other destructive transformations.
• ???: the data was converted into an encoding with no room for those characters. Those replaced characters cannot be recovered from these question marks alone; reload an intact source or backup. Literal question marks in unrelated data do not prove that replacement occurred.
• �, the replacement character: a decoder met bytes that are not valid UTF-8 (here, a syllable cut off after its first byte) and substituted a placeholder. Recoverable only if the original bytes still exist.

At ingestion, decode source bytes using the agreed source encoding into Unicode text. When writing text back out, encode it as UTF-8 if that is the destination contract. Decoding and encoding are opposite operations. Retain source bytes and encoding metadata when recovery matters, and quarantine undecodable or mixed-encoding input instead of silently replacing characters.

Numbers: integers have ceilings and floats have gaps

Fixed-width integers have a range set by their bit width and signedness. A signed 32-bit integer ranges from −2,147,483,648 to 2,147,483,647. Python’s ordinary int grows as needed, but NumPy int32 does not. Database and library behavior at the boundary differs; this NumPy array example wraps around:

import numpy as np

ids = np.array([2_147_483_647], dtype=np.int32)
print(ids + 1)
# [-2147483648]

The array wrapped without an exception. Choose a counter’s width from its expected lifetime range, including headroom; 64-bit integers are a common choice. Identifiers with leading zeros or nonnumeric syntax belong in text. Numeric database keys can remain integers when every consumer supports their range.

Fractions are stored differently. A floating-point number (FLOAT, DOUBLE) holds a value in binary, and most decimal fractions have no exact binary form, just as 1/3 has no exact decimal form. The stored value is the nearest representable one, and arithmetic on nearest values drifts:

print(0.1 + 0.2)
# 0.30000000000000004
print(0.1 + 0.1 + 0.1 == 0.3)
# False
print(round(2.675, 2))
# 2.67

The first two results show that exact equality is unsuitable for many floating-point calculations. The third also involves a rounding policy: Python’s round uses ties-to-even, and the stored binary value of 2.675 is slightly below the decimal value. A financial system must specify both representation and rounding rules.

For monetary amounts, use decimal arithmetic or integer minor units, with an explicit currency, scale, and rounding policy. Decimal arithmetic can represent decimal inputs exactly, but division and limited precision can still require rounding. Construct Decimal from text rather than an already approximated float. This example explicitly chooses ROUND_HALF_UP; it is not a universal accounting rule:

from decimal import Decimal, ROUND_HALF_UP

print(Decimal("0.1") + Decimal("0.2"))
# 0.3
print(Decimal("2.675").quantize(Decimal("0.01"), rounding=ROUND_HALF_UP))
# 2.68

Floats fit approximate measurements such as latency or temperature. Compare them using a tolerance justified by the measurement or calculation. Financial reconciliation instead needs an agreed decimal scale and rounding rule; see Python’s decimal documentation.

A binary floating-point value combines a sign, a significand (the stored significant digits), and a power of two. Python’s usual binary64 float has 53 bits of significand precision for normal values. The gaps between representable numbers grow with magnitude; 64 bits do not provide 64 decimal digits or a fixed decimal step. Infinity and NaN are special values, so a storage contract must also say whether they are allowed and how they are represented.

For very small magnitudes, underflow can round a result to a subnormal value or to zero. Binary64 can still represent values below its smallest positive normal value (about 2.225 × 10⁻³⁰⁸); those subnormal values have reduced relative precision. A positive product that rounds to zero can no longer be recovered by taking its logarithm. Compute in log-space before multiplying when the application can use a log result:

import math

print(1e-310 > 0)
# True
print(1e-200 * 1e-200)
# 0.0
log_product = 2 * math.log(1e-200)
print(round(log_product, 3))
# -921.034
print(math.exp(log_product))
# 0.0

The first result is nonzero despite being below the normal range. The product rounds to zero, while its logarithm remains finite. Converting that log result back with exp still rounds to zero. Log-space preserves a useful representation; it does not expand the float format’s range. See NumPy’s floating-point limits.

Time: a number that needs a unit and a zone

An epoch timestamp counts units from a defined origin. Here we use Unix time: seconds since 1 January 1970 at 00:00 UTC, ignoring leap seconds. Other systems use different origins or units. A Unix timestamp identifies an instant without storing a local zone; a local wall-clock label needs additional context.

from datetime import datetime, timezone

print(datetime.fromtimestamp(1_757_200_000, tz=timezone.utc).isoformat())
# 2025-09-06T23:06:40+00:00
seconds_per_year = 365.2425 * 86_400
print(round(1970 + 1_757_200_000_000 / seconds_per_year))
# 57653

• Unit. 1757200000 is a moment in September 2025 if it is seconds. The same digits with three zeros appended is that moment in milliseconds, and reading it as seconds lands it in the year 57,653, so far out that the standard library refuses to build a date from it. For dates near these examples, 10 digits suggest seconds and 13 suggest milliseconds. This is a sanity check, not a substitute for a declared unit.
• Zone. 2026-03-14 09:00:00 is a local clock label, not a uniquely identified instant without a zone or offset. For observed events, resolve that context before storing an instant in UTC. Preserve a named zone when local rules matter; a future “09:00 every day” schedule needs the local time and zone rule, not just one UTC instant.
• Daylight saving. Where clocks move by one hour, local times can repeat or be skipped, producing 23-hour and 25-hour local days. Not every zone changes clocks or uses a one-hour shift. UTC has no such gaps.
• Whose clock. The time a phone recorded and the time a server received are different numbers, sometimes hours apart, because devices have wrong clocks. Keep both.

The installed time-zone database gives the following elapsed durations for the Los Angeles transition dates in 2026. Convert local midnights to epoch timestamps to measure elapsed time:

from zoneinfo import ZoneInfo

la = ZoneInfo("America/Los_Angeles")
for start, end in (((2026, 3, 8), (2026, 3, 9)), ((2026, 11, 1), (2026, 11, 2))):
    elapsed = datetime(*end, tzinfo=la).timestamp() - datetime(*start, tzinfo=la).timestamp()
    print(start, elapsed / 3600)
# (2026, 3, 8) 23.0
# (2026, 11, 1) 25.0

The code converts each midnight to an epoch timestamp before subtracting. Subtracting two aware datetimes that share a zone object skips the offsets and reports 24 hours for both days, which measures the wall-clock difference rather than the elapsed duration needed here.

For text interchange, an ISO 8601 form such as 2026-03-14T09:00:00Z makes UTC explicit. Lexical order matches time order when every value uses the same fixed-width format, precision, and offset, as below. Mixed offsets or precision can break that property. Python’s datetime documentation also explains aware-datetime subtraction.

stamps = ["2026-03-14T09:00:00Z", "2025-12-31T23:59:59Z", "2026-01-02T00:00:00Z"]
print(sorted(stamps))
# ['2025-12-31T23:59:59Z', '2026-01-02T00:00:00Z', '2026-03-14T09:00:00Z']

Serialization: turning a record into a row of bytes

A customer record in memory may contain runtime-specific layouts and references. Serialization represents its logical values in a format that can be stored or exchanged, rather than copying arbitrary memory addresses. Deserialization reconstructs values from that representation. The two programs need compatible format and schema rules, not identical memory layouts.

A serialization format defines how values and record boundaries are represented. A schema adds field types and expectations such as required fields, units, and allowed nulls. Some formats embed a schema; others require a separate contract. A parseable record can still violate that contract.

A storage type and a data value’s meaning are separate. An integer customer ID is a category label, so averaging it does not yield an average customer. A data model also describes relationships: an order’s customer ID refers to a customer record, not merely to another integer. Types and a schema help state those expectations but do not establish that the records are accurate or complete.

“Structured” usually means records follow defined fields and relationships; “semi-structured” often describes tagged or nested records whose fields can vary. Images, audio, and free text are often called unstructured relative to a table model, but their files still have encodings or format rules. The distinction is about the organization of the content, not whether bytes have any structure.

Format	Human-readable	Schema lives	Typical use
CSV	yes	external contract; a header names columns	small exchanges, spreadsheets
JSON	yes	field names in objects; no full schema by default	APIs, configuration, nested data
Avro	no	object container header; separate for raw messages	row-by-row event storage
Protobuf	no	in a separate .proto file	service-to-service messages
Parquet	no	in the file footer	analytical tables

CSV has no built-in column types. A reader that infers numbers can turn 007 into 7 and remove leading zeros from identifiers:

import io
import pandas as pd

csv_text = "order_id,zip,phone\n007,02138,01012345678\n"
print(pd.read_csv(io.StringIO(csv_text)).iloc[0].to_dict())
# {'order_id': 7, 'zip': 2138, 'phone': 1012345678}
print(pd.read_csv(io.StringIO(csv_text), dtype=str, keep_default_na=False).iloc[0].to_dict())
# {'order_id': '007', 'zip': '02138', 'phone': '01012345678'}

The first read silently changed three identifiers. Explicit text types preserve them; keep_default_na=False also prevents default missing markers such as NA from being treated as null in this example. CSV does have documented quoting rules: RFC 4180 describes quoting fields containing commas or line breaks and doubling embedded quotes. Implementations still vary. Agree on the dialect, encoding, types, and null markers; otherwise an empty string can be confused with a missing value. With that contract and a proper parser, CSV can round-trip data reliably.

JSON describes structure honestly and pays for it in size: every record repeats every field name, so a thousand records repeat "customer_id" a thousand times. A second problem is subtler. JSON has no integer type, only “number”, and any parser that reads numbers as 64-bit floats (JavaScript’s among them) cannot represent every integer outside the interoperable exact range −(2⁵³−1) through 2⁵³−1. The final line below simulates such a conversion; Python’s default JSON parser itself preserves integer tokens as Python integers:

import json

record = {"customer_id": 1, "country": "KR", "amount": 12.5}
print(len(json.dumps(record)), len(json.dumps([record] * 1000)))
# 51 53000
print(float(9007199254740993))
# 9007199254740992.0

The converted identifier has changed. RFC 8259 discusses this interoperability boundary. Use strings for identifiers when consumers cannot preserve their numeric range. Field names alone are not a schema: JSON does not tell a reader whether amount is dollars or cents, whether it is required, or whether a missing field differs from explicit null.

Rows and columns

Consider a table of 10 million orders with 40 fields, and the question “what was total revenue last month?”, which needs 2 of the 40.

A row-oriented file keeps each order’s fields together. In a plain CSV scan without an index, reading 2 fields still requires scanning the record text to find them. This does not mean every row-oriented system reads the whole table: indexes, partitions, or block metadata can avoid unrelated records. Column selection and row selection are different opportunities to skip work.

A column-oriented layout groups values by column. Parquet does this within row groups: each group contains column chunks, which contain pages. A reader can select the needed chunks using file metadata. With similarly sized columns, selecting 2 of 40 can greatly reduce data bytes read, though metadata and row-group selection also matter.

Grouping similar values also helps dictionary encoding and compression. Low-cardinality country codes can be represented by compact dictionary indices, though the exact saving depends on the data. The Parquet file-format overview shows the layout.

Row formats keep their place. When events are written one at a time and later read as whole records, rows are the better fit, which is why event logs are often Avro or JSON while analytical tables are Parquet.

Compression, and whether a file can be split

With lossless compression, decompression reconstructs the original bytes exactly. A smaller file is useful only if the reader can also interpret those restored bytes correctly.

Here compression means lossless compression: decompression restores exactly the input bytes. Codecs exploit redundancy and encode common patterns compactly; no lossless method can make every possible input smaller. Repetitive data often shrinks well, while already compressed or random-like data can grow because of format overhead:

import zlib

repetitive = b"KR,paid,2026-03-14\n" * 100_000
noise = np.random.default_rng(0).integers(0, 256, size=len(repetitive), dtype=np.uint8).tobytes()
print(len(repetitive), len(zlib.compress(repetitive)), len(zlib.compress(noise)))
# 1900000 4653 1900586

The repetitive buffer shrank four hundred times over; the random one of the same length grew slightly, because the format’s own bookkeeping was added to bytes that could not be reduced. Real tables sit between the two, and results depend on the data, codec, compression level, and implementation. The table gives broad tendencies rather than a benchmark:

Codec	Ratio	Speed	Common use
Snappy	low	very fast	data queried constantly, where CPU time matters more than bytes
Zstd	high	fast	a candidate when both space and speed matter
Gzip	high	slow	exchange files and archives, where broad reader support matters

Parallel reads need independently decodable pieces and a reader that can locate them. An ordinary gzip stream uses decoding state from earlier bytes, so starting at an arbitrary offset is not enough. The next example passes the middle of a zlib-wrapped DEFLATE stream to a decoder expecting its header. That error alone does not prove the history-state requirement; it shows that an arbitrary slice is not a complete stream. Independent blocks or an index with restart information need separate reader support:

compressed = zlib.compress(repetitive)
try:
    zlib.decompress(compressed[len(compressed) // 2:])
except zlib.error as exc:
    print("cannot start in the middle:", exc)
# cannot start in the middle: Error -3 while decompressing data: incorrect header check

With an ordinary unsplittable CSV+gzip input, many distributed readers assign one file to one input task; downstream processing can still run in parallel. Parquet normally compresses pages independently and supplies metadata for locating column chunks. Parallelism still depends on row groups, reader support, and task planning: one file does not guarantee many useful tasks. Wrapping the entire Parquet file in an external gzip stream removes ordinary random access. Codec, container layout, and reader support must be considered together.

The fix is therefore a different layout: many medium-sized files instead of one large one (a few hundred megabytes each is a common target), or a format such as Parquet that stays splittable.

Follow one record there and back

JSON serialization builds text with record structure. UTF-8 encodes that text as bytes. Lossless compression transforms those bytes into another byte sequence. Reading reverses the order: decompress, decode, then deserialize. Binary serializers combine structure and byte representation without a separate text step. A tiny compressed record may be larger than its input; compression is not encryption.

import json
import zlib

record = {"name": "김민수", "amount_minor": 1250, "currency": "USD"}
serialized = json.dumps(record, ensure_ascii=False)
encoded = serialized.encode("utf-8")
compressed = zlib.compress(encoded)
restored_bytes = zlib.decompress(compressed)
restored = json.loads(restored_bytes.decode("utf-8"))
print(restored_bytes == encoded)
# True
print(restored == record)
# True

The first check tests exact byte recovery; the second tests this JSON-compatible record’s values. Neither validates the business schema. A reader must still agree that amount_minor is in USD cents, and a schema change must be tested against existing readers. Avro’s specification distinguishes schemas embedded in object container files from separately supplied writer schemas for binary messages.

Where bytes live, and what a request costs

The following numbers are illustrative orders of magnitude for discussing access latency, not measurements of this laptop or guarantees for a provider. Hardware, request size, locality, cache state, and concurrency change them:

Layer	Time to reach data	Relative
CPU cache	~1 nanosecond	1×
Main memory (RAM)	~100 nanoseconds	100×
Local SSD	~100 microseconds	100,000×
Another machine on the network	~500 microseconds	500,000×
Cloud object storage (S3, GCS, ADLS)	tens of milliseconds per request	~10,000,000× and up

Suppose a request has 30 ms of fixed latency. Making 100,000 requests sequentially then adds 3,000 seconds (50 minutes) of waiting; making 100 adds 3 seconds. These figures exclude data transfer, parsing, retries, and other work. Parallel requests can overlap latency, and provider pricing determines request charges. The calculation illustrates per-request overhead, not total runtime for either dataset.

This per-file overhead contributes to the small-file problem. Metadata, file opening, and scheduling also cost work on local storage. Local timings cannot be extrapolated directly to object storage.

Decisions to make once

• Encoding. Decode source bytes to Unicode using the source encoding; encode outgoing text as agreed. Preserve source metadata and quarantine invalid input.
• Money. DECIMAL or integer minor units. A float’s error can be hidden by display rounding or tolerant checks, and may surface only when a reconciliation compares amounts exactly.
• Time. Store resolved event instants in UTC, retain zone rules for local schedules, and declare the unit (event_time_ms).
• Format. Consider Parquet for analytical scans and Avro or JSON for records; CSV needs an explicit dialect and schema contract.
• File size. Balance request overhead, available parallelism, and memory. Hundreds of megabytes can be a starting point to measure, not a fixed rule.

Agreeing on these early can reduce rework, although the effort depends on the environment; format and file-size choices in particular need measurement. Decided late, they can show up as a pipeline that never fails and quietly produces something slightly wrong.