Data Fundamentals: Structures, File Formats, Databases, and Analytical Systems

Data appears throughout modern business, science, technology, and everyday life. Transactions, sensor readings, customer records, photographs, emails, and social media posts are all forms of data.

However, data does not arrive in a single structure or format. Its structure affects how it should be collected, transferred, stored, queried, and analyzed.

This article introduces four essential areas of data fundamentals:

• Structured, semi-structured, and unstructured data
• Common data sources
• File formats used to exchange data
• Relational and non-relational databases
• The distinction between OLTP and OLAP workloads

What Is Data?

Data consists of recorded facts, observations, measurements, symbols, or representations that can be processed to produce information.

Examples include:

• A customer’s name and address
• The temperature recorded by a sensor
• A completed financial transaction
• An image captured by a camera
• The text of an email
• A product review
• A sequence of website clicks

Raw data does not automatically provide useful insight. It must usually be organized, validated, interpreted, and analyzed within an appropriate context.

The structure of the data is one of the first characteristics to consider.

The Three Main Data Structures

Data is commonly divided into three broad categories:

1. Structured data
2. Semi-structured data
3. Unstructured data

These categories are useful, but their boundaries are not always absolute. A single source can contain more than one type of data.

Structured Data

Structured data follows a predefined model. It is commonly organized into rows and columns, with each field assigned a defined meaning and data type.

Consider this customer table:

Each row represents a customer, while each column represents a particular attribute.

The schema might specify that:

• customer_id must be an integer.
• name must contain text.
• city must contain text.
• signup_date must be a date.

Because the structure is known in advance, this data can be searched, filtered, aggregated, and validated efficiently.

Examples of Structured Data

Common sources include:

• Relational database tables
• Transaction records
• Inventory systems
• Point-of-sale systems
• Standardized online forms
• Sensor measurements with fixed fields
• Well-organized spreadsheet tables

A spreadsheet can contain structured data when it uses consistent columns and data types. However, spreadsheets do not automatically enforce a rigorous schema. Merged cells, inconsistent headings, formulas, notes, and multiple unrelated tables can make spreadsheet data less structured than a database table.

Advantages of Structured Data

Structured data generally provides:

• Consistent organization
• Straightforward querying
• Efficient aggregation
• Easier validation
• Compatibility with established analytical tools
• Clear relationships between fields

Its main limitation is that a predefined schema may be inconvenient when records vary substantially or requirements change frequently.

Unstructured Data

Unstructured data does not follow a predefined tabular model that can be represented naturally as rows and columns.

Examples include:

• Photographs
• Audio recordings
• Videos
• PDF documents
• Free-form text
• Presentation files
• Scanned documents
• Social media content
• Customer-service conversations

A photograph, for example, cannot be reduced naturally to a few conventional database columns. It contains complex visual information that may require image-processing or machine-learning techniques.

Similarly, the message body of an email may contain topics, opinions, names, dates, and requests without assigning them to predefined fields.

Unstructured Does Not Mean Meaningless

Unstructured data can still contain substantial organization and meaning. The word unstructured means that the content does not conform to a fixed model that conventional database systems can process directly.

Specialized techniques can extract structure from it. Examples include:

• Natural language processing for documents
• Speech recognition for audio
• Computer vision for images
• Optical character recognition for scanned pages
• Metadata extraction from media files

A web page illustrates the distinction particularly well. Its HTML markup has a defined structure, but the article text, images, videos, and other content embedded in that structure may be unstructured.

Semi-Structured Data

Semi-structured data falls between structured and unstructured data.

It does not normally use a rigid table in which every record has exactly the same columns. Instead, it uses keys, tags, attributes, or hierarchical relationships to identify and organize values.

Common examples include:

• JSON documents
• XML documents
• Emails
• Application logs
• Web pages
• NoSQL documents
• Event messages

JSON Example

{
  "customer_id": 1001,
  "name": "Maria Chen",
  "interests": [
    "statistics",
    "data engineering"
  ],
  "contact": {
    "email": "maria@example.com"
  }
}

{
  "customer_id": 1001,
  "name": "Maria Chen",
  "interests": [
    "statistics",
    "data engineering"
  ],
  "contact": {
    "email": "maria@example.com"
  }
}

This document has an identifiable structure, but it is not a conventional table. It contains nested objects and an array.

JSON is a lightweight data-interchange format based primarily on name-value objects and ordered arrays. These structures can be nested, making JSON suitable for hierarchical data and web APIs. JSON’s specification and basic structures are summarized at JSON.org.

XML Example

<customer id="1001">
    <name>Maria Chen</name>

    <interests>
        <interest>statistics</interest>
        <interest>data engineering</interest>
    </interests>

    <email>maria@example.com</email>
</customer>

<customer id="1001">
    <name>Maria Chen</name>

    <interests>
        <interest>statistics</interest>
        <interest>data engineering</interest>
    </interests>

    <email>maria@example.com</email>
</customer>

XML uses tags, attributes, and nested elements to represent relationships.

Email as Semi-Structured Data

An email combines structured and unstructured components.

Structured fields include:

• Sender
• Recipient
• Subject
• Date
• Message ID

The message body is generally free-form text and therefore more appropriately treated as unstructured content.

Comparing the Three Structures

The appropriate classification depends partly on how the data is represented and used.

Common Sources of Data

Organizations obtain data from many internal and external sources.

Operational Systems

Business applications generate data while supporting routine operations.

Examples include:

• Customer relationship management systems
• Enterprise resource planning systems
• Human resources platforms
• Banking systems
• E-commerce platforms
• Point-of-sale systems

Flat Files and Spreadsheets

Organizations frequently exchange data through:

• CSV files
• TSV files
• Excel workbooks
• Google Sheets
• Plain-text files

These formats are convenient for smaller data exchanges but may not provide the controls, concurrency, or scalability of a database system.

APIs and Web Services

APIs allow applications to request or submit data through defined interfaces.

API responses are frequently delivered in:

• JSON
• XML
• CSV
• Binary formats

Because an API provides a documented interface, it is generally preferable to web scraping when both options are available.

Web Scraping

Web scraping extracts selected information from web pages when no suitable API or downloadable dataset is available.

The process normally involves:

1. Downloading the page
2. Parsing its HTML
3. Locating the required elements
4. Converting their contents into structured records

Data Streams

Streaming sources continually generate new events.

Examples include:

• IoT sensors
• Application logs
• Financial market events
• Website clicks
• GPS devices
• Industrial equipment
• Social media activity

Streaming systems often require continuous ingestion and near-real-time processing.

Common File Formats for Data Exchange

A file format defines how information is represented in a file. Selecting an appropriate format affects portability, storage, data types, validation, and processing performance.

Delimited Text Files

Delimited files store records as lines and separate fields using a designated character.

The two most common examples are:

• CSV: comma-separated values
• TSV: tab-separated values

Example CSV:

customer_id,name,city
1001,Maria Chen,Phoenix
1002,David Smith,Chicago

customer_id,name,city
1001,Maria Chen,Phoenix
1002,David Smith,Chicago

Delimited files are widely supported and easy to inspect. However, they have limitations:

• Data types are not strongly preserved.
• Nested data is difficult to represent.
• Missing values may be ambiguous.
• Escaping delimiters and quotation marks requires care.
• Different programs may interpret dates and numbers differently.

The acronym CSV means comma-separated values, not “comma-separated variables.”

Spreadsheet Files

Spreadsheet formats such as XLSX can contain:

• Multiple worksheets
• Formulas
• Charts
• Formatting
• Named ranges
• Pivot tables
• Comments

Spreadsheets are useful for interactive analysis and business reporting. However, complex formatting and formulas can make them unsuitable as dependable machine-to-machine exchange formats.

A spreadsheet application can export a worksheet as CSV, but an XLSX workbook and a CSV file are fundamentally different:

• XLSX can contain multiple worksheets and formatting.
• CSV stores a single plain-text table.
• CSV does not preserve formulas, charts, colors, or workbook relationships.

JSON

JSON is commonly used by APIs, web applications, configuration files, and event systems.

Advantages include:

• Human-readable syntax
• Nested objects and arrays
• Wide programming-language support
• Natural representation of many application objects

A limitation is that large JSON documents can require more storage and parsing work than optimized binary formats.

XML

XML uses named tags and attributes to organize information.

It supports:

• Hierarchical structures
• Namespaces
• Validation schemas
• Detailed document-oriented representations

XML is still used in enterprise integrations, document systems, configuration files, financial reporting, and legacy services.

Compared with JSON, XML is frequently more verbose but can provide sophisticated document and validation capabilities.

Data Repositories

A data repository is a system or storage location used to collect, organize, retain, and make data available.

Examples include:

• Relational databases
• Non-relational databases
• Data warehouses
• Data marts
• Data lakes
• Object storage
• Distributed big-data systems

A repository should be selected according to factors such as:

• Data structure
• Data volume
• Read and write patterns
• Query requirements
• Consistency requirements
• Response-time expectations
• Scalability
• Security
• Governance
• Cost

There is no single repository that is best for every workload.

Relational Databases

A relational database organizes data into tables consisting of rows and columns. Tables can be connected through keys that represent relationships between records.

Consider two tables.

Customers

Orders

The customer_id field connects each order to the appropriate customer.

A query can combine the tables:

SELECT
    customers.customer_name,
    orders.order_id,
    orders.amount
FROM customers
JOIN orders
    ON customers.customer_id = orders.customer_id;

SELECT
    customers.customer_name,
    orders.order_id,
    orders.amount
FROM customers
JOIN orders
    ON customers.customer_id = orders.customer_id;

Relational Database Characteristics

Relational databases generally provide:

• Defined schemas
• Tables, rows, and columns
• Relationships implemented with keys
• Constraints for data integrity
• SQL querying
• Transaction support
• Indexes for faster retrieval
• Controlled concurrent access

Common relational database management systems include:

• PostgreSQL
• MySQL
• Microsoft SQL Server
• Oracle Database
• IBM Db2

Relational systems are particularly effective when the data has clear relationships and operations require strong consistency.

Non-Relational Databases

Non-relational databases are commonly grouped under the term NoSQL, meaning “not only SQL.”

Rather than requiring every dataset to fit a relational table model, these systems provide other ways to represent and access information.

Four common categories are:

1. Document databases
2. Key-value databases
3. Wide-column databases
4. Graph databases

Document Databases

Document databases store records as documents containing fields, nested objects, and arrays.

MongoDB is a prominent example. Its flexible model allows documents within one collection to contain different fields or field types, although production systems should still design and govern their schemas carefully. MongoDB’s data-modeling documentation recommends structuring documents around application access patterns.

Key-Value Databases

A key-value system associates each unique key with a value.

Example:

"user:1001" → user session data

"user:1001" → user session data

These systems are useful for:

• Caching
• Session storage
• User preferences
• Fast lookups

Redis is a well-known example.

Wide-Column Databases

Wide-column databases organize data into flexible column families and are designed for distributed workloads.

Examples include:

• Apache Cassandra
• Apache HBase

They are often used for high-volume event, time-series, and distributed application data.

Graph Databases

Graph databases represent data as nodes and relationships.

They are useful when connections are central to the analysis, including:

• Social networks
• Fraud detection
• Recommendation systems
• Network management
• Knowledge graphs

Neo4j is a prominent graph database.

Relational and Non-Relational Databases Compared

NoSQL databases are not limited to unstructured data. They can store highly organized information, and many implement validation rules and transaction capabilities.

Similarly, relational databases are not limited exclusively to rigid scalar values. Modern relational systems may support JSON, arrays, spatial values, documents, and other advanced types.

The correct choice depends on the workload rather than a simple structured-versus-unstructured rule.

OLTP Systems

OLTP stands for Online Transaction Processing.

OLTP systems support frequent operational transactions such as:

• Placing an order
• Recording a payment
• Updating inventory
• Booking a flight
• Transferring money
• Changing a customer address

These workloads normally require:

• Fast inserts and updates
• Short queries
• Many concurrent users
• Reliable transactions
• Strong integrity controls
• Current operational data

Relational databases are frequently used for OLTP because their transaction and constraint features match these requirements. However, some non-relational systems also support transactional workloads.

OLAP Systems

OLAP stands for Online Analytical Processing.

OLAP workloads focus on analyzing data across many records, dimensions, and periods.

Typical questions include:

• How did sales vary by product and region?
• Which customer groups had the highest retention?
• What was the quarterly revenue trend?
• Which campaigns produced the strongest returns?
• How have operational costs changed over time?

Analytical workloads generally involve:

• Large scans
• Historical data
• Aggregations
• Complex joins or multidimensional operations
• Fewer data modifications
• Reporting and business intelligence

Data warehouses are commonly designed for these workloads.

OLTP and OLAP Compared

OLTP and OLAP describe workload patterns, not simply two exclusive database products.

A relational database can support either type, depending on its design and configuration. Modern analytical environments can also use relational warehouses, columnar platforms, data lakes, lakehouses, and distributed processing systems.

Choosing an Appropriate Storage Solution

The following questions can guide repository selection:

• Is the data structured, semi-structured, or unstructured?
• Are relationships between records important?
• Must transactions be processed reliably?
• How often will data be inserted or updated?
• What kinds of queries will be performed?
• Does the system need real-time responses?
• How quickly will the data volume grow?
• Is horizontal distribution required?
• What security and compliance controls apply?
• Will analysts need current, historical, or raw data?

For example:

• A payment system may benefit from a relational OLTP database.
• A product catalog with highly variable attributes may benefit from a document database.
• A recommendation network may benefit from a graph database.
• Historical enterprise reporting may require a data warehouse.
• Raw data in many formats may be retained in a governed data lake.

In practice, organizations frequently combine multiple repositories rather than selecting only one.

Key Takeaways

• Data consists of recorded facts, observations, measurements, symbols, and representations.
• Structured data follows a predefined model, commonly using rows and columns.
• Semi-structured data uses keys, tags, or hierarchies without requiring a rigid table.
• Unstructured data includes free-form text, images, audio, video, and documents.
• CSV and TSV are simple delimited text formats.
• XLSX workbooks can preserve formulas, formatting, and multiple worksheets.
• JSON and XML can represent hierarchical data and are widely used for data exchange.
• Relational databases organize related data into tables.
• Non-relational databases include document, key-value, wide-column, and graph systems.
• NoSQL does not mean that the data has no structure.
• OLTP systems support operational transactions.
• OLAP systems support complex analysis and reporting.
• Repository selection should be based on workload, structure, scale, governance, and access requirements.

Conclusion

Understanding data structure is fundamental to effective data engineering and analytics. Structured, semi-structured, and unstructured data each require different approaches to storage and processing. File formats determine how data moves between systems, while repositories determine how it is retained, governed, queried, and made available.

Relational databases remain essential for structured relationships and transactional integrity. Non-relational databases offer alternative models for flexible, connected, or distributed data. OLTP and OLAP further distinguish systems built for daily operations from those optimized for analytical decision-making.

One-sentence summary: Data fundamentals connect the structure and format of information with the database and processing systems best suited to store, exchange, and analyze it.