Modern organizations collect data from a diverse and constantly changing set of sources. Information may originate in an internal transactional database, arrive as a file from an external provider, be requested through an API, extracted from a public webpage, or produced continuously by sensors and applications.

Understanding these sources helps data engineers select appropriate methods for ingestion, validation, storage, and processing.

Common data-source categories include:

1. Relational databases and data warehouses
2. Flat files and spreadsheets
3. XML and other semi-structured datasets
4. APIs and web services
5. Web scraping
6. Data streams and event feeds
7. RSS and other subscription feeds

Each source presents different requirements for connectivity, security, reliability, frequency, and data quality.

1. Relational Databases

Organizations use operational applications to manage activities such as:

• Customer transactions
• Sales
• Inventory
• Human resources
• Billing
• Accounting
• Supply-chain operations
• Internal workflows

Many of these applications store their data in relational database management systems.

Common relational database platforms include:

• Microsoft SQL Server
• Oracle Database
• MySQL
• PostgreSQL
• IBM Db2

Relational databases organize structured data into tables connected through defined relationships.

Using Operational Databases as Sources

Operational data can be extracted for reporting and analysis.

For example:

• Retail transactions can support regional sales analysis.
• Customer relationship management data can support sales projections.
• Inventory data can be used to identify stock shortages.
• Human-resources records can support workforce planning.
• Billing data can be used to analyze revenue and payment behavior.

Methods for Extracting Database Data

Data engineers may retrieve data through:

• SQL queries
• Scheduled exports
• Database connectors
• Replication
• Change data capture
• Transaction logs
• Application APIs

The appropriate method depends on how frequently the data must be updated and how much impact the extraction can have on the source system.

Protecting Operational Performance

Analytical queries can consume substantial computing, memory, and storage resources. Running large queries directly against a production transactional database can affect the performance of the application it supports.

Organizations may therefore:

• Extract data during low-activity periods
• Use read replicas
• Capture incremental changes
• Copy data into a warehouse or lake
• Restrict resource-intensive queries
• Monitor extraction workloads

The objective is to acquire analytical data without disrupting normal business operations.

2. Data Warehouses

A data warehouse may also serve as a source for downstream analysis.

Unlike an operational database, a warehouse is specifically designed for analytical workloads. It commonly contains:

• Integrated data from multiple systems
• Historical records
• Curated analytical tables
• Consistent business definitions
• Aggregated metrics

Analysts, BI tools, data scientists, and downstream data products may all retrieve information from a warehouse.

A warehouse is therefore both a destination for upstream pipelines and a source for downstream consumers.

3. External and Third-Party Datasets

Organizations do not rely exclusively on internally generated data. External information can provide additional context and improve decision-making.

Sources may include:

• Government agencies
• Academic institutions
• Industry associations
• Open-data portals
• Commercial data providers
• Research organizations
• Financial-data services
• Weather services
• Geographic-data providers

Examples of external datasets include:

• Demographic statistics
• Economic indicators
• Weather observations
• Market prices
• Geographic boundaries
• Consumer research
• Point-of-sale information
• Industry benchmarks

Organizations may use this information to:

• Forecast demand
• Select business locations
• Plan inventory
• Assess market conditions
• Design promotions
• Analyze risk
• Enrich internal customer or product data

Evaluating External Data

Before using an external dataset, data professionals should evaluate:

• Source credibility
• Collection methodology
• Update frequency
• Coverage
• Definitions
• Missing data
• Licensing terms
• Permitted uses
• Privacy restrictions
• Historical consistency

External data should not be assumed to be accurate merely because it was published by a recognized provider.

4. Flat Files

A flat file typically stores records in a single, non-relational structure.

Delimited text files are a common form. Each line represents a record, while delimiters separate individual fields.

Example CSV file:

transaction_id,store_id,date,amount
5001,S102,2026-03-01,42.50
5002,S105,2026-03-01,79.95
5003,S102,2026-03-02,18.25

transaction_id,store_id,date,amount
5001,S102,2026-03-01,42.50
5002,S105,2026-03-01,79.95
5003,S102,2026-03-02,18.25

Common delimiters include:

• Commas
• Tabs
• Pipes
• Semicolons

Common Flat-File Formats

Examples include:

• CSV
• TSV
• Pipe-delimited text
• Fixed-width text files
• Plain log files

Flat files are widely used because they are simple to produce and supported by many systems.

Advantages of Flat Files

Flat files offer:

• Broad compatibility
• Simple transfer
• Human-readable contents
• Easy generation
• Straightforward batch ingestion
• Low tooling requirements

Limitations of Flat Files

They may lack:

• Explicit data types
• Relationships between tables
• Strong validation rules
• Consistent metadata
• Update tracking
• Transaction management
• Efficient random access

File producers and consumers must agree on details such as:

• Character encoding
• Delimiter
• Header presence
• Quote rules
• Date format
• Decimal separator
• Missing-value representation
• File-delivery schedule

5. Spreadsheet Files

Spreadsheets also organize data into rows and columns, but they should not be treated as ordinary flat-text files.

Spreadsheet formats such as XLSX can contain:

• Multiple worksheets
• Formulas
• Formatting
• Charts
• Pivot tables
• Hidden rows and columns
• Merged cells
• Named ranges
• Data-validation rules
• Images and annotations

Popular spreadsheet applications include:

• Microsoft Excel
• Google Sheets
• Apple Numbers
• LibreOffice Calc

Google Sheets is primarily a cloud-based spreadsheet service, although its data can be exported into formats such as XLSX or CSV.

Spreadsheets as Data Sources

Spreadsheets are often used for:

• Manually maintained reference data
• Small business datasets
• Survey results
• Budget planning
• Operational tracking
• One-time exchanges
• User-entered data

They are convenient for people but can be difficult to process automatically.

Common problems include:

• Inconsistent column names
• Multiple tables on one worksheet
• Empty header rows
• Merged cells
• Formulas instead of fixed values
• Manual notes within the data
• Unexpected worksheet names
• Mixed data types
• Hidden content

A production ingestion process should define exactly which workbook, worksheet, range, and fields are expected.

6. XML Datasets

XML is a text-based markup format that represents information with user-defined tags.

<transactions>
    <transaction id="5001">
        <store>S102</store>
        <date>2026-03-01</date>
        <amount currency="USD">42.50</amount>
    </transaction>
</transactions>

<transactions>
    <transaction id="5001">
        <store>S102</store>
        <date>2026-03-01</date>
        <amount currency="USD">42.50</amount>
    </transaction>
</transactions>

Unlike flat tables, XML can naturally represent hierarchical relationships.

Common XML Sources

XML may be used for:

• Enterprise application exchanges
• Bank statements
• Government data
• Online survey exports
• Configuration files
• Document standards
• Legacy web services
• Industry-specific messages

XML is generally considered semi-structured rather than unstructured because tags and attributes describe its organization.

Advantages of XML

XML provides:

• Hierarchical representation
• Human-readable text
• User-defined tags
• Formal schema validation
• Namespace support
• Broad enterprise compatibility

XML Ingestion Challenges

Data engineers may need to handle:

• Deeply nested elements
• Large document sizes
• Optional fields
• Multiple namespaces
• Schema versions
• Attributes and text values
• Secure parser configuration

XML schemas can improve reliability by defining permitted elements, relationships, and data types.

7. APIs and Web Services

An application programming interface allows one system to request data or functionality from another system through a defined interface.

APIs commonly return:

• JSON
• XML
• Plain text
• HTML
• Images
• Documents
• Other media files

Many modern web APIs use HTTP requests and return JSON responses.

API Request Example

A client might request weather information using a URL such as:

https://api.example.com/weather?city=Phoenix

https://api.example.com/weather?city=Phoenix

The service could return:

{
  "city": "Phoenix",
  "temperature": 31,
  "unit": "C",
  "observed_at": "2020-03-10T14:00:00Z"
}

{
  "city": "Phoenix",
  "temperature": 31,
  "unit": "C",
  "observed_at": "2020-03-10T14:00:00Z"
}

Common API Data Sources

APIs may provide:

• Financial information
• Weather observations
• Maps and geographic data
• Public government records
• Marketing-platform information
• Social-platform information
• Product catalogs
• Payment records
• Internal application data
• Address and postal-code validation

Availability and access conditions can change. Some providers restrict access by subscription level, user authorization, intended use, or request volume.

API Authentication

APIs may require:

• API keys
• Access tokens
• OAuth authorization
• Client certificates
• Signed requests
• User credentials

Credentials should be stored securely and should not be embedded directly in public code.

API Rate Limits

Providers commonly limit:

• Requests per second
• Requests per day
• Number of returned records
• Concurrent connections
• Historical access
• Download volume

Pipelines must respect these limits and handle responses indicating that requests should be delayed.

Pagination

An API may return only part of a dataset in each response. The client must follow page numbers, cursors, or continuation tokens to retrieve the remaining records.

Reliability Considerations

An API pipeline should handle:

• Temporary failures
• Timeouts
• Rate limits
• Authentication expiration
• Pagination
• Duplicate responses
• Schema changes
• Partial data
• Service outages
• Retry behavior

Retries should include limits and delays so that a failing service is not overwhelmed with repeated requests.

8. Web Scraping

Web scraping extracts information from web pages when a suitable structured dataset or authorized API is unavailable.

It is also called:

• Web data extraction
• Web harvesting
• Screen scraping

A scraper may retrieve a page and identify information using:

• HTML elements
• CSS selectors
• XPath expressions
• Embedded metadata
• Page structure

Common Uses of Web Scraping

Authorized scraping may be used for:

• Price comparisons
• Public product information
• Public research datasets
• News monitoring
• Public job listings
• Public business directories
• Academic research
• Collecting permissible training data

Common Web-Scraping Technologies

Popular tools include:

• Beautiful Soup
• Scrapy
• Selenium
• Playwright
• Browser automation tools

Pandas can read certain HTML tables and process scraped data, but it is primarily a data-analysis library rather than a general web-scraping framework.

Static and Dynamic Pages

A static page contains relevant information in the HTML returned by the server. Tools such as Beautiful Soup or Scrapy may be sufficient.

A dynamic page may load data through JavaScript after the initial page appears. Browser automation may be required, although inspecting an authorized underlying API can sometimes provide a more reliable source.

Legal and Ethical Considerations

The technical ability to scrape a website does not automatically grant permission to do so.

Before scraping, consider:

• Website terms of service
• Copyright
• Privacy
• Data-protection laws
• Authentication restrictions
• Access controls
• Licensing requirements
• Contractual limitations
• Robots directives
• Effect on server resources

A robots file communicates crawler preferences but is not, by itself, a complete statement of legal permission.

Scrapers should use reasonable request rates, identify themselves when appropriate, and avoid collecting restricted or sensitive information without authorization.

Operational Limitations

Web scraping is often less reliable than an API because page layouts can change without notice.

A minor modification to HTML structure may break the extraction process. Production scrapers therefore require:

• Monitoring
• Validation
• Error handling
• Rate control
• Change detection
• Maintenance

9. Data Streams

A data stream is a continuous or frequently updated sequence of events.

Streaming sources include:

• Applications
• Websites
• Financial markets
• IoT devices
• Industrial sensors
• Vehicles
• Network systems
• Social platforms
• Transaction-processing systems

A stream record commonly includes a timestamp. Some events may also include geographic coordinates, but geotagging is not a universal feature of streaming data.

Streaming Use Cases

Financial markets

Market events can support trading, monitoring, and risk analysis.

Retail transactions

Transaction streams can support fraud detection, demand forecasting, and inventory management.

Industrial sensors

Sensor events can identify equipment anomalies and support predictive maintenance.

Website clickstreams

User events can help monitor application performance and analyze navigation behavior.

Transportation

Flight, vehicle, or delivery events can support routing, rebooking, scheduling, and operational monitoring.

Security

Network and application events can be analyzed for suspicious behavior.

Video and surveillance

Video streams can support authorized safety monitoring, traffic analysis, and threat detection, subject to appropriate privacy and legal controls.

Technologies for Streaming Data

Technologies associated with stream ingestion and processing include:

• Apache Kafka
• Apache Flink
• Apache Spark Structured Streaming
• Apache Storm
• Cloud messaging and streaming services

These technologies have different functions. Kafka is commonly used for durable event streaming and messaging, while Flink, Spark, and Storm provide stream-processing capabilities.

Streaming Challenges

Streaming systems must address:

• Event ordering
• Duplicate events
• Late-arriving records
• Processing guarantees
• Checkpointing
• Backpressure
• Partitioning
• Consumer failures
• Retention
• Schema evolution
• Monitoring

Real-time processing is more complex than batch processing and should be adopted when its business value justifies that complexity.

10. RSS and Syndication Feeds

RSS commonly stands for Really Simple Syndication. It provides a standardized method for publishing updates from websites.

RSS feeds are often used by:

• News sites
• Blogs
• Podcasts
• Online publications
• Community forums

An RSS feed usually contains XML with entries such as:

• Title
• Link
• Publication date
• Author
• Summary
• Content reference

A feed reader periodically checks the feed and presents new entries to users.

RSS as a Data Source

Data pipelines can use RSS to monitor newly published content without repeatedly scraping complete webpages.

However, the feed may provide only a summary. Retrieving the complete article may require following its link and respecting the publisher’s access and usage conditions.

Choosing an Appropriate Source

When the same information is available through several sources, prefer the most stable and authorized option.

A general preference is:

1. Official database access or managed export
2. Official API
3. Published file or feed
4. Authorized web scraping

This ordering is not universal, but structured and documented interfaces are generally more reliable than extracting content from page layouts.

Important selection criteria include:

• Authority of the source
• Data quality
• Freshness
• Coverage
• Access method
• Security
• Usage rights
• Cost
• Stability
• Documentation
• Rate limits
• Operational complexity

Comparing Common Data Sources

Building a Reliable Ingestion Process

Regardless of the source, a production ingestion process should answer several questions:

• Who owns the source?
• How frequently does it update?
• How is access authenticated?
• What schema is expected?
• What happens when the schema changes?
• How are duplicates identified?
• How are failures retried?
• How is completeness verified?
• What privacy restrictions apply?
• How is the source documented?
• How is data lineage recorded?
• Who receives an alert when ingestion fails?

A pipeline that retrieves data successfully once is not necessarily reliable. Production systems must continue operating as sources, schemas, credentials, and workloads change.

Key Takeaways

• Modern data comes from internal and external systems in many structures and formats.
• Relational databases and warehouses are important sources of structured data.
• Flat files provide simple tabular exchange but usually lack strong schema information.
• Spreadsheet files contain richer features and should not be treated as ordinary flat text.
• XML represents hierarchical, semi-structured data.
• APIs provide programmatic data access but require authentication, pagination, rate-limit handling, and monitoring.
• Web scraping can extract public web data but must account for permission, ethics, privacy, and layout changes.
• Data streams provide continuously arriving events for low-latency applications.
• RSS feeds offer a standardized way to monitor published updates.
• The best source is usually the most authoritative, stable, documented, and appropriately licensed source available.

Conclusion

A modern data platform must connect to databases, files, APIs, websites, streams, feeds, and external providers.

Each source requires a different ingestion strategy. Databases demand careful query and extraction design, files require explicit format rules, APIs require resilient request handling, and streaming systems require continuous event processing.

Understanding these differences allows data engineers to build pipelines that collect information reliably while protecting its quality, security, and permitted use.

One-sentence summary: Modern data engineering collects information from databases, files, APIs, websites, streams, and feeds using ingestion methods tailored to each source’s structure, frequency, reliability, and access requirements.