pandas DataFrame

A DataFrame is the table at the centre of pandas: rows, named columns, and a row label called the index. Unlike a two-dimensional array, each column carries its own type, so one table can hold identifiers as text, durations as integers, and flags as booleans at the same time. Unlike a list of dictionaries, the columns are stored together, which is what makes whole-column arithmetic and filtering possible in one expression.

Three properties answer most questions about a DataFrame you have just loaded. shape gives rows and columns, in that order. dtypes gives the type pandas inferred for each column, which is where a numeric column silently arriving as text becomes visible. info() adds the non-null count per column, so gaps show up before they reach an average. Reading those three before anything else is the habit that catches most import problems.

When a DataFrame is constructed from lists without an explicit index, it receives a RangeIndex. Filtering retains selected labels, which may leave gaps; concat can repeat labels. Label alignment is useful only when those labels have the same meaning across objects. Two separately numbered exports can share labels while describing different tickets. Check label meaning and uniqueness, or merge on an explicit entity key.

A DataFrame lives in memory, and that sets its limits. Reading a file without chunking builds the whole table at once, and the memory used depends on types, parsing, and temporary copies rather than on file size alone. When a table stops fitting, the options are to push work to the database, process in chunks or batches, or use an engine built for larger data. That is a different decision from whether pandas is convenient, and worth measuring rather than guessing.

References: pandas DataFrame API, pandas data structures. See it in use in Pandas Foundations: Tables, Filtering, Grouping, and Joins.