1) What CRISP-DM is and why it matters

CRISP-DM (Cross-Industry Standard Process for Data Mining) is a process framework published in 1999 to standardize how organizations run data mining and analytics projects. It is widely viewed as the most commonly used end-to-end methodology for data mining and many modern data science projects. In practice, teams often get the best outcomes by using CRISP-DM as the “analytic lifecycle” while managing work with agile delivery methods (e.g., Scrum or Kanban).

2) The 6 phases of CRISP-DM (with key tasks)

I. Business Understanding

Purpose: ensure the work is anchored to real business needs and success criteria (the “foundation” of the project).

Key tasks:

  • Determine business objectives: understand what the customer truly wants; define business success criteria.
  • Assess situation: evaluate resources, constraints, risks, contingencies; perform cost–benefit analysis.
  • Determine data mining goals: translate business goals into measurable technical objectives.
  • Produce project plan: choose tools/technologies and plan each phase.

Why it matters: teams often rush this phase, but weak business understanding is a common root cause of misaligned modeling and “successful” models that do not deliver business value.

II. Data Understanding

Purpose: identify, collect, and examine the data relevant to the goals.

Key tasks:

  • Collect initial data: acquire and load data into an analysis environment.
  • Describe data: document basic properties (schema, fields, record counts, formats).
  • Explore data: query, visualize, and identify patterns/relationships.
  • Verify data quality: assess missingness, noise, inconsistencies, and limitations.

Primary outcome: a clear view of what data exists, what it means, and whether it is fit for the intended use.

III. Data Preparation

Purpose: create the final modeling dataset(s). This is often the most time-consuming portion of a project (commonly cited as ~80%).

Key tasks:

  • Select data: decide what to include/exclude and justify those decisions.
  • Clean data: correct, impute, or remove erroneous/missing values; reduce “garbage in, garbage out.”
  • Construct data: engineer features (e.g., deriving BMI from height and weight).
  • Integrate data: merge multiple sources into coherent datasets.
  • Format data: transform types/encodings so data is usable for modeling and analysis.

Primary outcome: analysis-ready features and datasets with documented preparation logic.

IV. Modeling

Purpose: build and compare candidate models using suitable techniques.

Key tasks:

  • Select modeling techniques: choose algorithms (e.g., regression, tree-based models, neural networks).
  • Generate test design: define evaluation strategy (train/validation/test split, cross-validation, etc.).
  • Build model: train models and implement pipelines.
  • Assess model: compare results using success criteria and domain understanding.

Practical note: the guide suggests iterating until the “best” model is found, but real teams often iterate until a “good enough” model is achieved, then deliver and improve later.

V. Evaluation

Purpose: confirm the solution meets business needs and decide what to do next.

Key tasks:

  • Evaluate results: check alignment with business success criteria; choose model(s) to approve.
  • Review process: ensure nothing critical was missed; summarize findings and corrective actions.
  • Determine next steps: deploy, iterate further, or start a related initiative.

Difference vs. “Assess model”: Modeling assessment is primarily technical; Evaluation is business + operational readiness.

VI. Deployment

Purpose: make results accessible and usable by stakeholders; transition toward operations.

Key tasks:

  • Plan deployment: define how results/models will be delivered (report, dashboard, API, batch job, etc.).
  • Plan monitoring and maintenance: monitoring, drift detection, retraining triggers, incident response.
  • Produce final report: communicate outcomes, assumptions, limitations, and recommendations.
  • Review project: retrospective on what worked, what didn’t, and how to improve.

Important caveat: CRISP-DM does not fully specify post-deployment operations, so teams must explicitly plan monitoring, maintenance, and periodic tuning.

3) Is CRISP-DM Agile or Waterfall?

CRISP-DM can resemble either, depending on implementation.

  • Waterfall-style implementation:
    • heavy up-front planning and documentation
    • limited iteration between phases
    • “big bang” delivery at the end
      This is often described as horizontal slicing: progressing across phases for all deliverables in parallel.
  • Agile-style implementation:
    • frequent iteration across phases
    • incremental delivery and feedback loops
    • continuous refinement
      This aligns with CRISP-DM’s statement that phase order is not rigid and back-and-forth movement is expected. Agile CRISP-DM is commonly executed via vertical slicing: delivering one end-to-end slice of value (from data to deployment) at a time.

Recommendation emphasized in the text: when possible, use agile + vertical slicing so stakeholders receive value sooner, feedback arrives earlier, and model performance/risks are surfaced quickly.

4) Popularity of CRISP-DM (how the text argues it)

The text claims CRISP-DM is the most commonly used approach based on:

  • historical KDnuggets polls (2002–2014) where CRISP-DM was the top methodology
  • a 2020 poll (on the author’s site) where CRISP-DM won nearly half of votes
  • Google search volume comparisons indicating strong interest in CRISP-DM relative to alternatives

It also notes that definitive, rigorous research across the whole industry is limited, so these are proxies rather than conclusive evidence.

5) Should you use CRISP-DM?

The text presents a balanced view.

Benefits

  • Generalizable: applicable beyond classic data mining; starts with business needs and ends with deployment.
  • Common-sense structure: aligns with how many teams naturally work when given no methodology.
  • Adoptable: can be implemented with minimal organizational disruption (similar to Kanban).
  • Strong start: reduces risk of building technically impressive solutions that miss business objectives.
  • Strong finish: explicitly addresses deployment planning and transition considerations.
  • Flexible (when implemented loosely): supports iterative cycles and learning-driven refinement.

Weaknesses / Challenges

  • Can be rigid: if followed “by the book,” it can behave like waterfall and slow iteration.
  • Documentation-heavy: extensive reporting requirements may reduce delivery speed.
  • May not fit modern big data contexts by default: critics argue it predates big data concerns and needs augmentation.
  • Not a full project management methodology: lacks sufficient guidance for large-team coordination, stakeholder communication, and delivery governance.

6) Practical recommendations to make CRISP-DM work well today

The text recommends combining CRISP-DM with agile project management and modern engineering practices:

  • Iterate quickly and deliver thin vertical slices end-to-end.
  • Document enough, not excessively: prioritize what supports reproducibility, auditability, and stakeholder clarity.
  • Include modern tooling: cloud architectures, version control (Git), CI/CD pipelines when appropriate.
  • Set expectations and communicate frequently: CRISP-DM under-specifies stakeholder management.
  • Combine with a coordination framework: Scrum, Kanban, Data-Driven Scrum, etc.

7) Alternatives mentioned and how they compare

SEMMA (SAS)

  • Phases: Sample, Explore, Modify, Model, Assess
  • More narrowly focused on technical modeling steps
  • Often omits Business Understanding and Deployment
  • Popularity has declined (very low usage in the cited poll)

KDD (Knowledge Discovery in Databases)

  • Older and generally narrower than CRISP-DM
  • Typical phases: Select, Pre-Processing, Transformation, Data Mining, Interpretation/Evaluation
  • Similar to the “middle” of CRISP-DM but often lacks explicit business/deployment emphasis

KDDS (Knowledge Discovery in Data Science)

  • Proposed in 2016 as an end-to-end model addressing big data concerns
  • Phases: assess, architect, build, improve; stages: plan, collect, curate, analyze, act
  • Described as having limited adoption