# CRISP-ML(Q) CRISP-ML(Q) stands for "CRoss-Industry Standard Process model for the development of Machine Learning applications with Quality assurance methodology". It's a process model for ML projects, proposed by Studer et al. in 2020 and made popular by the INNOQ team behind ml-ops.org. It extends CRISP-DM, the data mining process model from the late 1990s. The (Q) is the important part: every phase comes with its own quality assurance steps. Why bother with a new model? Because CRISP-DM treats deployment as the finish line. With machine learning, deployment is where the trouble starts. Data drifts, models go stale, and the hardware and software under them keep changing. So CRISP-ML(Q) adds monitoring and maintenance as a phase of its own, and the process never reaches "done". ## The six phases 1. **Business & Data Understanding**: scope, feasibility, KPIs and success criteria at three levels (business, ML and economic). You set a simple non-ML heuristic as the benchmark to beat, fill in the ML Canvas, document the properties of the data and build a proof of concept. Business and data understanding are ONE phase here, because you can't scope the problem without knowing your data (and the other way around) 2. **Data Engineering**: feature and data selection (filter, wrapper and embedded methods), class imbalance, cleaning, unit tests for data, feature engineering, augmentation, normalization. Everything runs in reproducible pipelines 3. **ML Model Engineering**: define quality measures beyond accuracy (robustness, fairness, scalability, interpretability, complexity, resource demand), pick a baseline, inject domain knowledge, then train. [[Transfer learning]], compression and ensembles are all on the table. Capture the metadata, validate across several random seeds and document the model with a Model Card 4. **Model Evaluation**: test on a held-out set, check how the model holds up against noisy inputs, check explainability, then decide whether to deploy. That decision can be automatic or made by a domain expert, and it gets documented 5. **Deployment**: integrate the model into the software system, through a pre-production environment first. Choose the [[ML Deployment Patterns|deployment mode]] early (batch or online, dashboard, precomputed predictions, plugin, web service) along with the inference hardware. Run A/B tests and user acceptance tests, keep a fallback plan, and roll out with canary or blue/green releases 6. **Monitoring & Maintenance**: watch for staleness and drift, look at how changes to the hardware and software stack affect the model, keep evaluating it, retrain when needed, and automate the loop with CI/CT/CD (CT is continuous training) ## Quality assurance in every phase Each phase runs the same four steps: 1. Define the requirements and constraints 2. Derive the tasks 3. Identify the risks (e.g., bias, overfitting, irreproducibility) 4. Apply QA methods to mitigate them (e.g., cross-validation, documentation) The themes that keep coming back are [[ML Reproducibility|reproducibility]], documentation everywhere, and iteration. The process is NOT linear: you jump back to an earlier phase whenever you learn something that invalidates it. The step I find most useful is the non-ML benchmark in phase one. If a dumb heuristic already hits the target, you may not need a model at all... and you've saved yourself a lot of maintenance. PMI's [[PMI Certified Professional in Managing AI (PMI-CPMAI)|CPMAI]] methodology also derives from CRISP-DM, but it's aimed at project managers. CRISP-ML(Q) stays much closer to the engineering, which makes it a good checklist for [[MLOps]] work. ## References - https://ml-ops.org/content/crisp-ml (INNOQ: Larysa Visengeriyeva, Anja Kammer, Isabel Bär, Alexander Kniesz, Michael Plöd; CC BY 4.0) - Studer et al., "Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology": https://arxiv.org/abs/2003.05155 ## Related - [[MLOps]] - [[ML Reproducibility]] - [[ML Deployment Patterns]] - [[Model Registry]] - [[Continuous Integration (CI)]] - [[Continuous Delivery (CD)]] - [[Machine Learning Project Workflow]] - [[Machine Learning (ML)]] - [[Machine Learning (MoC)]] - [[PMI Certified Professional in Managing AI (PMI-CPMAI)]]