AI & Data Project Management

How to scope, execute, and deliver AI/ML projects. The discipline that bridges data science and production systems. Directly applicable to IEIA, SijiLab Assistant, Idarati RAG, and all NOTQIN Research papers.

Why AI Projects Fail Differently

Traditional software projects fail primarily due to scope creep, technical complexity, and schedule pressure. AI projects fail for additional structural reasons:

Failure Mode	Description	Frequency
Wrong problem framing	Solving an ML problem when a rule-based system would work	Very common
Data doesn’t exist	Assuming data is available and clean	Common
Training/serving skew	Model trained on data that doesn’t match production data	Common
No clear success metric	Can’t tell if the model is “good enough”	Common
Accuracy ≠ business value	95% accurate model that isn’t useful in practice	Common
Overfit to training data	Works in lab, fails in prod (SijilPharma ML: R²=0.98 train / 0.31 test)	Very common
Concept drift	World changes, model degrades silently over time	Common post-deployment
Ethical/legal issues	Bias, privacy, regulatory compliance discovered late	Increasing

Andrew Ng’s AI Project Lifecycle

Andrew Ng (DeepLearning.AI, Coursera “AI for Everyone”) provides the most practical framework for non-researchers managing AI projects:

1. SCOPING
   → Define the project
   → What is X? What is Y?
   → Is it feasible? Is it valuable?
   → Key metrics to optimize
   ↓
2. DATA
   → Define data and establish baseline
   → Label and organize data
   → Is data clean? Consistent? Sufficient?
   → Edge cases, ambiguous labels
   ↓
3. MODELING
   → Select and train model
   → Perform error analysis
   → Improve model and data
   ↓
4. DEPLOYMENT
   → Deploy in production
   → Monitor system
   → Maintain and retrain

Key insight: Steps 1–2 are 80% of the work but get 20% of the attention.

The ML Project Scoping Canvas

Before starting any ML project, answer:

Question	Why it matters
What is the input X?	Defines data requirements
What is the output Y?	Defines the target metric
What does “good enough” mean? (metrics)	Prevents infinite tuning
What is the business value of getting Y right?	Justifies investment
What is the cost of being wrong? (types of errors)	Determines acceptable error rate
Does useful data exist? How much?	Feasibility check
What is the baseline performance?	Reference point (human-level, rule-based)
How often does the world change? (drift)	Determines retraining frequency
What are the ethical risks?	Legal and reputational

Applied to NOTQIN IEIA anomaly detection:

X: energy consumption + maintenance logs + shift schedule time series
Y: anomaly label (is this hour anomalous? what’s the root cause?)
Good enough: precision ≥ 80%, recall ≥ 70% (higher recall preferred — missing anomaly is worse than false alarm)
Business value: MAD 15,000/month energy waste identified per factory
Cost of error: false alarm → operator annoyance; missed anomaly → equipment failure → MAD 50,000 downtime
Baseline: current human review catches ~40% of anomalies
Ethical risks: none critical, but wrong CBAM data could cause legal issues

CRISP-DM — Cross-Industry Standard Process for Data Mining (1999)

Still the most widely used ML project process model:

                ┌──────────────┐
                │   BUSINESS   │
            ┌──▶│ UNDERSTANDING│◀──┐
            │   └──────┬───────┘   │
            │          ↓           │
            │   ┌──────────────┐   │
            │   │     DATA     │   │
            │   │ UNDERSTANDING│   │
            │   └──────┬───────┘   │
            │          ↓           │
            │   ┌──────────────┐   │
            │   │     DATA     │   │
            │   │ PREPARATION  │   │
            │   └──────┬───────┘   │
            │          ↓           │
            │   ┌──────────────┐   │
            │   │   MODELING   │──▶│
            │   └──────┬───────┘   │
            │          ↓           │
            │   ┌──────────────┐   │
            │   │  EVALUATION  │───┘
            │   └──────┬───────┘
            │          ↓
            │   ┌──────────────┐
            └──▶│  DEPLOYMENT  │
                └──────────────┘
                (feedback loop)

Note the arrows going backward — CRISP-DM is iterative. You often need to go back to data preparation after modeling reveals issues.

MLOps — Machine Learning Operations

MLOps applies DevOps principles to ML systems. The goal: reliable, scalable, reproducible ML in production.

The MLOps Maturity Levels

Level	Description	Characteristics
0	Manual process	Ad hoc, Jupyter notebooks, manual deployment
1	ML pipeline automation	Automated training, feature store, model registry
2	CI/CD for ML	Automated testing, deployment, monitoring pipelines

NOTQIN Research current state: Level 0 (Jupyter notebooks, manual validation). NOTQIN IEIA: closer to Level 1 (FastAPI serving, structured training). Target: Level 1 for IEIA, Level 0→1 for research papers before submission.

MLOps Components

Component	Purpose	Tools
Feature Store	Central repository of reusable features	Feast, Hopsworks, Tecton
Model Registry	Versioned model storage + metadata	MLflow, Weights & Biases, DVC
Model Serving	Deploy models as APIs	FastAPI, BentoML, TorchServe, Triton
Monitoring	Track data drift, model drift, performance	Evidently, WhyLabs, Prometheus
Experiment Tracking	Track hyperparameters, metrics across runs	MLflow, W&B, Comet
Pipeline Orchestration	Automate data + training pipelines	Apache Airflow, Prefect, Dagster
Data Versioning	Track changes to datasets	DVC, LakeFS
CI/CD for ML	Automated testing and deployment	GitHub Actions + custom

Your Current Stack (IEIA)

Model serving: FastAPI ✅
Monitoring: Prometheus + OpenTelemetry ✅
Model registry: None (add MLflow)
Experiment tracking: None (add MLflow or W&B)
Feature store: None (PostgreSQL acts as de facto store)
Data versioning: None (add DVC)

AI Project Types & Their Management Approaches

Type 1: Rule-Based AI (Expert Systems)

No learning, explicit logic. When to use: known rules, high interpretability needed, small dataset.

Example: Anomaly detection rules in NOTQIN Core (22 machine types × 74 rules)
Management: traditional software development (tests, versioning, code review)
Key risk: rule maintenance as edge cases multiply

Type 2: Classical ML (Supervised/Unsupervised)

Learn patterns from labeled data. When to use: labeled data available, interpretable results needed, structured tabular data.

Example: SijilPharma demand forecasting (XGBoost), IEIA anomaly classification
Management: CRISP-DM, MLflow, train/val/test split discipline
Key risk: overfitting (your SijilPharma issue: R²=0.98 train, 0.31 test)

Type 3: Deep Learning / Neural Networks

Large datasets, complex patterns. When to use: unstructured data (images, text, audio), very large datasets.

Example: NOTQIN Research NILM (Non-Intrusive Load Monitoring)
Management: more compute, longer training, W&B for experiment tracking
Key risk: interpretability (black box), data hungry, expensive

Type 4: LLM Integration / Agentic AI

Use pre-trained LLMs (Claude, GPT-4) via API. When to use: natural language tasks, code generation, reasoning, document analysis.

Example: SijiLab Assistant (Claude API), NOTQIN IEIA analyst tool, Idarati RAG
Management: prompt engineering, eval frameworks, token cost management, latency
Key risk: hallucination, cost overrun, prompt injection (security)

Type 5: RAG Systems (Retrieval-Augmented Generation)

LLM + vector search over proprietary documents. When to use: domain-specific Q&A, document analysis, factual queries over your data.

Example: Idarati RAG (Moroccan administrative documents)
Management: chunking strategy, embedding model choice, retrieval evaluation, generation evaluation
Key risk: retrieval quality (garbage in → garbage out), latency, document freshness

Evaluating AI Systems — Metrics by Type

Problem Type	Primary Metrics	Secondary Metrics
Binary Classification	Precision, Recall, F1, AUC-ROC	Confusion matrix, calibration
Regression	R², MAE, RMSE, MAPE	Residual analysis
Anomaly Detection	Precision, Recall, F1 (per class)	False alarm rate, detection latency
RAG / LLM	Faithfulness, Answer Relevance, Context Precision	Latency, token cost, hallucination rate
LLM Agents	Task success rate, tool call accuracy	Number of turns, cost per task

The Confusion Matrix (for classification)

                    PREDICTED
                Positive  │  Negative
ACTUAL Positive    TP     │    FN     → Recall = TP/(TP+FN)
       Negative    FP     │    TN     → Specificity = TN/(TN+FP)
                          │
                          ↓ Precision = TP/(TP+FP)

F1 Score = 2 × (Precision × Recall) / (Precision + Recall)

When to optimize for Recall: missing anomaly is worse than false alarm (IEIA anomaly detection, medical diagnosis) When to optimize for Precision: false alarm is costly (CBAM compliance report should not overstate emissions)

Responsible AI — The Management Dimension

The AI Ethics Principles (EU AI Act, 2024 — context)

Principle	Definition	Your Projects
Fairness	Model should not discriminate against protected groups	SijiLab: lab results not influenced by patient demographics
Transparency	Users should understand how AI makes decisions	IEIA: explainable anomaly reasoning (Claude provides it)
Accountability	Clear ownership of AI decisions	All projects: Ahmed is accountable
Privacy	Personal data protected	SijiLab: CNSS/CNOPS data under Loi 09-08 (CNDP Maroc)
Robustness	Model performs reliably under distribution shift	IEIA: tested on multiple machine types
Human oversight	Humans can override and audit AI decisions	IEIA: operator dashboard, Claude explains recommendations

EU AI Act Risk Categories (applies if selling to EU clients)

Risk Level	Examples	Requirements
Unacceptable	Social scoring, real-time facial recognition	Banned
High	Medical devices, critical infrastructure, employment	Conformity assessment, documentation, human oversight
Limited	Chatbots, deepfakes	Transparency requirements
Minimal	Spam filters, recommendations	No specific requirements

IEIA anomaly detection for industrial safety = potentially High Risk under EU AI Act if sold to EU clients.

Data Privacy in Morocco — CNDP Loi 09-08

SijiLab processes patient data (health) → requires CNDP authorization
Health data is “sensitive” category under Loi 09-08
Must have explicit patient consent, data minimization, right to access/delete
CNDP registration required before processing

LLM Cost Management

For Claude API and similar:

Optimization	Impact	How
Prompt caching	90% cost reduction for repeated prefixes	Anthropic prompt caching
Model selection	Use smaller model when capability is sufficient	Claude Haiku for simple extraction, Sonnet for complex reasoning
Output length control	Token cost = input + output	Set max_tokens, instruct to be concise
Batch processing	Reduce API calls	Batch multiple analyses in one call
Caching responses	Cache deterministic outputs	Redis cache for stable queries
Streaming	Improves perceived latency (not cost)	Use for user-facing features

Token cost estimate (Claude Sonnet 3.5 as reference):

IEIA energy report: ~2,000 tokens input + ~1,000 output ≈ $0.01 per report
At 1,000 reports/month/factory: $10/month/factory in Claude API cost
Factor into pricing: if customer pays MAD 5,000/month (€450), Claude API = 2% of revenue ✅

AI Project Timeline Reality

Common timeline underestimation in AI projects:

Phase	Expected	Reality
Data collection + cleaning	2 weeks	6–12 weeks (data doesn’t exist, is dirty, requires labeling)
First model	1 week	2–4 weeks (features not obvious, baseline needed)
Improving model	2 weeks	4–8 weeks (each improvement brings new failure modes)
Production deployment	1 week	4–8 weeks (serving infrastructure, monitoring, security)
Monitoring + maintenance	Ongoing	Plan for 20% ongoing effort forever

Rule of thumb: multiply your AI project estimate by 3–4×.

Agile for AI/ML — Sprints with ML Nuance

Regular Agile sprints work but need adaptations:

AI spike sprint: 1 sprint dedicated to feasibility/prototyping (no guaranteed deliverable)
DoD for ML: model achieves target metric on test set + deployed to staging + monitored
Data sprint: separate sprint for data pipeline and labeling
Model sprint: train/evaluate/select
Deployment sprint: serving, monitoring, rollout
Maintenance sprint: retraining, drift handling

Applied to Your Projects

Project	Key Management Gaps	Recommended Action
SijilPharma ML	Overfitting (train R²=0.98, test R²=0.31)	Regularization, cross-validation, feature selection. Add MLflow to track experiments.
NOTQIN IEIA	No model registry, no drift monitoring	Add MLflow for model versioning. Add Evidently for drift detection.
NOTQIN Research	Papers need real data validation (Paper 4)	Source real industrial dataset (ENGIE, UCI ML repository, or factory pilot)
Idarati RAG	Retrieval quality unknown	Implement RAGAS evaluation framework (faithfulness, relevance metrics)
SijiLab Assistant	No eval framework for Claude	Build a golden test set (20 Q&A pairs), run eval before each new prompt version

NOTQIN — OT Track

Explorer

AI & Data Project Management

AI & Data Project Management

Why AI Projects Fail Differently

Andrew Ng’s AI Project Lifecycle

The ML Project Scoping Canvas

CRISP-DM — Cross-Industry Standard Process for Data Mining (1999)

MLOps — Machine Learning Operations

The MLOps Maturity Levels

MLOps Components

Your Current Stack (IEIA)

AI Project Types & Their Management Approaches

Type 1: Rule-Based AI (Expert Systems)

Type 2: Classical ML (Supervised/Unsupervised)

Type 3: Deep Learning / Neural Networks

Type 4: LLM Integration / Agentic AI

Type 5: RAG Systems (Retrieval-Augmented Generation)

Evaluating AI Systems — Metrics by Type

The Confusion Matrix (for classification)

Responsible AI — The Management Dimension

The AI Ethics Principles (EU AI Act, 2024 — context)

EU AI Act Risk Categories (applies if selling to EU clients)

Data Privacy in Morocco — CNDP Loi 09-08

LLM Cost Management

AI Project Timeline Reality

Agile for AI/ML — Sprints with ML Nuance

Applied to Your Projects

See Also

Graph View

Table of Contents

Backlinks