NOTQIN Grid

1. Project Summary

A standalone smart meter energy forecasting + anomaly detection pipeline — ingests 15-minute electricity readings via Kafka, stores in TimescaleDB, clusters meters by consumption pattern (K-Means), trains per-cluster LSTM + Prophet models, and detects abnormal peaks via model residuals. Originally a course project (Projet 16, ENSA Berrechid), designed for NOTQIN integration.

---

2. Architecture

Smart Meters (simulator or real)
  │ 15-min readings (JSON)
  ▼
Kafka 7.6 (Confluent CP)
  │
  ▼
kafka_to_timescale.py (Python consumer)
  │
  ▼
TimescaleDB (PostgreSQL 16 + TimescaleDB extension)
  ├── meter_readings         ← raw 15-min hypertable
  ├── meter_hourly           ← continuous aggregate (1h)
  ├── meter_daily            ← continuous aggregate (1d)
  ├── meter_predictions      ← LSTM/Prophet forecasts
  ├── anomaly_events         ← detected spikes
  └── meters                 ← meter metadata + cluster_id
  │
  ├─► ml/clustering.py       → K-Means on daily profiles → cluster_id per meter
  ├─► ml/lstm_model.py       → TensorFlow LSTM per cluster (adaptive windowing)
  ├─► ml/prophet_model.py    → Facebook Prophet per cluster (seasonal + trend)
  ├─► ml/anomaly_detector.py → Z-score on (actual - predicted) → anomaly_events
  └─► ml/incremental_train.py → Daily cron re-training
  │
  ▼
Grafana (3001)
  ├── Load Map Dashboard
  ├── Forecast vs Actual (time series)
  ├── Anomaly Events (table)
  └── Cluster Distribution (bar chart)

Monitoring:
  ├── Prometheus (9091)
  └── Kafka UI (8080)

---

3. Tech Stack

Layer	Technology
Message Queue	Kafka 7.6 (Confluent CP)
Database	TimescaleDB 1.14 (PostgreSQL 16 extension)
ML — Deep Learning	TensorFlow 2 (LSTM)
ML — Time Series	Facebook Prophet
ML — Clustering	scikit-learn K-Means
Orchestration	Docker Compose + k8s/ (future)
Monitoring	Prometheus + Grafana
Data	Pandas, NumPy, scikit-preprocessing

---

4. ML Models

LSTM Forecasting

• Input: 96 timesteps (24h at 15-min intervals — adaptive 12h–72h window)
• Horizon: 4 timesteps (1 hour ahead)
• Architecture: 2-layer LSTM (64→32 units) + Dropout(0.2) + Dense
• Loss: MSE + MAE metric, 20 epochs, batch 32
• Scaler: MinMaxScaler (per-meter normalization)

Adaptive Windowing:

if recent_std / global_std > 1.5: window × 2 (max 72h)
if recent_std / global_std < 0.5: window / 2 (min 12h)

Prophet

• Seasonal decomposition (yearly + weekly)
• Trend component + Morocco-specific holidays
• 80% + 95% interval forecasts

Anomaly Detection

• Residuals: |actual - predicted|
• Z-score: (residual - mean) / std
• Flag if z_score > 3 (configurable threshold)
• Written to anomaly_events table

Datasets

• Primary: London Smart Meters dataset (Zenodo) — cleaned 15-min readings
• Secondary: UCI household power consumption
• Optional: REFIT dataset (appliance-level)

---

5. Current Status

Stage: Production-ready (course project complete)

Last commit: Apr 20, 2026

Full Makefile automation:

make up           # Start stack (Kafka, TimescaleDB, Grafana, Prometheus)
make simulate     # Generate 15-min meter readings
make ingest       # Push to Kafka → TimescaleDB
make cluster      # K-Means clustering
make train        # Train LSTM + Prophet per cluster
make detect       # Run anomaly detection
make eda          # Exploratory data analysis

---

6. NOTQIN Integration Path

NOTQIN machines expose power_watts per loom via ESP32
  → MQTT → Kafka bridge
  → SmartGrid treats each loom as a "smart meter"
  → Per-loom forecasting + anomaly detection
  → Complements Flink's real-time anomaly detection with historical ML forecasting

From docs/architecture.md: “SmartGrid is designed as a future integration layer for NOTQIN looms.”

---

7. Next Actions

• Build MQTT → Kafka bridge to feed NOTQIN loom data into SmartGrid
• Test per-loom LSTM forecasting with real NOTQIN telemetry
• Deploy to k8s/ manifests (already in repo)
• Integrate forecasts into IEIA scenario computation (predicted vs actual)