NOTQIN Architecture Playbook

Codified from ~70MB of conversation logs across the notqin repo (Apr 11 → May 7 2026). Companion to NOTQIN Ecosystem Map and NOTQIN Ecosystem — Modular Integration Architecture. Use this as the single source of truth for what is actually built, what’s deployed where, and the decisions behind it.

1. Canonical pipeline (multi-sector)

Machine PLC/sensor
  → (OPC UA | Modbus TCP | Sparkplug B | plain MQTT JSON | CSV/SQL)
  → ingestion-service
  → MQTT broker (Mosquitto/EMQX, TLS 8883, plain 1883 internal)
  → Telegraf
  → InfluxDB measurement `machine_telemetry`
  → Grafana / anomaly-service / state-service (Redis) / orchestrate-service / analyst-service (Claude)

Legacy ESP32 textile path (parallel): ESP32 → MQTT notqin/telemetry → NiFi enrichment → InfluxDB → Flink BearingAnomalyJob → notqin/enriched | notqin/alerts → Grafana.

Telemetry envelope (uniform across 5 protocols): Sample{ts, source, tag, value, quality} + Quality enum.

2. Deployment topology

Layer	Host	What runs there
Dev	Windows laptop	Vite frontends, Python ingestion, Docker Desktop compose stack
Build	WSL	rsync/scp shell, build artifacts
Demo control plane	monmachine droplet (DigitalOcean fra1)	forge-api, forge-app, factory-api, factory-app, orchestrate, provisioner, ERP postgres, sector simulators, Mosquitto, NiFi, InfluxDB, Grafana, Prometheus, Telegraf — public via host nginx at `*.notqin.ahmedbenahmed.com`
Per-customer	$6 s-1vcpu-1gb fra1 droplets	`deploy/site-stack/docker-compose.site.yml`, nginx + LE cert
DNS	Porkbun	Root `ahmedbenahmed.com`; subdomain field `<slug>.notqin`; specific A overrides wildcard `*.notqin` → monmachine
Cloud	none (yet)	Phase-1 on-prem-first per ADR-002

Phase gates (ADR-002): on-prem Compose now → hosted SaaS at 3+ customers → hybrid edge+cloud at 10+. Multi-tenant SaaS deferred. Rationale: regulated industries (Safran/HACCP), data sovereignty.

3. Modules / services

Module	Path	Purpose
`factory-app`	`apps/factory-app`	Plant operator UI (Vite+React, blue theme); 9 sections grouped CONNECT / OPERATE / ADMIN
`factory-api`	`apps/factory-api`	NestJS+Fastify+Prisma, Keycloak JWT, audit interceptor, 10 modules
`forge-app`	`apps/forge-app`	Internal meta-app (amber theme); Catalog / Composer / Generator / Plugins / Knowledge / Sites
`forge-api`	`apps/forge-api`	NestJS site lifecycle (Draft → Validated → Deployed); pack-reader.service reads YAML packs
`provisioner-service`	`apps/provisioner-service`	FastAPI; orchestrates DO droplet creation, Porkbun DNS, SSH bootstrap, TLS, smoke probes; 11 named steps
`orchestrate-service`	`apps/orchestrate-service`	Python pipeline runtime; `/data-health`, `/dead-letters`, ERP context, UNS payload validation
`ingestion-service`	`apps/ingestion-service`	Python adapters: opcua, modbus_tcp, sparkplug, csv_db, retrofit
`anomaly-service`	`apps/anomaly-service`	paho-mqtt subscriber → 74 threshold rules across 22 machine types → publishes `notqin/alerts/{sector}/{type}/{id}`
`state-service`	`apps/state-service`	FastAPI+Redis, sub-ms `/machines/{id}` lookups; per-machine digital twin state
`analyst-service`	`apps/analyst-service`	Claude-agent (OpenRouter) tools: `query_machine_telemetry`, `get_fleet_status`, `query_oee`
`simulator`	`simulator/sectors/{sector}/machines/*.py`	4 sectors, 101 machines, 1Hz physics-based MQTT publishers
`notqin-physics`	sibling repo	LLM-orchestrator over SymPy/SciPy; physics verification engine; SSE event stream
`notqin-agent-v2` (IEIA)	sibling repo	Energy intelligence agent; NestJS+Python, ONEE tariff, structured SSE

4. Sector pack model — the moat

Layout at packages/sector-packs/{sector}/:

pack.yaml — manifest (schemaVersion 1.0, version e.g. 1.2.0, indexes)
uns-conventions.yaml — ISA-95 enterprise/site/shop/cell/machine/tag + topic template notqin/{sector}/{site}/{shop}/{machine_id}/{tag}
kpis.yaml, alert-rules.yaml, recipes.yaml, compliance.yaml, dashboards.yaml, ingestion-templates/*.yaml
Asset templates: required signals as {name, unit, range:[low,high], description}
- Schema gotcha: tuple form is non-canonical for JSON Schema 2020-12 — proper form is prefixItems. Fix is queued.
_schema/ holds 6 JSON Schema 2020-12 files; validate.mjs uses ajv8 + ajv-formats + js-yaml

Status: 4 packs validate (textile production-ready 1.2.0; aeronautics, agrifood, cement). Demo packs cover laboratories. 26 plugin manifests in packages/plugin-manifests/{drivers,transformers,widgets} (15 production, 11 with demo: true).

Adding a new sector:

Copy textile/
Replace asset templates from schemas/{sector}/*.json
Write recipes.yaml (ISA-88 phases)
Point ingestion-templates/ at OPC UA / Modbus / Sparkplug
Run node packages/sector-packs/_schema/validate.mjs

→ Skill candidate: sector-pack-extractor (used 4× successfully). See NOTQIN Reusable Skills.

5. Site lifecycle + Forge token flow

POST /api/v1/sites                     → Draft  (name → slug, JWT signed by SITE_TOKEN_SECRET)
PATCH /api/v1/sites/:id                → fill signal mappings
POST /api/v1/sites/:id/validate        → Validated  ({valid:true})
POST /api/v1/sites/:id/deploy          → Deployed   (digest)
GET  /api/v1/sites/active/spec         → consumed by orchestrate (else local-fallback)

Token model: short-lived JWT, name-derived slug (uniqueness check, not numbered). Provisioner endpoints behind Bearer (FORGE_DEV_API_TOKEN); nginx layer injects the header so token never reaches the JS bundle.

6. Native OPC UA integration

Library: asyncua (from asyncua import Client)
Modes: Subscription (default — create_subscription + DataChange handler → asyncio queue) and Polling (poll_interval_sec)
Auth: username/password + security policy Basic256Sha256
Quality mapping: StatusCode.is_good() → Quality.GOOD/BAD
Path: apps/ingestion-service/notqin_ingestion/adapters/opcua.py
Real example: Siemens S7-1500 exposing MachineState, RecipeID, BathTemperature, BatchID, AlarmActive
Gotcha: subscription model has no mid-session-drop detection → Wave 9 reconnect test redesigned to validate connect-failure backoff.

7. PLC simulation environment

101 machines across 4 sectors at simulator/sectors/:

Sector	Customer fiction	Machines
Aeronautics	Safran Nacelles	21 (autoclaves, 5-axis CNC, cleanroom AHUs, robots, test benches)
Textile	Hallotex	48 (air-jet looms, OE spinners, circular knitters, jacquards, dyeing, stenters)
Agri-food	Extralait	17 (pasteurizers, yogurt lines, butter, refrigeration, CIP, filling)
Cement	Novacim	15 (OK mills, VRM drives, rotary kiln, fans, crusher, packing)

Common base: TelemetryPayload, MachineSimulator, SectorRunner, physics helpers gaussian_walk, StateMachine, bernoulli(rng, prob).

Gotcha: bernoulli argument order trips agents — always bernoulli(rng, prob), not bernoulli(prob, rng). Memorize.

Start: docker compose build && docker compose up -d boots Mosquitto (1883/8883), Telegraf MQTT→InfluxDB machine_telemetry, 4 simulator containers. ESP32 path: python -m simulator.publish_fleet --cafile ./mosquitto/certs/ca.crt --user notqin-simulator --password $MQTT_PASS.

8. Site Composer wizard

8-step Composer.jsx: pack → basics → assets → UNS tree → signals → dashboards → alerts → review. Autosaves to localStorage, emits Site Spec JSON. fillDemoValues populates required signals per template:

air-jet-loom: picks_per_min, air_pressure_bar, alarm_active
dyeing-machine: bath_temp_c, bath_ph
OE-spinner: rotor_speed_rpm, end_breaks_per_hour

“Save to Forge” → saveSite → POST /api/v1/sites. Generator produces 6 artifacts: docker-compose.yml, .env, Keycloak realm.json, mosquitto.conf, ingestion-template.yaml, README.md.

9. Production Intelligence (OEE pipeline)

influxdb/tasks/03_oee.flux → oee_telemetry measurement (Availability × Performance × Quality per 8h shift)
Downsampling: 1-min after 7d (01_downsample_1min.flux), 1-h after 30d (02_downsample_1h.flux)
Grafana datasources InfluxDB-1m, InfluxDB-1h provisioned for direct query
14 alert rules in grafana/provisioning/alerting/notqin_alert_rules.yaml; food-safety rules at 30s evaluation
Source data: simulator machine_telemetry (tags: sector/company/site/machine_type/machine_id/status/protocol)

10. Wave-driven delivery — actual state

Wave	Title	Status	Delivered
0	DigiTPME factory-app	✅	5 sections + 4 stunning (DependencyGraph, PredictiveMaintenance, ProductionIntelligence, FloorMap)
1	Forge Catalog + PackDetail	✅	8-tab pack detail, 4 packs
2	Site Composer	✅	8-step wizard, autosave, JSON export
3	Deployment Generator	✅	6 artifacts
4	Plugin Registry	✅	6 types, 26 manifests, 10 demo
5	Knowledge Graph	✅	alarm precision/recall, learnings → pack PRs
15	anomaly + OEE + downsampling + state-service + k8s + EMQX-HA + Kafka + JSON Schemas	✅	All 15 implemented
Sprint 1 wire-up	Prisma migrations, IEIA bridge, Composer Save→API, Sites.jsx live	✅	TanStack Query, Keycloak, axios
TLS+auth	acme.sh + Porkbun DNS-01, Bearer auth	✅	LE cert, HTTP/2, real prod URL
Provisioner	$6 droplet auto-provision	✅	4m15s end-to-end with DNS+TLS

Queued / blocked

Runtime config injection for per-droplet Vite (VITE_* is build-time → bakes monmachine URL into customer bundles)
Auto-trigger provisioning from Forge Deploy button (currently separate click)
TLS rate-limit hardening on retry (LE production limits)
5th sector decision (proposed: automotive — Renault Tangier, Stellantis Kenitra, ~30% reuse from aeronautics)
Apply prefixItems schema fix
Tests: Vitest + RTL coverage, one Playwright e2e (Composer → Generator → bundle)
Multi-tenant SaaS deferred to phase 3
ISO 27001 audit interceptor → quarterly evidence exporter

11. HighByte capability benchmark (scrape 2026-05-27)

Use this as an external product benchmark, not as a copy target. HighByte’s v4.x release notes and product pages show what mature enterprise Industrial DataOps buyers may expect.

Benchmark capability	HighByte signal	NOTQIN backlog decision
Callable pipelines / API layer	v4.1 REST Data Server + API Builder; callable pipelines can return values or raise errors.	Add `notqin-api-layer`: read-only customer/agent API over curated UNS, sector-pack outputs, and evidence objects.
Pipeline debug mode	v4.1 Pipeline Debug compares before/after events through stages.	Add operator/developer troubleshooting for orchestrate pipelines before production pilots.
Model validation stage	v4.0 model validation and model reshaping stages.	Mirror as deterministic JSON Schema validation for sector-pack events and CBAM proof objects.
Namespaces + Smart Query	v4.0 Namespaces; v4.3 namespace browsing; v4.4 federated namespace / central data direction.	Evaluate a UNS query layer while keeping Morocco UNS path rules canonical.
HA config/state sync	v4.3 synchronizes project configuration and state between Intelligence Hubs.	Add production-readiness requirement for multi-node pilots after first paid deployment.
OTel and pipeline statistics	v4.2 OpenTelemetry; v4.3 pipeline event-rate stats.	Define observability baseline for broker, connectors, pipelines, and agents.
Store-and-forward	v4.4 performance improvements and throttling.	Treat edge buffering as mandatory for unreliable brownfield plant connectivity.
OPC UA NodeSet import as models	v4.3 imports OPC UA NodeSets as Models.	Future ingestion feature for asset model bootstrapping and knowledge-spine indexing.
Ignition Module	v4.0 direct Ignition integration and subscriptions.	Track as partner/interoperability requirement if Moroccan prospects run Ignition.

Concrete follow-up: NOTQIN Factory API Layer Spec 2026 turns the callable pipeline / REST / i3X benchmark into a scoped NOTQIN product surface: read-only UNS browse/read, KPI reads, signed evidence objects, and export generation.

12. Architectural Decision Records (in spirit)

#	Decision	Rationale
InfluxDB > Timescale	1Hz × 101 machines = 8.7M pts/day; native Flux tasks; Grafana-native datasource
NiFi for ESP32 + Telegraf for sector sims	NiFi for branch/route/enrich on legacy textile schema; Telegraf for simple `machine_telemetry` ingest
Sample envelope + Quality enum	Uniform contract across 5 protocols
Java 8 for Hadoop+Hive+HBase	Hive 3.1.3 dies on Java 9+ (`URLClassLoader` removal)
Phase-gated deployment (ADR-002)	Regulated industries, data sovereignty
Heritage + composition + abstraction	`AssetTemplate` abstract; `SectorPack` has templates+KPIs+rules+recipes; UNS = universal contract
Sector pack as moat	Domain knowledge baked into asset templates + KPIs + alert rules + recipes + UNS conventions
Plain MQTT 1883 internal + TLS 8883 external	Simulator/Telegraf use anonymous internal; gateway uses authenticated TLS
Bearer auth at nginx, never in JS	`envsubst` filtered to `^FORGE_` keeps `$uri` literal
acme.sh + Porkbun DNS-01 over HTTP-01	Works during droplet bootstrap before nginx is up; allows wildcard-future
Short-lived JWT > long-lived API key	Cleaner rotation
Name-derived slug, uniqueness-checked	Not numbered

13. Hard-won gotchas (don’t lose these)

Python / SymPy

sympify("sqrt") returns a Symbol, not a function — must from sympy import sqrt as sympy_sqrt, pi as sympy_pi
Pydantic EmailStr rejects .local — use admin@notqin.io not admin@ieia.local

LLM / API

Gemini OpenAI-compat rejects required: [], additionalProperties, null in assistant messages — build a clean dict, don’t msg.model_dump()
Gemini free tier = 20 req/day → fall back to OpenRouter openai/gpt-4o-mini for tool-call workflows

Big-data stack

Hive 3.1.3 + Java 11 = broken (URLClassLoader.getURLs removed) → install Temurin Java 8
Hadoop 3 refuses to run as root by default → set HDFS_NAMENODE_USER=root in hadoop-env.sh

DigitalOcean / provisioning

First-boot apt lock — unattended-upgrades concurrent; bootstrap must wait up to 5min
apt-get install --no-install-recommends docker.io doesn’t ship CLI binary → use static binary from download.docker.com
monmachine had no SSH private key, only authorized_keys → had to generate ed25519 keypair to act as initiator

Windows / shell

PowerShell Set-Content injects CRLF into bash scripts → round-trip via git-bash tr -d '\r'; commit scripts to repo, not Temp
PowerShell heredoc not supported by remote shells → use Write tool → scp pattern

Vite / nginx

VITE_* env vars are build-time constants baked into JS — for per-customer URLs need runtime config injection
Nginx host config bypasses container nginx — adding /api/provisioner/ route required editing /etc/nginx/sites-enabled/notqin-demo directly via install script
Mount conflict when binding a file into a read-only parent → keep one canonical location

Porkbun

acme.sh’s dns_porkbun.sh reads PORKBUN_API_KEY and PORKBUN_SECRET_API_KEY (uppercase, API_KEY not Api_Key) — older docs are wrong
API access is per-domain toggle (Domain Management → Details dropdown), NOT on the API page
Root domain ≠ subdomain — calls target ahmedbenahmed.com with subdomain field set to <slug>.notqin

Fastify

500 on POST with Content-Type: application/json and empty body — only set the header when body is present

Process

feedback_codex_hangs.md: Codex hangs after work is done → detect via mtime gap >5min + verify build/tests, don’t wait for the never-coming notification

13. Reusable prompt patterns (verbatim, proven)

“don’t stop until you finish them all even if it will take you hours” → full autonomy
“if you need to make any decision just do the best production ready one” → unblocks decisions
“save notes apart and tell me where I will find them” → produces ENHANCEMENT_NOTES.md, PROGRESS.md
“split the prompt to output small chunks” → recovery from 32k token cap
“go ahead, I will let you work, ask me anything you need right now (credentials, api keys)” → pre-bundles secrets in one round-trip; rule: put secrets in server .env, never in chat
“based on what’s added what’s can be added to the [other repo]” → 7-item gap analyses (cross-pollination)
“document that decision” → ADR pattern at docs/architecture-decision-records/00X-*.md
“what’s still missing or what do you suggest as enhancement?” → 15-item ranked list
“what are the next steps?” → sprint-1/sprint-2/sprint-3 plans

14. Files to revisit when starting any next project

apps/provisioner-service/ — reusable for any per-customer droplet pattern
packages/sector-packs/_schema/validate.mjs — JSON Schema 2020-12 with ajv reference
simulator/sectors/common/ — physics helpers (gaussian_walk, StateMachine, bernoulli)
deploy/site-stack/tls-bootstrap.sh — acme.sh + Porkbun reference
VISION.md, ENHANCEMENT_NOTES.md, PROGRESS.md
docs/architecture-decision-records/002-deployment-model.md

NOTQIN — OT Track

Explorer

NOTQIN Architecture Playbook

NOTQIN Architecture Playbook

1. Canonical pipeline (multi-sector)

2. Deployment topology

3. Modules / services

4. Sector pack model — the moat

5. Site lifecycle + Forge token flow

6. Native OPC UA integration

7. PLC simulation environment

8. Site Composer wizard

9. Production Intelligence (OEE pipeline)

10. Wave-driven delivery — actual state

11. HighByte capability benchmark (scrape 2026-05-27)

12. Architectural Decision Records (in spirit)

13. Hard-won gotchas (don’t lose these)

13. Reusable prompt patterns (verbatim, proven)

14. Files to revisit when starting any next project

See also

Graph View

Table of Contents

Backlinks