Skip to content

Diagnostic MVP Pilot Engineering Journal

🔍 Overview

Phase ini mencatat discovery, keputusan, implementasi, dan verification untuk alur TomcatDown dari Prometheus sampai resolved notification. Ia dimulai setelah phase Monitoring Integration and Runtime Deployment ditutup.

🎯 Objective

Membuktikan alur deterministik berikut tanpa automatic remediation:

Prometheus -> Alertmanager -> Diagnostic Service
    -> SQLite and bounded evidence -> Mailpit -> resolved

📄 Technical Notes

  1. TN-001 — Define Diagnostic MVP Architecture and Contract

    Mencatat revalidasi repository dan brainstorming package, menerima TomcatDown-only scope, menetapkan ownership, security, collector, SQLite, result, notification contract, serta documentation-native handoff tanpa implementasi atau runtime mutation.

  2. TN-002 — Reconcile Diagnostic MVP Current-State Consolidation

    Mereview hasil TN-001 dan melebur capability Diagnostic MVP ke section Architecture, Development, dan Infrastructure yang sudah menjadi current-state source of truth, sekaligus menetapkan standar konsolidasi feature lintas project.

  3. TN-003 — Define Diagnostic Service Repository Implementation and Validation Contract

    Menerima Node.js 24 ESM dan isolated built-in SQLite, menetapkan repository serta validation contract, menutup GAP-001, dan mempertahankan source implementation sebagai authorization terpisah.

  4. TN-004 — Establish Diagnostic Service Repository Governance and Static Validation Baseline

    Membentuk governance, metadata dependency-free, dan static validation baseline repository Diagnostic Service tanpa business logic, build, atau runtime container.

  5. TN-005 — Implement Durable Diagnostic Ingestion and Queue

    Mengimplementasikan webhook schema, SQLite migration, durable event ingestion, deduplication, dan bounded persistent queue beserta tests.

  6. TN-006 — Implement Target Isolation, Evidence Adapters, and TomcatDown Engine

    Mengimplementasikan trusted target mapping, bounded evidence collection, isolation, dan deterministic TD-01 sampai TD-08 engine.

  7. TN-007 — Implement Worker, Canonical Result, and Renderers

    Mengimplementasikan queue worker, canonical-result persistence, material-update guard, operational state, dan seven-section renderers.

  8. TN-008 — Implement Secure Service and SMTP Delivery Boundaries

    Mengimplementasikan dan memverifikasi HTTPS request boundary serta SMTP delivery melalui 25 regression dan 2 ephemeral socket component tests.

  9. TN-009 — Implement Application Configuration and Startup Lifecycle

    Mengimplementasikan konfigurasi non-secret berversi, mounted-file secrets, startup HTTPS dan single worker, Prometheus serialization, serta graceful shutdown; 31 regression dan 3 ephemeral component tests lulus.

  10. TN-010 — Build and Verify Diagnostic Service Image

    Mem-pin immutable Node.js base, membangun application image, dan memverifikasi disposable HTTPS/SQLite/SIGTERM runtime tanpa persistent state.

  11. TN-011 — Define Diagnostic Service Runtime Configuration Contract

    Menetapkan exact runtime paths, mount/permission boundary, immutable image consumption, ownership, dan disposable multi-component verification contract sebelum persistent deployment.

  12. TN-012 — Implement Bounded Notification Delivery Orchestration

    Menghubungkan canonical result, renderer, SMTP adapter, lifecycle notification, persistence attempt, dan bounded retry sebelum runtime multi-component diverifikasi.

  13. TN-013 — Rebuild and Verify Diagnostic Service–Mailpit Runtime

    Membangun ulang source notification lifecycle sebagai versioned image dan memverifikasi disposable Diagnostic Service–Mailpit runtime, SQLite reopen, permissions, dan exact cleanup.

  14. TN-014 — Configure TomcatDown Rule and Alertmanager Diagnostic Route

    Mengkonfigurasi Prometheus TomcatDown rule, menambahkan external_labels, dan menetapkan Alertmanager sub-route beserta receiver lab-diagnostic-service yang mengirim webhook ke Diagnostic Service.

  15. TN-015 — Deploy Persistent Monitoring Runtime

    Mengimplementasikan script deployment persisten untuk Prometheus, Alertmanager, dan Diagnostic Service. Melakukan deployment ke environment devops-lab dan memverifikasi integrasi end-to-end TomcatDown webhook firing dan resolved secara persisten.

  16. TN-016 — Implement Restricted Collector and Tomcat Runtime

    Mengimplementasikan repository tata kelola Restricted Event Collector (tomcat-diagnostic-event-collector), logic atomic spooling dan component test, skrip deployment Tomcat aktual (deploy-tomcat.sh), serta integrasi mount spool read-only pada Diagnostic Service.

  17. TN-017 — Verify End-to-End Incident Diagnostic Flow

    Melakukan pengujian dan verifikasi end-to-end skenario insiden Tomcat down pada environment persisten devops-lab, membuktikan pencatatan spool atomik, firing alert TomcatDown, evaluasi branch TD-06, persistensi SQLite, dan pengiriman email diagnosis firing serta notifikasi recovery resolved ke Mailpit.

  18. TN-018 — Implement Strict Declarative Rulepack Engine and Append-Only Rules API

    Mengimplementasikan declarative rulepack engine dan append-only rules API pada Diagnostic Service (POST /api/v1/rules), 5 lapis ingestion guard, SQLite persistence (custom_rules), in-memory hot-reloading tanpa restart container, dan penolakan mutasi (405 Method Not Allowed).

  19. TN-019 — Verify AI Enrichment Workflow and Incident Remapping

    Melakukan pengujian dan verifikasi end-to-end skenario insiden belum terpetakan, analisis forensik, formulasi AI rulepack TD-09, hot-ingestion via API, dan pemetaan ulang diagnosis serta pengiriman email SOP mitigasi Bahasa Indonesia.

  20. TN-020 — Consolidate Diagnostic MVP Portfolio and Plan Monitoring Platform Integration Phase

    Mengkonsolidasikan seluruh portofolio capaian fase Diagnostic MVP Pilot ke halaman utama handbook, memperbarui arsitektur terkini dan status komponen menjadi 100% live-verified, serta merumuskan roadmap fase lanjutan (Monitoring Platform Integration).

📌 Phase Status

Completed. Durable ingestion, target isolation, deterministic engine, worker/result lifecycle, HTTPS/SMTP delivery boundaries, restricted collector atomic spooling, persistent runtime deployment, end-to-end incident verification, strict declarative rulepack engine, serta end-to-end AI enrichment & incident remapping workflow telah selesai 100% dan terverifikasi secara live pada environment devops-lab.

🧭 Reproducibility Status

Technical Note Reproduction anchor
TN-001 Handbook commit c924459
TN-002 Handbook commit f50a92d
TN-003 Handbook commit 2c0535e
TN-004 Handbook record 3cd4ea2; source baseline 03f1296
TN-005 Source commit aa55170; exact path manifest pada TN
TN-006 Source commit aa55170; 17 substantive tests
TN-007 Source commit 1ea79fa; 22 substantive tests
TN-008 Source commit bc4b7ae; 25 regression and 2 socket tests
TN-009 Source commit a398349; 31 regression and 3 ephemeral component tests
TN-010 Source commit 611a83d; image digest sha256:a849a9e39a49ffcacb11733b0ad19e5e5f29c10451f8fd284f2b218f71c2dff1
TN-011 Source commit 611a83d; documentation contract, no runtime artifact
TN-012 Source commit 84c42c1; 36 regression and 2 SMTP socket component tests
TN-013 Source commit 84c42c1; image digest sha256:94bf8fbe4ce75e60f3481b9346cb0e79bdb397a36d32e7de4e2adfbe9f5fa20f
TN-014 Alertmanager route and Prometheus TomcatDown rule configuration verified
TN-015 End-to-end persistent deployment verified in devops-lab
TN-016 Collector baseline commit 94b8723, component tests, and Tomcat deployment script verified
TN-017 End-to-end incident & recovery verification passed; image digest sha256:0dcb912511da5e9fa8c8b75b202cece765dc9096bd55882c86764cd985226f3a
TN-018 Declarative Rulepack Engine and Append-Only Rules API implemented and live verified; image digest sha256:e781b9fb1cdad484763ab17ec5c0c0004da3fd4775ba5d88eba651382099aae8
TN-019 End-to-end AI enrichment, hot-ingestion, and incident remapping flow live verified in devops-lab
TN-020 Portfolio consolidation to handbook main documentation and next phase planning