# OKF-Compatible Bundle And Derived Retrieval Index Architecture V01 Issue: [#285](https://github.com/pinklon/ai-capability-discipline/issues/285) Parent reference: [#271](https://github.com/pinklon/ai-capability-discipline/issues/271) Related references: - [#269 Source Registry Admin Layer v01](https://github.com/pinklon/ai-capability-discipline/issues/269) - [#286 Google New SDLC candidate-source evaluation](https://github.com/pinklon/ai-capability-discipline/issues/286) - [#287 Queue-backed multi-agent work engine evaluation](https://github.com/pinklon/ai-capability-discipline/issues/287) ## Governance Boundary This artifact is a design and architecture decision for future knowledge-bundle and retrieval-index work. It is not package authority, source activation, runtime source-set configuration, database implementation, MCP implementation, provider approval, model approval, Cloudflare Access/WAF approval, Production approval, Preview approval, or enterprise workflow approval. This note does not add arbitrary web retrieval, user-provided URL retrieval, browser-side provider calls, browser-side current-source calls, durable telemetry, uploads, private-data workflow, source registry administration, or an implemented database/vector layer. ## Problem The repository already uses Git-controlled artifacts, generated context-pack files, source profiles, extraction runs, citation anchors, validation receipts, authority lanes, and package/current-public separation. That model is auditable, but the source-grounded assistant and future Open Brain / SourceMesh / ArtifactPlane work need a more explicit architecture for portable semantic bundles and derived retrieval projections. The decision is whether the repo should adopt OKF-compatible knowledge bundles and whether database/vector indexes should become derived retrieval projections while Git remains canonical authority. ## Decision Drivers - Preserve Git as the governed source of truth for authored artifacts, generated manifests, receipts, and review history. - Keep context-pack JSONL useful for current static package consumption while allowing future projections. - Make source profiles, extraction runs, citation anchors, validation receipts, authority lanes, and package/current-public separation survive indexing. - Avoid designing #269 Source Registry Admin Layer around an undersized representation. - Account for the current Cloudflare Pages runtime path for GhostMesh. - Account for Tony already having a paid Supabase subscription and intended Open Brain / SourceMesh / ArtifactPlane use. - Treat vector retrieval as complementary ranking and recall infrastructure, not a replacement for deterministic manifests, citations, receipts, and source-lane enforcement. - Prefer deterministic script-first generation and validation over LLM-only corpus restructuring. - Keep OKF and MCP complementary rather than collapsing either into authority. ## Current Constraints - Git is the repository source of truth. - `context-pack/` is the canonical AI-consumable governed source package, with `docs/context-pack/` as the Cloudflare Pages published mirror. - Current/public retrieval remains gated, source-governed, and server-side. - Package-only remains the default authority posture. - Package + current context must keep package citations separate from current/public context citations. - Existing source registries include candidate-only entries, owner authorization records, extraction profile work, and validation receipts. - Runtime proof work must not add browser-side provider calls or browser-side current-source calls. - #269 must not start until the representation decision is made. - Remaining #271 extraction-profile work should pause until this issue decides whether profiles should map into OKF-compatible concepts, database tables, MCP resources, or a layered model. ## Candidate Architectures | Option | Summary | Strengths | Limits | Decision | |---|---|---|---|---| | 1. Git/context-pack only | Keep authored docs plus generated JSONL and manifests as the only retrieval substrate. | Lowest complexity, strongest current auditability, no new runtime dependency. | Does not prepare a durable semantic bundle model, relational query model, or future index projection. | Rejected as the only target, retained as the current baseline and fallback. | | 2. Git-first OKF-compatible bundle with Cloudflare D1/R2/Vectorize derived retrieval | Generate OKF-compatible files from Git-controlled artifacts, then compile Cloudflare-native metadata, payload, and vector indexes for GhostMesh runtime. | Fits current Cloudflare Pages product runtime, keeps deployment path compact, aligns with Worker-side retrieval. | Weaker long-term relational knowledge posture for Open Brain / SourceMesh / ArtifactPlane if used alone. | Accepted for near-term runtime prototype projection only. | | 3. Git-first OKF-compatible bundle with Supabase/Postgres/pgvector derived retrieval | Generate OKF-compatible files from Git-controlled artifacts, then compile Postgres metadata and pgvector retrieval projections. | Stronger relational model, audit queries, cross-project knowledge reuse, and alignment with Tony's existing paid Supabase subscription. | Adds a separate runtime/service path from current GhostMesh Cloudflare deployment. | Accepted for longer-term Open Brain / SourceMesh / ArtifactPlane knowledge path. | | 4. Dual-path model | Use the same Git/OKF canonical bundle to feed Cloudflare runtime projections and Supabase relational knowledge projections. | Preserves current runtime momentum while avoiding a dead-end schema for broader knowledge systems. | Requires strict projection manifests and validation so two indexes do not drift. | Recommended. | ## Recommendation Adopt a Git-first, OKF-compatible bundle model as a portable semantic layer, then treat context-pack JSONL, Cloudflare D1/R2/Vectorize, Supabase/Postgres/pgvector, and optional MCP resources/tools as generated projections from that controlled bundle. Recommended target: ```text Git-controlled authored artifacts and receipts -> deterministic OKF-compatible bundle projection -> generated context-pack JSONL projection -> generated Cloudflare D1/R2/Vectorize runtime projection -> generated Supabase/Postgres/pgvector relational knowledge projection -> optional read-only MCP resource/tool exposure -> policy-filtered runtime retrieval with canonical citations ``` Cloudflare should remain the near-term GhostMesh product-runtime path because the production domain, Pages project, proof route, static context-pack mirror, and server-side retrieval adapter already live there. Supabase should become the longer-term Open Brain / SourceMesh / ArtifactPlane relational knowledge path because the intended use is broader than a single runtime proof and Tony already has a paid Supabase subscription. The shared rule is that both paths consume the same Git-controlled bundle. Neither path becomes authority by itself. ## Rejected Alternatives ### Database As Canonical Authority Rejected. A database may serve query, filtering, retrieval, analytics, and projection needs, but database contents must be rebuildable from Git commit SHA, manifest hashes, source profile records, extraction receipts, and validation receipts. Database edits outside the compiler path are uncontrolled authority and must not ground package answers. ### OKF As Runtime Authority By Itself Rejected. OKF-compatible files can define semantic concepts, routing, metadata, and graph links, but the runtime answer boundary still depends on source-lane policy, package/current-public separation, citation validation, and compiled manifests. ### MCP As Package Authority Rejected. MCP can expose read-only OKF resources and controlled tools later, but tool execution returns dynamic state. Dynamic resource/tool output is not package authority unless it is separately captured, reviewed, committed, and validated through Git-controlled artifacts. ### Vector-Only Retrieval Rejected. Vector search can improve recall and ranking, but it cannot replace source profiles, exact citation anchors, authority lanes, deterministic manifests, receipts, chunk hashes, or rebuild validation. ### LLM-Only Corpus Restructuring Rejected. LLMs may assist with summaries or classification review, but structural conversion to OKF-compatible bundles must be script-first, deterministic, diffable, and validator-backed. ## Canonical Authority Model Canonical authority remains in Git: - authored package, product-architecture, source-governance, runbook, template, and receipt files - generated context-pack artifacts and published mirror hashes - source registry candidates and owner authorization records - extraction profile definitions and extraction-run receipts - validation scripts and validation receipts - issue and PR trail for review and close semantics Projection state must record: - `source_commit` - canonical source path - canonical reader target - artifact id - authority lane - source profile id when applicable - extraction run id when applicable - citation anchor id - chunk hash - generator version or script identity - manifest hash - validation receipt path Any projection row without those fields is retrieval-only and must not be treated as package authority. ## OKF Bundle Model The repo should generate an OKF-compatible bundle rather than hand-author a parallel corpus first. The bundle should use directory structure, Markdown files, and YAML frontmatter to represent atomic concepts. `type` should remain the required OKF field, with repo-specific governance extensions for authority and provenance. Recommended frontmatter sketch: ```yaml type: source_profile title: OWASP GenAI LLM Top 10 2025 artifact_id: current-source-profile-owasp-genai-2025 authority_lane: current_public approval_status: candidate_only retrieval_enabled: false canonical_source_path: docs/product-architecture/current_source_owner_authorization_owasp_genai_2025_v01.json reader_target: docs/product-architecture/current_source_owner_authorization_owasp_genai_2025_v01.json source_profile_id: owasp_genai_2025_bounded_html_extract_v01 resource: aicd://source-profiles/owasp_genai_2025_bounded_html_extract_v01 validation_receipt: validation/receipts/example.md source_commit: ``` The `resource` field must use allowlisted internal schemes such as `aicd://`, `git://repo-path/`, or future explicitly approved MCP resource names. It must not become arbitrary user-provided URL retrieval. Directory-level `index.md` files should act as deterministic routing tables. They should reduce blind recursive traversal, document local graph edges, and list the concept files in that directory. ## Database/Vector Projection Model Database and vector indexes should be generated projections with three classes of data: | Projection class | Examples | Authority posture | |---|---|---| | Relational metadata | artifact id, source path, authority lane, profile id, extraction run id, citation anchor id, chunk hash, receipt path, source commit | Derived from Git, auditable, rebuildable | | Object payload | extracted text, normalized Markdown, generated OKF files, raw extraction payloads when approved | Derived from Git-controlled source profiles and extraction receipts | | Vector embedding | embedding vector, model label, chunk hash, projection build id | Retrieval aid only, not source authority | Each projection build must produce a manifest that records the Git commit, generator command, included artifacts, skipped artifacts, hash totals, validation command results, and rollback pointer. ## Cloudflare Path Cloudflare remains the near-term GhostMesh runtime path. Recommended Cloudflare projection: - D1 stores compact metadata tables for artifacts, source profiles, authority lanes, chunks, citations, and projection builds. - R2 stores larger generated payloads when they should not live entirely in D1. - Vectorize stores embeddings keyed by chunk hash and projection build id. - Workers enforce package/current-public lane separation before retrieval output reaches the provider boundary. - The static site continues to publish `docs/context-pack/` through Cloudflare Pages. This path should stay prototype-oriented until the projection compiler, validation manifest, rollback procedure, and runtime smoke gates exist. It must not mutate Preview or Production source sets inside this issue. ## Supabase Path Supabase/Postgres/pgvector should be the longer-term relational knowledge path for Open Brain / SourceMesh / ArtifactPlane. Recommended Supabase projection: - Postgres stores normalized artifacts, concepts, source profiles, extraction runs, citation anchors, receipts, package/current-public lanes, and relationship tables. - pgvector stores embeddings keyed by deterministic chunk hash. - SQL views support audit queries such as "which chunks cite this source profile", "which retrieval rows came from this Git commit", and "which rows are candidate-only". - Row-level security and API boundaries can be evaluated later, but they are not implemented by this issue. Tony already has a paid Supabase subscription, which reduces adoption friction for the long-term relational path. It does not make Supabase package authority, and it does not require moving current GhostMesh runtime off Cloudflare. ## OKF + MCP AI Context Stack Model Recommended model: ```text OKF = static semantic/context definition layer MCP = dynamic execution/resource/tool access layer database/vector index = optional derived retrieval projection Git = canonical governed source of truth ``` OKF should describe concepts, metadata, relationships, routes, and governed resource identifiers. MCP can later expose read-only OKF resources and controlled retrieval/admin tools. A future agent flow could be: ```text user request -> OKF semantic lookup -> extract governed resource URI -> invoke approved MCP resource/tool if needed -> return structured result -> synthesize answer with citations to canonical Git-controlled sources ``` This does not assume OKF has native first-class support in current coding agents or model runtimes. It also does not assume Google BigQuery-oriented reference tooling fits this repo without adaptation. ## Source Profile Mapping Source profiles should become OKF concepts and relational projection rows. Required mapping fields: - source profile id - source type - authority lane - approval status - retrieval enabled flag - source URL or canonical source identifier - extraction method - extraction bounds - normalization method - exact source version or retrieval timestamp when applicable - owner authorization status - validation receipt path - rollback path Candidate-only source profiles must remain candidate-only after OKF generation and indexing. Indexing must not promote them to package authority or enable runtime retrieval. ## Citation And Chunk Model Chunks should be deterministic and cite canonical anchors. Required chunk fields: - chunk id - artifact id - source path - reader target - section id or anchor - byte, line, heading, or structural anchor when available - source profile id when applicable - extraction run id when applicable - authority lane - normalized text hash - chunk hash - projection build id - citation display label Vector rows should point back to chunk hashes. Runtime answers should cite canonical files, reader targets, source URLs, or receipt anchors rather than vector row ids. ## Authority-Lane Enforcement Authority lanes must be explicit metadata in OKF frontmatter, context-pack JSONL, relational tables, vector metadata, and runtime retrieval output. Minimum lanes: - `package` - `current_public` - `validation_receipt` - `runtime_boundary` - `source_profile` - `operating_model` Rules: - Package answers may use `package` lane material only unless Package + current context mode is explicitly active. - Current/public material must remain separately cited and labeled. - Validation receipts can support provenance and build evidence, but they do not become package guidance by themselves. - Runtime boundary material can explain implementation posture, but it does not approve source activation. - Source profiles define retrieval eligibility and constraints, not answer authority by themselves. - Operating-model artifacts guide repo execution, not enterprise policy approval. ## Validation And Rebuild Model The generation path should be deterministic: ```text Git commit -> OKF-compatible bundle generator -> OKF validators -> context-pack projection -> database/vector projection manifests -> runtime smoke and boundary checks ``` Required validators should eventually cover: - frontmatter schema - required `type` - required authority metadata - index coverage - internal link integrity - resource URI allowlist - concept atomicity - source profile mapping - citation anchor integrity - chunk hash determinism - projection manifest completeness - package/current-public separation Every projection build must be reproducible from a Git commit and should fail closed when required metadata is missing. ## Rollback Model Rollback should restore the prior projection by Git commit and projection manifest: 1. Identify last accepted Git commit and projection build id. 2. Rebuild OKF bundle, context-pack projection, and any database/vector projection from that commit. 3. Compare hashes and projection manifests. 4. Deploy the previous Cloudflare or Supabase projection only after validation passes. 5. Record rollback receipt with source commit, projection build id, validation results, and runtime/source boundary confirmation. No manual database editing should be treated as rollback unless the database is restored to a build manifest that maps back to Git. ## Migration Path 1. Keep current Git/context-pack path as the baseline. 2. Add an OKF-compatible exporter that reads existing controlled artifacts and emits a candidate bundle. 3. Add OKF validators for frontmatter, index coverage, links, authority metadata, and resource URI allowlist. 4. Regenerate context-pack JSONL from either the current generator or the OKF-compatible source model after parity checks exist. 5. Prototype a Cloudflare projection for GhostMesh runtime retrieval without source activation. 6. Prototype a Supabase projection for Open Brain / SourceMesh / ArtifactPlane relational queries. 7. Compare projection manifests, query results, rollback behavior, and citation integrity. 8. Decide whether #269 admin workflows operate on Git/OKF packets first, then compile projections, or whether a controlled admin UI creates Git-backed change proposals. ## Sequencing Impact For #269, #271, #286, And #287 ### #269 Source Registry Admin Layer Pause implementation until the OKF/exporter and projection model is accepted. #269 should be redesigned as a Git/OKF-first admin workflow that creates reviewed source-profile packets and projection manifests rather than directly editing runtime database rows. ### #271 Extraction Profiles Continue to pause broad extraction-profile implementation until the representation is decided. Existing completed child work remains valid evidence. Remaining profiles should likely change representation so extraction profiles map into OKF concepts plus deterministic projection rows. ### #286 Google New SDLC Candidate Evaluation #286 should remain a candidate-source and operating-model gap evaluation. Its findings may inform OKF metadata, context engineering language, MCP cautions, and validation hooks, but #286 should not be blocked by database implementation and should not promote external guidance to package authority without separate approval. ### #287 Queue-Backed Multi-Agent Work Engine #287 should treat OKF and MCP as related context and execution layers. Work payloads may later reference OKF bundle paths or MCP resources, but queue/state-machine implementation should remain a separate decision and should not be started by this issue. ## Follow-Up Child Issues Recommended child issues: - OKF compatibility generator/exporter v01. - OKF frontmatter and authority-lane validator v01. - OKF index and link integrity validator v01. - Resource URI allowlist and MCP resource naming evaluation v01. - Cloudflare D1/R2/Vectorize projection prototype v01. - Supabase/Postgres/pgvector projection prototype v01. - Context-pack to OKF parity and migration plan v01. - Projection manifest and rollback validator v01. - Source Registry Admin Layer representation update for #269. ## Acceptance Boundary This issue should close when the architecture decision is documented, discoverable, validator-backed, and reflected in generated context-pack artifacts. Closing this issue does not approve database implementation, MCP implementation, source activation, current/public retrieval expansion, runtime source-set mutation, provider/model changes, package authority changes, Preview mutation, Production mutation, Access/WAF changes, or `codex-automerge`.