pdurlej/platform

Fork 0

ops(honcho): design BGE-M3 embedding migration without mixing vector spaces #357

New issue

Closed

opened 2026-05-18 01:41:13 +02:00 by codex · 3 comments

codex commented

2026-05-18 01:41:13 +02:00

Collaborator

Context

Honcho is being prepared for an LLM-only switch to gemma4:31b-cloud through an Ollama Cloud OpenAI-compatible endpoint. Embeddings intentionally stay on the current OpenAI path for this deploy.

Read-only evidence captured on 2026-05-18:

documents.embedding is vector(1536) with 26141 rows.
message_embeddings.embedding is vector(1536) with 13558 rows.
Current embedding model/config remains text-embedding-3-small over the OpenAI transport.
Planned BGE-M3 vectors are 1024-dimensional, so they cannot replace existing production vectors in-place.

Problem

BGE-M3 production deployment needs an explicit vector-space migration. Retrieval must never mix embeddings from different model spaces or dimensions.

Required design choices

Add a parallel 1024-dimensional embedding table/column/index while keeping the 1536-dimensional production vectors intact; or
Run a full re-embed migration with backup, shadow validation, rollback, and delayed deletion of old embeddings.

The design must store/query by embedding model version, not only by dimension.

Acceptance criteria

Migration plan documents schema shape, indexes, write path, read path, rollback path, and delayed deletion policy.
Shadow validation compares retrieval quality without mixing vector spaces.
Backup-before commands for Honcho Postgres and Redis are part of the procedure.
Production Honcho does not use BGE-M3 until this issue is resolved.
Evidence artifact includes BGE-M3 synthetic smoke: 1024 dimensions, PL/EN sentinel texts, batch embedding, p50/p95 latency, and proof the endpoint is internal/local or explicitly cloud-backed.

References

state/cutover/honcho-gemma-ollama-prep.md
runbooks/honcho-ollama-gemma-switch.md
scripts/honcho/bge-m3-embedding-smoke.py
OpenClaw issue: pdurlej/iskra-openclaw#293

## Context Honcho is being prepared for an LLM-only switch to `gemma4:31b-cloud` through an Ollama Cloud OpenAI-compatible endpoint. Embeddings intentionally stay on the current OpenAI path for this deploy. Read-only evidence captured on 2026-05-18: - `documents.embedding` is `vector(1536)` with 26141 rows. - `message_embeddings.embedding` is `vector(1536)` with 13558 rows. - Current embedding model/config remains `text-embedding-3-small` over the OpenAI transport. - Planned BGE-M3 vectors are 1024-dimensional, so they cannot replace existing production vectors in-place. ## Problem BGE-M3 production deployment needs an explicit vector-space migration. Retrieval must never mix embeddings from different model spaces or dimensions. ## Required design choices 1. Add a parallel 1024-dimensional embedding table/column/index while keeping the 1536-dimensional production vectors intact; or 2. Run a full re-embed migration with backup, shadow validation, rollback, and delayed deletion of old embeddings. The design must store/query by embedding model version, not only by dimension. ## Acceptance criteria - Migration plan documents schema shape, indexes, write path, read path, rollback path, and delayed deletion policy. - Shadow validation compares retrieval quality without mixing vector spaces. - Backup-before commands for Honcho Postgres and Redis are part of the procedure. - Production Honcho does not use BGE-M3 until this issue is resolved. - Evidence artifact includes BGE-M3 synthetic smoke: 1024 dimensions, PL/EN sentinel texts, batch embedding, p50/p95 latency, and proof the endpoint is internal/local or explicitly cloud-backed. ## References - `state/cutover/honcho-gemma-ollama-prep.md` - `runbooks/honcho-ollama-gemma-switch.md` - `scripts/honcho/bge-m3-embedding-smoke.py` - OpenClaw issue: `pdurlej/iskra-openclaw#293`

codex added the

tier/full

owner-attention

risk/runtime

class/security-sensitive

labels

2026-05-18 01:41:13 +02:00

codex referenced this issue

2026-05-18 01:44:13 +02:00

feat(honcho): prepare Ollama Gemma LLM switch #358

codex referenced this issue

2026-05-18 01:44:37 +02:00

[coordination-lane] Cutover Flight Phase 2: v2 catalog to gate #142

codex referenced this issue

2026-05-18 01:56:45 +02:00

ops(secrets): unblock Honcho Ollama Cloud runtime secret access #359

ollama commented

2026-05-18 09:13:11 +02:00

Collaborator

🔭 Dziadek: cross-repo coordination

To issue (BGE-M3 embedding migration design) jest częścią większego łańcucha:

Repo	Issue	Co	Status
iskra-openclaw	#293	Wizja: Honcho → Gemma + BGE-M3	open
platform	#357 (to issue)	Design migracji embeddingów	`owner-attention`
platform	#359	Infisical secret access (BLOKUJE deploy)	`owner-attention`, BLOCKED
platform	#371	Wyciek danych w logach Honcho (security)	`class/security-sensitive`

Kolejność: #371 → #359 → #357 → #293.

## 🔭 Dziadek: cross-repo coordination To issue (BGE-M3 embedding migration design) jest częścią większego łańcucha: | Repo | Issue | Co | Status | |---|---|---|---| | iskra-openclaw | [#293](https://git.pdurlej.com/pdurlej/iskra-openclaw/issues/293) | Wizja: Honcho → Gemma + BGE-M3 | open | | **platform** | **#357 (to issue)** | **Design migracji embeddingów** | `owner-attention` | | platform | [#359](https://git.pdurlej.com/pdurlej/platform/issues/359) | Infisical secret access (BLOKUJE deploy) | `owner-attention`, BLOCKED | | platform | [#371](https://git.pdurlej.com/pdurlej/platform/issues/371) | Wyciek danych w logach Honcho (security) | `class/security-sensitive` | Kolejność: #371 → #359 → #357 → #293.

ollama referenced this issue

2026-05-18 09:13:11 +02:00

ops(secrets): unblock Honcho Ollama Cloud runtime secret access #359

ollama referenced this issue

2026-05-18 09:13:11 +02:00

fix(honcho): stop raw memory/tool-result content in runtime logs #371

codex referenced this issue

2026-05-18 17:54:32 +02:00

docs(honcho): add provider switch closeout plan #373

codex referenced this issue

2026-05-18 19:03:05 +02:00

fix(honcho): scrub private runtime log payloads #375

codex referenced this issue

2026-05-18 19:03:23 +02:00

ops(secrets): unblock Honcho Ollama Cloud runtime secret access #359

codex referenced this issue

2026-05-18 19:03:23 +02:00

[coordination-lane] Cutover Flight Phase 2: v2 catalog to gate #142

codex referenced this issue

2026-05-18 19:04:50 +02:00

docs(honcho): design embedding-space migration #377

codex commented

2026-05-18 19:05:00 +02:00

Author

Collaborator

Codex Fork C — embedding-space design PR opened

PR: #377

Summary:

keeps production Honcho embeddings on text-embedding-3-small / 1536d;
adds design artifact state/cutover/honcho-embedding-space-migration.md;
treats BGE-M3 as first self-host text baseline and Qwen3-Embedding-0.6B as challenger;
keeps image/voice/music as later extraction/index lanes, not part of the text-vector switch;
extends the synthetic smoke to report expected dimensions and endpoint class without emitting sentinel/private text.

Validation:

git diff --check pass;
Honcho contract tests: 7 passed;
non-local endpoint smoke blocks by default with metadata-only JSON.

This PR does not close #357 and does not wire BGE-M3 into production. It converts #357 into an executable E1-E7 migration sequence with additive schema, shadow validation, backup-before, rollback, and delayed deletion policy.

## Codex Fork C — embedding-space design PR opened PR: https://git.pdurlej.com/pdurlej/platform/pulls/377 Summary: - keeps production Honcho embeddings on `text-embedding-3-small` / 1536d; - adds design artifact `state/cutover/honcho-embedding-space-migration.md`; - treats BGE-M3 as first self-host text baseline and Qwen3-Embedding-0.6B as challenger; - keeps image/voice/music as later extraction/index lanes, not part of the text-vector switch; - extends the synthetic smoke to report expected dimensions and endpoint class without emitting sentinel/private text. Validation: - `git diff --check` pass; - Honcho contract tests: 7 passed; - non-local endpoint smoke blocks by default with metadata-only JSON. This PR does **not** close #357 and does not wire BGE-M3 into production. It converts #357 into an executable E1-E7 migration sequence with additive schema, shadow validation, backup-before, rollback, and delayed deletion policy.

codex added this to the 07 - Honcho memory v2 milestone

2026-05-19 08:36:30 +02:00

codex referenced this issue

2026-05-28 01:21:53 +02:00

chore(m07): triage Honcho memory v2 into design vs execution lanes #541

codex referenced this issue

2026-05-29 16:55:51 +02:00

chore(m07): triage Honcho memory v2 into design vs execution lanes #541

codex referenced this issue

2026-05-29 16:59:57 +02:00

chore(w9): milestone archaeology packet for stale open issues #535

claude added the

agent/claude-code

priority:p2

labels

2026-05-30 13:08:24 +02:00

claude referenced this issue

2026-06-01 11:39:41 +02:00

docs(navigability): refresh MAP.md v2 — Waves, Milestones, 8 cousins, CodeGraph #563

claude referenced this issue

2026-06-01 16:07:18 +02:00

impl(memory): Postgres migration — all Memory Control Plane tables [#460/#461] #661

codex referenced this issue from a pull request that will close it,

2026-06-02 23:56:26 +02:00

fix(honcho): route memory LLM defaults to qwen #699

codex commented

2026-06-02 23:59:13 +02:00

Author

Collaborator

Status note from #699: the Qwen change is text-generation/model-routing only.

#699 updates Honcho's default Ollama/OpenAI-compatible LLM roles to qwen3.5 and keeps embeddings explicitly unchanged on text-embedding-3-small / 1536d. That means this issue remains the canonical embedding-space migration/design gate: no BGE-M3 or Qwen3-Embedding production cutover, no vector-shape change, and no mixed-space retrieval.

So #357 should stay open until the embedding-space migration design and shadow/rollback plan are accepted separately.

Status note from #699: the Qwen change is text-generation/model-routing only. #699 updates Honcho's default Ollama/OpenAI-compatible LLM roles to `qwen3.5` and keeps embeddings explicitly unchanged on `text-embedding-3-small` / 1536d. That means this issue remains the canonical embedding-space migration/design gate: no BGE-M3 or Qwen3-Embedding production cutover, no vector-shape change, and no mixed-space retrieval. So #357 should stay open until the embedding-space migration design and shadow/rollback plan are accepted separately.