how i built org brain
how i built org brain
what it is
org brain is garden’s shared memory layer, one brain per workspace. it’s a clean way for an organisation to remember what it knows, and for agents to actually use that memory while they work instead of starting from zero every time.
people ask it questions and get answers with the file they came from attached. the agents can pull context mid run and a new teammate can easily access the brain and see durable decisions, owners, definitions, everything. all connected.
before i start explaining what org brain is, here are a couple of things it isnt:
- it’s not a google drive killer. moving off drive is a side effect of using it. if that happens, cool. it’s not the point.
- it’s not a doc editor. editing documents is a solved commodity. not interesting.
- it’s not a wiki with search. wikis rot. pages go stale, nobody updates them, search returns twelve near duplicates and a lie from 2023.
the whole design is me trying to not build those three things by accident.
garden workspace
│
┌───────────────┴───────────────┐
▼ ▼
people chat agents at work
ask questions pull context mid run
│ │
└───────────────┬───────────────┘
▼
┌───────────────────┐
│ org brain │
│ graph + vectors │
│ + keyword search │
│ one per workspace│
└─────────┬─────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
helixdb r2 raw postgres
nodes + edges bytes folders +
+ embeddings telemetry
the obvious approach is wrong
every company brain startup does the same thing. pipe everything into a vector store. slack, drive, github, all of it, embed it, search it.
i think that’s a trap. it produces two failures.
one, stale info. a doc from february and its replacement from june both sit in the store with equal weight. the model confidently quotes the old one. nobody deletes anything because nothing knows what superseded what.
two, confusion between two kinds of truth. what the org decided last quarter is not the same as what an issue’s status is right now. if you dump both into one blob, memory gets volatile and live tools get context poor.
so org brain keeps a hard line between durable knowledge and live state.
┌────────────────────────┐ ┌────────────────────────┐
│ org brain memory │ │ connected systems │
│ decisions │ │ current issue status │
│ owners │ │ latest email │
│ project context │ │ live deployment │
│ definitions │ │ recent messages │
└────────────────────────┘ └────────────────────────┘
durable live
slow, cited fast, current
written after work queried during work
a turn starts with the brain to know what “the migration” means, then hits the live system to check where it is. treating them as interchangeable breaks both.
four rules i refused to break
before any code, i wrote these down and made them non negotiable.
observe, don’t impose. the agent generates the graph itself. no fixed schema, no ontology you build up front that nobody maintains. kinds are free text. an agent can invent “policy” or “gotcha” and it just works. structure emerges or it doesn’t.
this matters because i watched microsoft viva topics die in feb 2025. it required a confirmed taxonomy before it gave any value, and the curation never kept up with reality. never require structure before value.
add and index are separate moments. adding a file and knowing its contents are two different events, and you choose when the second one happens. drop a file in, it’s in the grid, raw bytes in r2, costs nothing. index it and it becomes answerable. you can see the line between “we have this” and “we know this.”
identity is anchored but not merged. without a stable key you get three copies of the same company. so every item has a source_key. same key means update in place. different key means new item, even if it looks like a duplicate. when things genuinely are the same, i add a soft same_as link. i never auto merge. merging is destructive and i’m not doing destructive automatically.
writes are additive and traceable. nothing the brain writes edits existing knowledge in place. every item carries an origin: who made it, from what, and when. if an agent wrote it, you can trace it to the run that made it.
the data model
three node kinds.
file one uploaded file
section one chunk of a file
note one agent written fact
here’s everything an item carries.
id numeric in helix, branded string in typescript
tenantId the workspace, every query filters on it
kind free text, storage maps it to file | section | note
label short title
summary short human readable sum
r2Key path to raw bytes, this is the citation
canonical a type + value pair for finding the same thing again
indexed false on add, true once indexed
origin actor + source + timestamp
body full text, used for keyword match and embeddings
scope org | team | user, defaults to org
embedding 384 numbers
three edge types.
HAS_SECTION file -> its sections
MENTIONS item -> itself, with the text and where it came from
SAME_AS a soft link. never merges. just says "related"
and the indexes on helix.
vector on embedding for file, section, note cosine, 384 dims
keyword on body for file, section, note
equality on source_key so upserts are cheap lookups, not scans
why helixdb
i needed a graph, a vector store, and keyword search over one dataset. the normal answer is postgres plus pgvector plus neo4j plus elasticsearch. four systems, four backups, four ways to go out of sync.
helixdb gives me all three in one engine. it’s a native graph store with vectors and bm25 keyword search built in. the reason it matters for this design isn’t the storage, it’s that i can run a graph traversal, a vector search, and a keyword search in one round trip. and it’s open source, which matters a lot here. flow research does open work. the brain lives in the open garden repo, and its engine should be open too.
what other people do what i do
───────────────────── ────────
query pgvector ┐
query elasticsearch ├──────► one http call
query neo4j for links ┘ to helixdb
returns fused hits
i query it through helix’s ts dsl. no query strings, no helixql. i build with g(), vectorSearchNodes, textSearchNodes, readBatch, writeBatch, and it compiles to json bytes the server executes.
one readBatch
├─ vector search files
├─ vector search sections
├─ vector search notes
├─ keyword search files
├─ keyword search sections
└─ keyword search notes
local helix runs at localhost:6968. prod runs on a vps behind caddy, no public domain, no host port. the browser never talks to helix. only garden’s server does, through a thin auth proxy. the workspace auth stays in garden, caddy just checks and strips the shared secret. every request carries a 30s deadline and one retry on the transient planner error.
embeddings
384 dimensions. that means every piece of text becomes a list of 384 numbers that captures what it means. two texts about the same thing get number lists that point in a similar direction. search compares directions with cosine similarity. 384 is the small end. big enough to tell topics apart, small enough to store and compare cheaply at our scale.
prod @cf/baai/bge-small-en-v1.5 (workers ai)
local all-MiniLM-L6-v2 (xenova, offline)
both output 384 numbers, so a local test and a prod write live in the same space. index and search both cut text at 2000 characters before embedding. the query gets embedded once per search.
add then index
this is the part i care most about. adding and knowing are separate, and the whole flow runs on that.
ADD
───────────────────────────────────────────────────────────
upload file
-> check format (md pdf docx xlsx txt) and size (100 mb cap)
-> r2.put at brain/workspaces/<ws>/<item>/<name>
-> brain.addItem kind=file, indexed=false, origin=human
-> shows in the grid, findable by name, NOT answerable
INDEX
───────────────────────────────────────────────────────────
read bytes from r2
-> extract text (unpdf / mammoth / exceljs / decode)
-> chunk into sections (cap 1000)
-> embed in batches of 100
-> upsert file + sections into helix
-> link HAS_SECTION edges, idempotent
-> drop stale sections, mark indexed=true
once indexed it’s answerable. before that it’s just a file you own. that distinction is visible in the ui and it’s honest about what the system actually knows.
extraction and chunking
five formats.
markdown / txt plain decode, markdown keeps frontmatter title
pdf unpdf, merged pages, needs a fresh Uint8Array wrap
docx mammoth, flat 1000 char fallback
xlsx exceljs, one section per sheet, skips hidden sheets
no pptx, no images, no video. the system handles what it handles and says so.
chunking has two paths.
structured (md, xlsx)
split on # headings
keep the breadcrumb "helixdb basics > query model"
keep text before the first heading as preamble
flat (txt, pdf, docx)
pack paragraphs to 1000 chars
overlap 100 chars
every section gets its own source_key and its own embedding, so a reindex updates sections in place instead of duplicating them and leaving orphans behind.
search
one search, six streams, then a merge. each of the six pulls 50 rows.
i use reciprocal rank fusion. plain version. each list votes by rank.
rank 1 in a list -> 1 / (60 + 1) = 1/61
rank 10 in a list -> 1 / (60 + 10) = 1/70
add the votes from all six lists, sort high to low. no score scaling. the point is that a hit which is both semantically close and keyword exact beats a hit that’s only one of those. fusion without trusting any single engine’s score.
then freshness.
half life 90 days
floor 0.25
source occurredAt, falls back to origin timestamp
old facts sink but never disappear. a two year old policy still shows up, just below the fresh one.
the tool version takes k with default 5 and max 8, and bodies cut at 4000 chars. and results carry the citation. every hit has an r2Key, so the answer points straight at the raw file in r2, and readFile streams the whole file or just the byte range a section came from. the model never holds a 40 page doc in context to answer one question about it.
read injection
once per turn, before the model runs, garden takes your latest message, searches the brain once, and merges the result with anchors. anchors are the linked issue, the project, and the open doc. up to 4 anchors plus the search hits, hard cap of 8 items after merge, around 1200 to 1500 tokens total. this runs on all three surfaces. chat, issue runs, and automation runs all recall what the workspace already knows instead of re-deriving everything.
your message
│
▼
single search + anchors
│
├─ best hit < 0.02? -> inject nothing
├─ hit < 60% of top score? -> cut it
└─ else -> merge, cap 8, ~1200-1500 tokens
the floors are what keep small talk clean. this is why typing “yo” pulls nothing. the top hit scores below the absolute floor, so the brain stays quiet instead of hallucinating a reason to talk.
who sees what is scoped. chat searches the thread owner’s scope. issue and automation runs search the agent owner’s scope. every turn logs whether it found anything, plus the query and the scores, so i can see exactly why the brain did or didn’t speak.
if search fails, the turn just runs without memories. nothing breaks. a memory system that can take down a chat turn is worse than no memory system.
write paths
five tools agents can call.
brain_search read only, max 8 hits
add_to_brain write a fact straight through, no gate
brain_observe_mention attach a name (person, company, project) with a span
brain_link create a custom edge, SAME_AS is soft only
brain_neighborhood walk 1-2 hops, capped at 50 items and
100 edges per hop
these are wired into chat, issue runs, and automation runs.
add_to_brain writes straight through with no review. that’s deliberate. when an agent learns something mid run, stopping to ask permission is the wrong default. the gate lives on the other path, the post run writer.
write-back
when an issue run ends succeeded or blocked, or an automation ends completed, garden starts one write-back pass. one per run, max, fire and forget. uploads put knowledge in. write-back is what makes the brain a living thing, because work itself files knowledge back.
the input is the run’s work products, capped at 8000 chars, not the whole transcript. a small writer agent gets that summary and two tools, search and propose, and asks itself four questions.
1. useful in a month?
2. about the org, not just this task?
3. useful to a new teammate?
4. is it a fact, decision, rule, owner, definition, or gotcha?
saving nothing is a valid outcome. it can propose zero items. max three.
then a gate.
empty or nonsense -> skip
sensitive, or confidence < 0.75 -> human review (inbox card)
user scope personal -> always human review
user scope with no user attached -> skip, never widen to org
confident org knowledge -> write straight in
approving a proposal writes it at the stored scope. only the scoped user can approve their own. this is the line between “the agent can remember things” and “the agent can’t quietly rewrite what the company believes.”
the audit subagent
after a file is indexed mechanically, a second pass runs. one think turn, a 30 step budget, five tools. it adds mentions, adds links, and improves the graph around the new file. it never merges anything.
that split matters. the mechanical pass is deterministic and cheap. the audit pass is where a model gets to make judgment calls about what a document is really about.
scopes
i nearly shipped without this and i’m glad i didn’t.
org everyone in the workspace sees it (default)
team:<id> that team plus org
user:<id> that user only
every read checks scope in code. an edge only shows if both ends are visible. derived items take the narrowest scope of their sources, and if two sources can’t be compared, access fails closed.
missing scope means org. that’s the safe default in both directions, it matches how people actually treat org wide knowledge, and it fails closed rather than open when something’s wrong. personal facts stay personal. everything else belongs to the org. retrieval, proposals, and outcomes all log to postgres plus posthog, so the whole loop is measurable.
what it doesn’t do
honest boundaries, stated plainly.
- you can upload. you can’t crawl. no bulk drive, gmail, notion, or github ingest.
- no item grid, no entity dossier, no graph view in the ui. backend first, ui next.
- no live connector sync. on demand capture, not a mirror.
- no always on background extraction.
- no temporal validity beyond the 90 day freshness decay.
- no pptx, no image ocr, no video.
- docx chunking is a flat fallback.
tradeoffs
every design above buys something and pays for it. here’s the bill.
additive only. the graph never edits in place, so nothing is ever lost to a bad merge. the price is growth. paraphrases and conflicts sit side by side until something resolves them. that something is a cleanup digest. until then, dedupe is policy, not engine behavior.
free text kinds. agents can invent kinds without asking, which keeps momentum. the price is mess. without a cleanup pass the graph accumulates near duplicate concepts. i accept the mess early because the alternative, a fixed ontology nobody maintains, is worse.
one self hosted engine. helix replaces four systems and one query replaces four round trips. the price is ops. i run the binary, the backups, the auth proxy. for flow research that’s the right trade. open work on open infra.
fire and forget write-back. runs never wait on memory. the price is lag. a fact learned in a run lands after the run, not during it. mid run memory still comes from injection and direct tool calls.
floors over cleverness. the 0.02 absolute and 0.6 relative floors are dumb and they work. the price is missed recall at the margins. something genuinely relevant at 55 percent of the top score gets cut. i’d rather miss a memory than bloat every turn.
small embeddings. 384 dims are cheap to store and fast to compare. the price is nuance. fine distinctions between close documents blur. at our corpus size that’s fine. if it stops being fine, dims go up.
direct writes with no gate. add_to_brain mid run never asks. the price is review burden shifting to the post run gate and the inbox. capture first, judge after.
the living test is one question. ask the brain what the org currently believes about x, from this week’s work. if it answers with a citation, it’s alive. if it can’t, something in that chain above broke, and i’d rather find that out from a test than from a customer.