Edit the prompt. Ship. Hope.
- The playbook is a string in code, with no real versions.
- Feedback floats free. A thumbs-down cannot point at the wording.
- Or the agent edits itself, and the rules drift.
worked example · skill maintenance
A bad refund rule becomes a reviewable fix.
A fictional support assistant attaches a discount code to every refund reply—even when the company made the mistake. Three example outcomes expose the problem. The system drafts a narrower rule, but only a person can decide whether it becomes the active playbook.
Simulation · example walkthrough. The support cases, outcomes, and proposed playbook revision are fictional examples used to walk through the maintenance loop. They are not customer incidents or a live policy change.
one ID to remember · nothing promoted automatically
maintained_skill · candidate
awaiting review
pitfalls · drafted from 3 cited signals
− attach a discount code to every refund reply
+ when the charge was our own error, apologise and refund in full — no discount code
= steps, examples unchanged
your call
promote →
the assistant keeps running the old skill until you do
The model wrote the draft. It cannot ship it.
00 · the problem
Your assistant is wrong. Who edits its instructions?
Today: a person edits a prompt in the codebase, ships it, and waits to see whether the complaints stop. The instructions live in source control, the complaints live in a support tool, and nothing connects a bad reply to the exact wording that produced it.
Treat the playbook as data, and gate the change.
- The playbook is versioned records the agent reads as it answers.
- Each outcome is filed against that exact version.
- The model drafts; only a person ships.
-
the rule in force
The assistant follows a playbook. One line of it is too broad.
“Attach a discount code to every refund reply.”
-
the complaints arrive
Each one is filed against the exact version of the playbook that caused it.
“Being upsold immediately after our billing error felt tone-deaf.”
-
a narrower rule is drafted
The model writes the replacement. It cannot put it into service.
“When the charge was our own error, apologise and refund in full — no discount code.”
Until someone approves it, the assistant keeps running the old line. -
what you write
Report the outcome. Approve, or don’t.
await memseek.feedback.thumbs_down(use_id=use_id, note="Discount on a refund that was our own error.") await memseek.promote(...) # only a person runs this
1 rule narrowed, the rest untouched“Discount every refund” becomes “not when the error was ours”. Steps and examples carry forward.
0 automatic changesThe assistant keeps running the old playbook until someone approves the new one.
Every complaint points at a versionYou can show which wording caused which reply, months later.
New to Memseek? You write one small configuration file describing what your application should remember. After that your application only appends what happened; Memseek does the deriving, keeps every conclusion linked to the evidence underneath it, and hands your agent a bounded briefing instead of a pile of text. Everything below is that file and what it produces — how it works.
- skill, playbook
- The operating instructions an assistant follows: steps, pitfalls, examples.
- use ID
- The one short ID your app stores beside its reply. It pins exactly which version was in force.
- candidate
- A proposed new playbook: stored, cited, and not in use.
- promotion
- The single act that makes a candidate live. A person does it.
the rule in force
artifacts/skill.yamlthe thing being learned
The playbook is data with a version—not a string buried in application code.
Three keyed records make up the current skill: steps, pitfalls, and examples. Rendering the skill resolves their current versions and records exactly which heads the agent used.
active rule
pitfalls@v1 says “Attach a discount code to every refund reply.”name: maintained_skill
kind: skill
lifecycle: reviewed
blocks:
sections:
document:
collections: [skills]
template: |
{{sections}}
candidate_processor: skill
complete_keys: [steps, pitfalls, examples]lifecycle: reviewedGenerated replacements stop as candidates. The renderer keeps serving the currently promoted records.
documentResolve the current keyed skill records at render time, so every use can identify the exact versions it carried.
complete_keysA candidate must contain the whole skill contract. Reviewers never approve a mysterious partial patch.
writes now
An artifact-use receipt pins the rendered skill — pitfalls@v1, “Attach a discount code to every refund reply.” — along with its definition hash and learning target. No skill record changes.
the outcomes arrive
collections/learning.yamlthe feedback contract
Feedback names the exact rule use it is evidence about.
The application reports one task success plus a thumbs-down, a low evaluator score, and an operator correction. Each signal carries the artifact-use receipt, preventing feedback from being misattributed to whatever skill happens to be current later.
name: learning_signals
mode: event
schema:
required: [text, artifact_use, signal]
properties:
artifact_use:
required: [id, artifact, render_sha256]
signal:
required: [kind, source]
properties:
kind:
type: string
enum: [thumbs_down, correction,
task_failure, evaluation]
# plus success and exception kindsmode: eventKeep every selected outcome as immutable evidence. Do not edit history to make the new skill look better.
artifact_useBind the outcome to the precise render that influenced the reply, including its content hash.
signal.kind / sourceSeparate what happened from who judged it, so downstream policy can weigh an end user, evaluator, and application differently.
writes next
Four learning_signals records: one success, plus “Being upsold immediately after our billing error felt tone-deaf.” (thumbs down), an evaluation of 0.2, and an operator correction — “Apologize and fix company errors; reserve discounts for goodwill and retention.” The over-broad rule remains active while evidence accumulates.
the proposal—not promotion
derivations/skill.yamlthe guarded write
The model must return a complete, cited candidate.
The derivation compares new evidence with the current skill, first proposes the smallest supported patch, then materializes all three sections. The emission contract requires citations and marks the output for review.
active pitfalls rule
Always include a discount code for the customer’s next order before closing.
thumbs down end user
Being upsold immediately after our billing error felt tone-deaf.
evaluation score 0.2
The always-discount rule fires when the company caused the charge.
correction operator
Apologize and fix company errors; reserve discounts for goodwill and retention.
route evidence
skill derivation
skill derivation
candidate · steps carried forward
The working refund procedure remains intact.
candidate · pitfalls changed
Do not attach a discount when the company caused the error; use one only for a supported goodwill or retention case.
candidate · examples carried forward
The existing worked examples remain part of the complete draft.
active skill · unchanged
The candidate has an effect only after a person promotes it.
Visible effect: the problematic rule changes in the draft; the rest of the playbook is preserved; production behavior does not move yet.
sources:
new_evidence:
kind: changes
collections: [main, outcomes]
current_skill:
kind: current
collections: [skills]
tasks:
- id: proposal
- id: result
emit:
collection: skills
keys: [steps, pitfalls, examples]
complete: true
review: requiredproposal → resultSeparate diagnosis from writing: identify the smallest evidence-supported change, then construct the complete candidate snapshot.
complete: trueRequire all three keys in one coherent draft, even when only pitfalls changes.
review: requiredStore the output as a candidate. Only an explicit promotion moves the active heads.
final state
Continue into the complete feedback and promotion trail →
One cited candidate replaces “discount every refund” with “When the charge was our own error, apologise and refund in full — no discount code.” steps and examples are carried forward unchanged. The old skill still serves until approval.
01 · the skill
A playbook is three keyed records.
Not a prompt string in your codebase — records in a keyed collection, so the playbook has versions, provenance, and a current head the agent reads at render time.
skills · keyed · active nowone current version per key
stepsRead the charge history · confirm the error class · draft the reply · attach the creditv1
pitfallsAttach a discount code to every refund replyv1
examplesTwo worked refunds, one partial and one fullv1
why keyed, not a prompt file4 consequences
supersederecords→ rewrite ≠ overwrite
A new version of pitfalls supersedes the old one. The old one is still addressable, so "what was the playbook in March?" has an answer.
renderartifact→ maintained_skill@1
The agent never reads the collection directly. It renders an artifact, which pins the exact heads it used.
improvederivation→ skill
A derivation can propose a new version of any of the three keys — but only as a draft.
auditruns→ every change has a run
Each version names the run that wrote it and the evidence that run stood on.
02 · bind a use
One short ID is all your app remembers.
Rendering the prompt gives you text. Binding the render gives you a handle: the artifact identity, its content hash, and the exact promoted skill heads that were in force when the assistant ran. Store that ID beside your own result and you are done.
- 01the artifact declares its learning targetFeedback about this prompt is about the skill it carried — not the calendar, not the retrieved memory. The artifact says so, once.
- 02the bind resolves itYou get back a use ID plus the promoted skill version that produced this exact render.what comes backuse_9f2c41ab · maintained_skill@1 · heads in force: steps@v1, pitfalls@v1, examples@v1 — the version that said “attach a discount code to every refund reply”
- 03deliberately not observabilityA use holds no prompt, no response, no tool calls, no token counts, no spans. It expires on a retention setting.
- 04correlation stays optionalOpenTelemetry integration is a bounded map of scalar attributes and an optional dependency — the SDK works without it.
artifacts/agent_prompt.yamllearning target
artifacts: - name: daily_agent_prompt kind: prompt lifecycle: live blocks: skill: {document: {collections: [skills]}} calendar: {view: upcoming_calendar@1} memory: {view: agent_relevant_memory@1} # Feedback about a render of this prompt is about the # maintained skill it carried, not about the calendar or the # retrieved memory. Binding a use resolves this to the exact # promoted skill heads that were in force. learning: target_block: skill artifact: maintained_skill@1
03 · report what happened
Outcomes become ordinary records.
A thumbs-down, a correction, an evaluation score, a task result — each one lands as a learning_signals record naming the precise skill version that produced the bad reply. No special store, no privileged pipeline.
learning_signals · from one use IDcited to a version
Discount code attached to a refund that was our own billing error. Customer read it as an insult.
"We've refunded the full charge and fixed the cause. No code needed."
Tone mismatch on company-error refunds, third occurrence this month.
the one step memseek leaves to yourouting
signals landautomatic→ learning_signals
Submitting an outcome against a use ID always writes a record. That part needs no decision.
what counts as evidenceyours→ your routing rule
Whether a single thumbs-down is worth revising a playbook — or whether it takes three, or a score below a threshold — is a product judgement, so the catalog does not guess it. Your application routes the signals it considers evidence into the collection the skill pipeline watches.
nothing implicitby design→ no auto-promotion
A signal never changes the skill on its own, and never promotes anything.
04 · the draft, and the gate
The model proposes. You dispose.
The shipped skill derivation fires on new evidence, under a cooldown, and emits all three sections at once as a candidate. Because the reviewed artifact names the complete key set, a partial draft cannot be promoted over a working playbook.
what the gate guarantees3 things
complete_keyscontract→ [steps, pitfalls, examples]
A candidate must cover all three, so promotion can never leave the playbook half-updated.
stale guardcompare-and-set→ heads re-checked at promote
If the live skill moved while the draft sat in review, activation is rejected rather than clobbering it.
declining is freeno-op→ drafts stay in the audit trail
Decline and the skill simply never changed — with a permanent record of what was proposed and refused.
derivations/skill.yamlshipped
name: skill trigger: write: collections: [main, outcomes] types: [skill_brief, outcome, observation, exception, feedback] cooldown_s: 600 # don't redraft on every signal sources: new_evidence: {kind: changes, max_records: 100} current_skill: {kind: current, keys: [steps, pitfalls, examples]} model: strong limits: max_tasks: 2 max_llm_calls: 4 max_total_tokens: 50000 # artifacts/skill.yaml — the reviewed contract artifacts: - name: maintained_skill kind: skill lifecycle: reviewed # not live — promotion required candidate_processor: skill complete_keys: [steps, pitfalls, examples]
05 · from your app
Bind, report, review, promote.
Four calls across the whole loop, and only the last one is a decision.
app.pypublic SDK
# 1 · render the prompt your assistant will run on, and bind it prompt = memseek.artifact("daily_agent_prompt") use = await prompt.bind(entity="agent:support") use["id"] # store this one ID beside your own reply use["learning_target"] # the exact promoted skill heads in force # 2 · later, report what the reply actually caused await memseek.feedback.thumbs_down( use_id=use["id"], note="Discount code on a refund that was our own billing error.", ) await memseek.feedback.correction( use_id=use["id"], expected="We've refunded the full charge and fixed the cause.", ) # 3 · draft a revision from the evidence, then read the candidate queued = await memseek.run_processor("skill", entity="agent:support") job = await memseek.job(queued["job_id"]) # poll to a run id review = await memseek.run(job["successful_run_id"]) candidate = review["run"]["content"]["candidate_set"] candidate["covered_keys"] # ["steps", "pitfalls", "examples"] candidate["divergence"] # what actually moved, per key # 4 · the decision. this is the only line that changes the skill. if approved_by_a_human(candidate["divergence"]): await memseek.promote( entity="agent:support", source_run_id=job["successful_run_id"], artifact="maintained_skill", )
The artifact's learning: declaration, the skill derivation's trigger and limits, and the maintained_skill reviewed contract are the checked-in catalog; the loop is examples/skill_maintenance.py. The drafted wording is representative of a real provider run, not a captured transcript.
start building
Run the loop, then refuse the draft.
The script installs a skill, binds a real render, reports outcomes against it, drafts a cited revision — and then stops and asks you. Declining is the interesting path: the skill never changes, and the proposal stays on the record.
# postgres, the api, and a worker
$ make database && source .env.sh
$ uv run memseek migrate
$ uv run uvicorn memseek.api:app &
$ uv run memseek worker &
# then the loop
$ uv run python examples/skill_maintenance.py
promote this candidate? [y/N]