Keep metadata outside the document

External metadata is a YAML manifest that supplies a fixed set of frontmatter keys for named documents. manni meta merges the values into each page’s metadata before resolving its schema, so every command sees one object. The page never holds the value. The contract on it stays with the page, in the same schema set, checked by the same run, with the same exit code.

Three things people use it for. Each has a worked path on this page, and all three share one manifest format and one set of rules.

1. Lean documents. A page carries what it is about, and nothing else. The design note it was written from, the ticket that tracks it, its owner and its review state live in the manifest. A contributor fills in fewer fields. A reader, or an agent that ingests the page, sees none of the bookkeeping. The run still sees all of it. Start at A lean docset.
2. Private values for public documents. The manifest lives in a private repository. Public CI never sees it, and the private run validates the public pages against it. See Private values for public documents.
3. Across repositories. One repository’s pages take keys from a manifest in another, by URL, public or private. See Across repositories.

A manifest belongs to a collection, the named document set every manni tool reads. externalMetadata: sits inside the collection entry, beside the globs that select the pages the manifest describes. The two are declared together, so neither can drift from the other.

What this page covers

The journey. The contract, meaning every rule, message and exit code, lives in the configuration reference.

Coming from a 0.x config?

This was called a sidecar, and it was meta.sidecars in the config. In 1.0.0 it is externalMetadata on a collection, and the old key is refused with a message naming the new one. The two finding ids are external:owned and external:duplicate. If you have a baseline recorded before the upgrade, regenerate it. Migrating a 0.x config is the before-and-after.

A lean docset

The manifest sits beside the config, in the same repository as the pages. Nothing here is secret. The point is what the page does not have to carry.

The worked example is test/fixtures/external-metadata/, the corpus this feature’s tests run. Four pages, one manifest, one schema. The schema’s file name is the fixture’s, and plays no part:

manni.config.yaml
docs-meta.yaml         # the manifest
private.schema.json    # requires the keys the manifest supplies
docs/auth.md
docs/billing.md
docs/new.md
docs/ops.md

1. Write the manifest. One entry per page, keyed by the page’s path relative to the config file. Paths are exact. There are no globs. A page that moves often can be keyed by a frontmatter field instead; see Survive renames with a field join.

   docs-meta.yaml

   docs/auth.md:
     source: internal/auth-design.md
     jira: PLAT-412
   docs/billing.md:
     source: internal/billing.md
     jira: PLAT-388

2. Write the schema. It requires the keys the manifest supplies, beside the keys the page carries itself. Leave additionalProperties unset, or the page’s own keys fail it.

   private.schema.json

   {
     "type": "object",
     "required": ["title", "jira"],
     "properties": {
       "title": { "type": "string" },
       "source": { "type": "string" },
       "jira": { "type": "string", "pattern": "^PLAT-[0-9]+$" }
     }
   }

3. Declare the collection, and hang the manifest off it. paths: selects the pages. externalMetadata.keys: names what the manifest owns. A page carrying one of those keys itself is a finding, and an entry setting any other key is an error. The override points at the collection by name, so the glob is written once.

   manni.config.yaml

   collections:
     - name: pages
       paths: ["docs/**/*.md"]
       externalMetadata:
         - file: ./docs-meta.yaml
           keys: [source, jira]
   
   meta:
     overrides:
       - collection: pages
         schemas: [./private.schema.json]

4. Run it. A bare manni meta validate reads every collection, which is what makes the orphan check run.

   npx -y @hawkeyexl/manni meta validate

The page itself carries its title and nothing else:

docs/auth.md

---
title: Auth
---

# Auth

The run sees the whole object. From the fixture directory, manni meta get jira docs/auth.md prints docs/auth.md: jira=PLAT-412, and a corpus check can phrase a rule over jira like any other key. So can a query over the collection’s own view: SELECT _path FROM pages WHERE jira IS NULL.

What a failure looks like

The fixture’s manifest is not the clean one above. One page has a malformed ticket in the manifest, one has no entry, and one carries jira: in its own frontmatter:

✓ docs/auth.md
✗ docs/billing.md
    /jira  must match pattern "^PLAT-[0-9]+$"  (docs-meta.yaml:6)  [./private.schema.json]
✗ docs/new.md
    (root)  must have required property 'jira'  (line 1)  [./private.schema.json]
✗ docs/ops.md
    /jira  "jira" is owned by manifest docs-meta.yaml (collection pages); remove it from the document  (line 3)  [external:owned]

4 files checked, 1 passed, 3 failed, 3 errors

Each line points at the place to fix. billing.md’s bad value is at docs-meta.yaml:6, so the manifest is what to edit. new.md needs an entry. ops.md needs its jira: line removed, because the manifest is the only place that key may be set. The message names the collection whose manifest owns it. All three are exit 1, and --format github annotates the manifest line for the first and the page line for the other two.

A renamed page whose entry was not updated is different. On a run that reads the whole corpus it is exit 2. An entry naming a page the run did not load is a named input that is not there. The same fixture directory carries a second config whose manifest names a page that is gone:

$ manni meta validate -c manni.orphan.config.yaml
manni: Manifest docs-meta.orphan.yaml:3 names "docs/gone.md", which this run did not load. Fix the entry, or remove it.

Naming a path skips that check; naming a collection does not

Naming a path tells manni you chose to look at part of the corpus, so entries for the rest are expected rather than orphaned. manni meta validate docs/auth.md passes without mentioning docs/gone.md.

--collection is different. It chooses a corpus rather than reshaping one, because the run loads all of every collection it names. So manni meta validate --collection pages still reports the orphan, while entries of collections it did not name are left alone. A CI job that gates one collection keeps this check.

Either way the page’s manifests still merge, because membership is decided per file however the file reached the run.

Private values for public documents

The same manifest, moved. Your docs repository is public. Some of the metadata its pages must carry is not, and in the public frontmatter it leaks. In a private spreadsheet it is never validated, and the whole point of a jira: key is that CI refuses a page without one.

The manifest goes in a private repository, with the config and the schema. The public repository never holds the value, and the private CI run validates the public pages against a schema that requires it. By the end you will have two repositories, one manifest, and a private CI job. The job fails when a public page has no ticket or a ticket is misspelled. It also fails when someone writes the ticket into the public page by hand.

The layout

The private repository owns the config, the manifest, and the private schema. The public repository is a git submodule inside it:

private-repo/
  manni.config.yaml     # the collection, its manifest, and the override
  docs-meta.yaml        # the manifest
  schemas/private.json  # requires the private keys
  public/               # git submodule: the public docs repository
    manni.config.yaml   # plain; never mentions external metadata
    docs/guides/auth.md

Two sibling checkouts also run, and report worse. SARIF locates every finding relative to the nearest repository root and cannot represent a finding in another repository’s file. From a sibling layout every document finding is therefore dropped from the log. A path that climbs out of the config’s directory is also a member of no collection, so it would take no manifests at all. The submodule keeps every document under the private root. The reasons are in the reference.

Set it up

1. Add the public repository as a submodule. From the private repository:

   git submodule add https://github.com/your-org/public-docs public

2. Write the manifest. The entries are the ones from the lean docset, keyed through the submodule:

   docs-meta.yaml

   public/docs/guides/auth.md:
     source: internal/auth-design.md
     jira: PLAT-412
   public/docs/guides/billing.md:
     source: internal/billing.md
     jira: PLAT-388

3. Write the private schema. It is the lean docset’s schema, kept where the public repository cannot see it. Leave additionalProperties unset, or the public keys fail it.

4. Declare the collection in the private config.

   manni.config.yaml

   collections:
     - name: public-pages
       paths: ["public/docs/**/*.md"]
       externalMetadata:
         - file: ./docs-meta.yaml
           keys: [source, jira]
   
   meta:
     overrides:
       - collection: public-pages
         schemas: [./schemas/private.json]

   The public repository’s own manni.config.yaml stays as it is. It never mentions external metadata, so no public run warns about a private file it cannot see.

5. Run it from the private checkout.

   npx -y @hawkeyexl/manni meta validate

The two workflows

The public repository keeps its ordinary gate. Nothing about the manifest reaches it:

public-docs/.github/workflows/manni.yml

name: Validate metadata
on: [push, pull_request]
jobs:
  manni:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: hawkeyexl/manni@v2

The private repository checks out its submodule and runs from its own root. submodules: true is what brings the public pages in. If the public repository is public, the default token can clone it. If it is only public to your organization, pass a token that can read it:

private-repo/.github/workflows/manni.yml

name: Validate metadata
on: [push, pull_request]
jobs:
  manni:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true
          token: ${{ secrets.PUBLIC_DOCS_READ_TOKEN }}
      - uses: hawkeyexl/manni@v2

The Action passes no paths, so the targets come from the private config’s collections: and the orphan check runs. Bump the submodule pointer when the public docs change, or the private run validates the pages as they were.

Across repositories

file: accepts an https:// URL as well as a path. The manifest is fetched at the start of every run, never cached, and merged exactly as a local one is. Use it when the run cannot check the manifest out. The public form comes first because it needs nothing else.

A public catalog

Nothing here is private. One public repository keeps a catalog, say which team owns each page, and another public repository’s pages are judged by it. The docs run never clones the catalog. A plain URL, and no token anywhere:

docs-site/manni.config.yaml

collections:
  - name: pages
    paths: ["docs/**/*.md"]
    externalMetadata:
      - file: https://raw.githubusercontent.com/your-org/docs-catalog/main/owners.yaml
        keys: [team, owner]

meta:
  overrides:
    - collection: pages
      schemas: [./schemas/owned.json]

The ordinary public workflow runs it unchanged. There is no secret to pass, and the public log shows only public values. A finding on a fetched value names the URL and the manifest line, so the catalog is what to edit.

A private manifest

The manifest lives in a private repository the job does not check out, such as a platform team’s catalog that serves several docsets. This is the private values case with the manifest moved. The run still starts from a private checkout, with the public pages as its submodule. Add tokenEnv, the name of an environment variable holding a bearer token:

private-repo/manni.config.yaml

collections:
  - name: public-pages
    paths: ["public/docs/**/*.md"]
    externalMetadata:
      - file: https://raw.githubusercontent.com/your-org/docs-catalog/main/docs-meta.yaml
        keys: [source, jira]
        tokenEnv: PRIVATE_DOCS_TOKEN

meta:
  overrides:
    - collection: public-pages
      schemas: [./schemas/private.json]

The workflow passes the secret as that variable. The token never appears in the config, a message, or a report:

private-repo/.github/workflows/manni.yml

name: Validate metadata
on: [push, pull_request]
jobs:
  manni:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true
          token: ${{ secrets.PUBLIC_DOCS_READ_TOKEN }}
      - uses: hawkeyexl/manni@v2
        env:
          PRIVATE_DOCS_TOKEN: ${{ secrets.PRIVATE_DOCS_TOKEN }}

Both hosts accept a bearer token on their raw-file route, so the same tokenEnv serves either:

GitHub  https://raw.githubusercontent.com/OWNER/REPO/REF/PATH
GitLab  https://gitlab.com/api/v4/projects/ID/repository/files/PATH/raw?ref=REF

Host	Token
GitHub	A fine-grained personal access token with read access to the repository’s contents.
GitLab	A project or personal access token with the read_repository scope. PATH is URL-encoded, so docs/docs-meta.yaml is written docs%2Fdocs-meta.yaml.

Fetch a private manifest from the private side

A public repository’s CI logs are public. A private manifest fetched into a public run puts its values in that log. When the values are private, use the URL form from the private side, when the private run cannot check the manifest out. Never use it to feed a public run with private values. A public catalog has nothing to leak.

A run that cannot reach the manifest stops before any page is judged, and says why. The status is named and the token never is:

$ PRIVATE_DOCS_TOKEN= manni meta validate
manni: Manifest https://raw.githubusercontent.com/your-org/docs-catalog/main/docs-meta.yaml: the environment variable PRIVATE_DOCS_TOKEN named by "tokenEnv" is not set.
                                                                                    exit 2
$ manni meta validate --offline
manni: Manifest https://raw.githubusercontent.com/your-org/docs-catalog/main/docs-meta.yaml is remote and the run is offline. Vendor it to a path, or drop --offline.
                                                                                    exit 2

Every rule, including the timeout, the body cap, the redirect behaviour and http://, is in the reference.

Survive renames with a field join

A path entry is orphaned the moment the page moves, and the next whole-corpus run is exit 2 until someone edits the manifest. That is the right default for a docset that rarely moves. It is the wrong shape for one that moves pages often. It is also wrong for a manifest kept by a team that does not watch the docs repository’s renames. The path is the one thing about a page the manifest author does not control.

join: keys the manifest by a top-level frontmatter field instead. The page carries the key with it, so a rename never orphans an entry. Any field works. A hand-written id, a Starlight slug, and a Docusaurus id are the same mechanism.

1. Name the field on the manifest entry. Everything else in the config stays.

   manni.config.yaml

   collections:
     - name: pages
       paths: ["docs/**/*.md"]
       externalMetadata:
         - file: ./docs-meta.yaml
           keys: [source, jira]
           join: id
   
   meta:
     overrides:
       - collection: pages
         schemas: [./private.schema.json]

2. Key the manifest by that field’s values. The value is compared as a string, wherever the page lives.

   docs-meta.yaml

   auth-guide:
     source: internal/auth-design.md
     jira: PLAT-412
   billing-guide:
     source: internal/billing.md
     jira: PLAT-388

3. Require the field in the schema, and give it a shape. A page without the field matches no entry, and nothing says so. What catches it is the required finding for the manifest’s key, which only fires when the schema demands both. A pattern keeps an id well-formed.

   private.schema.json

   {
     "type": "object",
     "required": ["title", "id", "jira"],
     "properties": {
       "title": { "type": "string" },
       "id": { "type": "string", "pattern": "^[a-z][a-z0-9-]*$" },
       "source": { "type": "string" },
       "jira": { "type": "string", "pattern": "^PLAT-[0-9]+$" }
     }
   }

Two pages sharing one id is a finding on both, at the field’s line. One entry matched both, and the manifest cannot tell them apart. Both pages still receive the entry’s values, so the schema judges what the site would publish. From test/fixtures/external-metadata-join/:

✗ docs/dup-a.md
    /id  2 documents carry id "shared"; docs-meta.yaml cannot tell them apart (docs/dup-b.md)  (line 3)  [external:duplicate]
✗ docs/dup-b.md
    /id  2 documents carry id "shared"; docs-meta.yaml cannot tell them apart (docs/dup-a.md)  (line 3)  [external:duplicate]

An entry whose id no page carries is still exit 2 on a whole-corpus run, checked once every page has been read:

$ manni meta validate -c manni.orphan.config.yaml
manni: Manifest docs-meta.orphan.yaml:3 names id "gone-guide", which no loaded document carries. Fix the entry, or remove it.

The join key is contributor-controlled, and that is the trade. A pull request that sets id: auth-guide on a new page inherits every assertion the manifest makes about the real one. Three things bound that. The duplicate finding turns the run red rather than green. The schema’s pattern keeps an id well-formed. And the manifest is reviewed on its own, in the private repository when there is one. An entry that suddenly matches two pages shows up there as a duplicate first. A path join has none of this exposure, which is why it stays the default.

query follows the same line. Renaming a field-joined page is allowed, which is the point. Changing its id is a write to the join, and is refused:

$ manni meta query "UPDATE docs SET id = 'other' WHERE _path = 'docs/auth.md'"
manni: "docs/auth.md": "id" is the field manifest docs-meta.yaml joins on, and this document has an entry; change the manifest first.

When a page is in two collections

Two collections may each have a manifest supplying owner. That is ordinary: a guides collection and a blog collection are separate sets with separate catalogs. It only becomes a question when one page is a member of both, which happens when their globs overlap:

manni.config.yaml

collections:
  - name: guides
    paths: ["docs/**/*.md"]          # includes docs/api/
    externalMetadata:
      - file: ./guides-meta.yaml
        keys: [owner]
  - name: api
    paths: ["docs/api/**/*.md"]      # a subset of the above
    externalMetadata:
      - file: ./api-meta.yaml
        keys: [owner]

docs/api/auth.md is in both, and both own owner. There is no honest winner, so the run stops rather than picking one:

manni: docs/api/auth.md: "owner" is owned by manifests in two of its collections, guides (guides-meta.yaml) and api (api-meta.yaml); a key has one manifest per file. Narrow one collection's paths or exclude.

Two fixes, both in the config where the overlap was declared. Narrow guides so it stops covering docs/api/. exclude: ["docs/api/**"] does it, since a collection’s exclude decides membership. Or move owner out of one of the two manifests, so only one owns the key for that page.

Overlapping collections are otherwise fine, and useful. A page in both appears in both views, and still resolves exactly one schema set by first-match-wins.

Editing the values

A key the manifest owns is readable by every command, and every writer writes it into the manifest. The page is left alone. Again over the fixture corpus:

$ manni meta get jira docs/auth.md
docs/auth.md: jira=PLAT-412

$ manni meta query "UPDATE docs SET jira = 'PLAT-1' WHERE _path = 'docs/auth.md'" --dry-run
docs/auth.md: jira: PLAT-412 -> PLAT-1  [docs-meta.yaml]
1 change across 1 file — dry run; run again without --dry-run to apply

The bracket names the manifest the change lands in. fill writes a proposal for an owned key into the page’s entry the same way, and derive stamps a managed field there, owner or last-reviewed included. Each splice keeps the manifest’s comments. A corpus check still sees every key the manifest supplies. So “every guide has a ticket in the open state” is a SQL rule like any other.

Three writes are still refused. query refuses each before any file is written (exit 2):

• A URL manifest. It is fetched, so nothing can be written to it: "owner" is owned by manifest https://example.com/owners.yaml, which is fetched and cannot be written; set it in that repository.
• A join field on a page that has an entry. Changing it would point the page at another entry, as the field join section shows.
• ALTER TABLE … RENAME COLUMN of an owned key, because the manifest’s keys: would need renaming too.

Moving keys in or out

Adding a key to keys: by hand is only half the job. Every page that still carries the key then fails external:owned until its value is copied across. manni meta relocate does both halves: it edits keys: and moves every value, in either direction. A schema that marks a property x-manni-location tells it which way each key goes. It also creates what a move needs, and the config edits keep comments and key order:

Your config	What relocate writes
The page’s collection has a local manifest	The key is appended to that manifest’s keys:.
The collection has no manifest	<collection>.metadata.yaml beside the config file, and an externalMetadata: entry that owns the keys.
No collections: at all	A collection named default whose paths: are the targets you typed, and default.metadata.yaml.
One collection, and the page is outside it	The target is appended to the collection’s paths:.
Several collections, and the page is in none	Nothing. The value stays, and the run exits 1.
A key moves back into the pages and empties keys:	The externalMetadata: entry is removed and the manifest file is left on disk, with a line saying so.

Keep maintainer metadata out of delivered pages walks through each case.

Where to go next

External metadata in the configuration referenceEvery rule, message and exit code, and the additionalProperties caveat.

Collections in the configuration referenceThe document set every tool reads: paths, exclude, url, membership, and --collection.

Gate on rules that span filesPhrase a rule over the manifest's keys as SQL, and wire it in beside the gate.