Cache Pipeline (Notion + Search + Build Artifacts)

This document explains how cache and generated artifacts move through the build pipeline in this repository.

Why This Exists

The project separates content generation (local, before commit) from site build (Netlify, on push):

  • reduce Notion API pressure during static generation
  • avoid rate-limit bursts (429) in production builds
  • keep search/navigation data static and fast at runtime
  • generate redirect maps once per deploy, then serve them at edge/CDN speed

Static export exists because rendering Notion on Netlify Functions was too costly. Visitors never hit Notion; Netlify only reads committed artifacts.

Official vs unofficial Notion API

  Official Unofficial (page bodies)
Host api.notion.com/v1 app.notion.com/api/v3 (via notion-client)
Auth NOTION_API_KEY (Connection / integration) NOTION_TOKEN (token_v2 cookie)
Used by build:search, build:metadata, optional component Notion patches build:content, local next dev page fetch
Writes search-index.json, components metadata public/.notion-cache/*.json (recordMaps for react-notion-x)

Netlify prebuild / next build call neither API.

Shared unofficial client options live in notion-private-api.mjs: non-empty User-Agent + app.notion.com base URL. Required since Aug 2026 when Cloudflare began blocking Node requests without a UA (react-notion-x#710).

Two Pipelines

1) Content publish (local only)

Run when Notion content changes:

npm run publish:content

Order (from scripts/publish-content.mjs):

  1. npm run build:search — official
  2. npm run build:metadata — official + Gemini (incremental, no --force)
  3. npm run build:search — re-index after metadata sync
  4. npm run build:content -- --refresh-shell — unofficial page cache + homepage gallery merge
  5. npm run build:artifacts — official patches optional + Gemini
  6. npm run validate:artifacts — report
  7. npm run components:seeds:upload — ship new seeds to Netlify Blobs (seed_exists skipped)

Commit local outputs, then push. Runtime Export/Preview reads Blobs (git TSX is stripped from out/). See Publishing Notion Content and Components Seeds Pipeline.

2) Site prebuild (every npm run build, including Netlify)

From package.json prebuild hook — no Notion fetches:

  1. npm run build:redirects
  2. npm run build:edge-data
  3. npm run validate:redirects
  4. npm run build:mcp-discovery
  5. npm run build:agent-skills

Then next build runs.

Low-level generators (build:search, build:content, build:metadata, build:components) are not part of prebuild. Use them directly only for targeted debugging.

Artifact Map

1) public/search-index.json

Generated by: scripts/build-search-index.mjs

Purpose:

  • canonical static index of Notion pages
  • search payload consumed by the client
  • source of truth for menu grouping and route metadata
  • source input for redirects and edge redirect data

Read by:

  • src/lib/search.ts (client-side search fetch)
  • src/app/layout.tsx (header/menu data)
  • src/lib/notion-cover.ts (cover/tag/link lookup artifacts)
  • scripts/build-content-cache.mjs (list of pages to cache)
  • scripts/build-redirects.mjs and scripts/build-edge-data.mjs

2) public/.notion-cache/*.json

Generated by: scripts/build-content-cache.mjs

Purpose:

  • per-page recordMap cache for Notion pages
  • used during Next production build instead of live Notion calls

Read by:

  • src/lib/notion-server.ts when NEXT_PHASE === "phase-production-build"
  • all static page generators that call getBuildCachedNotionPage(...)

Behavior:

  • sequential downloads with retry/backoff
  • skip existing files by default
  • --refresh-shell to always refresh homepage + menuGroup container pages
  • merges patterns gallery collection_query into homepage cache after build
  • --force to refresh all
  • optional --allow-failures mode

3) public/components/components-metadata.json + public/components/code/*.tsx

Generated by: scripts/build-artifacts-cache.mjs (via publish:content)

Purpose:

  • AI-enriched metadata layer per pattern
  • generated component TSX artifacts per pattern
  • generation status and manifest reports

Read by:

  • optional UI enrichment paths
  • debug/artifacts inspection pages

Notes:

  • Local TSX under public/components/code is committed for next dev / /debug and as upload input.
  • Production Export/Preview reads Netlify Blobs, not published out/components/code (stripped on build).
  • Normal path: npm run publish:content (generates + uploads new seeds), then commit public/components/.
  • Escape hatch: PATTERN_PUBLISH_SKIP_SEEDS_UPLOAD=1 skips Blobs upload.

4) public/_redirects

Generated by: scripts/build-redirects.mjs

Purpose:

  • Netlify redirect rules for high-value legacy URL patterns
  • mainly folder + Notion ID -> canonical slug redirects

Notes:

  • constrained to Netlify free-tier redirect limits
  • non-core redirect cases are handled by edge function logic

5) netlify/redirect-data.json

Generated by: scripts/build-edge-data.mjs

Purpose:

  • edge lookup maps for redirector function
  • slug map and normalized Notion ID map

Read by:

  • netlify/edge-functions/redirector.ts

6) .next/

Generated by: next build

Purpose:

  • intermediate Next.js build output used before export

7) out/

Status in this repo:

  • ignored in git
  • Netlify publish target in this repo (publish = "out")
  • generated from next build when static export mode is enabled

Runtime Usage Summary

At runtime, the app mostly uses prebuilt static files:

  • search/menu/covers from public/search-index.json
  • pre-cached Notion recordMaps during build phase from public/.notion-cache
  • metadata/components enrichment from public/components/* when local artifacts are present
  • redirect maps from public/_redirects and edge redirect data

Netlify Cache Persistence

Netlify cache persistence for public/.notion-cache is disabled in this repo.

  • no build plugin restores content cache before build
  • no build plugin saves content cache after build
  • deploy output is produced from repository-tracked artifacts and current build inputs only

Environment Variables (Most Relevant)

Search/build-index and Notion fetch behavior:

  • NOTION_API_KEY
  • NOTION_TOKEN (if needed for private content)
  • NOTION_FETCH_RETRY_ATTEMPTS
  • NOTION_FETCH_BASE_DELAY_MS
  • NOTION_FETCH_MAX_DELAY_MS
  • NOTION_FETCH_TIMEOUT_MS
  • NOTION_FAIL_FAST

Metadata/components generation specific (local-only flow):

  • GEMINI_API_KEY
  • PATTERN_METADATA_MODEL
  • PATTERN_METADATA_GEMINI_TIMEOUT_MS
  • PATTERN_METADATA_NOTION_TIMEOUT_MS
  • PATTERN_METADATA_RETRY_ATTEMPTS
  • PATTERN_METADATA_NOTION_DELAY_MS
  • PATTERN_METADATA_GEMINI_DELAY_MS

Content-cache specific:

  • NOTION_CONTENT_DELAY_MS
  • NOTION_CONTENT_RETRY_ATTEMPTS
  • NOTION_CONTENT_TIMEOUT_MS
  • NOTION_CONTENT_429_DELAY_MS

Local Commands

Publish everything (recommended):

npm run publish:content

Run only search index:

npm run build:search

Generate/sync local AI artifacts:

npm run build:artifacts

Validate local AI artifacts snapshot:

npm run validate:artifacts

Build/update Notion content cache:

npm run build:content

Refresh only shell pages (homepage + menuGroup containers):

npm run build:content -- --refresh-shell

Force-refresh all cached pages:

npm run build:content -- --force

Run deploy-time prebuild only:

npm run prebuild

Failure Modes and Fixes

  1. search-index missing
  • Symptom: menu/search empty, redirect build fails
  • Fix: run npm run publish:content or npm run build:search
  1. notion-cache misses during build
  • Symptom: [notion-cache] miss ... logs and potential 404 in prerender
  • Fix: run npm run publish:content or npm run build:content before next build
  1. metadata descriptions not visible on front pages
  • Symptom: pattern pages still show baseline descriptions from search-index
  • Fix: run npm run publish:content
  1. homepage grid missing new patterns
  • Symptom: pattern works at /patterns/slug but not on homepage
  • Fix: run npm run publish:content (includes --refresh-shell + gallery merge)
  1. too many Notion retries/timeouts
  • Symptom: slow or failing publish
  • Fix: increase timeout/retry env values, keep sequential content-cache mode
  1. redirect count too high
  • Symptom: warning/error in redirect build
  • Fix: keep only essential static redirects and rely on edge maps for scale

Quick Mental Model

  • publish:content — one local command for Notion + AI artifacts and new Blobs seed upload
  • build-search-index — discover and normalize content metadata
  • build-artifacts-cache — generate AI metadata and component artifacts
  • build-content-cache — snapshot full page content for build-time rendering
  • prebuild — deploy-time redirect/discovery plumbing only
  • next build — compile app using committed artifacts