Cache Pipeline (Notion + Search + Build Artifacts)

This document explains how cache and generated artifacts move through the build pipeline in this repository.

Why This Exists

The project prebuilds Notion and route data before Next.js compilation to:

  • reduce Notion API pressure during static generation
  • avoid rate-limit bursts (429) in production builds
  • keep search/navigation data static and fast at runtime
  • generate redirect maps once, then serve them at edge/CDN speed

Pipeline Order

The prebuild step runs in this order (from package.json):

  1. npm run build:search
  2. npm run build:metadata (optional/manual today, enabled when metadata sync is needed)
  3. npm run build:content
  4. npm run build:redirects
  5. npm run build:edge-data
  6. npm run validate:redirects

Then next build runs.

Artifact Map

1) public/search-index.json

Generated by: scripts/build-search-index.mjs

Purpose:

  • canonical static index of Notion pages
  • search payload consumed by the client
  • source of truth for menu grouping and route metadata
  • source input for redirects and edge redirect data

Read by:

  • src/lib/search.ts (client-side search fetch)
  • src/app/layout.tsx (header/menu data)
  • src/lib/notion-cover.ts (cover/tag/link lookup artifacts)
  • scripts/build-content-cache.mjs (list of pages to cache)
  • scripts/build-redirects.mjs and scripts/build-edge-data.mjs

2) public/.notion-cache/*.json

Generated by: scripts/build-content-cache.mjs

Purpose:

  • per-page recordMap cache for Notion pages
  • used during Next production build instead of live Notion calls

Read by:

  • src/lib/notion-server.ts when NEXT_PHASE === "phase-production-build"
  • all static page generators that call getBuildCachedNotionPage(...)

Behavior:

  • sequential downloads with retry/backoff
  • skip existing files by default
  • --force to refresh all
  • optional --allow-failures mode

3) public/metadata-cache.json

Generated by: scripts/build-metadata-cache.mjs

Purpose:

  • AI-enriched metadata layer per pattern (description + UX/UI signals)
  • safe Notion sync helper for Description and Version
  • stores VotesUp / VotesDown ingestion for quality loops
  • keeps generation stats (processed, skippedByDelta, geminiFailures, etc.)

Read by:

  • metadata/debug workflows and local quality checks
  • future snippet and recommendation layers

Input dependencies:

  • public/search-index.json (base pattern inventory)
  • Notion database rows (properties and sync target)

Notes:

  • This artifact is additive and does not replace search-index.json or .notion-cache.
  • Build flow to propagate descriptions to front surfaces remains:
    1. npm run build:metadata
    2. npm run build:search
    3. npm run build:content

4) public/_redirects

Generated by: scripts/build-redirects.mjs

Purpose:

  • Netlify redirect rules for high-value legacy URL patterns
  • mainly folder + Notion ID -> canonical slug redirects

Notes:

  • constrained to Netlify free-tier redirect limits
  • non-core redirect cases are handled by edge function logic

5) netlify/redirect-data.json

Generated by: scripts/build-edge-data.mjs

Purpose:

  • edge lookup maps for redirector function
  • slug map and normalized Notion ID map

Read by:

  • netlify/edge-functions/redirector.ts

6) .next/

Generated by: next build

Purpose:

  • deployable Next.js build output
  • Netlify publish target in this repo (publish = ".next")

7) out/

Status in this repo:

  • ignored in git, but not current deploy output
  • kept as a standard Next export artifact path only

Runtime Usage Summary

At runtime, the app mostly uses prebuilt static files:

  • search/menu/covers from public/search-index.json
  • pre-cached Notion recordMaps during build phase from public/.notion-cache
  • metadata enrichment from public/metadata-cache.json when metadata tooling is enabled
  • redirect maps from public/_redirects and edge redirect data

This shifts expensive/variable work to prebuild and keeps user-facing requests fast and deterministic.

Netlify Cache Persistence

Plugin: netlify/plugins/notion-cache/index.js

  • restores public/.notion-cache before build
  • saves it after successful build
  • also saves partial cache on build error

This makes subsequent deploys incremental and avoids re-fetching unchanged pages.

Environment Variables (Most Relevant)

Search/build-index and Notion fetch behavior:

  • NOTION_API_KEY
  • NOTION_TOKEN (if needed for private content)
  • NOTION_FETCH_RETRY_ATTEMPTS
  • NOTION_FETCH_BASE_DELAY_MS
  • NOTION_FETCH_MAX_DELAY_MS
  • NOTION_FETCH_TIMEOUT_MS
  • NOTION_FAIL_FAST

Metadata generation specific:

  • GEMINI_API_KEY
  • PATTERN_METADATA_MODEL
  • PATTERN_METADATA_GEMINI_TIMEOUT_MS
  • PATTERN_METADATA_NOTION_TIMEOUT_MS
  • PATTERN_METADATA_RETRY_ATTEMPTS
  • PATTERN_METADATA_NOTION_DELAY_MS
  • PATTERN_METADATA_GEMINI_DELAY_MS

Content-cache specific:

  • NOTION_CONTENT_DELAY_MS
  • NOTION_CONTENT_RETRY_ATTEMPTS
  • NOTION_CONTENT_TIMEOUT_MS
  • NOTION_CONTENT_429_DELAY_MS

Local Commands

Run only search index:

npm run build:search

Generate/sync pattern metadata descriptions:

npm run build:metadata

Metadata dry-run:

npm run build:metadata:dry

Build/update Notion content cache:

npm run build:content

Force-refresh all cached pages:

npm run build:content -- --force

Run full prebuild pipeline:

npm run prebuild

Failure Modes and Fixes

  1. search-index missing
  • Symptom: menu/search empty, redirect build fails
  • Fix: run npm run build:search
  1. notion-cache misses during build
  • Symptom: [notion-cache] miss ... logs and potential 404 in prerender
  • Fix: run npm run build:content before next build
  1. metadata descriptions not visible on front pages
  • Symptom: pattern pages still show missing/old descriptions after metadata run
  • Fix: run npm run build:search after npm run build:metadata, then rebuild/deploy
  1. too many Notion retries/timeouts
  • Symptom: slow or failing prebuild
  • Fix: increase timeout/retry env values, keep sequential content-cache mode
  1. redirect count too high
  • Symptom: warning/error in redirect build
  • Fix: keep only essential static redirects and rely on edge maps for scale

Quick Mental Model

  • build-search-index: discover and normalize content metadata
  • build-metadata-cache: generate and optionally sync AI description intelligence
  • build-content-cache: snapshot full page content for build-time rendering
  • build-redirects + build-edge-data: generate canonical redirect plumbing
  • next build: compile app using those prepared artifacts