> ## Documentation Index > Fetch the complete documentation index at: https://api.nektir.com/llms.txt > Use this file to discover all available pages before exploring further. # Import snapshots and follow observed changes > A runnable TypeScript reference for atomic imports, restartable processing, and gap recovery. Use exports as a complete baseline and observed changes as an **invalidation inbox**. The change feed is not a lossless database replication stream. In particular, an unchanged inventory check can advance retrieval timestamps without producing an event, and store-directory edits are not covered by product events. For those invalidations, use the separate resource update stream below. Periodically import a new complete snapshot as a reconciliation boundary. Neither stream establishes when DABS updated its source. ## Follow resource updates Use `GET /v1/updates/sync` when keeping cached product, store, or inventory views up to date. Use `/v1/changes/sync` for observed value differences and history. Their checkpoints are independent and cannot be interchanged. ```http theme={"dark"} GET /v1/updates/sync GET /v1/exports GET /v1/updates/sync?checkpoint=COMPLETED_RESOURCE_CHECKPOINT&limit=100 GET /v1/updates/sync?cursor=RESOURCE_PAGE_CURSOR&limit=100 ``` Capture the initial checkpoint **before** importing a baseline. Save only a final page's completed checkpoint, atomically with handling the invalidations. The same frozen-window, retry/deduplication, cursor-replay and checkpoint-gap rules described below apply. The existing file-based reference consumer follows product changes; it does not automatically process this separate stream. Each record identifies one resource with `resource.type`, its stable Nektir UUID, and `resource.source_key`. For `inventory`, the ID is the product UUID and the key is its six-digit DABS SKU. For `product`, the key is also its six-digit SKU; for `store`, it is the four-digit inventory store code. Leading zeros are preserved. An inventory publication produces one invalidation per product, not one per store row. Product and directory publications, including editorial and presence-only updates, invalidate their respective views. A record means **re-fetch this view**, not that its values necessarily changed. It is not a restock, sale, or stock guarantee. Optionally filter by `resource_type=product|inventory|store` and `resource_id=UUID`. An ID requires a type; filters are bound into the checkpoint. `limit` defaults to 25 and is capped at 100. A product filter does not include inventory records: use the unfiltered stream to maintain all resource types. There is no store-scoped inventory stream; an inventory invalidation identifies the whole product snapshot. Coalesce repeated IDs within a window and re-fetch in bounded batches: products through `/v1/products?product_ids=...`, inventory through `/v1/inventory`, and stores through `/v1/stores?store_ids=...`. Inventory updates can also change store, nearby, family and comparison result lists, including products that were previously absent from them. Revalidate those cached queries or expire them with a bounded TTL; invalidating only rows already displayed can miss a newly matching product. `times.retrieved_at_min` and `times.retrieved_at_max` preserve the range of newly published observations. Product times refer to the catalog source; store times to directory retrieval, so an editorial edit can retain older retrieval times. `times.published_at` is Nektir publication time. `times.source_updated_at` remains null unless supplied; mixed or partially unknown inventory source times are null. A new invalidation does not prove recent source data or complete store coverage. Re-fetch the resource to inspect its coverage and per-observation timestamps. The lightweight stream retains seven days from publication, independently of longer product history. It starts with publications made after installation, with no historical backfill. Failed refreshes, rolled-back and idempotently repeated publications emit nothing. Retention deletion or a restored/reset stream returns `410 CHECKPOINT_EXPIRED`; establish a new baseline. An expired page cursor instead returns `410 CURSOR_EXPIRED`; replay from the last completed checkpoint. Responses are `no-store`, and neither stream triggers DABS collection. ## Run the reference consumer The repository includes `api/scripts/examples/consumer-sync.ts`, a standalone Node.js TypeScript example with no database dependency. After `npm ci`, run: ```bash theme={"dark"} npm run api:consumer:sync -- https://API_HOST /tmp/nektir-consumer bootstrap npm run api:consumer:sync -- https://API_HOST /tmp/nektir-consumer page ``` Replace `API_HOST` with the operator-provided API origin; `api.nektir.com` currently hosts documentation, not the data service. Use one writer per output directory. The example does not persist credentials or follow redirects to other origins. `bootstrap` first captures a durable change checkpoint, **before** requesting the export manifest. It then downloads all three files sequentially into a staging generation. It validates schema version, resource, publication revision, record count, NDJSON framing, and every row's envelope. Only after all files validate does an atomic rename switch `checkpoint.json` to the new generation. Interrupted or malformed downloads leave the last complete baseline active. Node fetch decompresses HTTP gzip automatically; the files on disk are plain NDJSON. `page` commits one change page and its next cursor together. Repeat while the command reports `has_more: true`; on the next scheduled poll, invoke `page` again to resume from the last completed publication checkpoint, not the head of retained history. Re-running in a new process resumes the same cursor. Event UUIDs deduplicate retries. The local checkpoint binds the origin, SKU scope, and page size so a different query cannot silently reuse it. The example processes one page per invocation to bound work. It does not contain an infinite retry loop. On a transient `503`, retain the checkpoint and retry the same operation after `Retry-After` with jitter. Persistent export-limit errors need operator attention. An export `410` during import leaves the old generation active; start a new `bootstrap`. `410 CURSOR_EXPIRED` clears only the temporary page cursor and replays from the last completed checkpoint, preserving the baseline and deduplicating any repeated events. `410 CHECKPOINT_EXPIRED` imports a new baseline and clears the cursor and inbox. This restores current-state coverage; it does **not** recover every historical event lost across a retention gap. The command reports `rebootstrapped: true` and `recovery_reason: "checkpoint_expired"` so an operator can detect that boundary instead of assuming uninterrupted replay. Cursor-only recovery instead reports `restarted_window: true` and `recovery_reason: "cursor_expired"`. ## Durable checkpoint requests Start with `GET /v1/changes/sync` using your intended filters. The response has no events and supplies `meta.sync.checkpoint` at the current publication. Capture it before downloading your baseline: a publication during import may overlap the baseline and replay, but cannot fall into a gap after it. ```http theme={"dark"} GET /v1/changes/sync?sku=006291 GET /v1/exports GET /v1/changes/sync?sku=006291&checkpoint=COMPLETED_CHECKPOINT&limit=100 GET /v1/changes/sync?sku=006291&cursor=NEXT_PAGE_CURSOR&limit=100 ``` The token names above are placeholders; use the opaque strings returned by the API. `checkpoint` and `cursor` are mutually exclusive. The service freezes each window's upper publication and returns events in ascending publication/event order. Newer publications wait for the next poll. A final page, even an empty filtered one, returns the next completed checkpoint. Intermediate pages return a null checkpoint: do not advance your durable position before the window completes. Commit event processing and progress atomically in your storage. Delivery can repeat after an interruption, so deduplicate by event UUID and make downstream side effects idempotent. The API is stateless per consumer; there is no server-side subscription to create or acknowledge. Supported filters are `product_id` or six-digit `sku`, `store_id`, and `types`. Tokens bind those filters; type ordering and page size do not change server-side scope. Changing filters requires a new initial checkpoint and baseline strategy. Family and retrieval-time filters are not supported by this synchronization route. The ordinary `GET /v1/changes` remains the newest-first browsing/history endpoint. Completed checkpoints have no short page TTL and survive API process restarts. Their safety depends on retained history and stream identity, not retained product version projections. A deletion watermark conservatively invalidates any checkpoint that might have unread deleted events, even if the deleted event was outside a particular filter. Resetting the event stream also invalidates old checkpoints. All such gaps return `410 CHECKPOINT_EXPIRED`, never a silently shortened result. Restoring a database starts a new lineage too: the supported restore command rotates stream identities before completing. Operators using raw database restore tools must reset those identities before serving the restored copy; copying an old stream UUID is not proof of continuous history. Keep the signing secret persistent across service instances. A secret rotation invalidates signed tokens with `400 CHECKPOINT_INVALID`; establish a new baseline. Tokens are synchronization positions, not access credentials. Responses use `Cache-Control: no-store` so an intermediary cannot replay an old synchronization position. None of these operations requests fresh data from DABS. ## Apply invalidations in your application Read the checkpoint's `baseline.directory` to locate the three pinned files. Their flat export schema is not identical to the public product/detail DTO schema; keep baseline rows and independently fetched application views separate. For each inbox event, use `product.id` and the optional `store.id` to invalidate affected application views. Batch product re-fetches with `GET /v1/products?product_ids=...` and inventory re-fetches with the bounded matrix endpoint, `GET /v1/inventory`. Store the revision and observation times returned by each response. Fetching several views sequentially does not produce a single coherent statewide snapshot. Do not replay event payloads as though they contain every field needed to rebuild all product and inventory rows. Do not infer zero inventory from a missing event or a missing row. Preserve null source-update timestamps, missing observations, restricted purchase eligibility, and reported zero quantities as distinct states. Do not set `since` to the last poll's wall-clock time. That filter uses retrieval time: an older observation published later could be skipped. Publication-based checkpoints avoid that problem and read only the new event interval. This is still not every-retrieval or directory CDC: use the separate resource update stream for published view invalidations, a transactional inbox for large integrations, and periodically reconcile complete snapshots and current views. ## Bounds and durability This is a small reference consumer, not a production synchronization daemon. The default inbox holds 10,000 distinct events and stops without advancing when full. The library permits up to 50,000, supports page sizes 1–100, and optionally scopes events to a six-digit SKU. Archive/process events transactionally before adapting it for a larger integration; do not drop deduplication IDs and claim exactly-once delivery. Inbox delivery here is replay-safe, not exactly-once business-side-effect execution. For a selected-product invalidation inbox, pass a six-digit SKU as the final CLI argument, using the same scope for bootstrap and subsequent pages. Set `NEKTIR_SYNC_PAGE_SIZE` if you need smaller pages; keep it unchanged while resuming. SKU scoping limits events only, not the complete baseline export. Each export is limited to two million records, 1 GiB decoded bytes, and 8 MiB per record. Request deadlines are 150 seconds for exports and 15 seconds for change pages. Failed staging directories are removed. Previous successful generations are retained so existing readers are not disrupted; prune inactive generations only after readers release them and budget disk space accordingly. Data files and checkpoint files are synced before publication and checkpoint replacement is atomic on the same filesystem. Restart tests cover process interruption, not filesystem corruption or sudden host power loss. Production consumers should use their own transactional storage and backup policy. ## Verification `api/tests/consumer-sync.test.ts` covers partial imports, replay deduplication, restart, capacity limits, transient failure, malformed pagination, and expired cursor recovery. `api/tests/consumer-sync.postgres.test.ts` runs against a compiled API and disposable seeded PostgreSQL database, including a delayed-publication observation and actual cursor-expiry rejection. CI runs that integration test without contacting DABS. `api/tests/change-sync.postgres.test.ts` also checks exact summary boundaries, rollback/deletion handling, filter binding, frozen publication windows, process restart, independence from version-history TTL, and conservative retention gaps. The local reference checkpoint format is now version 2; if you used version 1, explicitly run `bootstrap` once rather than reusing its old feed position. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.