infinity-fetch

infinity-fetch

Configurable recursive fetch for paginated APIs. Works in Node.js and browsers.

Automatically re-invokes a fetch function across pages until a stop condition is met — accumulating all results into a single array. Zero dependencies.

---

How it works

                    ┌─────────────────────────────────────────────────────┐
                    │                   infinityFetch                     │
                    └────────────────────────┬────────────────────────────┘
                                             │
                                             ▼
                    ┌────────────────────────────────────────┐
                    │          fetcher(params)               │
                    └────────────────┬───────────────────────┘
                                     │
                    ┌────────────────▼───────────────────────┐
                    │             Response                   │
                    └──────┬─────────────────────┬───────────┘
                           │                     │
                    isLastPage?                  no
                           │                     │
                          yes            getNextParams()
                           │                     │
                           │                     ▼
                           │        ┌────────────────────────┐
                           │        │   fetcher(nextParams)  │  ← repeats
                           │        └────────────────────────┘
                           │
                           ▼
              ┌──────────────────────────────────────┐
              │   return { items[], pages: number }  │
              │   items  → all pages accumulated     │
              │   pages  → total iterations done     │
              └──────────────────────────────────────┘

Three convenience wrappers sit on top of infinityFetch:

Helper	Pagination style
pagedFetch	Offset-based — { start, limit, isLastPage, nextPageStart }
cursorFetch	Cursor/token-based — getCursor returns the next cursor or null
infinityFetch	Generic — you supply isLastPage and getNextParams

---

Installation

npm install infinity-fetch

---

Usage

pagedFetch — offset-based pagination

If your API returns { values, isLastPage, nextPageStart, size, limit, start }, use the built-in helper:

import { pagedFetch } from 'infinity-fetch';

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100, // items per page, defaults to 100
});

console.log(`${items.length} commits fetched across ${pages} pages`);

With loading state, progress tracking, and a delay between pages:

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100,
  maxPages: 20,
  delay: 200,
  onStart: () => setLoading(true),
  onEnd:   () => setLoading(false),
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} commits`);
  },
});

---

cursorFetch — cursor-based pagination

Use this when your API returns a cursor (or token) to request the next page, and null/undefined when there are no more pages:

import { cursorFetch } from 'infinity-fetch';

const { items, pages } = await cursorFetch({
  fetcher: ({ cursor }) => github.issues({ cursor, perPage: 50 }),
  getCursor: (r) => r.pageInfo.endCursor ?? null,
  getItems:  (r) => r.data,
});

With full options:

const { items, pages } = await cursorFetch({
  fetcher: ({ cursor }) => stripe.charges.list({ starting_after: cursor ?? undefined }),
  getCursor: (r) => r.has_more ? r.data.at(-1)?.id ?? null : null,
  getItems:  (r) => r.data,
  maxPages: 50,
  delay: 100,
  onEnd: ({ items, pages }) => console.log(`${items.length} charges in ${pages} pages`),
  retry: { maxRetries: 2 },
});

---

infinityFetch — generic, fully configurable

Use this when your API has a different response shape:

import { infinityFetch } from 'infinity-fetch';

const { items, pages } = await infinityFetch({
  fetcher: (params) => github.issues.list(params),
  initialParams: { page: 1, per_page: 50 },
  isLastPage: (response) => response.data.length < 50,
  getNextParams: (_response, currentParams) => ({
    ...currentParams,
    page: currentParams.page + 1,
  }),
  getItems: (response) => response.data,
  maxPages: 100,
  delay: 200,
  retry: {
    maxRetries: 3,
    delay: (attempt) => attempt * 500,
    retryWhen: (error) => error instanceof Response && error.status >= 500,
  },
  onStart: () => setLoading(true),
  onEnd: ({ items, pages }) => {
    setLoading(false);
    console.log(`Done: ${items.length} items across ${pages} pages`);
  },
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} items`);
  },
});

---

Cancellation with AbortSignal

All three helpers accept a signal option. When the signal fires, pagination stops immediately and returns whatever items were collected up to that point — no error is thrown.

const controller = new AbortController();

setTimeout(() => controller.abort(), 3000); // cancel after 3 seconds

const { items, pages, aborted } = await cursorFetch({
  fetcher: ({ cursor }) => api.items({ cursor }),
  getCursor: (r) => r.nextCursor ?? null,
  getItems:  (r) => r.data,
  signal: controller.signal,
});

if (aborted) {
  console.log(`Cancelled — got ${items.length} items across ${pages} pages`);
}

aborted is true in the result only when the signal fired. When pagination completes normally, the field is absent.

---

Error handling with InfinityFetchError

When a fetch fails (after exhausting retries), infinityFetch throws an InfinityFetchError with full context about where the failure occurred:

import { infinityFetch, InfinityFetchError } from 'infinity-fetch';

try {
  const { items } = await infinityFetch({ /* ... */ });
} catch (error) {
  if (error instanceof InfinityFetchError) {
    console.error(`Failed on page ${error.pageIndex}`);
    console.error(`Params at failure:`, error.params);
    console.error(`Items collected before failure:`, error.itemsSoFar);
    console.error(`Root cause:`, error.cause);
  }
}

Property	Type	Description
pageIndex	number	Zero-based index of the page that failed
params	TParams	Parameters that were passed to the fetcher
itemsSoFar	TItem[]	All items collected from pages before the failure
cause	unknown	The original error thrown by the fetcher
message	string	"infinity-fetch failed on page N: <cause message>"

---

API Reference

pagedFetch<TItem>(config)

Option	Type	Default	Description
fetcher	(params: PagedParams) => Promise<PagedResponse<TItem>>	required	Function that fetches one page
limit	number	100	Items per page
maxPages	number	Infinity	Maximum pages to fetch (safety limit)
delay	number	—	Milliseconds to wait between each page fetch
retry	InfinityFetchRetryConfig	—	Retry failed page fetches
signal	AbortSignal	—	Cancel pagination and return partial results
onStart	() => void	—	Called once before the first fetch
onEnd	(result: InfinityFetchResult<TItem>) => void	—	Called once after all pages are done, or when cancelled — receives { items, pages, aborted: true } on cancellation
onPage	(items, response, pageIndex) => void	—	Called after each individual page

PagedParams

{ start: number; limit: number }

PagedResponse<TItem> — expected response shape:

{
  values: TItem[];
  isLastPage: boolean;
  nextPageStart?: number;
  size: number;
  limit: number;
  start: number;
}

Returns: Promise<InfinityFetchResult<TItem>>

Throws: InfinityFetchError on fetch failure. Also throws if a non-final page response is missing nextPageStart.

---

cursorFetch<TResponse, TItem>(config)

Option	Type	Default	Description
fetcher	(params: CursorParams) => Promise<TResponse>	required	Function that fetches one page
getCursor	(response: TResponse) => string | null | undefined	required	Returns the next cursor, or null/undefined on the last page
getItems	(response: TResponse) => TItem[]	required	Extracts items from a response
maxPages	number	Infinity	Maximum pages to fetch (safety limit)
delay	number	—	Milliseconds to wait between each page fetch
retry	InfinityFetchRetryConfig	—	Retry failed page fetches
signal	AbortSignal	—	Cancel pagination and return partial results
onStart	() => void	—	Called once before the first fetch
onEnd	(result: InfinityFetchResult<TItem>) => void	—	Called once after all pages are done, or when cancelled — receives { items, pages, aborted: true } on cancellation
onPage	(items, response, pageIndex) => void	—	Called after each individual page

CursorParams

{ cursor: string | null }  // null on the first request (no prior cursor)

Returns: Promise<InfinityFetchResult<TItem>>

Throws: InfinityFetchError on fetch failure (after retries are exhausted).

---

infinityFetch<TResponse, TParams, TItem>(config)

Option	Type	Default	Description
fetcher	(params: TParams) => Promise<TResponse>	required	Function that fetches one page
initialParams	TParams	required	Parameters for the first request
isLastPage	(response: TResponse) => boolean	required	Returns true to stop iteration
getNextParams	(response: TResponse, currentParams: TParams) => TParams	required	Computes params for the next page
getItems	(response: TResponse) => TItem[]	required	Extracts items from a response
maxPages	number	Infinity	Maximum pages to fetch (safety limit)
delay	number	—	Milliseconds to wait between each page fetch
retry	InfinityFetchRetryConfig	—	Retry failed page fetches
signal	AbortSignal	—	Cancel pagination and return partial results
onStart	() => void	—	Called once before the first fetch
onEnd	(result: InfinityFetchResult<TItem>) => void	—	Called once after all pages are done, or when cancelled — receives { items, pages, aborted: true } on cancellation
onPage	(items, response, pageIndex) => void	—	Called after each individual page

Returns: Promise<InfinityFetchResult<TItem>>

Throws: InfinityFetchError on fetch failure (after retries are exhausted).

---

Shared types

type InfinityFetchResult<TItem> = {
  items: TItem[];     // all items collected across every page
  pages: number;      // total number of pages fetched
  aborted?: true;     // present only when an AbortSignal fired
};

InfinityFetchRetryConfig

Field	Type	Default	Description
maxRetries	number	0	Extra attempts per page. With 0, any failure throws immediately. Total attempts = maxRetries + 1.
delay	number | (attempt, error) => number	—	Wait between retries of the same page (attempt is 1-based). Different from config.delay, which separates consecutive pages.
retryWhen	(error, attempt) => boolean | Promise<boolean>	retry all errors	If it returns false, the error is thrown immediately without exhausting maxRetries. attempt is 1-based.

class InfinityFetchError<TParams, TItem> extends Error {
  readonly pageIndex: number;   // zero-based index of the failed page
  readonly params: TParams;     // params passed to the fetcher
  readonly itemsSoFar: TItem[]; // items collected before the failure
  readonly cause: unknown;      // original error
}

Type exports

All public types are available as named imports:

import type {
  CursorFetchConfig,
  CursorParams,
  InfinityFetchConfig,
  InfinityFetchResult,
  InfinityFetchRetryConfig,
  PagedFetchConfig,
  PagedParams,
  PagedResponse,
} from 'infinity-fetch';

---

Compatibility

Environment	Support
Node.js 18+
Node.js 20+
Modern browsers
Deno / Bun
ESM
TypeScript	(types included)

---

Contributing

Commits must follow the Conventional Commits spec — this drives automatic versioning and changelog generation via semantic-release.

Commit prefix	Triggers
fix:	Patch release (0.0.x)
feat:	Minor release (0.x.0)
feat!: / BREAKING CHANGE:	Major release (x.0.0)
chore:, docs:, test:	No release

git commit -m "feat: add onPage callback to pagedFetch"
git commit -m "fix: handle missing nextPageStart gracefully"
git commit -m "feat!: rename items field to data"

Changelog

See CHANGELOG.md for the full release history.

---

License

MIT