Node script for importing from an API keeps getting 429 after about 100 requests

Promise.all over 5000 ids fires 5000 requests at once, the setTimeout doesn't change that. 60/min is one per second, so this import takes ~85 minutes no matter what. Accept that and make it robust:

for (const id of ids) {
  const res = await fetchWithRetry(id);
  await save(res);
  await new Promise(r => setTimeout(r, 1100));
}

async function fetchWithRetry(id, tries = 5) {
  const res = await fetch(url(id), { headers });
  if (res.status === 429 && tries > 0) {
    const wait = Number(res.headers.get("retry-after") ?? 10) * 1000;
    await new Promise(r => setTimeout(r, wait));
    return fetchWithRetry(id, tries - 1);
  }
  return res.json();
}

Save progress as you go so a crash doesn't restart from zero. And no key rotation, you're right that it violates most ToS.

Check if they have a list/bulk endpoint first. Lots of APIs that limit per request also let you fetch 100 records per page. 50 requests instead of 5000.

If you do this kind of import more often, bottleneck on npm handles "N per minute" limits declaratively (reservoir + reservoirRefreshInterval) so you don't have to hand-roll the sleeps.