Silent 429 rate limit erased 40,000 imported Instagram posts; postmortem prescribes async imports

A developer writing on dev.to has published a postmortem of a failure that cost their platform 40,000 imported Instagram posts. A third-party API started returning silent 429 rate-limit errors while the frontend loading spinner kept animating, so users had no signal that their migration had stalled or died.

How the import failed

According to the dev.to post, the team had built the feature as if it were a simple file upload: a drag-and-drop zone pointed at a synchronous endpoint, on the assumption that the external platform's API would happily stream large media libraries. In reality, the upstream API was heavily paginated, throttled and prone to sudden schema shifts.

A synchronous request would time out after roughly thirty seconds, the browser would cut the connection, and the user would be left staring at a blank screen with no idea whether their content had arrived or vanished. Partial failures compounded the damage in the database: orphan records, videos without thumbnails, captions truncated by encoding mismatches, and authentication tokens expiring mid-batch. Support staff ended up manually patching broken user accounts in production.

The author argues the worst cost is trust. Social media managers and creators spend hours organising their libraries before migrating to a new tool, and a silent failure within the first minutes of onboarding can permanently destroy their confidence in the product.

The proposed fix: resumable background jobs

The redesign centres on decoupling the user's browser session from the heavy lifting of ingestion. Instead of waiting on a monolithic API response, every import becomes an asynchronous, resumable, observable background task handled by a worker pool.

When a user kicks off an import from Instagram, TikTok or YouTube, the frontend hands a payload reference to the backend and renders a persistent background drawer. That drawer does not depend on a fragile long-lived HTTP request; it subscribes to real-time status updates via WebSockets or polling backed by a Redis state cache.

If the network drops, the user refreshes the page, or the external API throttles requests, the system pauses rather than fails. It logs the exact cursor position, schedules a retry with exponential backoff, and shows the user a calm banner explaining that import speed has been temporarily throttled by platform limits, along with an estimated time remaining that adapts to observed throughput.

The implementation pattern

The post sketches the stack in TypeScript. A BullMQ worker processes imports in controlled batches, and when a response indicates more pages exist, the worker enqueues a follow-up job carrying the next cursor, with a deliberate two-second delay as rate-limit mitigation. Each chunk is small and immutable, so a transient failure only affects one batch rather than the entire migration, and request timeouts effectively disappear.

On the client, a React hook opens a Server-Sent Events stream and tracks a compact state machine — idle, processing, paused, completed, failed — alongside progress and error fields. A companion Express route enqueues the first job with a batch size of fifty and immediately returns HTTP 202 Accepted with a job identifier, keeping the initial response fast regardless of how large the import turns out to be.

The article closes with a section cataloguing further mistakes to avoid, though the published text is truncated partway through that discussion.

Why it matters

Any product that imports data from third-party APIs inherits their flakiness. The transferable lesson here is that synchronous requests plus indeterminate spinners convert upstream throttling into silent data loss — and users will blame your product, not the platform that returned the 429.

Modelling imports as chunked, cursor-based, resumable jobs turns a hard failure into a pause, and a UI that reflects real pipeline state (including throttling) turns confusion into an explainable wait. The same pattern applies well beyond social media imports: billing syncs, CRM migrations, bulk exports and any other pipeline that pulls large volumes of data across a rate-limited boundary benefit from the same architecture.