How to Build an AI Video Pipeline With a NestJS Backend

A generate button is a promise, and that promise is expensive to keep. Someone types a sentence, picks a look, presses the button, and expects a moving image within a minute or two. Behind that press sit input validation, model selection, a long-running render, storage of large files, a human approval step, and delivery to wherever the video will eventually live.

A NestJS backend is a reasonable home for all of it: modules, dependency injection, guards, interceptors, and background processors in one TypeScript codebase, with a testing story that does not require a second language. This guide is about the plumbing rather than the prompt craft. It covers how to shape the workflow, where to keep state, how to queue renders without blocking HTTP requests, how to keep a character looking like the same person across six shots, and which mistakes cause the most rework. Examples assume NestJS, PostgreSQL, a Redis-backed queue, and S3-compatible object storage, but the reasoning survives if you swap any of those pieces.

Model the workflow before you write a module

The first design artifact should be a state machine, not a folder structure. Video generation is asynchronous and partly external: your system submits work to a model provider, waits, may retry, may receive a callback, and eventually ends up with one approved asset out of several attempts. When the states are vague, every module invents its own vocabulary and the dashboard shows the word processing for four different situations.

States that earn their place

A workable set for most products looks like this:

State	What it means	Who moves it forward
draft	Scene exists with a prompt but nothing has been sent	User or planner
queued	Job accepted, waiting for a worker	Queue worker
rendering	Provider is actively producing output	Status poller or webhook
post_processing	Trim, caption, audio normalization, thumbnail	Media worker
needs_review	Output is ready for a human decision	Reviewer
approved	Asset selected for delivery	Reviewer
rejected	Asset discarded with a reason	Reviewer
failed	Terminal error after retries	Worker
cancelled	Stopped at user request	Worker or API

Two rules keep this honest. First, terminal states are terminal: moving out of failed or rejected requires a new attempt, not a quiet edit of the old row. Second, every transition writes an event with a timestamp, actor, and reason. When a client asks why a clip looks different from yesterday, the event log answers the question faster than any amount of log searching.

A worked example: the thirty-second product spot

Imagine a marketing team generating a thirty-second spot with six scenes. They upload a reference frame of the product, write six scene descriptions, and press generate. In a well-built system the intake endpoint rejects the request only if something is genuinely impossible, such as a nine-by-sixteen render at four minutes long on a plan capped at thirty seconds. It then creates six scenes, one generation job per scene, and returns a batch identifier with a 202 response. Planning assigns each scene a shot type: macro insert, medium product on desk, wide office establishing shot, hands interacting, close-up detail, logo end card. The render queue processes two at a time because concurrency is capped. Scene three fails on a provider timeout, retries once, and succeeds. Scene five comes back with a character wearing the wrong jacket, so a reviewer rejects it and requests a revision with the note that the jacket must stay charcoal. Delivery assembles the approved clips, adds captions, and returns signed URLs that expire in an hour.

Nothing in that story required microservices. It required clear states, one owner per transition, and a queue that could survive a provider hiccup.

Structuring a NestJS app around video jobs

The most common structural mistake is a single service that validates input, calls three providers, writes files, sends notifications, and updates the database. It works until the first provider change, then becomes impossible to test.

Modules with one reason to change

Split by workflow boundary rather than by technical layer alone:

• ProjectsModule owns projects, scene ordering, and project-level settings.
• ScenesModule owns shot descriptions, references, and prompt versions.
• GenerationModule owns job creation and state transitions.
• ProviderModule owns model adapters and capability checks.
• AssetsModule owns uploads, signed URLs, checksums, and metadata.
• ReviewModule owns comments, approvals, rejections, and revision requests.
• UsageModule owns quotas, concurrency caps, and rate limits.
• WebhookModule owns callback verification, deduplication, and enqueueing.

Keep dependency injection pointing inward. Controllers depend on application services; application services depend on domain rules and provider interfaces; infrastructure modules implement those interfaces. A queue processor should call the same application service as an HTTP endpoint so that there is exactly one place where a job moves from queued to rendering. Duplicating that transition in two places is how dashboards start lying.

Provider adapters and capability negotiation

Define an interface with four responsibilities: describe capabilities, submit, check status, cancel. Each adapter translates your internal request into whatever the provider expects and maps the response back into your shared job model. Capabilities should be data, not assumptions: supported aspect ratios, maximum duration, whether reference images are accepted, whether audio is generated or must be added later, whether the provider returns a seed.

That data drives routing. A scene that needs a talking presenter goes to one adapter; a scene that needs a slow camera push through fog goes to another. Routing rules belong in configuration with explicit fallbacks, so a failed primary does not silently produce a wildly different look. If you are comparing external options while designing this layer, the AI video generator alternatives page is a useful way to see how capability sets differ before you hard-code a decision.

Queue design that keeps HTTP requests short

A render can take minutes. No HTTP request should wait for it. Return 202 Accepted with a job identifier, then let the client poll a status endpoint or subscribe to server-sent events.

Payloads, idempotency, and queue separation

Separate queues by workload because their concurrency profiles differ. Planning is cheap and fast. Rendering is heavy with low concurrency. Post-processing is CPU-bound but predictable. Delivery is mostly file and metadata operations and can run at higher parallelism. One shared queue means a burst of renders starves thumbnail generation.

A render job payload should carry the project identifier, scene identifier, prompt version, provider key, requested duration, aspect ratio, reference asset identifiers, and an idempotency key built from the scene plus prompt version. The worker claims the job, writes rendering, submits the provider request, persists the provider job identifier, schedules a follow-up check, and returns. It must not sit in a sleep loop holding a worker slot for four minutes. A separate poller or the provider webhook advances the job.

Retries, backoff, and dead letters

Classify errors before deciding whether to retry. Rate limits and timeouts deserve exponential backoff with jitter. Invalid parameters and content-policy rejections should fail immediately, because retrying them burns money and delays the user. Authentication failures should also fail fast and raise an alert, since a rotated key is an operational problem, not a transient one.

When retries are exhausted, move the job to a dead-letter queue with the failure class, the last provider summary, and the attempt count. A job with no visible failure reason is a support ticket waiting to happen. Use a circuit breaker per provider so one unhealthy integration does not drain the whole queue while healthy providers sit idle.

Cancellation and honest progress

Users change their minds, and abandoned renders consume capacity that other projects need. Provide a cancel endpoint that behaves in two steps: mark the job cancelled locally, then ask the provider to stop if it supports cancellation. If it does not, let the render finish and discard the output rather than pretending it stopped.

Progress reporting is a trust surface. Map provider stages onto a small set of internal states and show a percentage only when the provider actually supplies one. Fabricated progress bars that sit at ninety-two percent for three minutes teach users to distrust every other number in the interface.

Data model, storage, and signed delivery

Relational tables for state, object storage for bytes. Do not put video files in the database, and do not put the only copy of a job identifier in the browser.

A minimal schema covers projects, scenes, prompt_versions, generation_jobs, generation_attempts, assets, and reviews. Every row should carry a tenant or owner identifier, created and updated timestamps, and a status. Attempts should record provider, model, a hash of the request payload, latency, error class, and the seed if one exists. Assets should record MIME type, dimensions, duration, checksum, and the attempt that produced them.

Use transactions where a single user action touches several tables. Approving a review might update the review row, mark one asset selected, and create a delivery record; if those can partially fail, the project becomes inconsistent in a way that is hard to explain. Index the columns the dashboard filters on constantly: project identifier, scene identifier, status, and creation time.

Store binaries in an S3-compatible bucket with a key convention that encodes tenant, project, scene, and attempt. Serve them through short-lived signed URLs scoped to a single object rather than permanent public links. Keep checksums so duplicate uploads can be detected, and set lifecycle rules that expire intermediate renders while retaining approved masters. Retention policy is a product decision, but write it down before storage costs surprise anyone.

Consistency across shots: characters, style, and seeds

Consistency is where most AI video workflows break. A model produces a beautiful first shot and a slightly different face in the second, and the resulting sequence reads as a montage of strangers. The backend can help by treating consistency as data instead of a prompt-writing habit.

Build a project bible

Store character sheets, wardrobe notes, location descriptions, palette rules, and camera conventions at the project level. When a scene is generated, compose the final prompt from the scene description plus the relevant project references, then version the composed prompt. When someone edits the character sheet, new renders pick up the change while old renders remain explainable.

Teams that want a starting point for shot planning can browse Orelon video templates to see how shot lists are structured before designing the intake form around them.

Version seeds and references

Record the seed, reference image identifiers, model version, and style strength for every attempt. If a shot works, the team should be able to clone its settings onto the next shot with one action. Providers that do not expose seeds should be documented as such, and the closest reproducible inputs should still be stored so a re-render is at least explainable.

Run a cheap continuity check before an expensive one

Before submitting six renders, compare character names, wardrobe tags, location identifiers, and time of day across adjacent scenes. Flag contradictions: a night interior followed by a noon exterior with no transition, a character who changes hair color between shots three and four, a prop that disappears. This is a lint step, not a creative director. It costs milliseconds and prevents re-renders that cost minutes.

A shared prompt library helps too, because reusable scene language reduces the drift that appears when six people write six slightly different descriptions of the same room.

The API contract and the human review loop

A predictable API makes the frontend simpler and the backend testable. Keep resource names boring and status codes honest about asynchronous work.

Endpoint	Purpose	Notes
POST /projects	Create a project	Returns project identifier
POST /projects/:id/scenes	Add a scene	Validates order and references
POST /scenes/:id/generations	Start a render	Returns 202 and job identifier
GET /jobs/:id	Read status	State, progress, asset identifiers
POST /jobs/:id/cancel	Request cancellation	Best effort with provider
POST /reviews/:id/approve	Approve an asset	Creates a delivery record
POST /reviews/:id/reject	Reject with reason	Preserves revision history

Require an idempotency key on generation creation so a double click cannot create two expensive jobs. Return a stable error shape with a machine code, a human message, and a request identifier. For review, store comments as immutable events and derive current state from them, or keep a state table plus an audit log. Either way, never overwrite history to make the interface simpler.

The review loop deserves more design attention than it usually gets. Reviewers need a side-by-side view of attempts, the prompt that produced each one, and the ability to request a narrow revision such as changing only the camera height. When a rejection reason is structured rather than free text, you can later measure which reason dominates and fix the upstream prompt template instead of blaming the model.

Security, quotas, and cost control

Ownership checks belong on every project, scene, job, asset, and review. Unguessable identifiers are not authorization. Enforce tenant filters in repository queries, verify webhook signatures, deduplicate replayed events, and validate uploads by content type and size before they reach storage.

Quotas protect both margins and fairness. Set per-user and per-project rate limits, concurrency caps so one account cannot occupy every render worker, and maximum duration or resolution by role if the product needs tiers. Enforce all of it server-side; a disabled button in the client is a suggestion, not a control.

Cost control is a design feature. Estimate the cost of a batch before submitting it and show that estimate in the confirmation step. Cache capability metadata instead of calling providers on every page load. Cancel aggressively when a project is abandoned. And keep moderation checks before submission rather than after, with an appeal path for false positives so users are not stuck with a silent failure.

Observability, testing, and failure drills

You cannot debug a distributed render pipeline with console logs. Attach a correlation identifier at intake and pass it through HTTP calls, queue jobs, provider requests, and webhook handlers. Log the job identifier, project identifier, attempt number, provider, state transition, and latency. Never log full prompts if they may contain client secrets; log a hash or a redacted summary instead.

Track metrics that reveal health rather than activity: queue wait time, render duration by provider, failure rate by error class, retry count, cancellation rate, review turnaround, and delivery errors. Rising queue wait usually means concurrency is too low or a provider is throttling you. Rising validation failures usually mean the interface allows settings the backend rejects.

Scenario tests that catch real bugs

Use a fake provider adapter with fixture responses and simulated delays so the suite runs in seconds, and keep one small live smoke test for the real integration. Then write tests around the scenarios that break production: a user double-clicks generate, a provider times out after accepting the job, a webhook arrives twice, a reviewer rejects and requests a revision, an upload fails halfway, a cancellation races with a completion. Load-test with realistic clip lengths, because a one-second test clip hides every queue pressure problem that a thirty-second sequence exposes.

Common mistakes and decision criteria

A short list of failures worth designing against:

• Storing only the final video URL, so nothing can be reproduced or explained.
• Retrying non-retryable errors and quietly doubling spend.
• Holding a worker slot while a remote model renders.
• Reporting invented progress percentages.
• Letting a single service own validation, providers, storage, and notifications.
• Hard-coding a provider choice instead of routing by capability.
• Treating review as an afterthought with no structured rejection reasons.
• Skipping webhook deduplication, then wondering why jobs complete twice.

When deciding how far to go, use three criteria. Build a separate service only when scaling pressure, team ownership, or isolation requirements justify the operational cost; otherwise a modular monolith with workers is enough. Add a second provider only when capability gaps or outage risk cost more than the extra adapter maintenance. Invest in prompt and seed versioning before investing in fancy analytics, because reproducibility is what makes every later optimization measurable.

FAQ

Do I need microservices for an AI video backend? No. A modular NestJS application with background workers covers most teams. Split services when scaling, ownership, or isolation demands it.

Should the browser call model providers directly? No. Direct calls expose keys, bypass validation, and make job tracking inconsistent. Route everything through your backend and return a job identifier.

How many queues should I run? Four is a good starting point: planning, rendering, post-processing, and delivery. Each has a different concurrency profile, and separating them prevents one workload from starving another.

What happens when a provider goes down mid-batch? The circuit breaker opens, in-flight jobs fail with a classified error, and queued jobs pause rather than burn retries. Show users a clear status and resume when the provider recovers.

How do I keep a character consistent across many shots? Store character sheets and reference assets at the project level, version the composed prompt and seed for every attempt, reuse settings from approved shots, and run a continuity lint before rendering.

How do I test without waiting for real renders? Use a fake adapter that returns fixtures with simulated delays for the bulk of the suite, and keep one live smoke test to catch contract drift with the real provider.

When should a job be marked failed? After retries are exhausted, or immediately when the error is non-retryable. Always persist the reason and the last provider summary so support can answer questions quickly.

Orelon is built for cinematic ideas in motion: a place to turn a script, a reference frame, and a clear visual direction into a generated video. When your backend is ready for a creative front end, start with the Orelon AI video generator, explore the Orelon blog for more production patterns, or read how teams compare tooling before they commit.