TypeScript Patterns for AI Video App Development That Scale

An AI video application is three products sharing one repository: a creative surface, a job scheduler, and a distributed pipeline that occasionally drops a message at the worst possible moment. TypeScript earns its keep on the second and third, and those two decide whether the first ever feels effortless.

Teams usually learn this the hard way. A prompt leaves a browser tab, becomes a database row, becomes a queue message, becomes an inference call in another region, becomes a file in object storage, and finally becomes a player element somebody is watching. That is six handoffs. Each one is a place where a renamed field turns into a black frame and an apologetic support reply instead of a stack trace you can follow.

The patterns below are the small ones that survive production: a discriminated union for job state, a runtime schema at every boundary, an idempotent submission key, an ordered progress stream, a capability registry that lets you swap a rendering backend without rewriting the product around it, and a test strategy built around ordering rather than timing. None of them are clever. All of them are cheaper than the incident they prevent.

Start With the Job Lifecycle, Not the Endpoint

Endpoints change weekly. A job lifecycle barely changes at all, so it is the better thing to model first, because everything else will end up conforming to it anyway.

The reason a video product rewards strict typing more than a typical content app is that its central object has a handful of states with wildly different shapes. A job is queued, running, succeeded, or failed, and those states share almost no fields. Without types you end up with one object holding a dozen optional properties plus a comment insisting that the asset URL only appears on success. Nobody reads that comment during a Friday deployment.

type StageName = 'queued' | 'sampling' | 'upscaling' | 'encoding';

type JobState =
  | { status: 'queued'; requestedAt: string; priority: number }
  | { status: 'running'; startedAt: string; stage: StageName; progress: number }
  | { status: 'succeeded'; assetUrl: string; durationMs: number; fps: number }
  | { status: 'failed'; reason: string; retryable: boolean };

With a discriminated union keyed on the status field, the interface layer narrows the object for you. Reading an asset URL from a failed job becomes a build error rather than a runtime surprise. Add an exhaustiveness guard and any new state you introduce later refuses to compile until it is handled everywhere.

function assertUnreachable(value: never): never {
  throw new Error('Unhandled state: ' + JSON.stringify(value));
}

Identifier confusion is the second cheap win. Video codebases juggle job identifiers, provider references, asset identifiers, tenant identifiers, and webhook identifiers. They are all strings, so nothing stops you from passing one where another was expected. A wrapper type costs one property and removes the ambiguity entirely.

type JobId = { readonly value: string };
type AssetId = { readonly value: string };
type BackendId = { readonly value: string };

Construct these wrappers immediately after schema validation, at the edge of the system, and let the wrapped values flow inward. Teams that adopt this see one specific bug class vanish: the retry that submits a provider reference where a local job identifier belongs, producing a duplicate render and a puzzled support thread.

Then validate at the boundary rather than in the middle. A single schema for a generation request documents the product surface, produces the static type, and rejects the low-quality input that burns expensive compute.

import { z } from 'zod';

const GenerationInputSchema = z.object({
  prompt: z.string().min(3).max(2000),
  aspect: z.enum(['16:9', '9:16', '1:1', '2.39:1']),
  durationSeconds: z.number().int().min(2).max(30),
  seed: z.number().int().optional(),
  referenceAsset: z.string().url().optional(),
});

// The static type is derived from this schema, so validation and typing never drift.

Reuse that schema for provider responses and queue payloads too. One definition, three consumers, zero drift. When a limit changes you edit one number and every layer follows.

One more habit belongs here: store every transition as an append-only record rather than overwriting a single row. A lifecycle table with timestamps for queued, started, and finished gives you queue latency, render duration, and failure attribution without a separate analytics pipeline. It also makes a support conversation trivial, because you can replay exactly what a specific job did, in order, months later.

Organise the Backend Around Domains

Most teams organise by technical layer, with controllers, services, and repositories sitting in separate trees, and then wonder why every feature touches every folder. Organising by domain keeps AI concerns where they belong and makes the expensive parts replaceable.

A module map that survives growth

A layout that holds up for a generation product usually separates four things:

• Generation handles model selection, request shaping, retry policy, and the adapter that talks to an inference provider.
• Jobs owns lifecycle transitions, queue publishing, worker handlers, and progress emission.
• Media covers object storage paths, signed URL issuance, transcoding, and cleanup of abandoned assets.
• Accounts covers tenancy, plan limits, and usage accounting that records what each job actually consumed.

Each module owns its types and exposes a narrow public surface. The generation module knows nothing about HTTP. The jobs module knows nothing about the database engine. The media module knows nothing about prompts. When a provider deprecates an endpoint, exactly one folder changes.

Two rules keep that structure honest. First, never pass a database entity from one module to another; map to a domain type at the boundary so a column rename cannot ripple across the codebase. Second, put every validation schema next to the module that owns the data, not in a shared folder that everyone edits and nobody reviews.

Dependency injection without framework lock-in

You do not need a decorator-heavy framework to wire this together, though one is fine if the interface comes first. Define the contract, then the wiring.

interface RenderBackend {
  readonly id: BackendId;
  readonly maxDurationSeconds: number;
  submit(input: GenerationInput): Promise; // resolves to the provider reference
  status(reference: string): Promise; // resolves to a JobState
  cancel(reference: string): Promise; // resolves to void
}

A service that depends on a named interface rather than on a concrete vendor package is trivial to fake in tests, which matters enormously when the real backend takes forty seconds per call. Wire it with a factory function in a small container module, or with constructor injection plus a token if your framework prefers that style. Either way, keep the container at the application edge so domain code imports interfaces and never implementations.

The payoff arrives when a provider is retired. Because nothing outside the adapter imports the vendor package, the migration is a new file plus a routing rule, not a hunt through forty call sites and a weekend of regression testing.

Treat the Queue as a Versioned Contract

Generation is slow, expensive, and failure-prone, which makes the queue the most important component you will write. Treat the message as a contract between two deployables rather than as a transport detail of whichever broker you picked this year.

Payload shape and payload size

Publish the payload type from a shared package that both the API and the worker import. When you add a field, both sides fail to compile until they agree, which is the cheapest possible defence against a producer that stops sending a field while a worker keeps silently defaulting it.

interface RenderJobPayload {
  jobId: JobId;
  prompt: string;
  aspect: '16:9' | '9:16' | '1:1' | '2.39:1';
  durationSeconds: number;
  seed: number | null;
  attempt: number;
  schemaVersion: number;
}

Keep messages small. Prompts, references, and settings travel in the message; binaries belong in object storage with a URL in the message. Fat messages slow the broker, bloat the dead-letter queue, and make every retry more expensive than it needs to be. Include a schema version field so a worker can reject or migrate a payload it does not understand instead of guessing, and treat a version bump as a deployment event with a drain period behind it.

Idempotency instead of retry guards

A retry that submits the same prompt twice doubles your inference spend and can hand one user two near-identical clips for a single request. Derive a deterministic key from normalised inputs and use it both as a deduplication key and as part of the storage path.

import { createHash } from 'node:crypto';

function submissionKey(input: GenerationInput): string {
  const parts = [
    input.prompt.trim().toLowerCase(),
    input.aspect,
    String(input.durationSeconds),
    input.seed === undefined ? 'random' : String(input.seed),
  ];
  return createHash('sha256').update(parts.join('|')).digest('hex');
}

Store the key on the job record behind a unique constraint. A duplicate submission then collapses into a lookup instead of a second render, and the user sees the same clip rather than two variants to choose between. This single constraint is the highest-value line of SQL in the whole schema, and it also makes an accidental double-click on a submit button harmless.

Backpressure, priority, and dead letters

Without limits, one enthusiastic user occupies the entire worker pool while everyone else watches a spinner. The controls that earn their keep:

• Per-tenant concurrency caps enforced at the queue level rather than in application code.
• Priority lanes so a short preview clip is not stuck behind a long cinematic render.
• Exponential backoff with jitter when a provider returns rate-limit responses.
• Timeouts set above the slowest realistic job, not above the average one.
• A dead-letter queue with an alert attached, because a growing dead-letter queue is the earliest signal of a contract mismatch.
• Worker-side refusals, so a payload whose job was cancelled never spends compute at all.

Managed brokers all support these primitives in slightly different words. Choose by where your workers already run, and keep orchestration portable by depending on your own payload type rather than on a broker-specific API.

Watch queue depth as a product metric rather than an infrastructure metric. Waiting time plus render time is the number users actually feel. A rising wait with a flat render time tells you to add workers, while a rising render time with flat waits tells you a provider has degraded.

Realtime Progress That Survives a Reconnect

Users tolerate a long render when they can watch it move. They lose patience when the bar stalls at nine percent and then jumps to done. A typed event stream is step one; ordering discipline is step two.

type ProgressEvent =
  | { type: 'stage'; jobId: JobId; stage: StageName; seq: number }
  | { type: 'percent'; jobId: JobId; value: number; seq: number; etaSeconds?: number }
  | { type: 'preview'; jobId: JobId; frameUrl: string; seq: number }
  | { type: 'done'; jobId: JobId; assetUrl: string; seq: number }
  | { type: 'error'; jobId: JobId; message: string; retryable: boolean; seq: number };

The details that separate a demo from a product:

• A monotonically increasing sequence number so a reconnecting client can discard anything older than what it already rendered.
• Authentication on the stream plus a check that the subscriber owns the job before a single frame is sent.
• A heartbeat during long inference steps so proxies do not close an idle connection.
• Server-side persistence of the last known state, with the stream treated as a convenience rather than as the source of truth.
• A single request that lets a reloaded tab reconstruct the progress bar exactly, including the stage label.

One-way or two-way

One-way event streams are usually the right fit for progress. They cost far less to operate than a fleet of sockets, reconnects are handled by the browser, and there is no session state to clean up when a laptop closes. Reach for full duplex when the client also needs to send commands mid-job, such as cancelling a render, changing a camera move, or extending a clip by a few seconds. Even then, keep the command surface tiny and validated, because it is an open door into your expensive workers.

Cancellation and cleanup

Cancellation is where progress systems usually fall apart. Define what cancel means before you build the button: mark the job cancelled, stop emitting events, refuse the payload at the worker, and delete the partial asset on a schedule. Without that last step, abandoned clips accumulate quietly and storage grows in a way no dashboard explains.

Return a clear outcome to the interface as well. A cancelled job that still shows a spinning indicator is worse than a failed job with an honest message, because the user cannot tell whether to wait or retry.

Type Every Model Boundary With a Schema

The hardest failures to debug in a generation product are not crashes. They are successful renders that produce nothing, caused by a response field that quietly changed name. Schema-first parsing at every boundary is the habit that prevents them.

Validate responses, not only requests

Parse provider responses with a schema that allows unknown keys and logs every unrecognised field. The first time a provider renames something you want a log line rather than a support ticket. Derive static types from those schemas so validation and typing cannot drift apart, and keep the raw response alongside the parse error so you can inspect the envelope that broke you.

The same discipline applies to websocket messages, webhook payloads, and cached JSON. Anything that arrives from outside the process gets parsed. Anything inside the process can trust its types.

A capability registry instead of hard-coded picks

Hard-coding one provider means every deprecation becomes a release. A registry of typed adapters keeps that decision in one place.

interface BackendCapabilities {
  maxDurationSeconds: number;
  supportedAspects: string[];
  supportsReferenceImage: boolean;
  supportsSeed: boolean;
}

interface BackendAdapter {
  readonly id: BackendId;
  readonly capabilities: BackendCapabilities;
  readonly status: 'active' | 'degraded' | 'retired';
  submit(input: GenerationInput): Promise; // resolves to the provider reference
}

Route by capability rather than by name: find a backend that supports the requested aspect and duration, prefer active over degraded, and fall back in a defined order. When you retire a provider, add a migration test that renders the same input through both adapters so you can judge the visual difference before shifting traffic.

Record what produced each asset

Provenance is not paperwork. Store the prompt, seed, duration, aspect, adapter identifier, and model version with every asset. Six months later, when someone asks why a clip looks different, the answer is a lookup rather than a guess. It also lets you reproduce a render for a bug report without asking the user to describe what they typed.

Treat provenance as a first-class record with its own schema, not as a loose JSON blob. Once it is typed, you can build a comparison view, a cost report, or a re-render button on top of it without another migration.

Choose Where Each Piece Runs

There is no single runtime for a video pipeline. There is a correct place for each step, and the usual mistake is putting a heavy step where it does not fit.

Edge compute is a legitimate home for light work: prompt screening, request routing, aspect-ratio arithmetic, thumbnail selection, cache lookups, and signed URL generation. Sampling a ten-second clip is not edge work; memory and compute keep it on GPU machines behind long-lived workers. The API layer owns tenancy, plan limits, and the product surface.

Typing across runtimes takes discipline. Node built-ins such as the filesystem and streaming compression modules do not exist in an edge worker, so use separate configuration files with appropriate library settings and keep shared domain types in a package that imports nothing runtime-specific. Provider packages written for Node belong behind an adapter so they never get pulled into the edge bundle by accident.

Caching is the other edge win that teams underuse. A rendered asset with a content-addressed path can be served from cache indefinitely, which means the edge answers repeat traffic and the GPU workers only ever see genuinely new requests. Combine that with signed URLs that expire quickly and you get both speed and control over who can watch a private clip.

A short decision rule helps when a new step appears: if it can fail without invalidating the render, it can run at the edge. If it produces the frames, it cannot.

Deterministic Assembly of Clips

Generation is half the job. Assembling clips into something coherent is the other half, and that layer benefits from precise types too. Model a timeline as ordered shots with explicit ranges and transitions, and constrain the values the encoder actually cares about.

type Transition = 'cut' | 'cross-dissolve' | 'whip-pan' | 'fade';

interface Shot {
  readonly startMs: number;
  readonly endMs: number;
  readonly assetId: AssetId;
  readonly transitionIn?: Transition;
}

function buildEncoderArgs(shots: Shot[], outPath: string): string[] {
  // Pure function: trivial to unit test, easy to log when a render misbehaves.
  return [];
}

Two habits pay off here. First, keep the argument builder pure and have it return an array of strings, so a snapshot test catches accidental encoder changes. Second, derive durations from a single source of truth. When the interface computes a clip length one way and the encoder computes it another, you get audio drift: invisible in the editor, obvious to every viewer.

If you want the interface itself to feel cinematic rather than mechanical, a shared vocabulary helps. Model shot size, camera move, and lighting as typed enums instead of free text, then let the prompt builder translate them into prose. Your schema becomes a creative tool: it prevents the incoherent combination where a wide establishing shot carries an intimate close-up lighting note.

When you are iterating on prompt-driven shots, a curated starting point beats inventing test cases from scratch. Browsing the video templates and the prompt library gives you concrete inputs to push through your typed schema, which is a fast way to discover a field you forgot, such as a camera move, a lighting note, or a seed value you meant to expose.

Testing an Async Pipeline Without Flakiness

Generated-video pipelines fail in ways unit tests alone will not catch, because most of their failures are about time and ordering rather than logic.

Capture real provider responses once, store them as fixtures, and replay them in contract tests. When a provider reshapes a response envelope overnight, the test fails in continuous integration instead of in front of a user. Fixtures also make parser changes reviewable, because the diff shows exactly which field moved.

Inject a fake clock and a fake queue into the worker, then assert ordering properties rather than timings. A job that fails twice and succeeds on the third attempt must produce exactly one asset and exactly one success event. That single test catches more duplicate-render bugs than any amount of manual clicking, and it never turns flaky on a busy build machine.

Generate one trace identifier at the HTTP edge, attach it to the queue payload, and forward it to provider calls as a correlation header. Structured logs keyed by that identifier turn a vague report about a hung render into a timeline you can read in ten seconds.

Finally, drill failures on purpose. Schedule an exercise in which a provider returns rate-limit responses, the storage bucket rejects writes, and a worker is killed mid-job. Each drill should end with written answers to two questions: what does the user see, and what recovers without human intervention? A pipeline that has rehearsed its worst day behaves very differently from one that has only ever seen happy traffic.

Keep the suite fast enough that people run it before pushing. A slow suite becomes a suite that runs only on the main branch, which is the same as not having it.

Mistakes That Turn a Demo Into an Incident

Most pain in generation backends comes from a short list of recurring decisions:

• Untyped values entering at the provider boundary and spreading inward through the rest of the codebase.
• Optional chaining used to silence missing data instead of validating it at the edge.
• Boolean flags such as isPreview and isFinal instead of a discriminated union, which lets impossible combinations compile.
• One sprawling types file that everybody edits and nobody owns.
• Provider response shapes left unvalidated, so a renamed field becomes a silent no-op rather than a loud failure.
• Progress events emitted without sequence numbers, producing a bar that moves backwards on reconnect.
• Retry logic with no idempotency key, which quietly doubles inference spend during an incident.
• Hard-coded provider selection, which forces a release every time a vendor changes its terms.
• Long-running work placed on a runtime with a short request timeout, which guarantees a mystery failure under load.
• Partial assets left behind after cancellations, which inflate storage until someone investigates.

The common thread is optimism. Each of these assumes that the happy path will be the only path, and video pipelines are exactly where that assumption fails, because they combine slow work, external providers, and impatient users in the same request.

FAQ

Should the whole backend be TypeScript?

For orchestration, queues, realtime, and the product API, yes, because the type system pays for itself quickly where correctness is mostly about shapes crossing boundaries. For training loops and research notebooks, a data-science language is still the better tool. Mature teams run both, joined by a versioned contract with validation on each side.

Do I need a heavy framework to get these benefits?

No. A minimal router with explicit wiring gives you the same boundaries and a smaller dependency surface. Frameworks mainly help by nudging teams toward modules that can be replaced independently.

How do I type a streaming model response?

Treat it as an async sequence of chunks and parse newline-delimited JSON inside the iterator, validating each chunk against a schema. Never assume a chunk boundary lines up with a message boundary; buffer partial lines or you will drop events exactly when traffic is highest.

What is the cleanest way to retire a provider?

Keep a registry where each entry declares capabilities, latency profile, and status, then route by capability rather than by name. Add a migration test that renders the same input through the old and new adapters so you can judge the visual difference before switching traffic.

Is edge inference realistic for video generation?

For light steps, yes. For sampling full clips, no. Design the interface so a step can move between runtimes without rewriting its callers, and you keep that option open without betting the architecture on it.

How large should a queue payload be?

Small. Prompts, references, and settings travel in the message; binaries live in object storage behind a URL. Large messages slow brokers and make every retry cost more than it should.

What if a provider changes its schema without warning?

Pin a fixture of the last known good response, parse with a schema that allows unknown keys, and log every unrecognised field. The first time a vendor renames something you want a log line, not a support ticket.

How do I keep spending predictable?

Validate inputs before enqueueing, cap duration and resolution per plan, deduplicate identical submissions, and record the estimated cost of every job next to its actual outcome. The gap between those two numbers is your real optimisation backlog.

Does any of this matter for a small team?

More than for a large one. A small team has no dedicated platform engineer, so the compiler has to do the guarding. Discriminated unions, schema parsing, and idempotency keys are cheap to add on day one and painful to retrofit after the first duplicate-render complaint.

How do I keep the schema from becoming bureaucracy?

Only type what crosses a boundary or appears in a support conversation. Internal helper functions can rely on inference and stay loose. If a type does not prevent a real bug or answer a real question, delete it.

Turn the Idea Into Motion With Orelon

Every pattern in this guide points at one discipline: make the slow, expensive, asynchronous parts of your product explicit in types, then let the compiler guard the seams. Do that and swapping a rendering backend, adding an aspect ratio, or introducing a preview pass stops being a project and becomes a small, reviewable change.

If you would rather spend your hours on story beats, camera language, and pacing, Orelon handles the generation layer for cinematic ideas in motion. Start with the AI video generator, shape a concept from a still made in the AI image generator, and iterate on shots until the sequence reads the way you imagined. Keep the Orelon blog open for workflow notes when you want a second opinion on pacing before you commit to a final render.