How splitting VOD archives cut costs and improved livestream reliability

When Shiftbloom Studio first launched its livestream archive system, it relied on a single, always-on recorder to capture every moment. While this approach guaranteed no missed content, it also generated unnecessary costs and complexity. Months into operation, the team realized that not all workloads required the same level of urgency.

Recognizing the inefficiency in monolithic recording

The core challenge wasn’t the lack of tools but the architecture itself. A continuous recorder consumes resources 24/7, even during idle periods. Backfill tasks—such as downloading VODs, re-encoding clips, or repairing corrupted files—can tolerate delays. Yet, they were forced to compete for the same infrastructure as time-sensitive live capture.

This mismatch led to bloated costs and operational friction. Critical live recordings required constant uptime, while background processing clogged the system with unnecessary latency. The need for a different approach became clear: separate the live from the backlog.

Designing a split-architecture for media workflows

The new system introduced two distinct roles, each tailored to its specific demands.

The control plane: A lightweight orchestrator

At the heart of the solution is a minimal control plane called the Mothership. This small cron job acts as the decision-maker, continuously monitoring queue sizes, active live channels, and observer cell status. Based on these inputs, it dynamically adjusts the number of harvest cells needed and assigns observer cells to channels requiring live recording.

The Mothership operates with a single source of truth: a shared database. This simplicity reduces overhead and ensures consistency across the entire system.

Observer cells: Specialized for live capture

Each observer cell is dedicated to recording just one livestream. It starts automatically when assigned a channel and begins writing HLS segments to object storage. The cell remains active for a brief standby window after the stream ends to capture any late reconnections, preventing fragmented VOD files.

Observer cells receive their assignments through environment variables:

OBSERVER_VOD_ID=12345
OBSERVER_CHANNEL_ID=67890
OBSERVER_CHANNEL_LOGIN=streamer_name
OBSERVER_CHANNEL_NAME="Creative Streamer"

This approach ensures minimal resource usage while guaranteeing that no critical moment is missed.

Harvest cells: Background processing at scale

Harvest cells handle all deferred tasks—downloading VODs, re-encoding files, recovering corrupted segments, and more. Unlike observer cells, they don’t need to run continuously. They can operate on burst capacity, a small VPS, or even a spare laptop, as long as they have outbound access to Postgres and object storage.

This flexibility allows the system to scale efficiently. During periods of high backlog, additional harvest cells can be spun up instantly without impacting live recording performance.

Measurable improvements after the architectural shift

The results were immediate and substantial. While the old system struggled with mixed workloads, the new split-architecture delivered measurable gains:

• Cost efficiency: No longer was the full recorder running unnecessarily during idle periods.
• Scalability: Harvest cells could be deployed on demand, accelerating backlog processing.
• Reliability: Live streams no longer competed with background tasks, reducing the risk of dropped frames.
• Storage clarity: Archived content could be easily segregated from active recordings, simplifying cold storage migration.

One of the most telling examples was the migration of 15.5 TB of historical backfill data. The entire process completed in just 36 hours without interrupting a single live stream. Under the previous monolithic setup, such a task would have been impossible without significant downtime.

A scalable pattern for modern media workflows

The key insight wasn’t about choosing the right tools but recognizing the fundamental difference between time-critical and throughput-based workloads. By drawing a clear line between these two types of work, Shiftbloom Studio transformed its infrastructure from a cumbersome monolith into a nimble, adaptable system.

This architectural shift reflects a broader trend in media technology: the move toward component-based, scalable systems that only activate when necessary. Whether for small studios or large platforms, treating live capture and backlog processing as separate entities can unlock significant efficiency gains.

For teams grappling with similar challenges, the lesson is clear: specialization breeds reliability. By separating live and background tasks, media workflows can achieve both cost savings and operational resilience—without sacrificing performance.