Nucleus: A lightweight, Nix-native runtime for secure AI workloads

The push for simpler, more secure container runtimes has led to Nucleus, a Rust-based tool designed to run ephemeral AI workloads and NixOS services with hardened isolation. Unlike Docker, Nucleus drops image management entirely, opting for a single-binary approach that emphasizes reproducibility and security over distribution. Its target users aren’t those seeking a Docker replacement, but rather teams needing auditable, fail-closed environments for untrusted or short-lived processes.

Built for isolation: no layers, no registries, just control

Nucleus abandons Docker’s image layers, registries, and persistent storage in favor of two distinct modes. In agent mode, workloads run from a tmpfs-backed directory, ideal for ephemeral AI agents. Production mode mounts a Nix-built closure read-only, ensuring deterministic deployments. The runtime enforces deep isolation through a multi-layered security model:

• Capability stripping: All Linux capabilities are dropped by default, with only ~100 syscalls allowed via seccomp (compared to Docker’s ~300).
• Namespace lockdown: Up to 8 namespaces, including time and cgroup isolation, are enabled per service.
• Path-level controls: Landlock Linux Security Module (LSM) applies deny-by-default file access policies, configurable via TOML files.
• Network segregation: Outbound traffic is blocked unless explicitly permitted by CIDR ranges or DNS-resolved domains, enforced via namespace-local iptables rules.

These defaults target defense-in-depth, reducing attack surfaces while maintaining flexibility for controlled exceptions.

Security policies as code: verifiable and auditable

Nucleus externalizes security policies into separate, hash-pinned files, decoupling them from root filesystem builds. This design allows teams to:

• Generate minimal seccomp profiles dynamically using nucleus seccomp generate, which traces syscalls in a zero-trust environment.
• Define capabilities and Landlock ACLs in TOML, enabling version-controlled, reviewable security policies.
• Verify rootfs integrity with SHA-256 manifests, checked at startup via attestation.

The production path leverages Nix’s declarative ecosystem. The nucleus.lib.mkRootfs function builds locked-down closures, while a NixOS module integrates Nucleus into systemd-managed services. This Nix-native approach ensures reproducible environments and simplifies auditing for compliance-driven workflows.

gVisor integration and formal verification

Nucleus embeds gVisor as a first-class runtime, offering additional isolation layers without the complexity of standalone setups. Unique network modes, including a gVisor-host separation mode, provide fine-grained control over host interactions. The team has also invested in formal methods:

• TLA+ specifications model isolation, resource limits, filesystem access, and security subsystems, checked with the Apalache model checker.
• Property-based testing validates the Rust implementation against these specifications, reducing the risk of implementation drift.

Cold-start latency is a standout feature, with benchmarks showing ~12ms in native runtime. Performance tests with PostgreSQL 18’s pgbench indicate results within noise levels compared to bare metal, suggesting minimal overhead for production workloads.

Tradeoffs and practical considerations

Nucleus isn’t a universal replacement for Docker. Its deliberate limitations reflect its design goals:

• Platform lock: Linux x86_64 only, with no plans to support macOS, Windows, or BSD.
• Single-host focus: No CNI, overlay networks, or cluster orchestration. The nucleus compose tool uses a TOML-based DAG over systemd, not Kubernetes or Swarm.
• Storage philosophy: Ephemeral-by-default; persistence requires explicit --volume binds.
• Agent mode flexibility: Security mechanisms in agent mode are applied best-effort, with warnings on failures. For strict isolation, the --service-mode strict-agent flag enforces fail-closed policies, while production mode is recommended for long-running services.

For teams prioritizing security and reproducibility over Docker’s ecosystem, Nucleus offers a compelling alternative. Its minimalist design and Nix-native approach align well with modern DevSecOps practices, particularly in AI and research environments where auditability is critical.

Looking ahead, the runtime’s emphasis on formal verification and policy-as-code could set a new standard for secure containerization. As AI workloads grow more complex, tools like Nucleus may become essential for balancing performance with security in heterogeneous environments.