A non-generative Joint-Embedding Predictive Architecture learning the physical dynamics of the world from continuous video streams.
Traditional computer vision relies on static human labels or wastes computational capacity predicting noisy pixels. Prometheus abandons pixel reconstruction entirely. By learning purely in latent feature space across continuous video streams, it masters intuitive physics, geometric continuity, and spatial reasoning from scratch.
Predicting in latent space bypasses the high-frequency pixel noise that cripples generative models, focusing 100% of capacity on structural invariants.
Comparison: Generative pixel reconstruction (left) spends capacity on unpredictable detail. Prometheus JEPA (right) optimizes for semantic physical representations.
Every component of Prometheus is designed for real-time spatiotemporal comprehension and zero-collapse stability.
Unified Conv2D (16x16) and Conv3D (2x16x16) tubelet embeddings with dedicated learnable modality tokens.
3-axis Rotary Position Embeddings allocate head dimensions across Depth (time), Height, and Width with GPT-J interleaved phases.
Constrains latent distributions to an isotropic Gaussian via sliced Epps-Pulley test, eliminating dimensional collapse without heuristics.
Video volumes are transformed into 3D tubelet tokens and augmented with rotary phases along all spatial and temporal dimensions.
Modality-specific projection pipelines for single images (Conv2D) and multi-frame videos (Conv3D) producing uniform transformer tokens.
Decomposing attention head dimensions into orthogonal temporal depth, height, and width subspaces enables seamless resolution generalization.
Interactive explorer: Manipulate temporal depth and spatial coordinates to inspect live 3D phase rotation and transformation matrices.
Hierarchical short-range and long-range block masking forces the model to predict complex trajectories across time rather than interpolating local pixels.
Interactive simulator: Adjust masking ratio and compare 3D continuous tubelet occlusion with standard 2D independent random noise.
The student encoder processes visible context tokens while the EMA teacher produces stable target embeddings across the full video volume.
Complete end-to-end dataflow: Student encoder, Predictor with mask tokens, EMA Teacher target, Dense Predictive Loss, and SIGReg.
Visible context tokens are supervised with a square-root Euclidean distance weight to the nearest masked patch, maintaining local context fidelity.
Mathematical formulation of the Dense Predictive Loss and context distance curve w_j = sqrt(d_min).
By testing sliced 1D projections against an isotropic Gaussian distribution, SIGReg prevents dimensional collapse without stop-gradients or BatchNorm.
Interactive inspector: Adjust regularization weight λ to observe the phase transition from representation collapse to an isotropic Gaussian N(0, I).
Muon orthogonalized momentum updates combined with a Warmup-Stable-Decay schedule accelerate convergence and enable maximal update parameter scaling (muP).
Pretraining convergence curves comparing Muon + WSD against standard AdamW with cosine annealing.
Prometheus seamlessly powers real-time detection by fusing single-scale ViT tokens with a lightweight convolutional detail stem into multi-scale feature pyramids (C3, C4, C5).
Spatial Tuning Adapter (STA) multi-scale pyramid integration feeding the D-FINE NMS-free detector in Argus.
Autoregressively predicts future physical states from actions for robotic trajectory planning.
Block-causal action-conditioned predictor architecture for robotic trajectory simulation and real-time MPC control.
Key architectural parameters, pretraining configuration, and inference profile.