Tracking and identity locking

The pipeline combines person detection and face recognition to keep the crop locked onto a specific subject rather than just the most visible person in frame.

• YOLOv8-Nano detects person bounding boxes efficiently through ONNX Runtime
• ArcFace compares cropped face embeddings against the provided reference image using cosine similarity
• the configured --threshold value is used consistently across both CLI and GUI workflows
• tracking includes relock bias from the last known position to improve recovery after brief occlusion
• identity checks are throttled adaptively once lock-on is established to reduce unnecessary compute

This makes the tracker more stable in crowded performance footage where multiple people may appear and disappear across frames.

Identity discovery pass (GUI)

The desktop app includes a pre-tracking discovery flow designed to make target selection more reliable before rendering begins.

• sampled frames are scanned first to propose likely identity candidates
• the UI can accept an expected member count and trigger a smarter rescan if duplicates or count mismatches are detected
• users can manually review candidates before rendering:
  • exclude false positives
  • resolve duplicates
  • confirm low-confidence matches
• once validated, the selected member card is used as an additional tracking prior alongside the reference image

The Tauri backend persists scan sessions and validates review state server-side before allowing a render to begin.

Scan session lifecycle

To make the review and render flow more robust, scan sessions track explicit lifecycle states:

• proposed
• validated
• tracking
• completed
• failed

Audit events are recorded through the session lifecycle, and run_fancam enforces that a validated session and selected identity match exist on the backend side, not just in the UI.

The GUI also supports manual split requests per identity, with a split-rescan path that refreshes candidate clustering when the initial grouping is not clean enough.

Smoothing and motion stability

To avoid shaky or jumpy crops, the render path uses a 2D Kalman filter to smooth subject motion across frames.

This helps with:

• reducing abrupt camera jumps
• keeping the framing more natural
• preserving momentum when the target is briefly lost
• simulating the feel of a human camera operator rather than a raw detector box snap

If the subject becomes occluded, the filter predicts the next likely position based on previous motion until visual confirmation is regained.

Performance pipeline

The core processing path is built around a 3-thread decode / inference / encode pipeline with bounded channels.

Performance-oriented behavior includes:

• recognition throttling before and after lock-on to reduce CPU stalls
• capping ArcFace checks to top-confidence person candidates per frame
• detection downscale for faster large-video processing
• parallel tensor preparation where possible
• fast SIMD face preprocessing
• render buffer and resizer reuse to reduce per-frame allocations
• periodic per-stage timing logs for detect, identify, and render

These optimizations are aimed at keeping the pipeline responsive and practical for longer videos without turning the whole thing into a heater-core cosplay.

Rendering behavior

Rendering is optimized for vertical fancam output while remaining resilient when tracking quality changes.

• automatic 1080x1920 framing for vertical output
• SIMD-accelerated resize path with fast_image_resize
• Lanczos3 upscaling when the subject is distant in frame
• fallback letterboxing when the target is lost or visibility drops too far

This keeps output usable even when the tracker cannot confidently maintain a tight crop for every frame.

Interfaces

The project supports two main usage paths:

Desktop app

The Tauri desktop application is intended for interactive use:

• scan identities visually
• validate candidates
• select a target
• run render jobs without touching the command line

CLI

The CLI is better suited for:

• direct runs
• repeated experiments
• scripting and batch workflows
• debugging model thresholds and pipeline behavior

Cross-platform scope

The project is designed to run across Windows, macOS, and Linux, with a shared Rust processing core and a Tauri-based desktop frontend.