# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

VIGS-SLAM is a Visual Inertial Gaussian Splatting SLAM system that takes RGB frames + IMU readings as input to produce camera trajectories and 3D Gaussian maps. It combines deep learning-based visual odometry (DroidNet) with classical geometric optimization and 3D Gaussian Splatting for reconstruction.

## Build & Setup Commands

```bash
# Clone and setup
git clone --recursive [repo_url]
git checkout release
git submodule update --init --recursive

# Create environment (RTX 5090 with CUDA 12.8 / PyTorch 2.8)
conda env create -f environment_5090.yaml
conda activate imugs_slam

# Install Python dependencies
pip install thirdparty/simple-knn
pip install thirdparty/diff-gaussian-rasterization
python -m pip install --no-binary=:all: -U sophuspy
pip install --no-binary=:all: --no-build-isolation --no-cache-dir -v torch-scatter

# Compile CUDA extensions (~10 minutes)
python setup.py install

# Download pretrained Omnidata models
wget https://zenodo.org/records/10447888/files/omnidata_dpt_normal_v2.ckpt -P pretrained_models
wget https://zenodo.org/records/10447888/files/omnidata_dpt_depth_v2.ckpt -P pretrained_models
```

## Running the System

```bash
# Single sequence demo
python demo.py \
  --imagedir /path/to/rgb/images \
  --calib calib/rpngar.txt \
  --config config/rpng.yaml \
  --imufile /path/to/imu.txt \
  --undistort \
  --IMU_poseinit_after 20 \
  --output outputs/demo \
  --gsmapping \
  [--gsvis] [--droidvis] [--rerunvis]

# Batch evaluation (edit dataset paths in scripts)
python eval_euroc_mono.py      # Euroc dataset
python eval_rpng_mono.py       # RPNG AR Table dataset
python eval_utmm_mono.py       # UTMM dataset
python eval_fastlivo_mono.py   # FAST-LIVO2 dataset

# Rendering evaluation
python scripts/eval_rendering.py

# Replay visualization
rerun outputs/demo/rerun_stream.rrd
```

## Architecture

### Core Components (vigs/)

- **vigs.py**: Main controller orchestrating all components via multiprocessing queues
- **depth_video.py**: Shared memory buffers for poses, depths, features, IMU states across processes
- **motion_filter.py**: Feature extraction (DroidNet encoders) + depth/normal priors (Omnidata)
- **track_frontend.py**: Local optimization window (25 frames), GRU-based iterative refinement, IMU initialization
- **track_backend.py**: Global loop closure and bundle adjustment
- **factor_graph.py**: Correlation volume matching, edge management, Schur complement solver
- **gs_backend.py**: 3D Gaussian Splatting training and rendering
- **pgo_buffer.py**: Pose graph optimization buffer
- **imu.py**: IMU pre-integration with bias Jacobians

### Neural Networks (vigs/modules/)

- **droid_net.py**: DroidNet with context encoder (`cnet`), feature encoder (`fnet`), and GRU update operator
- **trt_runner.py**: TensorRT inference wrapper for acceleration

### Geometry (vigs/geom/)

- **ba.py**: Bundle adjustment linear system construction
- **projective_ops.py**: Camera projection/unprojection operations
- **chol.py**: Choleski solver for optimization

### Depth Priors (vigs/midas/)

- **omnidata.py**: Omnidata depth/normal prediction (PyTorch)
- **omnidata_trt.py**: TensorRT-accelerated version

### Processing Flow

```
RGB + IMU → MotionFilter (features, priors) → TrackFrontend (local BA + IMU fusion)
                                                    ↓
                                              FactorGraph (optimization)
                                                    ↓
TrackBackend (loop closure) ← DepthVideo (shared state) → GSBackEnd (Gaussian mapping)
                                                    ↓
                                              Output: poses, depth maps, 3D Gaussians
```

## Configuration

Dataset configs in `config/` (rpng.yaml, euroc.yaml, fastlivo.yaml, utmm.yaml):

- **IMU section**: frequency, noise parameters, cam-to-IMU transform (Tcb_np)
- **Tracking.motion_filter**: keyframe creation thresholds
- **Tracking.frontend**: optimization window size, edge distance thresholds, `imu_late_init_from`
- **Tracking.backend**: loop closure thresholds
- **Training**: Gaussian update intervals, densification parameters
- **opt_params**: learning rates for pose, position, features, opacity, scaling

## GPU-Specific Build

setup.py auto-detects GPU:
- **RTX 5090**: Uses `src_5090/`, `src_bamf_5090/`, `thirdparty/lietorch_5090/` with compute_120
- **Other GPUs**: Uses `src/`, `src_bamf/`, `thirdparty/lietorch/` with compute_60-86

## Profiling

```bash
# Python line profiling
LINE_PROFILE=1 python demo.py [args]
python -m line_profiler -rtmz profile_output.lprof

# NVIDIA profiling
nsys profile -o report1 -t cuda,osrt python demo.py [args]
```

## Key Parameters

- `frontend_window`: Local optimization window size (default 25)
- `IMU_poseinit_after`: Number of keyframes before IMU initialization (typically 20-30)
- `keyframe_thresh`: Motion threshold for new keyframe creation
- `backend_thresh`: Distance threshold for loop closure edges
- Buffer size: 1500 keyframes by default
