<!-- PROJECT LOGO -->

<p align="center">

  <h1 align="center">VIGS-SLAM: Visual Inertial Gaussian Splatting SLAM</h1>
  <p align="center">
    <a href="https://zzh2000.github.io"><strong>Zihan Zhu</strong></a>
    ·
    <a href="https://www.ifp.uni-stuttgart.de/en/institute/team/Zhang-00004/"><strong>Wei Zhang</strong></a>
    ·
    <a href="https://moyangli00.github.io/"><strong>Moyang Li</strong></a>
    ·
    <a href="https://www.ifp.uni-stuttgart.de/institut/team/Haala-00001/"><strong>Norbert Haala</strong></a>
    ·
    <a href="https://people.inf.ethz.ch/pomarc/"><strong>Marc Pollefeys</strong></a>
    ·
    <a href="https://cvg.ethz.ch/team/Dr-Daniel-Bela-Barath"><strong>Daniel Barath</strong></a>
</p>
<p align="center"> <strong>arXiv 2025</strong></p>
  <h3 align="center"><a href="https://arxiv.org/abs/2512.02293">Paper</a> | <a href="https://www.youtube.com/watch?v=vj4oCx0AU_U">Video</a> | <a href="https://www.youtube.com/watch?v=djl4hZuGdiE">Real-Time Demo</a> | <a href="https://vigs-slam.github.io/">Project Page</a></h3>
  <div align="center"></div>
</p>
<p align="center">
    <img src="./media/teaser.jpg" alt="teaser_image" width="100%">
</p>

<p align="center">
Given a sequence of RGB frames and IMU readings, our VIGS-SLAM robustly tracks the camera trajectory while reconstructing a high-fidelity 3D Gaussian map.
</p>
<br>

<!-- TABLE OF CONTENTS -->
<details open="open" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'>
  <summary>Table of Contents</summary>
  <ol>
    <li>
      <a href="#getting-started">Getting Started</a>
    </li>
    <li>
      <a href="#run-demo">Run Demo</a>
    </li>
    <li>
      <a href="#iphone-capture--run">iPhone Capture & Run</a>
    </li>
    <li>
      <a href="#dataset-preparation">Dataset Preparation</a>
    </li>
    <li>
      <a href="#batched-run--tracking-evaluation">Batched Run & Tracking Evaluation</a>
    </li>
    <li>
      <a href="#rendering-evaluation">Rendering Evaluation</a>
    </li>
  </ol>
</details>


## Getting Started
NOTE: A Docker version is in progress — please stay tuned.

1. Clone the repo with submodules
```bash
git clone --recursive XXXXX
git checkout release
git submodule update --init --recursive
```

2. Create a new Conda environment and then activate it. Here we provide the environment for RTX 5090. Please note that we use the PyTorch version 2.8 compiled by CUDA 12.8 in the `environment_5090.yaml` file.
```bash
conda env create -f environment_5090.yaml
conda activate vigs-slam-5090
pip install thirdparty/simple-knn
pip install thirdparty/diff-gaussian-rasterization
python -m pip install --no-binary=:all: -U sophuspy
pip install --no-binary=:all: --no-build-isolation --no-cache-dir -v torch-scatter
```

3. Compile the CUDA kernel extensions (takes about 10 minutes). Please note that this process assume you have CUDA 12.8 installed. To look into the installed CUDA version, you can run `nvcc --version` in the terminal.
```bash
python setup.py install
```

4. Download the pretrained weights of Omnidata models for generating depth and normal priors
```bash
wget https://zenodo.org/records/10447888/files/omnidata_dpt_normal_v2.ckpt -P pretrained_models
wget https://zenodo.org/records/10447888/files/omnidata_dpt_depth_v2.ckpt -P pretrained_models
```

5. (Optional) Build IMU C++ module

The IMU C++ module (`vigs/imu_cpp/`) significantly accelerates IMU factor computation compared to the pure Python implementation. If not built, the system falls back to pure Python automatically.

<details>
<summary><b>Prerequisites & Build Instructions</b></summary>

Install the following system packages if not already present:
```bash
# C++ compiler and CMake build system
sudo apt update
sudo apt install build-essential cmake

# Eigen3 linear algebra library (header-only)
sudo apt install libeigen3-dev

# OpenMP runtime (usually bundled with GCC; install explicitly if using Clang)
sudo apt install libomp-dev
```

> **Note:** `pybind11` and `Sophus` are provided by the conda environment set up in step 2 — no separate installation needed. Make sure the environment is activated before running `cmake`.

```bash
cd vigs/imu_cpp
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
```

The compiled module `imu_integrator_cpp*.so` will be placed in `vigs/imu_cpp/build/`. VIGS-SLAM auto-detects and loads it at runtime — no config change needed.

</details>

6. (Optional) TensorRT Acceleration

TensorRT accelerates neural network inference by optimizing and compiling models into GPU-specific execution engines, delivering significantly lower latency than standard PyTorch.

<details>
<summary><b>Prerequisites & Build Instructions</b></summary>

Since TensorRT engines are device-specific, they must be built on each machine.
Install TensorRT following the [official NVIDIA installation guide](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html). Make sure the version is compatible with your CUDA and PyTorch versions. We use **TensorRT 10.13.0.35** with CUDA 12.8 on the RTX 5090.

Once installed, first export the pretrained Omnidata models to ONNX, then convert them to TensorRT `.engine` files:

**Step 1 — Export to ONNX:**
```bash
python scripts/export_omnidata.py
python scripts/export_droidnet.py
```

**Step 2 — Simplify ONNX graphs** (requires `pip install onnxsim`):
```bash
cd pretrained_models
python -m onnxsim omnidata_depth_512.onnx  omnidata_depth_512_simplified.onnx  --skip-fuse-bn
python -m onnxsim omnidata_normal_512.onnx omnidata_normal_512_simplified.onnx --skip-fuse-bn
```

**Step 3 — Build TensorRT FP16 engines:**

- `--memPoolSize=workspace:N` — scratch memory (MB) TensorRT can use during engine build and inference. Increase if build fails; decrease if GPU memory is limited.
- `--minShapes` / `--optShapes` / `--maxShapes` — define the range of input shapes for dynamic engines. TensorRT optimizes for `optShapes` and supports any shape within `[min, max]`. Set these based on the actual input shapes your sequences will produce — too wide a range reduces optimization quality.

```bash
cd pretrained_models
# Omnidata depth & normal
trtexec --onnx=omnidata_depth_512_simplified.onnx \
        --saveEngine=omnidata_depth_512_simplified_fp16.engine \
        --memPoolSize=workspace:24576 --fp16

trtexec --onnx=omnidata_normal_512_simplified.onnx \
        --saveEngine=omnidata_normal_512_simplified_fp16.engine \
        --memPoolSize=workspace:24576 --fp16

# DroidNet feature encoder (dynamic resolution)
trtexec --onnx=droidnet_fnet.onnx \
        --saveEngine=droidnet_fnet_fp16.engine \
        --memPoolSize=workspace:24576 --fp16 \
        --minShapes=input:1x1x3x328x328 \
        --optShapes=input:1x1x3x368x584 \
        --maxShapes=input:1x1x3x656x656

# DroidNet update module — frontend window
trtexec --onnx=update_module_partial.onnx \
        --saveEngine=update_module_partial_fp16.engine \
        --memPoolSize=workspace:24576 --fp16 \
        --minShapes=net:1x1x128x41x41,inp:1x1x128x41x41,corr:1x1x196x41x41,flow:1x1x4x41x41 \
        --optShapes=net:1x24x128x43x77,inp:1x24x128x43x77,corr:1x24x196x43x77,flow:1x24x4x43x77 \
        --maxShapes=net:1x60x128x82x82,inp:1x60x128x82x82,corr:1x60x196x82x82,flow:1x60x4x82x82

# DroidNet update module — backend / PGBA / final BA (larger window)
trtexec --onnx=update_module_partial.onnx \
        --saveEngine=update_module_partial_pgba_fp16.engine \
        --memPoolSize=workspace:24576 --fp16 \
        --minShapes=net:1x1x128x41x41,inp:1x1x128x41x41,corr:1x1x196x41x41,flow:1x1x4x41x41 \
        --optShapes=net:1x85x128x43x77,inp:1x85x128x43x77,corr:1x85x196x43x77,flow:1x85x4x43x77 \
        --maxShapes=net:1x120x128x82x82,inp:1x120x128x82x82,corr:1x120x196x82x82,flow:1x120x4x82x82
```

The generated `.engine` files will be placed in `pretrained_models/`. VIGS-SLAM auto-detects them at runtime and falls back to PyTorch if they are not found.

</details>

## Run Demo

The demo uses the first 1000 frames of the RPNG AR Table (`table_01`) sequence. Results are saved to `outputs/demo/`, including estimated camera poses, the Gaussian map, and renderings. Optional flags: `--gsvis` to watch the Gaussian map build in real time, `--droidvis` to inspect intermediate depth and point cloud estimates.

First, download the demo data:

```bash
bash scripts/prep_demo.sh
```

Then run:

```bash
python demo.py \
--imagedir data/demo/rgb \
--calib calib/rpngar.txt \
--config config/rpng.yaml \
--imufile data/demo/imu.txt \
--undistort \
--IMU_poseinit_after 20 \
--output outputs/demo \
[--gsmapping]    # Optional: enable Gaussian mapping
[--pure_online]  # Optional: online-only mode — skips final BA, trajectory filling, and Gaussian map refinement
[--input_vis]    # Optional: show input image in a cv2 window
[--gsvis]        # Optional: enable Gaussian map display (OpenGL window)
[--droidvis]     # Optional: enable point cloud display (OpenGL window)
[--rerunvis]     # Optional: live visualization in the Rerun web viewer
[--rerun_record] # Optional: save a .rrd file for later replay with Rerun
```

> **Note:** `--rerunvis` and `--rerun_record` are mutually exclusive — if both are given, live streaming takes priority and no `.rrd` file is saved.

**Live visualization with Rerun** (`--rerunvis`): once the run starts, open the printed URL in your browser (e.g. `http://127.0.0.1:9876?url=...`). If running on a remote server, forward both ports via SSH first:
```bash
ssh -L 9876:127.0.0.1:9876 -L 9877:127.0.0.1:9877 <your_server>
```

**Replay a saved Rerun recording** (`--rerun_record`):
```bash
rerun outputs/demo/rerun_stream.rrd
```


## iPhone Capture & Run

### 1. Install the iOS recorder app

A custom iOS recorder app is required to stream RGB frames and IMU data to the capture server. The source code and a short installation instruction video are available at the [PhoneStreamer](https://github.com/MoyangLi00/PhoneStreamer) repository — see its README for installation instructions.

### 2. Capture a session

Make sure your iPhone and your machine are on the **same local network**. Find your machine's local IP:

```bash
ifconfig
```

Enter that IP in the iOS app. The port is preset to `9999` on both sides.

Then start the capture server on your machine:

```bash
python scripts/stream_data.py
```

In the iOS app, tap **Start**. The first ~1 second is used for auto-focus and locking the intrinsics, after which the app begins streaming RGB frames and IMU data. Tap **Stop** when done.

This saves a timestamped session folder under `data/iphone/`, containing:
- `images/` — JPEG frames named by timestamp (nanoseconds)
- `imu.csv` — synchronized IMU data
- `calib.txt` — camera intrinsics (`fx fy cx cy`)
- `camera.json` — full camera metadata

### 3. Run Sequence

Open `run_iphone_mono.py` and add your session folder name to `SEQS`:

```python
SEQS = [
    "20260309_170728",   # replace with your session folder name
]
```

Also review the other options at the top of the file (`DATA_ROOT`, `output_folder`, `gsmapping`, etc.), then run:

```bash
python run_iphone_mono.py
```

Results (trajectory, Gaussian map) will be saved under `outputs/output_iphone/`.

### Alternative: Live SLAM Streaming

Instead of the two-step capture-then-run flow above, `demo_stream.py` runs VIGS-SLAM **in real time** as the iPhone streams. No data is saved to disk first — SLAM starts immediately after the first frame arrives.

Make sure your iPhone and machine are on the same local network, then run:

```bash
python demo_stream.py \
--config config/rpng.yaml \
--output outputs/stream \
--IMU_poseinit_after 25 \
[--gsmapping]         # Optional: enable Gaussian mapping
[--gsvis]             # Optional: Gaussian map display (OpenGL window)
[--droidvis]          # Optional: point cloud display (OpenGL window)
[--rerunvis]          # Optional: live visualization in the Rerun web viewer
```

Open the iOS app, enter your machine's local IP and port `9999`, then tap **Start**. SLAM begins automatically once intrinsics and an initial IMU backlog are received. Tap **Stop** to end the session.

When the iPhone disconnects, the system runs a final bundle adjustment and saves:
- `traj_kf_beforeBA.txt` — keyframe trajectory before final BA
- `traj_kf_afterBA.txt` — keyframe trajectory after final BA
- `3dgs_before_final.ply` / `3dgs_final.ply` — Gaussian map before and after final BA
- `intrinsics.npy` — scaled camera intrinsics used during the session

> **Note:** Unlike the offline pipeline, `demo_stream.py` receives camera intrinsics directly from the iPhone — no `--calib` file is needed.

---


## Dataset Preparation
> **Note:** By downloading any of the following datasets, you agree to the terms and conditions set by the respective dataset providers. Please review each dataset's license before use.
### Euroc Dataset
Download the dataset from the [Euroc](https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets) website. No preprocessing is needed.

### RPNG AR Table Dataset
Download the processed [RPNG AR Table Dataset](https://github.com/rpng/ar_table_dataset) using the script below
```bash
bash scripts/prep_rpng.sh
```

### UTMM Dataset
Download the processed [UTMM](https://huggingface.co/datasets/neel1302/UT-MM/tree/main) dataset using the script below
```bash
bash scripts/prep_utmm.sh
```

### FAST-LIVO2 Dataset
Download the processed FAST-LIVO2 dataset using the script below
```bash
bash scripts/prep_livo2.sh
```

### VIGS-SLAM Dataset
Download our self-captured VIGS-SLAM dataset using the script below
```bash
bash scripts/prep_vigs.sh
```



## Batched Run & Tracking Evaluation

> Each script runs all sequences in batch. Set the dataset paths, output directories, and whether to enable Gaussian Mapping at the top of each file before running.
>
> Due to massive refactoring, cleaning, and optimization, results may not be exactly the same as in the paper, but are generally better. Differences across environments and GPU models have also been observed.

### Euroc

> **Note:** Place all sequences directly under a single base folder (e.g. `data/euroc/MH_01_easy/`, `data/euroc/V1_01_easy/`). Do **not** nest them in sub-categories such as `Machine Hall/` or `Vicon Room 1, Vicon Room 2/` — the evaluation script expects a flat layout.

```bash
python eval_euroc_mono.py
```

### RPNG AR Table Dataset
```bash
python eval_rpng_mono.py
```

### UTMM Dataset
```bash
python eval_utmm_mono.py
```

### FAST-LIVO2 Dataset
```bash
python eval_livo2_mono.py
```

### VIGS-SLAM Dataset
```bash
python eval_vigs_mono.py
```
