To See All: LiDAR-Guided Active Reconstruction via Online Calibration and Gimbal Planning To See All：基于在线标定与云台规划的激光引导主动三维重建
Overview 项目概述
A wide-field-of-view LiDAR observes scene geometry beyond the camera frustum, leaving many measured surfaces without color or texture. This project records those appearance gaps in the LiDAR map and actively points a gimbal camera toward them. The gimbal encoder supplies an immediate camera pose for planning; the Tri-Axis Principal-Alignment Network then corrects its residual 6-DoF error from the RGB image and projected LiDAR depth for accurate reconstruction. The MINCO planner maintains a voxelized appearance-coverage frontier, orders useful yaw–pitch targets as a motion-budgeted Orienteering Problem with arrival-time reward decay, and refines the result into a smooth trajectory. The accompanying COOKIE dataset provides 50,006 hardware-synchronized image–scan pairs from 20 sequences and 32 scenes. Evaluation separates calibration, observation coverage, colored-point-cloud quality and downstream 3D Gaussian Splatting quality.
大视场激光雷达能够观测相机视锥以外的场景几何，因此许多已测表面仍缺少颜色或纹理信息。本项目在激光地图中显式记录这些外观覆盖空缺，并主动控制云台相机补充图像观测。规划阶段直接采用云台编码器给出的即时相机位姿；重建阶段则由 Tri-Axis Principal-Alignment Network 融合 RGB 与投影深度，校正编码器外参中的 6-DoF 残差。MINCO 规划模块维护体素化外观覆盖前沿，将有效的偏航—俯仰视角排序建模为带运动预算和到达时间奖励衰减的 Orienteering Problem，并进一步优化为平滑轨迹。配套 COOKIE 数据集包含 20 条序列、32 个场景和 50,006 对硬件同步图像—扫描数据；评估分别考察标定精度、观测覆盖、彩色点云质量和下游 3D Gaussian Splatting 质量。
Key Contributions 主要贡献
- Online residual calibration. Principal-point-anchored horizontal, vertical and polar sampling exposes axis-specific misalignment cues; cross-modal attention and cascaded re-projection jointly refine the 6-DoF camera–LiDAR residual.
- Coverage-aware gimbal planning. The MINCO planner turns camera-unobserved LiDAR surfaces into a utility field, orders views under a motion budget and uses receding-horizon replanning as the carrier moves.
- Smooth, executable trajectories. MINCO-based refinement converts discrete yaw–pitch targets into continuous gimbal commands.
- Dataset and end-to-end evaluation. COOKIE contains 50,006 synchronized pairs across 32 scenes, and the evaluation traces improvements from calibration and image coverage through colored point clouds and 3DGS reconstruction.
- 在线残差标定。围绕图像主点进行水平、竖直和极坐标采样以提取轴向错位线索，再通过跨模态注意力与级联重投影联合细化 6-DoF 相机—激光外参残差。
- 覆盖感知的云台规划。MINCO 规划模块将尚未被相机观测的激光表面转化为效用场，在运动预算内排序视角，并随载体运动进行滚动重规划。
- 平滑可执行的轨迹。基于 MINCO 的连续优化将离散偏航—俯仰目标转化为可执行的云台指令。
- 数据集与端到端评估。COOKIE 覆盖 32 个场景、包含 50,006 对同步数据；评估从标定与图像覆盖一直延伸到彩色点云和 3DGS 重建质量。
Future Work 后续工作
MINCO planning selects viewpoints with an explicit utility function and then optimizes a smooth gimbal trajectory. In ongoing work, we are studying a learned alternative: we formulate gimbal control as a continuous-control problem and train a Soft Actor-Critic (SAC) policy. At each 0.1 s decision step, the policy outputs yaw and pitch angular accelerations and receives the number of newly colored surface voxels as its reward.
MINCO 规划先根据显式设计的效用函数选择视角，再优化平滑的云台轨迹。在后续工作中，我们尝试另一条思路：将云台控制建模为连续控制问题，并训练一个 Soft Actor-Critic（SAC）策略。策略每 0.1 秒输出偏航和俯仰两轴的角加速度，以该时间步内新增的着色体素数作为奖励。
To account for vehicle motion and gimbal response, the policy observes visibility-gain maps for the present and 1, 2 and 4 s into the future. For each candidate yaw–pitch direction, a map records how many currently known, uncolored voxels could be observed if that direction were maintained over the corresponding horizon. The state also includes the carrier’s predicted displacement and the remaining flight time. The replay below compares the SAC policy with MINCO planning on the same recorded 60 s flight. In this example, the SAC policy reaches 64.2% coverage with 2,468° of cumulative gimbal rotation; MINCO planning reaches 58.1% with 4,821°.
为使策略能够考虑载体运动和云台响应，观测中加入当前及未来 1、2、4 秒的可见性增益图。对每个偏航–俯仰候选方向，图中记录的是：若在相应时间范围内保持该方向，当前地图中有多少尚未着色的已知体素可被相机观测。状态还包含载体的预测位移和剩余航程。下面的可视化在同一段 60 秒录制航迹上回放 SAC 策略与 MINCO 规划。在这个案例中，SAC 策略以 2,468° 的累计云台转角达到 64.2% 的着色覆盖率；MINCO 规划则以 4,821° 达到 58.1%。
Coverage over time着色率随时间
Cumulative gimbal travel累计云台行程
Grey voxels have been mapped by LiDAR; blue voxels have also been observed by the camera. The time at which each voxel turns blue is read directly from the recorded pipeline output. The amber line is the flight path, the black arrow shows the vehicle heading, and the violet frustum shows the camera’s 81°×63° field of view up to the 15 m coloring range. The strip below records the gimbal direction relative to the vehicle heading. Both methods replay the same flight and the same 46,637 mapped voxels; only the gimbal commands differ. 灰色体素表示激光雷达已完成建图，蓝色体素表示该区域也被相机观测到。每个体素变蓝的时刻直接读取自系统回放结果。琥珀色曲线为飞行轨迹，黑色箭头表示机头朝向，紫色视锥表示相机 81°×63° 的视场及 15 米着色距离。下方条带记录云台相对机头的朝向。两种方法回放同一段飞行，使用相同的 46,637 个地图体素，区别仅在于云台控制指令。
- Observation design. Four visibility maps summarize the currently known, uncolored surfaces that can be observed over horizons of 0, 1, 2 and 4 s.
- Control and reward. The SAC policy maps the observation directly to two-axis angular acceleration, with newly colored voxels as the training reward.
- Gimbal motion in this replay. The SAC policy covers a 134° yaw range at an average of 22.8 °/s, compared with 248° and 58.8 °/s for MINCO planning; none of its samples reaches the rate limit.
- Coloring delay. The SAC policy colors 36.4% of its voxels within 1 s after they are mapped, compared with 22.4% for MINCO planning. The shares colored after 4 s are 15.4% and 33.3%, respectively.
- 观测设计。四张可见性图分别汇总当前及未来 1、2、4 秒内可被观测的已知未着色表面。
- 控制与奖励。SAC 策略根据观测直接输出两轴角加速度，并以新增着色体素数作为训练奖励。
- 该段回放中的云台运动。SAC 策略的偏航范围为 134°，平均偏航角速度为 22.8 °/s；MINCO 规划分别为 248° 和 58.8 °/s。SAC 策略没有出现达到角速度上限的采样。
- 着色延迟。SAC 策略有 36.4% 的体素在建图后 1 秒内完成着色，MINCO 规划为 22.4%；超过 4 秒才着色的比例则分别为 15.4% 和 33.3%。