Why are interpolated video_paths of HyperNeRF data not utilized?

[cmh1027]

    def generate_video_path(self):
        self.select_video_cams = [item for i, item in enumerate(self.all_cam_params) if i % 1 == 0 ]
        self.video_path, self.video_time = smooth_camera_poses(self.select_video_cams,10)
        self.video_path = self.video_path[:500]
        self.video_time = self.video_time[:500]

    def load_video(self, idx):
        if idx in self.map.keys():
            return self.map[idx]
        camera = self.all_cam_params[idx]
        w = self.image_one.size[0]
        h = self.image_one.size[1]
        time = self.video_time[idx]
        R = camera.orientation.T
        T = - camera.position @ R
        FovY = focal2fov(camera.focal_length, self.h)
        FovX = focal2fov(camera.focal_length, self.w)
        image_path = "/".join(self.all_img[idx].split("/")[:-1])
        image_name = self.all_img[idx].split("/")[-1]
        caminfo = CameraInfo(uid=idx, R=R, T=T, FovY=FovY, FovX=FovX, image=self.image_one_torch,
                              image_path=image_path, image_name=image_name, width=w, height=h, time=time, mask=None
                              )
        self.map[idx] = caminfo
        return caminfo

These are functions in hyper_loader.py and it seems that only interpolated timestamps are used for video and interpolated poses are discarded. Could you explain me about the reason for it?

[guanjunwu]

you can also use it，but rendered video seems not good :(

[cmh1027]

@guanjunwu
I'm also curious about implementation of function smooth_camera_poses

def smooth_camera_poses(cameras, num_interpolations=5):
    """对一系列相机位姿进行平滑处理，通过在每对位姿之间插入额外的位姿"""
    smoothed_cameras = []
    smoothed_times = []
    total_poses = len(cameras) - 1 + (len(cameras) - 1) * num_interpolations
    time_increment = 10 / total_poses

    for i in range(len(cameras) - 1):
        cam1 = cameras[i]
        cam2 = cameras[i + 1]

        # 将旋转矩阵转换为四元数
        quat1 = rotation_matrix_to_quaternion(cam1.orientation)
        quat2 = rotation_matrix_to_quaternion(cam2.orientation)

        for j in range(num_interpolations + 1):
            t = j / (num_interpolations + 1)

            # 插值方向
            interp_orientation_quat = quaternion_slerp(quat1, quat2, t)
            interp_orientation_matrix = quaternion_to_rotation_matrix(interp_orientation_quat)

            # 插值位置
            interp_position = linear_interpolation(cam1.position, cam2.position, t)

            # 计算插值时间戳
            interp_time = i*10 / (len(cameras) - 1) + time_increment * j

            # 添加新的相机位姿和时间戳
            newcam = deepcopy(cam1)
            newcam.orientation = interp_orientation_matrix
            newcam.position = interp_position
            smoothed_cameras.append(newcam)
            smoothed_times.append(interp_time)

    # 添加最后一个原始位姿和时间戳
    smoothed_cameras.append(cameras[-1])
    smoothed_times.append(1.0)
    return smoothed_cameras, smoothed_times

In the line "interp_time = i*10 / (len(cameras) - 1) + time_increment * j", 10 looks like num_interpolation which is hard-coded. But why is that necessary? Time should be in the range of 0~1 so it should be ""interp_time = i / (len(cameras) - 1) + time_increment * j" without 10

[guanjunwu]

Hi, initially, I want to interpolate between poses and and timestamp between poses, so I write the code, but now there maybe a problem as you said (in fact you are right). If you can help me to fix the bug and submit a New PR, I'll merge your PR.