Clarification regarding feature depth updates

[baliika]

Hello,
I'd like to use ICE-BA with a PNP-style monocular frontend, in which I have metric camera poses, and landmarks with known depth.

1. I feed the 1st keyframe to ICE-BA with the corresponding IMU measurements, 3D camera pose prior, 2D features and their 3D priors.The LBA optimized camera pose roughly remains the same (as expected), while the 3D landmarks try to minimize the re-projection error in the given frame, hence their 3D position change significantly. This is clear. I verified that the old re-projection errors are in the order of 10^-2 (on the frontend), while the ICE-BA new re-projection errors drop below 10^-5 (only for the frame inserted).
2. Then I go ahead, and add a 2nd frame (or more), let it be either a keyframe or a regular frame, having overlapping feature measurements with the 1st frame. What I notice is that the camera pose again roughly remains the same as the input (as expected), but the updated landmarks' 3D position result in large re-projection error.

How do you explain this latter behavior?