[Question]: 目标点像素超出图像范围

[zhu-bingo]

Question

[09:56:28.088277] step_id: 38 output text: 4605 231
[09:56:28.089322] [WARN] pixel out of bounds: (u=4605, v=231), depth shape=(480, 640)
[09:56:28.089370] [WARN] invalid pixel_goal=[231, 4605], depth_shape=(480, 640), fallback to safe action
[09:56:28.089377] [WARN] fallback to STOP action

[yuqiang-yang]

请问输入图像是什么呢，在什么场景进行评测的。在我们的仿真和真机中，几乎碰不到这个情况。可以给一些示例图片

[zhu-bingo]

请问输入图像是什么呢，在什么场景进行评测的。在我们的仿真和真机中，几乎碰不到这个情况。可以给一些示例图片

运行的是habitat_dual_system_cfg.py，未启用flash_attn，完整报错如下：
[07:35:14.574003] sources 113 [{'from': 'human', 'value': "You are an autonomous navigation assistant. Your task is to Enter the dining room and walk to the other end. Go through the double doors on the right. Walk straight across the walkway and wait outside on the mat. Where should you go next to stay on track? Please output the next waypoint's coordinates in the image. Please output STOP when you have successfully completed the task. These are your historical observations: \n\n\n\n\n\n\n\n. in front of you is ."}, {'from': 'gpt', 'value': ''}]
[07:35:14.574052] step_id 113 messages: [{'role': 'user', 'content': [{'type': 'text', 'text': "You are an autonomous navigation assistant. Your task is to Enter the dining room and walk to the other end. Go through the double doors on the right. Walk straight across the walkway and wait outside on the mat. Where should you go next to stay on track? Please output the next waypoint's coordinates in the image. Please output STOP when you have successfully completed the task. These are your historical observations:"}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0291610>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0291E20>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D02C53D0>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D02AB700>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09C8700670>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09C8700A30>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0333550>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0333C70>}, {'type': 'text', 'text': '. in front of you is'}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0333A00>}, {'type': 'text', 'text': '.'}]}]
[07:35:15.981184] step_id: 113 output text: ↓
[07:35:15.981271] actions [5]
[07:35:16.055030] step_id 113 action 5
[07:35:16.161726] sources 113 [{'from': 'human', 'value': ' in your sight is .'}, {'from': 'gpt', 'value': ''}]
[07:35:16.161880] step_id 113 messages: [{'role': 'user', 'content': [{'type': 'text', 'text': "You are an autonomous navigation assistant. Your task is to Enter the dining room and walk to the other end. Go through the double doors on the right. Walk straight across the walkway and wait outside on the mat. Where should you go next to stay on track? Please output the next waypoint's coordinates in the image. Please output STOP when you have successfully completed the task. These are your historical observations:"}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0291610>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0291E20>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D02C53D0>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D02AB700>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09C8700670>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09C8700A30>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0333550>}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0333C70>}, {'type': 'text', 'text': '. in front of you is'}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=384x384 at 0x7F09D0333A00>}, {'type': 'text', 'text': '.'}]}, {'role': 'user', 'content': [{'type': 'text', 'text': 'in your sight is'}, {'type': 'image', 'image': <PIL.Image.Image image mode=RGB size=640x480 at 0x7F09D03336A0>}, {'type': 'text', 'text': '.'}]}]
[07:35:18.572539] step_id: 113 output text: 4605 278
[07:35:18.573034] [WARN] pixel out of bounds: (u=4605, v=278), depth shape=(480, 640)
[07:35:18.573056] [WARN] invalid pixel_goal=[278, 4605], depth_shape=(480, 640), fallback to safe action
Traceback (most recent call last):
File "/root/zzy/InternNav/scripts/eval/eval.py", line 48, in
main()
File "/root/zzy/InternNav/scripts/eval/eval.py", line 44, in main
evaluator.eval()
File "/root/zzy/InternNav/internnav/evaluator/distributed_base.py", line 78, in eval
local_metrics = self.eval_action() # dict[str, Tensor], each [N_local]
File "/root/zzy/InternNav/internnav/habitat_extensions/habitat_vln_evaluator.py", line 169, in eval_action
sucs, spls, oss, nes, _ = self._run_local_eval()
File "/root/zzy/InternNav/internnav/habitat_extensions/habitat_vln_evaluator.py", line 440, in _run_local_eval
return self.env.action_space.sample()
AttributeError: 'HabitatEnv' object has no attribute 'action_space'
Eval Epoch 0 Rank 0: 0%|▎ | 5/1822 [10:46<65:16:48, 129.34s/it]
[07:35:18:937092]:[Physics] PhysicsManager.cpp(54)::~PhysicsManager : Deconstructing PhysicsManager
[07:35:18:937210]:[Scene] SceneManager.h(21)::~SceneManager : Deconstructing SceneManager
[07:35:18:937218]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph
[07:35:18:937644]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor
[07:35:18:937672]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor
[07:35:18:937691]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph
[07:35:18:938622]:[Scene] SemanticScene.h(48)::~SemanticScene : Deconstructing SemanticScene
[07:35:18:958124]:[Gfx] Renderer.cpp(71)::~Impl : Deconstructing Renderer
[07:35:18:958224]:[Gfx] WindowlessContext.h(17)::~WindowlessContext : Deconstructing WindowlessContext
[07:35:18:966685]:[Sim] Simulator.cpp(69)::~Simulator : Deconstructing Simulator