ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long

[carmeloarci]

It is not possible to complete a simulation of a star cluser using SCOPESIM since python wants to save a string that is too long to be handled in a fits header keyword. It is not possible to save the fits image file.

In the example, we used 5 randomly placed stars using the scopesim_templates.stars tool. Specifically MICADO and MORFEO irdb were considered.

Attached the python script to reproduce the error
MORFEO_debug2.py.txt

Traceback (most recent call last):
File "/python/simcado/MORFEO_debug2.py", line 100, in
hdus_J_small_test_3DM = micado.readout(filename= "scopesim_debug.fits")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/utils.py", line 1053, in wrapper
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/optics/optical_train.py", line 328, in readout
hdul.writeto(fname, overwrite=True)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1006, in writeto
self.verify(option=output_verify)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/verify.py", line 82, in verify
errs = self._verify(opt)
^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1401, in _verify
result = hdu._verify(option)
^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 1162, in _verify
errs = super()._verify(option=option)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 599, in _verify
return super()._verify(option)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 1099, in _verify
errs.append(card._verify(option))
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1166, in _verify
keyword, valuecomment = self._split()
^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 838, in _split
image = self.image
^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 530, in image
self._image = self._format_image()
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1025, in _format_image
raise ValueError(
ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long

[hugobuddel]

Thanks. You can workaround this problem by disabling the source fits keywords:

micado["source_fits_keywords"].include = False

Nevertheless we should fix this. Unfortunately, I cannot reproduce your exact error. The file produced by your script for small n does not contain the SIM SRC0 amplitudes keyword if I run it.

Could you please share the output of scopesim.bug_report() with us; it should be right above the Traceback?

However, increasing n from 5 to 6 will lead to a related error:

ValueError: FITS header values must contain standard printable ASCII characters; "scopesim_templates.stellar.stars.stars(filter_name='J', amplitudes=<Quantity [18.46960607, 21.22000502, 16.16989011, 19.55818477, 19.90220094,\n           15.61757236] mag>, spec_types=['A0V', 'A0V', 'A0V', 'A0V', 'A0V', 'A0V'], x=array([0.43299294, 0.81188355, 0.86067838, 0.12246919, 0.35946662,\n       0.45763185]), y=array([0.89122441, 0.36107683, 0.12356588, 0.70669564, 0.16268664,\n       0.86055453]), library='pyckles', ra=0, dec=0)" contains characters not representable in ASCII or non-printable characters.

The error above is caused by the key SIM SRC0 function_call. We explicitly 'prevent' the error I got by this line in fits_headers.py:

hdu.header[f"FN{key.split()[1]}"] = hdu.header.pop(key)

for that key. That line will convert the card from a HIERARCH card to a normal one, which supports the CONTINUE keyword. And indeed for n=5 this key indeed does not appear in the headers. But this code is not reached for n=6 because the error occurs earlier.

We would need to fix that 'hack', but once fixed, we could perhaps employ something similar for the other keywords that can reach an arbitrary length like SIM SRC0 amplitudes.

For completeness, this is the output of scopesim.bug_report() for me:

Python:
3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0]

Installed Python packages:
           anisocado: 0.3.0
             astropy: 5.3.3
      beautifulsoup4: 4.12.2
            docutils: 0.20.1
                lxml: 4.9.3
          matplotlib: 3.7.2
      more-itertools: 10.1.0
               numpy: 1.25.2
               pooch: 1.7.0
              pyyaml: 6.0.1
            requests: 2.31.0
      requests-cache: 1.1.0
               scipy: 1.11.2
       scopesim_data: could not be loaded.
  scopesim_templates: 0.4.5a0
         skycalc-ipy: 0.3.0a0
             synphot: 1.2.1
                tqdm: 4.66.1

Installed IRDB packages:
                .git: version number not available.
             .github: version number not available.
           Armazones: 2023-07-11
                 ELT: 2023-07-11
                ERIS: version number not available.
                FORS: version number not available.
                 GTC: version number not available.
             HARMONI: version number not available.
               HAWKI: 2023-06-14
                 HST: version number not available.
                KMOS: version number not available.
                LFOA: 2022-04-12
             LaPalma: version number not available.
               MCIFU: version number not available.
               METIS: 2023-07-11
              MICADO: 2023-07-14.dev
          MICADO_Sci: version number not available.
                MUSE: version number not available.
                NACO: version number not available.
              OSIRIS: version number not available.
             Paranal: 2022-04-09
                UVES: version number not available.
               VISIR: version number not available.
                 VLT: version number not available.
            ViennaLT: version number not available.
                WFC3: version number not available.
            _REPORTS: version number not available.
                docs: version number not available.
                irdb: version number not available.
        test_package: 2023-07-20.dev
               .idea: version number not available.
       .pytest_cache: version number not available.
    _ZIPPED_PACKAGES: version number not available.
              MORFEO: 2023-07-11
  .ipynb_checkpoints: version number not available.

Operating System info:
   System: Linux
  Release: 5.4.0-167-generic
  Version: #184-Ubuntu SMP Tue Oct 31 09:21:49 UTC 2023
  Machine: x86_64

[carmeloarci]

MORFEO_debugSRC0.py.txt
In this version of the file you may reproduce the error, avoiding randimization.

(scope_test) [carmelo@dyno simcado]$ python MORFEO_debug.py
['A0V']
Observe
Save
ERROR - Unhandled exception occured, see log file for details.
Traceback (most recent call last):
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/utils.py", line 1053, in wrapper
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/optics/optical_train.py", line 328, in readout
hdul.writeto(fname, overwrite=True)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1006, in writeto
self.verify(option=output_verify)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/verify.py", line 82, in verify
errs = self._verify(opt)
^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1401, in _verify
result = hdu._verify(option)
^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 1162, in _verify
errs = super()._verify(option=option)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 599, in _verify
return super()._verify(option)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 1099, in _verify
errs.append(card._verify(option))
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1166, in _verify
keyword, valuecomment = self._split()
^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 838, in _split
image = self.image
^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 530, in image
self._image = self._format_image()
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1025, in _format_image
raise ValueError(
ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long
ERROR - Python:
3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0]

Installed Python packages:
anisocado: 0.3.0
astropy: 5.3.3
beautifulsoup4: 4.12.2
docutils: 0.20.1
lxml: 4.9.3
matplotlib: 3.8.0
more-itertools: 10.1.0
numpy: 1.26.0
pooch: 1.8.0
pyyaml: 6.0.1
requests: 2.31.0
requests-cache: 1.1.0
scipy: 1.11.3
scopesim_data: could not be loaded.
scopesim_templates: 0.4.4
skycalc-ipy: 0.2.0
synphot: 1.2.1
tqdm: 4.66.1

Installed IRDB packages:
ELT: 2023-07-11
Armazones: 2023-07-11
MICADO: 2023-07-14.dev
MORFEO: 2023-07-11

Operating System info:
System: Linux
Release: 5.14.18-100.fc33.x86_64
Version: #1 SMP Fri Nov 12 17:38:44 UTC 2021
Machine: x86_64

Traceback (most recent call last):
File "/python/simcado/MORFEO_debug.py", line 110, in
hdus_J_small_test_3DM = micado.readout(filename= "scopesim_debug.fits")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/utils.py", line 1053, in wrapper
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/optics/optical_train.py", line 328, in readout
hdul.writeto(fname, overwrite=True)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1006, in writeto
self.verify(option=output_verify)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/verify.py", line 82, in verify
errs = self._verify(opt)
^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1401, in _verify
result = hdu._verify(option)
^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 1162, in _verify
errs = super()._verify(option=option)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 599, in _verify
return super()._verify(option)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 1099, in _verify
errs.append(card._verify(option))
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1166, in _verify
keyword, valuecomment = self._split()
^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 838, in _split
image = self.image
^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 530, in image
self._image = self._format_image()
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1025, in _format_image
raise ValueError(
ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long

[hugobuddel]

Thanks for sharing the output of scopesim.bug_report(). You are using ScopeSim_Templates 0.4.4, with which I can indeed reproduce the error.

The latest ScopeSim_Templates is 0.5.0, in which this particular problem does not occur. However, that is a coincidence, because 0.5.0 does not store the default source parameters. It would probably still fail with ScopeSim_Templates 0.5.0 if you would have explicitly listed the desired amplitudes in the script.

The short term fix would be to default to not storing these source description fits keywords, since they don't work properly, and they are probably not actively used, so they cause more problems then they solve, see AstarVienna/irdb#150 .