UVES example doesn't normalise spectra correctly ("continuum" and "finalize" steps)

[regislachaume]

After a fresh install, I've run the UVES example and uncommented all the reduction steps without any other modification. The final products has strong artifacts that seem to arise from a normalisation gone completely wrong (see attached figure).

File run: original uves_example.py with comments removed:

import os.path
import pyreduce
from pyreduce import datasets

instrument = "UVES"
target = "HD[- ]?132205"
night = "2010-04-01"
mode = "middle"
steps = (
    "bias",
    "flat",
    "orders",
    "norm_flat",
    "wavecal",
    "curvature",
    "science",
    "continuum",
    "finalize",
)
base_dir = datasets.UVES("/data/PyReduce")
input_dir = "raw/"
output_dir = "reduced/{night}/{mode}"

config = pyreduce.configuration.get_configuration_for_instrument(instrument, plot=1)

pyreduce.reduce.main(
    instrument,
    target,
    night,
    mode,
    steps,
    base_dir=base_dir,
    input_dir=input_dir,
    output_dir=output_dir,
    configuration=config,
    order_range=(1, 21),
)

Figure script

from matplotlib import pylab as plt
from astropy.io import fits
import numpy as np

fig = plt.figure(1)
fig.clf()
ax1 = fig.add_subplot(311)
ax3 = fig.add_subplot(312)
ax2 = fig.add_subplot(313)

path = '/data/PyReduce/datasets/UVES/reduced/2010-04-01/middle'

with fits.open(f'{path}/UVES.2010-04-02T09_28_05.650.science.ech') as hdus:
    data = hdus[1].data
    sig, spec = data['SIG'], data['SPEC']

with fits.open(f'{path}/uves.2010-04-01_0.final.ech') as hdus:
    data = hdus[1].data
    sig2, cont2, spec2 = data['SIG'], data['CONT'], data['SPEC']

arrays = np.load(f'{path}/uves_middle.flat_norm.npz', allow_pickle=True)
blaze = arrays['blaze']
norm = arrays['norm']

for i in range(20):

    nonzero = sig[0, i, :] * spec[0, i, :] > 0
    wave = np.arange(4096 * i, 4096 * (i + 1))
    ax1.plot(wave[nonzero], spec[0, i, nonzero])
    ax1.text(25000, 10000, 'UVES.2010-04-02T09_28_05.650.science.ech')

    nonzero2 = sig2[0, i, :] * spec2[0,i,:] > 0
    ax2.plot(wave[nonzero2], spec2[0, i, nonzero2], '-',
             wave[nonzero2], cont2[0, i, nonzero2], 'k--')
    ax2.text(25000, 10000, 'uves.2010-04-01_0.final.ech')

    nonzero3 = blaze[i, :] > 0
    ax3.set_ylim(0, 1.35)
    ax3.plot(wave[nonzero3], blaze[i, nonzero3] / 1e7, '-',
             wave[nonzero3], norm[i, nonzero3], 'k--')
    ax3.text(25000, 1.15, "uves_middle.flat_norm.npz")

fig.savefig(uves.png)

The result is in the attached picture.

• Top panel: extracted spectrum, where you can clearly see the impact of blaze in each order.
• Middle panel: blaze (and its normalisation)
• Bottom panel: final spectrum, which should have been normalised, i.e. spectrum should be approximately flat. If it is not normalised it should feature something similar to previous steps with a typical blaze pattern (roughly bell shaped, like both panels above) but it's not the case. (Pixel number instead of wavelength to avoid overlapping for the sake of clarity.)

System:

• ubuntu 22.04 (Linux macondo 5.15.0-50-generic Helps creating guess lines #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux)
• python 3.10.6 with IPython 7.31.1 run within a virtual environment dedicated to pyreduce