Add items in Vocabulary

[bkhelifi]

This issue aims to collect items to add or to correct in the IVOA Vocabulary. All are welcome to add their proposal as comment..

[Bonnarel]

This issue aims to collect items to add or to correct in the IVOA Vocabulary. All are welcome to add their proposal as comment..

Which vocabularies Bruno ? any kind ? or dataproduct_type only ?

[bkhelifi]

@bkhelifi This issue aims to collect items to add or to correct in the IVOA Vocabulary. All are welcome to add their proposal as comment..

@Bonnarel Which vocabularies Bruno ? any kind ? or dataproduct_type only ?

I would say any kind.... But maybe one should make separated issues for ObsCore, DataLink, etc... Any preference?

[bkhelifi]

In https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html,
I propose to add:

Term	Label	Description	Parent
arf	Ancillary Response File	Contains the combined telescope/filter/detector areas ("effective area") and the quantum efficiency (QE) as a function of energy averaged over time. Dimension: [L**2 counts/photon]. See [1]	calibration
rmf	Redistribution Matrix File	Probability Density Function from energy space into detector pulse height (or position) space. Dimension : [none]. See [1]	calibration
psf	Point Spread Function	Probability Density Function of Spatial/angular spreading of incident photons from a point source caused by the instrument (detector and/or mirror and/or analysis). Dimension: [none]. See [2, 3]	calibration
aeff	Effective Area	Contains the detection efficiency of an instrument with the observable area. Dimension: [L**2]. See [3]	calibration
edisp	Energy Dispersion	Probability Density Function for the energy migration (Ereco/Etrue) as a function of true energy and spatial position. Dimension: [none]. See [3]	calibration
background	Background Rate	Rate of residual atmospheric cosmic-ray event. Dimension: [L2 L-2 M L2 T−2], ie [/sr/TeV]. See [3]	calibration

Reference:
[1] I.M. George, K.A. Arnaud and A.F. Tennant. The Calibration Requirements for Spectral Analysis. Technical Report OGIP/92-002, NASA/GSFC, Dec 1998. (https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_002/cal_gen_92_002.pdf)
[2] I.M. George & R. Yusaf. The OGIP Format For 2-D (image) Point Spread Function Datasets. Technical Report OGIP/92-027, NASA/GSFC, Nov 2011. (https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_027/cal_gen_92_027.pdf)
[3] Data Formats for Gamma-ray Astronomy, v0.3, 2018 (https://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/irf_components/index.html#)

[iannevans]

In https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html,
I propose to add:

Term	Label	Description	Parent
arf	Ancillary Response File	Contains the combined telescope/filter/detector areas ("effective area") and the quantum efficiency (QE) as a function of energy averaged over time. Dimension: [L**2 counts/photon]. See [1]	calibration
rmf	Redistribution Matrix File	Probability Density Function from energy space into detector pulse height (or position) space. Dimension : [none]. See [1]	calibration
psf	Point Spread Function	Probability Density Function of Spatial/angular spreading of incident photons from a point source caused by the instrument (detector and/or mirror and/or analysis). Dimension: [none]. See [2, 3]	calibration
aeff	Effective Area	Contains the detection efficiency of an instrument with the observable area. Dimension: [L**2]. See [3]	calibration
edisp	Energy Dispersion	Probability Density Function for the energy migration (Ereco/Etrue) as a function of true energy and spatial position. Dimension: [none]. See [3]	calibration
background	Background Rate	Rate of residual atmospheric cosmic-ray event. Dimension: [L2 L-2 M L2 T−2], ie [/sr/TeV]. See [3]	calibration

Reference:
[1] I.M. George, K.A. Arnaud and A.F. Tennant. The Calibration Requirements for Spectral Analysis. Technical Report OGIP/92-002, NASA/GSFC, Dec 1998. (https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_002/cal_gen_92_002.pdf)
[2] I.M. George & R. Yusaf. The OGIP Format For 2-D (image) Point Spread Function Datasets. Technical Report OGIP/92-027, NASA/GSFC, Nov 2011. (https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_027/cal_gen_92_027.pdf)
[3] Data Formats for Gamma-ray Astronomy, v0.3, 2018 (https://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/irf_components/index.html#)

Your definition of "background" is far too restrictive given that "background" is a very general term used in every waveband.

[bkhelifi]

If you agree on all the rest, this is a good point;)
One should take care that there is already a noise in the auxiliary (https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html).
So, here, I would like to propose residual background from phenomena coming from outside the detector/instrument/telescope that create events into our detector and that stay despite our analysis. They are gamma/neutrino-like events, "residual events", undistinguishable from real gamma/neutrino events, leading to the need of statistical analysis.
For us, this type of background is distinct from the "astrophysical background", e.g. X-ray diffuse emission from a galaxy cluster, large scale GeV/TeV emission created by CR-ISM interaction

Are you sharing this concept? If yes, one should find then a better definition...

[Bonnarel]

@bkhelifi This issue aims to collect items to add or to correct in the IVOA Vocabulary. All are welcome to add their proposal as comment..

@Bonnarel Which vocabularies Bruno ? any kind ? or dataproduct_type only ?

I would say any kind.... But maybe one should make separated issues for ObsCore, DataLink, etc... Any preference?

Vocabularies can be common to several spec:

• dataproduct_type basically took origin from obscore FIELD of the same name but can be used in the registry and also in the content_qualifier FIELD of datalink

• The so called "DataLink core" vocabulary (strange name) is actually the vocabulary for semantics FIELD.

• Question open for the vocabulary for irf types : addition to dataproduct_type ? addition to semantics DataLink core ? or new vocabulary ? ----> to be discussed. Could be used in DataLink content_qualifier FIELD ( this one is not required to be filled by a dataproduct_type term. Terms from other vocabularies are allowed)

• other small vocabularies scan_mode : is that enough to define it internally to radio extension ?

• tracking type : obviously already common to several specs.

I think we can create an issue for each of these vocabularies

[msdemlei]

On Fri, May 09, 2025 at 09:29:02AM -0700, Bruno Khélifi wrote: bkhelifi left a comment (ivoa/HighEnergyObsCoreExt#5) In https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html, I propose to add:

This looks like a very good start for a VEP; you can propose multiple related terms in one VEP, and if there's basic community agreement about these concepts, I'd say it would be a good idea to have them in one. Also, the explanations already help me a log resolving my arf puzzlement in the mail a few minutes ago :-) Oh, and for now I stand by my request to try and have a hierarchy in there. Do you think you could write up a VEP as per <https://ivoa.net/documents/Vocabularies/20230206/REC-Vocabularies-2.1.html#tth_sEc5.2.1>? If necessary, we could also have a little telecon on the technicalities of that effort; as usual, I'd *really* love if we didn't have to fake the used-in parts. Oh, and let me, just in case, pitch again my soapbox section on creating concepts: <https://ivoa.net/documents/Vocabularies/20230206/REC-Vocabularies-2.1.html#tth_sEc5.2.4>.

[Bonnarel]

I think we all agree that we have to define new terms for the instrument response functions.

From the discussion I see three different possible vocabulary integration for these terms

Beside this it's also important to distinguish creation/extension of vocabularies from the FIELDs where we use them.

And we have to find a solution which works for two different exposition strategies of the irf :

 - directly as observation in ObsCore
 - via DataLink, attached to a primary science dataproduct in ObsCore (event-list,  spectrum, etc...) 

1 ) One of the suggestion I have read so far (Bruno) is to refine the #calibration branch of "DataLink core" (https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html) to add your new terms for irf . Let's take the #psf term for example : if we set this term in the "semantics" FIELD of the DataLink response that means that we are facing the "#psf of the primary item in ObsCore". But I understood (from Ian) that sometimes you want to discover irf directly, as observations. And in the current status of ObsCore/SSA/SIA/DataLink it's not possible to use https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html outside the "semantics DataLink response" FIELD. Changing that would be cumbersome.

In addition we could imagine in DataLink that a psf (or any other irf ) link could be used for a material required to calibrate (process) the primary item (child of #calibration) or on the other side for material already used to calibrate the dataset discovered in ObsCore. (currently would be a child of #progenitor although I don't like it personally). So this would require some duplication of all the different irf terms ! one psf-for-calibration and one psf-used-tocalibrate !!! , etc ...

----> Probably we want to avoid that

2 ) Data Product type (https://www.ivoa.net/rdf/product-type/2024-05-19/product-type.html) : This vocabulary originally extracted from ObsCore is intended to be used in the dataproduct_type FIELD (in future version of ObsCore) and can also be used elsewhere. It can be used in the registry and in the content_qualifier FIELD of DataLink.
Considering that IRF are "special" observations, it's possible to create a new "irf" branch in this vocabulary with all irf children.
It could be used either in ObsCore dataproduct_type or in DataLink content_qualifier depending on the irf "exposition" strategy.
For the DataLink exposition strategy, this kind of combination (completed by content_type for the media-type/format) is what we experimented with the DataLink core #coderived and #counterpart terms in this context (example):

            primary item : an astronomical source in a catalog
                                 link1 : semantics=#coderived, content_qualifier=#time_series,content_type=text/csv
                                 link2 : semantics=#counterpart, content_qualifier=#image,content_type=application/fits

However there is a question: should all the types of irf be potentially exposed in ObsCore as "observations" ? If some of them are excluded it would be difficult to mark them as irrelevant : we don't have ways to exclude some terms from a vocabulary in a given context.
It's one reason why I think there is also a third solution to manage this "irf type" vocabulary.

3 ) create a new "irf type" IVOA vocabulary.
It can be used in "content_qualifier" FIELD of DataLink in combination with "semantics=#calibration" (or any other appropriate term from https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html)
This is perfectly valid because content_qualifier is not REQUIRED to be taken from "Data Product type" vocabulary. It's only the default ! And the combination between semantics , content_qualifier and content_type works as in 2)
The drawback is that in that case we cannot use these new irf terms directly in the dataproduct_type FIELD of ObsCore, if we want to expose them there.
The compromise could be to create a single new term in https://www.ivoa.net/rdf/product-type/2024-05-19/product-type.html : #irf
and when she finds this, the user/software has to look in another ObsCore field (maybe dataproduct_subtype ???) the accurate term (taken from this independent irf vocabulary) describing the irf

---

In my personal opinion , solution 1 should be rejected. It's not consistent with what we did when we creates #counterpart, #coderived and the content_qualifier FIELD.
2 or 3 ) can be further discussed. I personally prefer 3 because IRF may be either images, or spectra, or whatever, in such a way that what distinguish them from classical observation results is the object which is observed: "something in the sky" versus "measuring a response of the instrument". SO I am reluctant to add them to data product type vocabulary where so far the distinction between so far was "what axis is observed, what axis is sampled".

[bkhelifi]

Thank you for synthesis, that is very useful!

Indeed, IRFs have a quite variable dimensions. They are N-dimensional data cubes (N=4 or 5 won't be rare).

Having a new type in the IVOA vocabulary for datalink is needed at some stage. I understand that it is more wise to use these new terms for the content_qualifier field of datalink. The wording using 'irf' might be too oriented VHE, "response" could be more generic. And having a new optional content_qualifier_subtype with the different type of irfs (psf, arf, rmf, aeff, edisp, bkg) could be very appealing and interesting... This is very optional because they are use cases where all components of the irf are stored into the same ("response") file in different HDU. But it offers to the data producers any data decomposition, and this solution is then very generic.

[msdemlei]

On Thu, May 15, 2025 at 01:50:29AM -0700, François Bonnarel wrote: 1 ) One of the suggestion I have read so far (Bruno) is to refine the #calibration branch of "DataLink core" (https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html) to add your new terms for irf . Let's take the #psf term for

IMHO that's definitely where we want to go. The use case is clearly: collect all calibration data, and possibly within the calibration data everything that's like a response function, in a hideable or selectable branch. That means in particular that we need these terms in datalink/core. We *could* consider using using content_qualifier for "subtyping", but I see no benefit to that over nicely organising them in a hierarchy within datalink/core.

DataLink response that means that we are facing the "#psf of the primary item in ObsCore". But I understood (from Ian) that

Private mail with Ian suggests that at least for Chandra, obscore has nowhere near the metadata that would be required for meaningfully discovering response functions, and the discovery processes seem to be very instrument-specific anyway. I hence doubt there is room for a *standard* that says how to do that observation-independent response function discovery. I'd frankly expect that to be necessarily based on non-standard (but hopefully TAP-queriable) tables. But let's see. The one thing that's clear to me at this point is that obscore is not the right tool to publish something that is very much *not* an observation. Response functions are ancillary data, and for thta we have datalink in the standard case.

[iannevans]

Private mail with Ian suggests that at least for Chandra, obscore has
nowhere near the metadata that would be required for meaningfully
discovering response functions, and the discovery processes seem to
be very instrument-specific anyway.

I think there are a common set of response functions that one could consider. Some are waveband-dependent while others span multiple wavebands. For example, for HEA we have the ARF and RMF (following OGIP standards) or equivalently the IRF in the case of CTAO and some others. (In principle, the same formalism could be used in other wavebands but since the RMF at lower energies is very close to diagonal the LSF makes more sense there.) There is the PSF that is common to virtually all wavebands, and there are several others (Bruno has highlighted some). So response functions are quite common and in many cases they can be associated with "observations" where I put the term in quotes because they might vary with location/time/energy. If you consider "advanced data products", for example Chandra Source Catalog products associated with a source detection, then you can connect the responses to the detection event list because the event selection for the latter is determined and so the responses are localized and can be constructed.

The one thing that's clear to me at this point is that obscore is not
the right tool to publish something that is very much not an
observation. Response functions are ancillary data, and for thta we
have datalink in the standard case.

Here I have to disagree. Advanced data products such as spectra, light-curves, aperture photometry MPDFs, or position error MCMC draws (including response functions) associated with source detections are readily discoverable independently using their world coordinates or the combination of world coordinates and additional constraints such as obs_id. In this case, most users are interested in querying and retrieving specific dataproduct types for the detections.

The issue that arises in a few cases is the one we discussed where one might want to query a for dataproduct such as a PSF in a coordinate system that is not exposed through ObsCore. Here I agree that the information needed to perform the query is very likely dependent on the individual telescope/instrument combination. While some of these cases may be addressed though the addition of obs_mode and/or analysis_mode qualifiers as suggested in the HEA Extensions, those that depend on querying a facility-specific coordinate system are likely insoluble using the current ObsCore definition.

Nevertheless, even with this limitation if we can make available our advanced data products using ObsCore (and I believe we can) that would still be a tremendous advantage for data discovery and cover the vast majority of cases that the broader community would prioritize.

[Bonnarel]

On Thu, May 15, 2025 at 01:50:29AM -0700, François Bonnarel wrote:
1 ) One of the suggestion I have read so far (Bruno) is to refine
the #calibration branch of "DataLink core"
(https://www.ivoa.net/rdf/datalink/core/2022-01-27/datalink.html)
to add your new terms for irf . Let's take the #psf term for
IMHO that's definitely where we want to go. The use case is clearly:
collect all calibration data, and possibly within the calibration
data everything that's like a response function, in a hideable or
selectable branch. That means in particular that we need these terms
in datalink/core.

Markus, I am really surprised !!!
You were the one who convinced me several years ago (2019-2020) not to refine too much the datalink/core branches to tackle something like types, reason why we created content_qualifier in DataLink 1.1 to avoid such specialisation under coderived and counterpart !

I am sure that the different response function such as psf, will not be discovered ONLY in the case where we need to calibrate the initially discovered dataset.

At least they could be used as part of the #progenitor links for the initially discovered dataset like this

                  obscore : obs_publisher_did, dataproduct_type, s_ra, s_dec, em_min, em_max 
                                   ivo://aaa/id1 , spectrum, 35.7, 50.2, 10**-16, 10**-15, etc....
                                   DataLink ID, semantics, content_qualifier, 
                                      ivo://aaa/id1, #progenitor, event-list
                                      ivo://aaa/id1, #progenitor, irf_type:#psf
                                     .....

And could be also simply be used with semantics = #this , when the primary item is the response function itself, but requires various links.
And if the irf types are defined in datalink core it will not be possible to use them outside the semantics FIELD context. Other terms in there such as #proc, #cutout, #preview, #package, #documentation, #counterpart only make sense in this DataLink semantics context. So what to do with such terms elsewhere ?

[msdemlei]

On Fri, May 16, 2025 at 07:28:25AM -0700, Ian Nigel Evans wrote: iannevans left a comment (ivoa/HighEnergyObsCoreExt#5) > The one thing that's clear to me at this point is that obscore is not > the right tool to publish something that is very much *not* an > observation. Response functions are ancillary data, and for thta we > have datalink in the standard case. Here I have to disagree. Advanced data products such as spectra, light-curves, aperture photometry MPDFs, or position error MCMC draws (including response functions) associated with source detections are readily discoverable independently using their world coordinates or the combination of world coordinates and additional constraints such as obs_id. In this case, most users are

Ah, ok -- I had gotten a rather different impression from what we had discussed via mail. I am still not entirely convinced that, say, PSFs should come back from an Obscore query; that may make clients' lives a bit harder. But then they will in general constrain on dataproduct_type anyway, and as long as the PSFs are sufficiently far way from images and spectra in the pertinent semantics, they will probably not contaminate common obscore queries. That indeed, as François says, makes it a lot more attractive to think about just one term in datalink/core and try to work all the rest out in product-type. Who of the participants here will be in College Park?