Add items in the o_ucd list

[bkhelifi]

This issue aims to collect items to add in the IVOA o_ucd. All are welcome to add their proposal as comment..

[Bonnarel]

For o_ucd the current text reads.

Since the data content of event-
lists may vary significantly from facility to facility, meaningful discovery of
HEA datasets requires the user be able to query the UCDs of the set of
observables included in an event-list.
A natural way of doing this that is consistent with current usage would
be to extend o_ucd to allow specification of multiple observables for event-
lists, for example, o_ucd = pos.eq,time,phys.pulseHeight. Note that real
event-lists may include an extensive set of columns (e.g., a Chandra ACIS

If we have so much columns in the event list, our event list is close to a catalog. WE don't have so much catalogs in ObsCore (they should share the "measurements" data_product-type) and the columns are not described in them.
o_ucd so far was restricted to "dependant" variables, ie to variables which have a functional dependency to other variables
If we have so much variables (20, 25 is a number I heard yesterday during the workshop) it would be really painful to list all the ucds in o_ucd. IN addition what would be the names, units, types of these columns ?

However a feature exists in IVOA to describe such tables in advance, prior to dowload them, its the column element in VODataservice tableset or the columns table in a TAP schema. Below is an excerpt of the TAP specification describing this

So now if our service delivering ObsCore (+ extension) for He event-lists, also delivers a column-like table (just call it event-list-quantities by exemple) describing all the columns of all the event lists and the "table_name" column of this event-list-quantities table contains an identifier also present in the obscore table we can make a join query on both with potential constraints on ucds, units, types, in addition to the obscore attributes ones etc...

[iannevans]

I'm not convinced that it would be sensible to include UCDs for all of the columns in the event list, but this should be up to the data provider. Ultimately, we have up to 2 spatial, 1 spectral, 1 time, and possible polarization values as our set of physical observables and from a practical perspective UCDs that describe the "most" calibrated versions of those quantities are what a user is likely to want to query on.

Ignoring event lists, advanced data products are likely to require multiple UCDs, even if restricted to dependent variables, as they will likely include multiple related quantities. For example, a light curve data product might include both a "source + background" light curve and the associated "background" light curve and possibly estimated confidence intervals (particularly for HE where we are often close to the Poisson limit).

[bkhelifi]

From Karl Kosack:
Another thing for the UCD vocabulary: probably need some more gamma-ray bands: "hard" is now >500 KeV ...
See https://wiki.ivoa.net/internal/IVOA/IvoaUCD/NoteEMSpectrum-20040520.html

[bkhelifi]

Here is a proposal to update the ucd list:

Label	UCD word	Description
S	em.gamma.hard	Hard gamma ray (500 keV - 100 MeV)
S	em.gamma.he	High-Energy gamma ray (100 MeV - 10 GeV)
S	em.gamma.vhe	Very-High-Energy gamma ray (10 GeV - 100 TeV)
S	em.gamma.uhe	Ultra-High-Energy gamma ray (100 TeV - 10 PeV)
S	obs.event-list	To be added?
S	obs.spectrum	To be added?
S	phys.particle.electron	Related to electron/positron
S	phys.particle.cr	Related to cosmic rays particles
P	stats.error.minus	Negative statistical error
P	stats.error.plus	Positive statistical error
P	stat.confidenceLevel.ul	Level of confidence for a upper/lower limits computation

I have followed the label scheme of the current UCDs for these new ones.

A question:

• stat.max: "Maximum or upper limit" - how to know that this is a maximum or an upper limit?
• The same for stat.min

And for the document about the Electromagnetic spectrum description:

UCD designation	Wavelength	Frequency	Energy	Notes
Gamma Regime
em.gamma.soft	2.5-10pm	30-120EHz	120-500keV
em.gamma.hard	12-2500fm	120EHz-24ZHz	500keV-100MeV
em.gamma.he	0.1-12fm	24ZHz-2.4YHz	100MeV-10GeV
em.gamma.vhe	0.01am-0.1fm	2.4YHz-24200YHz	10GeV-100TeV
em.gamma.uhe	1.2e-4am-0.01am	24.2e3-2.4e6YHz	100TeV-10PeV

PS: note the previous typo on 500keV == 1200 EHz, which is wrong

[iannevans]

Here is a proposal to update the ucd list:

Label	UCD word	Description
S	em.gamma.hard	Hard gamma ray (500 keV - 100 MeV)
S	em.gamma.he	High-Energy gamma ray (100 MeV - 10 GeV)
S	em.gamma.vhe	Very-High-Energy gamma ray (10 GeV - 100 TeV)
S	em.gamma.uhe	Ultra-High-Energy gamma ray (100 TeV - 10 PeV)
S	obs.event-list	To be added?
S	obs.spectrum	To be added?
S	phys.particle.electron	Related to electron/positron
S	phys.particle.cr	Related to cosmic rays particles
P	stats.error.minus	Negative statistical error
P	stats.error.plus	Positive statistical error
P	stat.confidenceLevel.ul	Level of confidence for a upper/lower limits computation

I have followed the label scheme of the current UCDs for these new ones.

A question:

• stat.max: "Maximum or upper limit" - how to know that this is a maximum or an upper limit?
• The same for stat.min

And for the document about the Electromagnetic spectrum description:

UCD designation	Wavelength	Frequency	Energy	Notes
Gamma Regime
em.gamma.soft	2.5-10pm	30-120EHz	120-500keV
em.gamma.hard	12-2500fm	120EHz-24ZHz	500keV-100MeV
em.gamma.he	0.1-12fm	24ZHz-2.4YHz	100MeV-10GeV
em.gamma.vhe	0.01am-0.1fm	2.4YHz-24200YHz	10GeV-100TeV
em.gamma.uhe	1.2e-4am-0.01am	24.2e3-2.4e6YHz	100TeV-10PeV

PS: note the previous typo on 500keV == 1200 EHz, which is wrong

I suggest phys.particle.cosmicray instead of physical.particle.cr as spelling it out is more explicit for folks not working directly in the field.

Similarly, I suggest stat.error.positive and stat.error.negative instead of .plus/.minus for positive/negative confidence limits.

I'm not convinced stat.confidenceLevel.ul is needed since I think you can in principle combine stat.confidenceLevel and stat.max/stat.min to get there. However, I don't like stat.max/stat.min representing both a maximum/minimum statistic and an upper/lower limit since these are different concepts and it's not clear which one you mean when they're combined. I suggest restricting stat.max/stat.min to mean the maximum/minimum statistic and adding new terms stat.upperlimit/stat.lowerlimit for upper/lower limits (which would also be more clear terminology).

We also need a UCD for PHA/PI and I suggests phys.pulseHeight. As noted in the draft document, a current proposed solution that suggests using src.var.amplitude;src.var.pulse;stat.uncalib is not at all appropriate since the connection to src (“observed
source viewed on the sky”) is misleading and src.var.amplitude is defined as the “amplitude of variation” of the source which is a completely separate concept from an astronomical perspective.

Finally, I have a concern about the current draft "Electromagnetic spectrum description in UCD Version 0.3" since the document describes particular EM spectrum, regimes in Wavelength, Frequency, and Energy but the values provided are approximations, i.e., they don't translate precisely from wavelength to frequency to energy. Natural usage would suggest that in the radio and millimeter regimes the boundaries should be defined in terms of frequency (and indeed the names of the UCD designations include the frequency boundaries as part of the name, so anything else would be very confusing), while for the X-ray and gamma-ray regimes the boundaries would be defined in terms of energy, with wavelength being used for UV through IR. Regardless, this needs to be defined explicitly in the document.

[bkhelifi]

Thanks for your feedback. +1 for your proposal for cosmic rays and the errors (positive/negative).

About stat.confidenceLevel.ul, I think that this is mandatory because when publising an UL how to know that this is at 95%, 99% or 68%? Frequently, we give errors at 68% and UL at 95%, so the 2 confidence levels are necessary when both errors and UL are published. For me, this is a clear lack in the ucds... But I might have mistaken or misunderstood your point.
I fully share your comment about stat.max/stat.min and stat.upperlimit/stat.lowerlimit. I have even hesitated to make the proposal... but I don't in order to not trigger a "word revolution";) I will in any case follow your best choice as we are in line.

+1 for the PHA. Indeed, I forget to add your important add-on that you mentioned by email before. Sorry for that and thanks for the reminder;)

You're right about the approximations... In other end, these limits between domain are also artificial, from scratch. I see them as a very rough division of the EM spectrum, based on human conventions. Personally, such (real and numerical) approximations are of second-order compared to the division of the EM spectrum convention... But one can propose to not put the number in frequency and wavelength. I will not be hurt;)

[bkhelifi]

And what about the event-list? I am not sure whether it is needed...

[iannevans]

About stat.confidenceLevel.ul, I think that this is mandatory because when publising an UL how to know that this is at 95%, 99% or 68%? Frequently, we give errors at 68% and UL at 95%, so the 2 confidence levels are necessary when both errors and UL are published. For me, this is a clear lack in the ucds... But I might have mistaken or misunderstood your point.

I agree that for every quoted confidence limit one needs to know the confidence level for the former to be meaningful. I'm perhaps not understanding your use case. Are you saying you have 3 columns (I'm ignoring lower confidence limits for now)
<value> <+68%confidence> <+95%confidence> and you're calling the latter an upper limit? Because statistically that is not an upper limit, it's just a second upper confidence limit with a different confidence level. So you would quote a value column, plus a confidence limit column and a confidence level column for each confidence limit. The former confidence limit columns would have UCD stat.error and the confidence level columns would have UCD stat.confidenceLevel (or possibly stat.confidenceLevel;stat.percentile if you're quoting your confidence level as a percentage).

[bkhelifi]

The UL can be at 68%, 90%, 95%, any probability in fact....
When a producer gives a number as UL, the user should know the associated probability. Otherwise, it is meaningless.

The same for the confidence levels, they can be at 68%, 2sigma or 3, any probability. The user should be aware of the value.

For each item (confidence interval and UL and LL), one should have both the value(s) and their associated probability. So, statistically speaking and for a proper usage of such numbers, one should check that all information are present.

Is it more clear?

[iannevans]

The UL can be at 68%, 90%, 95%, any probability in fact....
When a producer gives a number as UL, the user should know the associated probability. Otherwise, it is meaningless.

The same for the confidence levels, they can be at 68%, 2sigma or 3, any probability. The user should be aware of the value.

For each item (confidence interval and UL and LL), one should have both the value(s) and their associated probability. So, statistically speaking and for a proper usage of such numbers, one should check that all information are present.

Is it more clear?

I understand that the user needs to know the associated confidence level of an upper (lower) limit.

My original comment was related to your statement that you needed a separate UCD (stat.confidenceLevel.ul) for the level of confidence for an upper/lower limits computation and I am not yet convinced that is appropriate. Upper (lower) limits for some statistical measures may require additional information for the user to be able to fully understand them (for example, if you are reporting an upper flux limit for a source that may or may not be detected then formally there is a detection threshold that depends on acceptable limits for both Type I and Type II errors). This information may need additional UCDs beyond those currently defined, but I don't think those UCDs are appropriately built on stat.confidenceLevel.

So I am trying to understand your definition of upper (lower) limit and the use case that lead you to propose stat.confidenceLevel.ul.