Use Case 6: discussion

[msdemlei]

Unless I misunderstand this use case, it proposes to allow embedding some sort
of executable code into the data format.

If that is true, I believe it the use case should be dropped, for at least the
following reasons:

(1) Security concerns: Even if you "sandbox" whatever code is executing
(which, of course, makes it more likely that the format's execution
facilities in the end will be too slow or restricted to be generally
useful), it's still going to be hard to control what apparently
innocuous files actually do (see Adobe's pain with Javascript in PDF).

(2) Ease of implementation: If we allow something like this, all
conforming implementations will have to include an interpreter for
whatever code this turns out to be. This will typically be a major
effort (or at least dependency) that's going to hurt adoption (not to
mention security concerns again). On the other hand, I've always wanted
to write a FORTH machine...

(3) Complexity considerations: As file formats are always at the "edges"
of computer systems, it's great if they are "verifiable" in some
sense (e.g., checking validity with a context free grammar). This
feature is deep, deep within the land of Turing complete languages with
all the related problems ("will this image halt?"). That's a fairly
fundamental flaw for something that sounds like a fairly exotic
application that would probably better be solved by local convention (a
pipeline manual might state: "look for the chunk labeled
'foo-execute', check for foo's signature via the foo-signature chunk,
and then just do it").

[embray]

I agree--if nothing else this use case needs clarification and/or narrowing. For example, there is a narrow sense in which this might be useful. For example for WCS or possibly other data reduction uses it would be possible to embed simple instructions for sequences of transformations and arithmetic functions to perform on some data in the file. But as currently written this use case read to me like stored procedures, as in a database, and that I think we want to avoid.

[brianthomas]

No, the intention here was not to embed any executable or compiled code; its basically what Erik wrote above. The idea was that some restricted set of notation/instructions would be adopted in the standard so that some parts of the data could be algorithmically described (and generated). Libraries, regardless of actual implementation language would have to support parsing, and executing, the instruction set.

[embray]

This one does need to be handled with care though. If we allow mathematical transformations on image data, why not allow, say, virtual tables created from joins of other tables, or other such database-like operations? I don't think we should have such a requirement, but why do we privilege one type of embedded data transformation over another? I'm not sure how to write this use case in such a way that addresses that slippery slope.

[msdemlei]

Hi,

On Mon, Jan 12, 2015 at 02:20:47PM -0800, Erik Bray wrote:

This one does need to be handled with care though. If we allow
mathematical transformations on image data, why not allow, say,
virtual tables created from joins of other tables, or other such
database-like operations? I don't think we should have such a

I'd maintain it's a question of the type of machine required to
execute the embedded specifications, and I'd say we should be "below
Turing" in some sense.

Now, for use cases like the specification of generalised transforms
evidently common mathematical expressions would be required, and
such expressions are, in themselves, probably not computable by
pushdown automata -- but I'd not be worried about these, accepting
"normal math" as elementary operations doesn't look dangerous to me.

Loops and function definitions are an entirely different beast. The
difference essentially is that it's easy to reason about what the
expressions do, whereas with loops and recursion it's at least hard
and in general impossible. Conditionals are a bit in between, but
something like SQL's CASE should be useful for many important
expressions (splines, say) while probably not poisoning the language
with computability problems.

The bottom line is that if we come up with a spec on this, we should
find some computability experts and ask them for their opinions...

Cheers,

     Markus

[nxg]

This approach, as Brian glosses it, is similar to the approach used by the AST library. That library provides general WCS support not by listing a number of algorithms and parameters, in the style of FITS-WCS, but by implementing a collection of general transformations which can be composed to provide complicated transformations on the data (and several of which are precomposed to provide the standard WCS mappings). That manifestly works in that case, and it's easy to see how it might work for a more general case of data transformations.

The transformations are specified within NDF files in (if I recall correctly; it's been a while) a not terribly readable form. One could imagine a little language which articulated them in a more naturally editable form.

[brianthomas]

I've tried to re-write this use-case a little based on the discussion here. I've dropped the idea of generation of theory datasets from the use case and focused it more on tablular and image transformation/value generations. Please take a look and feedback. I expect we'll still need to iterate.

[mdboom]

In my view, use case #6 still reads more like a feature in search of a use case than a use case. It would be helpful to understand the reasons why such a feature would be important, and why it must be part of a storage file format.

I understand why descriptions of coordinate transformations is essential: it allows for mapping between logical and physical coordinates without the problems that come with resampling the data. It could be done with a fixed lookup table (and HST had a history of that in some cases), but being able to tweak knobs of the transformation has proven very useful.

I'm not as sold on the reasons why algorithmically-generated data must be specified in the file format, rather than as an adjunct tool or extension for that purpose. Particularly given that the file format will support the storage of structured metadata, one could store a procedure in the file that could be understood by some domain-specific tool in the future. I don't think the file format should require anything like this, as it adds significantly to the implementation burden and has the potential to create many more security holes where otherwise there would be few.

[brianthomas]

There appears to be some confusion here still. Another attempt to explain this is that allowing simple mathematical formulae to describe the data is a good thing, if only from the standpoint of compression of large datasets. It also promotes long-term understanding of the data since you can see succinctly what the underlying formula (and perhaps scientific principle, as applicable) are behind that portion of the dataset.

I'm not as sold on the reasons why algorithmically-generated data must be specified in the file format, rather than as an adjunct tool or extension for that purpose.

I'd be all for more complex generation of data sitting in an (optional, outside of the spec) plugin which has compiled code. Where the line is between "simple mathematical formulae" and "complex generation" is another matter.