simple alternative needed

[BobHanson]

I'm not seeing how one would integrate this mmtf-java package into a working Java program such as Jmol. It has a very large set of dependencies, including

org.msgpack.jackson.dataformat
fasterxml.jackson.annotation
fasterxml.jackson.core
fasterxml.jackson.databind

This amounts to over 5 MB of code and over 500 classes.

The decoding task, at least, is not at all that difficult. I have implemented it in Jmol using a very simple class that has only three generic dependencies (a byte array converter, a binary document reader, and an efficient JavaScript-compatible StringBuffer equivalent). See Jmol's MessagePackReader

I offer this code as a possible very lightweight alternative to what is presently on this site (4 files total; under 20K total for either .class or .js files).

So perhaps just suggesting development of a similar "mmtf-java-decode-lite"

Bob Hanson

[josemduarte]

True, the dependencies have a lot of extra code and might seem complex but at the same time we get solid implementations where the experts in each of the topics (be it message pack or some other thing) have gone through a few release cycles, thinking about a good design and fixing bugs and issues that a much larger community has encountered along the road.

In general our philosophy is to use libraries and off-the-shelf components when those are available, in order to avoid getting into the same traps that others got in before. In my opinion, a larger package size is not a big price to pay for all that.

[pwrose]

I agree in general, however, we need to check if the jackson library is
serializable, otherwise it won't work in our Spark applications.

On Tue, Aug 9, 2016 at 5:12 PM, Jose Manuel Duarte <notifications@github.com

wrote:

True, the dependencies have a lot of extra code and might seem complex but
at the same time we get solid implementations where the experts in each of
the topics (be it message pack or some other thing) have gone through a few
release cycles, thinking about a good design and fixing bugs and issues
that a much larger community has encountered along the road.

In general our philosophy is to use libraries and off-the-shelf components
when those are available, in order to avoid getting into the same traps
that others got in before. In my opinion, a larger package size is not a
big price to pay for all that.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#22 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADuwELPh9xFwvz_4xJNS_xMXyyNAd7-sks5qeRdvgaJpZM4Jgn-2
.

Peter Rose, Ph.D.
Site Head, RCSB Protein Data Bank West (http://www.rcsb.org)
San Diego Supercomputer Center (http://bioinformatics.sdsc.edu)
University of California, San Diego
+1-858-822-5497

[BobHanson]

I understand. But these are really simple functions.

Maybe my use case -- a program that needs to be as compact as possible and
to run extremely efficiently in both Java and transpiled JavaScript -- is
unusual. I don't know. Jmol/JSmol has to be so efficient in both Java and
JavaScript in all respects that I rarely have the luxury of just pulling
code off the shelf and using.

In any case, if you would expose one class with a few simple Java methods
such as I am doing in Jmol, or one .js file with all that is needed, I
think it would be much appreciated. The specs are clear enough and so well
written that I was able to implement these without any reference code.
Although, even there it might be nice to put in code snippets to show
working examples. For example, Type 9 is:

public static float[] rldecodef(byte[] b, int n, float divisor) {
float[] ret = new float[n];
for (int i = 0, pt = 3; i < n;) {
int val = bytes4ToInt32(b, (pt++) << 2, true);
for (int j = bytes4ToInt32(b, (pt++) << 2, true); --j >= 0;)
ret[i++] = val / divisor;
}
return ret;
}

It might be hard to see that from what is written there.

If you do want to include all those libraries as they are, would it be
possible to explain to people exactly how to implement them? I could not
figure it out myself. What I saw was a huge spider web of methods that, in
the end, only needed to be about a dozen small methods. The needs are so
minimal for decoding -- one relatively simple binary decoder method along
with the 15 array codec methods.

Right?

​Bob

[arose]

If you can use a separate file for javascript, there is mmtf.js which includes everything for decoding and encoding in ~13KB (ungzipped).

[BobHanson]

And an unobscurified version of that?

On Thu, Aug 11, 2016 at 10:06 AM, Alexander Rose notifications@github.com
wrote:

If you can use a separate file for javascript, there is mmtf.js
https://github.com/rcsb/mmtf-javascript/blob/master/dist/mmtf.js which
includes everything for decoding and encoding in ~13KB (ungzipped).

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#22 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AQ7RWzfhUYmvWpD8ZH8DPQ6nRVF1jJ35ks5qezp3gaJpZM4Jgn-2
.

Robert M. Hanson
Larson-Anderson Professor of Chemistry
St. Olaf College
Northfield, MN
http://www.stolaf.edu/people/hansonr

If nature does not answer first what we want,
it is better to take what answer we get.

-- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900

[arose]

And an unobscurified version of that?

currently you have to build it yourself, opened an issue rcsb/mmtf-javascript#12

[andreasprlic]

Just to chime in here. I did some profiling and it seems that the fundamental decoding in mmtf-java is slow. I suspect that using jackson adds some overhead and inefficiencies. We should try to do something as simple as what mmtf-javascript is doing!

[pwrose]

A simple solution would have other benefits, too. Including: 1. we will be able to ignore custom records users may add to MMTF, and 2. we can make the method serializable, which is required to make it run on Spark in a multi-server environment.

…

On Wed, Jan 11, 2017 at 9:15 AM, Andreas Prlic ***@***.***> wrote: Just to chime in here. I did some profiling and it seems that the fundamental decoding in mmtf-java is slow. I suspect that using jackson adds some overhead an inefficiencies. We should try to do something as simple as what mmtf-javascript is doing! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADuwEIqZ1isvr7Xant-z_58zTkimTT3tks5rRQ5EgaJpZM4Jgn-2> .

-- Peter Rose, Ph.D. Site Head, RCSB Protein Data Bank West (http://www.rcsb.org) San Diego Supercomputer Center (http://bioinformatics.sdsc.edu) University of California, San Diego +1-858-822-5497

[iclkevin]

I fully agree with Bob on this note. MMTF would be much more accessible if the limited functionality from the external dependencies could be internally written in the MMTF jars, thereby making it incredibly lightweight. I know Jose Duarte suggested that the implementations from the open source community are credible and should be used, but they also add overhead for other uses for their other users, and having control over this functionality internally would allow you to custom tailor the functions to your goals. And also prevent a lot headaches for those of us trying to include it :).

Love the MMTF format, btw. It is great to see all those bond types loaded in and it is very fast!

[josemduarte]

@iclkevin note that since version 1.0.5 the default is to decode through a (slightly modified) version of @BobHanson 's code. The msgpack lib dependency is in any case still there. Decoding through the msgpack lib can be switched on with a flag.

[iclkevin]

In order to get the library to run at all, I need the following:

For input:

jackson-annotations-2.8.0
jackson-core-2.8.8
jackson-databind-2.8.8
jackson-dataformat-msgpack-0.7.0-M5

I don't know if all of those versions are compatible, but that was the latest for what I can find.

As for output, so far I have:

commons-lang-2.6
msgpack-core-0.8.11

Decoding works fine for me, but I haven't been able to find the right dependencies for encoding.

The above version of msgpack doesn't seem to be compatible (java.lang.NoSuchMethodError: org.msgpack.core.MessagePacker.(Lorg/msgpack/core/buffer/MessageBufferOutput;)V). Whatever that means...

Do you still plan on using the jackson libraries? Do you have a zip of the current dependencies so we can run the encoder? Do you plan on using your own functions for encoding as well? I definitely don't want to turn on msgpack for decoding.

Thanks,
Kevin Theisen

[josemduarte]

From the pom file I can see that mmtf-java depends currently on msgpack 0.7.1. Maven should take care of any sub-dependencies of that. Are you using maven?

[iclkevin]

Thanks, I see, we do not use Maven. We have a custom build system. Are there any other dependencies I should know about?

[josemduarte]

It's all in maven. If you really can't use maven then try something like mvn dependency:tree which should show the full dependency tree, then you can manually extract the dependencies from there. But that can be difficult.