future ideas for this project

[afgane]

I just came across this repo and was wondering if it would be desirable to channel the effort going into this to the new version of CloudLaunch that has basically the same feature: https://beta.launch.usegalaxy.org/public_appliances

[bgruening]

Definitely - can you read my mind? :)
At the current state it's just a test and I'm collection feedback. I wanted to discuss this at first with @tnabtaf before announcing it publicly and maybe moving it into the Galaxy organisation.

I have a few ideas how we can use these informations, e.g. for finding training resources (human and metal) for GTN or generating the Public-Galaxy Server wiki page out of it. One explicit aim is to include all Galaxy instances, not only public ones. So we can generate a knowledge map.

If this idea is taking off we can create some mini-ontology to automatically filter this json file for Cloudman, GTN, ELIXIR, de.NBI and so on. I can imagine ELIXIR is interested to see how many Galaxy instances are deployed in Europe.

I hope I have not produced something redundant here, I was not aware of a geojson collection of already known Galaxy servers.

[afgane]

I was hoping we can go the other way and replace the public galaxy servers page with a page like this one that's more descriptive and interactive. In addition to the ideas you listed, I'd really like to see a cross-Galaxy tool search. I keep wanting to chat to @martenson about this idea since he's been working with the search functionality (@martenson thoughts?).

I guess we can ingest data from here into the CloudLaunch at some point so effort wouldn't need to be duplicated. As the features are added to CloudLaunch, the idea is that any user can add their instance by just filling out a form there. I guess we'd then have tags to include public instance, themed instances, instances accessible only to the given user, etc.

[hexylena]

I think GRT is the solution to all of the problems :)

• Serves as a Galaxy directory
• Has a list of tools used in partcipating Galaxies
• (As of this morning) has code for collecting IP address and galaxy name / description

What more could we want? (Serious question)

By default galaxy ships with part of a GRT configuration. Cloudlaunch could automatically register an instance for users and automatically apply tags to that instance like "public" or "private", and "cloudlaunch" and maybe even infrastructure run on would be nice to have tagged.

[martenson]

@afgane more than a year ago I added this to the API: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/tools.py#L67 which allows to cheaply query remote Galaxy for a tool presence. It might be around that time when we can assume a decent number of public instances is updated to have this functionality and we can build some aggregated search UI.

My idea was that you search on top of the Tool Shed list of tools with fulltext and then with the tool_id(s) you query all known public Galaxies to see where you can run the tool(set of tools).

GTR has a bit different approach as it plans to fetch all tools and then perform the search locally (right, @erasche ?).

[hrhotz]

sounds very good, but what about Galaxies behind firewalls?

On Wed, Aug 10, 2016 at 7:57 PM, Eric Rasche notifications@github.com
wrote:

I think GRT is the solution to all of the problems :)

• Has a list of tools used in partcipating Galaxies
• (As of this morning) has code for collecting IP address and galaxy
  name / description

What more could we want?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#6 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKjM6i8hfNaKsYcA4IRGrFqL1e3Z3K9Zks5qehD0gaJpZM4JhIA5
.

[bgruening]

I was hoping we can go the other way and replace the public galaxy servers page with a page like this one that's more descriptive and interactive.

You have my vote on this, but I don't want to be so disruptive again ;)

In addition to the ideas you listed, I'd really like to see a cross-Galaxy tool search. I keep wanting to chat to @martenson about this idea since he's been working with the search functionality (@martenson thoughts?).

Many thoughts :) but I think this is a different discussion I think. The entire federation idea is awesome and something we should heading to, tool search would be the logical first step I think.

I currently try to push the ELIXIR registry to implement a Galaxy tool search so if you like I would try to organise a meeting with @joncison and see where we can join forces.

I guess we can ingest data from here into the CloudLaunch at some point so effort wouldn't need to be duplicated. As the features are added to CloudLaunch, the idea is that any user can add their instance by just filling out a form there. I guess we'd then have tags to include public instance, themed instances, instances accessible only to the given user, etc.

Sounds great!

[bgruening]

What more could we want? (Serious question)
By default galaxy ships with part of a GRT configuration. Cloudlaunch could automatically register an instance for users and automatically apply tags to that instance like "public" or "private", and "cloudlaunch" and maybe even infrastructure run on would be nice to have tagged.

I'm sceptical about this, haven't we learnt that any kind of automatically "call home" features, no matter how transparent it is - get's a bad reputation?
I don't want to force users/admins to submit data, and they will never be anonymous. Private tools, workflows and so on can tell a lot about your instance. Implementing filter rules can solve this, but do we want to invest so much time here and do we want everyone to configure this so extensively in the end?

For tool-search I don't think a pushing mechanism will work, we create a single-point of failure for such an important feature. I think crawling is the way to go here, Google is quite successful with this I have heard ;) - but again I had the same discussion with @joncison from ELIXIR-registry fame.

Don't get me wrong I like the original GRT idea and I think for this it is ideally suited. Some big, very public instances can collect data (and communicate this to their users) and we can improve Galaxy with it. I don't see anyone using this in the future, for this we have to many private and highly secure instances (without any internet connection, behind firewalls, etc..).

Registering (manually) pure Geo data and say "Here I'm, you can not see my Galaxy instance but I'm a Galaxy expert, get in touch!" is the lowest barrier what we can offer.

Just my 2cents, I'm completely fine with abandon this idea in favour of GRT or something else. It was a test and I appreciate the discussion - that was the aim before going public :)

[bgruening]

@martenson

My idea was that you search on top of the Tool Shed list of tools with fulltext and then with the tool_id(s) you query all known public Galaxies to see where you can run the tool(set of tools).

We could use the URL from the geojson file to get all public instances. We need to communicate this properly though,

[hexylena]

I'm sceptical about this, haven't we learnt that any kind of automatically "call home" features, no matter how transparent it is - get's a bad reputation?
I don't want to force users/admins to submit data, and they will never be anonymous.

Now that we also collect metadata, I'll add a flag for not submitting job/tool data. They shouldn't be forced to submit any data they don't want to, I don't want this either.

It could be that it is because I work on it and am biased in favour of GRT, but it feels much less like an automated call home, and much more like an opt-in registry. Perhaps the language needs to be updated to reflect this better.

Private tools, workflows and so on can tell a lot about your instance. Implementing filter rules can solve this, but do we want to invest so much time here and do we want everyone to configure this so extensively in the end?

Is this really such a concern? That other people learn that I have version X of tool Y? Are tool IDs and names really sensitive data? (Are there any examples of tools that someone really would do anything to not have exposed?)

For tool-search I don't think a pushing mechanism will work, we create a single-point of failure for such an important feature.

You create an SPOF either way. Either with a pulling crawler and its site to display results, or with the point that data is pushed to.

I do not see the difference until we move to a scale with multiple servers and failover/handoff logic. At that scale, neither of them suffer from the SPOF problem, but we are a long ways from that.

I think crawling is the way to go here, Google is quite successful with this I have heard ;) - but again I had the same discussion with @joncison from ELIXIR-registry fame.

A push model additionally gets us around the firewall problem. Internal-to-university galaxies can still advertise their tools (to their own users) despite the rest of the world not being able to access them.

Don't get me wrong I like the original GRT idea and I think for this it is ideally suited. Some big, very public instances can collect data (and communicate this to their users) and we can improve Galaxy with it. I don't see anyone using this in the future, for this we have to many private and highly secure instances (without any internet connection, behind firewalls, etc..).

I hear what you are saying, that there are high security instances with different priorities. In this case, GRT is no better than the manual galaxy-maps/registry + tool search engine approach here. So, we ignore these cases because they are not relevant to the discussion. For every other galaxy, that is public, that is willing to be indexed, that is willing to have a pin on a map, GRT is a fairly convenient web-form place to do this. (I'll add a pin selector today rather than just lat/lon fields, given how much people struggle with geojson's choices there.)

Registering (manually) pure Geo data and say "Here I'm, you can not see my Galaxy instance but I'm a Galaxy expert, get in touch!" is the lowest barrier what we can offer.

Registering just the location of a trainer (and not their associated galaxyies) seems to be different than what was originally proposed. Is this in scope?

Speaking of scope, should it be expanded to:

• include the identity of the admin/trainer?
• include contact information for them?

Just my 2cents, I'm completely fine with abandon this idea in favour of GRT or something else. It was a test and I appreciate the discussion - that was the aim before going public :)

My apologies on this front Björn, for attacking your idea outright, but yes, discussion is what we were going for, hope this did not come off too aggressively/in a mean spirit, not my intention.

[bgruening]

I'm sceptical about this, haven't we learnt that any kind of automatically "call home" features, no matter how transparent it is - get's a bad reputation?
I don't want to force users/admins to submit data, and they will never be anonymous.

It could be that it is because I work on it and am biased in favour of GRT, but it feels much less like an automated call home, and much more like an opt-in registry. Perhaps the language needs to be updated to reflect this better.

Not sure about the difference ;)

Private tools, workflows and so on can tell a lot about your instance. Implementing filter rules can solve this, but do we want to invest so much time here and do we want everyone to configure this so extensively in the end?

Is this really such a concern? That other people learn that I have version X of tool Y? Are tool IDs and names really sensitive data? (Are there any examples of tools that someone really would do anything to not have exposed?)

Yes. I know about people that strictly don't want to share anything not even what they are working on, because this can give others a clue on which new technologies they are working. I'm not allowed to fill github issues for tools I will work on and such stuff :(

For tool-search I don't think a pushing mechanism will work, we create a single-point of failure for such an important feature.

You create an SPOF either way. Either with a pulling crawler and its site to display results, or with the point that data is pushed to.

I do not see the difference until we move to a scale with multiple servers and failover/handoff logic. At that scale, neither of them suffer from the SPOF problem, but we are a long ways from that.

Searching on demand (crawling) does not need such a failover it just searches, no storing needed.

I think crawling is the way to go here, Google is quite successful with this I have heard ;) - but again I had the same discussion with @joncison from ELIXIR-registry fame.

A push model additionally gets us around the firewall problem. Internal-to-university galaxies can still advertise their tools (to their own users) despite the rest of the world not being able to access them.

Now you are talking about local GRTs, isn't it? Then it's the same for a local search crawlers.

Don't get me wrong I like the original GRT idea and I think for this it is ideally suited. Some big, very public instances can collect data (and communicate this to their users) and we can improve Galaxy with it. I don't see anyone using this in the future, for this we have to many private and highly secure instances (without any internet connection, behind firewalls, etc..).

I hear what you are saying, that there are high security instances with different priorities. In this case, GRT is no better than the manual galaxy-maps/registry + tool search engine approach here.

Let's not mix up the search discussion with the "Galaxy registry discussion" here.
Galaxy-maps or the wikipage with the public Galaxy servers are just there to indicate resources (human or metal), no matter if they are private, public, secure, non-secure ....

So, we ignore these cases because they are not relevant to the discussion.

For me they are. As stated in the readme, I really want to register and map instances that are not able to submit data to the public (no matter in which way :))

This projects aims in collection all Galaxy servers across the earth. We stricly encourage all Galaxy
Admins to add their server to this map, no matter if the server is public or not. We will use this map
to guide users to you, make people and funding agencies aware of your awesome service, connect
Admins between instances more closely and build a network of potential training instances.

For every other galaxy, that is public, that is willing to be indexed, that is willing to have a pin on a map, GRT is a fairly convenient web-form place to do this. (I'll add a pin selector today rather than just lat/lon fields, given how much people struggle with geojson's choices there.)

I don't think Galaxy should overtake the registry part, for this we have the ELIXIR registry, that does way more than the registration of Galaxy instances or tools and we should support them.

Registering (manually) pure Geo data and say "Here I'm, you can not see my Galaxy instance but I'm a Galaxy expert, get in touch!" is the lowest barrier what we can offer.

Registering just the location of a trainer (and not their associated galaxyies) seems to be different than what was originally proposed. Is this in scope?

Speaking of scope, should it be expanded to:

• include the identity of the admin/trainer?
• include contact information for them?

My initial aim was to collect instances and finally creating one giant map of Galaxies :)

[hexylena]

It could be that it is because I work on it and am biased in favour of GRT, but it feels much less like an automated call home, and much more like an opt-in registry. Perhaps the language needs to be updated to reflect this better.

Not sure about the difference ;)

The technical difference is non-existent. But clearly you had a negative perception of "automated call-home."

Yes. I know about people that strictly don't want to share anything not even what they are working on, because this can give others a clue on which new technologies they are working. I'm not allowed to fill github issues for tools I will work on and such stuff :(

I'm sorry to hear that, that is unfortunate. Science should not (have to?) be so secretive.

I think this is solved in both cases, the crawling scenario would only see the public api/tools list, the GRT scenario would use a blacklist.

Searching on demand (crawling) does not need such a failover it just searches, no storing needed.

Oh, wow, you mean dynamically searching? Whenever someone puts in a query for "bowtie", some services talks to every galaxy in the universe at once to find out who has bowtie? I'm sure the ELIXIR people have plans for this that are hopefully more sophisticated than this.

A push model additionally gets us around the firewall problem. Internal-to-university galaxies can still advertise their tools (to their own users) despite the rest of the world not being able to access them.

Now you are talking about local GRTs, isn't it? Then it's the same for a local search crawlers.

No, I wasn't. If you have internet access + a university firewall + no external access to inside, then GRT would continue to function since you can push metadata out to the central GRT with your galaxy's information.

And if you don't wish to send job logs, GRT supports just registering the name / location.

Let's not mix up the search discussion with the "Galaxy registry discussion" here.
Galaxy-maps or the wikipage with the public Galaxy servers are just there to indicate resources (human or metal), no matter if they are private, public, secure, non-secure ....

Sorry for this, it comes up with GRT since that encompasses both of these functionalities in one project.

Registering human resources not attached to a Galaxy would be a strong point in favour of a separate project to track human resources, but I am worried that it is the minority case. Most of us are admins + trainers and our training is integrally related to our administration and the presence of a public Galaxy instance on which we train people.

For me they are. As stated in the readme, I really want to register and map instances that are not able to submit data to the public (no matter in which way :))

This projects aims in collection all Galaxy servers across the earth. We stricly encourage all Galaxy
Admins to add their server to this map, no matter if the server is public or not. We will use this map
to guide users to you, make people and funding agencies aware of your awesome service, connect
Admins between instances more closely and build a network of potential training instances.

I will be curious to see how popular such a thing would be. My hypothesis is: If their galaxy is not internet connected, how often are they trying to direct people to them as a training resource? I believe that will be a minority of cases. People trying to do training would more likely have some form of public galaxy that their trainees can use, or would not have a galaxy they admin but instead just want to register them self as a human resource.

GRT supports this: registering, mapping (pub + priv), guiding users (website), central bragging point for their galaxy to funding agencies (badges with "we run #1 most jobs out of all galaxies" (a really important point, imo, GRT has user/job statistics, so it can share your ranking, and you can share that with funding agencies)), and (possibly) connecting admins. Especially if we added the admin's info to the GRT page.

I don't think Galaxy should overtake the registry part, for this we have the ELIXIR registry, that does way more than the registration of Galaxy instances or tools and we should support them.

ELIXIR was just tool registry, or ...? What new features are they adding?

I do think galaxy should handle galaxy registry. It seems very strange to be sending all of our galactic instance metadata to a completely separate organisation, separate funding, to track Galaxies across the universe.

[bgruening]

I think this is solved in both cases, the crawling scenario would only see the public api/tools list, the GRT scenario would use a blacklist.
Searching on demand (crawling) does not need such a failover it just searches, no storing needed.

Oh, wow, you mean dynamically searching? Whenever someone puts in a query for "bowtie", some services talks to every galaxy in the universe at once to find out who has bowtie? I'm sure the ELIXIR people have plans for this that are hopefully more sophisticated than this.

We are mixing things up here. Please let's move the search discussion to a different thread.
ELIXIR registry is a registry storing things with an ontology in a database, searchable for everyone, you can also search in it. The Galaxy Federation idea is different imho, and searching should be also different.

This is about registering Galaxy instances, private and public ones and reusable for what ever comes to our mind because it is structure in a standard format (geojson).

Now you are talking about local GRTs, isn't it? Then it's the same for a local search crawlers.

No, I wasn't. If you have internet access + a university firewall + no external access to inside, then GRT would continue to function since you can push metadata out to the central GRT with your galaxy's information.

Ah got it!

Let's not mix up the search discussion with the "Galaxy registry discussion" here.
Galaxy-maps or the wikipage with the public Galaxy servers are just there to indicate resources (human or metal), no matter if they are private, public, secure, non-secure ....

Sorry for this, it comes up with GRT since that encompasses both of these functionalities in one project.

Back to the SPOF :)

Registering human resources not attached to a Galaxy would be a strong point in favour of a separate project to track human resources, but I am worried that it is the minority case. Most of us are admins + trainers and our training is integrally related to our administration and the presence of a public Galaxy instance on which we train people.

An other argument is that a trainer not strictly can configure Galaxy to send this data or that you need to convince your admin to send data ...

I will be curious to see how popular such a thing would be. My hypothesis is: If their galaxy is not internet connected, how often are they trying to direct people to them as a training resource? I believe that will be a minority of cases. People trying to do training would more likely have some form of public galaxy that their trainees can use, or would not have a galaxy they admin but instead just want to register them self as a human resource.

I know a few instances that are restricted and not public, yet they advertise it and offering service for others. Required VPN access etc. But training aside, it's also to state that there is someone with Galaxy experience, maybe an Admin to whom I can talk to.

GRT supports this: registering, mapping (pub + priv), guiding users (website), central bragging point for their galaxy to funding agencies (badges with "we run #1 most jobs out of all galaxies" (a really important point, imo, GRT has user/job statistics, so it can share your ranking, and you can share that with funding agencies)), and (possibly) connecting admins. Especially if we added the admin's info to the GRT page.

Eric I'm not arguing against GRT here :) It is a great project and very useful for some use cases.

I don't think Galaxy should overtake the registry part, for this we have the ELIXIR registry, that does way more than the registration of Galaxy instances or tools and we should support them.

ELIXIR was just tool registry, or ...? What new features are they adding?

Workflows, Galaxy instances, they have a large ToDo list afaik.

I do think galaxy should handle galaxy registry. It seems very strange to be sending all of our galactic instance metadata to a completely separate organisation, separate funding, to track Galaxies across the universe.

They register services and tools, Galaxy is one service under many and this fits nicely.
But again we mixing topics here. Sending _all_ metadata will most likely not happen for many instances for given reasons. I'm not sure I'm allowed to do this and I consider myself as very liberal maintainer ;). Sending parts of very restricted metadata might happen, but this can happen also with other projects that have dedicated funding for it, with an ontology and people dedicated for this.

Not discussing the search idea here - what counts for me and some kind of map project is to get as many people on board as possible, as easily as possible. I don't see this happen with GRT quickly. Especially not if I assume that not everyone will activate this for given reasons. Do you intend to activate it by default?
GRT could also just import this geojson file and add instances that are not registered and vice-versa :)

[hexylena]

We are mixing things up here. Please let's move the search discussion to a different thread.
ELIXIR registry is a registry storing things with an ontology in a database, searchable for everyone, you can also search in it. The Galaxy Federation idea is different imho, and searching should be also different.

This is about registering Galaxy instances, private and public ones and reusable for what ever comes to our mind because it is structure in a standard format (geojson).

Ok, sure, that's fine. Not talking about federation either, that's completely separate.

Back to the SPOF :)

We have these everywhere. We are not talking about building and deploying infinitely scalable services to AWS + GCP with multi-region failover, why bring this up? Is this such a big concern?

An other argument is that a trainer not strictly can configure Galaxy to send this data or that you need to convince your admin to send data ...

But a trainer can register their galaxy. This does not require admin access to a galaxy, or server access at all. You can register your galaxy with whatever subset of data you want (name, description, location) and not send job logs. This is fine by GRT.

The data would have to be updated manually, but that is no different from galaxy-maps, just through a web interface instead of hand edited files and PRs.

I know a few instances that are restricted and not public, yet they advertise it and offering service for others. Required VPN access etc. But training aside, it's also to state that there is someone with Galaxy experience, maybe an Admin to whom I can talk to.

Very interesting! People do such strange things!

I would argue that this case is covered in GRT, through the "register through the website and do nothing else" case.

Eric I'm not arguing against GRT here :) It is a great project and very useful for some use cases.

I know this Björn, I just strongly believe that GRT completely covers this precise use case here, and possibly Cloudlaunch's as well.

They register services and tools, Galaxy is one service under many and this fits nicely.
But again we mixing topics here. Sending all metadata will most likely not happen for many instances for given reasons.

Sure. That's fine, we don't ask for all of their metadata. You opt-in to providing as much as you want during 1) registration, and 2) regular crontab sending, if and only if you are wishing to submit job run logs.

I'm not sure I'm allowed to do this and I consider myself as very liberal maintainer ;). Sending parts of very restricted metadata might happen, but this can happen also with other projects that have dedicated funding for it, with an ontology and people dedicated for this.

Not discussing the search idea here - what counts for me and some kind of map project is to get as many people on board as possible, as easily as possible. I don't see this happen with GRT quickly. Especially not if I assume that not everyone will activate this for given reasons. Do you intend to activate it by default?

This is the current state.

Again, no activation necessary, it's a website you can sign up at and register your galaxy.

GRT could also just import this geojson file and add instances that are not registered and vice-versa :)

I'm already using it as test-data ;)

[hexylena]

Update: https://oc.hx42.org/grt/galaxy/

Internally I've exposed this as an API endpoint in GRT which can show the geojson data for all or one galaxy (depending on which map you wish to place, no need for you to fetch data about the entire world). You could embed these maps or use the GeoJSON any other way that you want, much like you were suggesting.