Feature/annotator plugin #5
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s for easier run of roger helm chart.
Consolidated helm
Adding Dug First pass work on Annotate and Normalize to Develop
Roger dug extension
* Util class addition of dug file method, and a few modificaitons on graph building. * - biolink 1.5.0 support. Latest kgx files from automat - Unified edges and nodes in single file support in fetch - metadata update to use real kgx files - Data conversion utils to treat none values - Consise requirements.txt - BMT version of resolving leaf types. * adding some logging for debug * more debug logging * removing intro for debug * arg complete in requirments * adding missing requirements * parse dd xml files and build kg * fixed function call * make kg dd xml * make kg dd_xml * methods to get dd xml files * fixed error - get dd xml objects * return dd xml files and load step * modified edge labels * make convert_json_to_kg a static method * removed duplicate code * Some changes on node and edge predicates and labels. (#8) * make edges and nodes biolink model compatible * make edges and nodes biolink model compatible * Indexing changes * Dag util for common functions across dags * adding dug 1.0.3 in requirements.txt * Topmed files check move to dug utils * typo fix * config for elastic * fixing bad dir * path correction for topmed file detection. * fixing var identifier addition * some more mods * indexing error for dict keys being passed * Deployment overhaul * pushing temp pass for testing * data dir for test * returning back config, adding logs volume for k8s tasks * docker correction and dug-test new ver * back to stable dug * Common cli redis upgrade, * Delete dug_utils.cpython-38.pyc * install packages in airflow user home
* Config from env variables
* testing out general execption
* general exception testing
* using different prefix, ROGER_ seems to conflict with other vars.
* removing general exception catch block
* testing passing env as is to task runner pods
* adding name to container incase matching is done there.
* plain envs
* need to be dict ?
* trying something different
* testing plain list
* testing with password visible for testing
* making passwords hidden
* Testing secrets mounts
* typo
* removing config map ref and using cfg value from chart ("AIRFLOW__KUBERNETES__ENV_FROM_CONFIGMAP_REF")
* config prefix correction for secrets
* testing redis pass toggling visiblilty
* back to redacted password
* elastic search config, redis password and generic roger configmap.
* yaml ref's don't work with helm well, tranql redis password config ,
* reads concepts and elements pickle files. * Initial changes to make annoation work with dug develop latest * A little more cleanup towards collecting configs for dug in config.yaml, deleting duplicate logger.
* Update metadata.yaml * Update metadata.yaml update cde * first version that uses the disk based merge from the kg_utils package * updated as per Yaphet's suggestions * removed unneeded json.dumps and decode call when writing the nodes and the edges * added the bmt requirement and bumped the required version of kg-utils * Correct the bmt include * Hashed the subject, object and predicate to make an id field * updating bmt * update bmt * update bmt * added a comment * some debugging code to try to see why the create schema task is hanging * more debugging * less logging * more logging * Update requirements.txt * remove logging * Update core.py catch schema key exception * Update core.py revert * debug exception * schema for all keys * Update core.py --------- Co-authored-by: YaphetKG <45075777+YaphetKG@users.noreply.github.com> Co-authored-by: YaphetKG <kebedey@renci.org>
* renaming another util file to something more descriptive (logger) * renaming roger_db -> RedisGraph * renaming dag_util -> tasks * Deprecated DummyOperator -> EmptyOperator, imports.
Restructure code to break up core.py into submodules (within core and models), adjust tests accordingly, increase conformity to PEP8/pylint standards, and somewhat simplify code organization. * Data models have each been given their own submodule within the new `models` submodule * roger.core still exists as a module but is now a module folder which imports a limited set of objects from submodules * Top level roger.core elements now in `roger.core.base` * roger.core.Util has been converted from a static class into a plain module as roger.core.storage * Many long lines broken following PEP8 conventions, trailing whitespace removed from many files. Named lambdas changed to use `sub` instead of direct assignment (following PEP8) * Tests lightly refactored such that mocks comport to `storage` module rather than `Util` static class. Mock targets
…e tests restored to functioning
Restructure code to break up core.py into submodules (within core and models), adjust tests accordingly, increase conformity to PEP8/pylint standards, and somewhat simplify code organization.
Data models have each been given their own submodule within the new models submodule
roger.core still exists as a module but is now a module folder which imports a limited set of objects from submodules
Top level roger.core elements now in roger.core.base
roger.core.Util has been converted from a static class into a plain module as roger.core.storage
Many long lines broken following PEP8 conventions, trailing whitespace removed from many files. Named lambdas changed to use sub instead of direct assignment (following PEP8)
Tests lightly refactored such that mocks comport to storage module rather than Util static class. Mock targets moved from declaring contexts to calling contexts, as recommended by unittest.mock documentation.
This reverts commit b17ea15.
Converted core.py into full submodule, helm chart and old data removed, several minor style and pattern fixes
* commiting changes * remove newlines from descriptions of study variables * add asyncio run for index removal * some debug message * use common event tool * Update requirements.txt
* update dug * add things for mds run * add logging * fix dir to objects * correct format * more correction * remove extra task * removes airflow extra task * filter xml files remove mds stuff * remove import * add mds import task to be able to pull in mds files * trying out list of s3 objs * fix merge conflict * Update requirements.txt * Update requirements.txt * error debug * normalize category to array * normalize category to array * normalize category to array * normalize category to array * normalize category to array * normalize category to array * adding mapping config * fix typo * remove debug code * kg-utils 0.0.6 * dug latest
* dug version change * debug * more debug * more debug * more debug * rename config for list pick first * bump dug version
* roger cli preped for Merge Deploy * Update Makefile to work with python env * Update redisgraph-bulk-loader to fix issue with loading MODULE LIST * Revert "Update redisgraph-bulk-loader to fix issue with loading MODULE LIST" This reverts commit 7baf7ef. * Finalized dev deployment of dug inside Catapult Merge, deployment yamls, code changes and configurations * updated to reflect the Dug-Api updates to FastAPI * adding multi label redis by removing 'biolink:' on nodes, edges cannot be fixed after update so they need to be solved either by changing TranQl AND Plater or forking bulk-redisgraph to allow for colons to be added in the edges * Working multi label redis nodes w/ no biolink label * Latest code changes to deploy working Roger in Merge * biolink data move to '.' separator * updates to include new dug fixes, upgraded redis-bulk-loader and made changes to for biolink variables to specify it's domain with a 'biolink.' * adding test roger code * removed helm deployments * change docker owner * remove core.py * remove dup dev config * redis graph is not directly used removing cruft * remove print statement * remove logging files * update requriemtns * update requriemtns * add redis graph.py * fix import error for logger * adding es scheme and ca_path config * adding es scheme and ca_path config * adding debug code * removing debug * adding nodes args * adding biolink. * adding biolink. * Update requirements.txt * Update .gitignore * Update dug_utils.py Handle Error when curie not found in validate * Update __init__.py * Update config.yaml * Update dev-config.yaml * Update docker-compose.yaml * fixed docker-compose * adding back postgres volume to docker compose * env correction , docker compose updates --------- Co-authored-by: Nathan Braswell <nbraswell@rti.org> Co-authored-by: esurface <esurface@rti.org> Co-authored-by: braswent <braswent6@gmail.com>
Annotator dict for multiple annotator support
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added this code to use annotator plugins on the existing develop branch to implement addition annotators in the future