problem with option -p and finding rack (branch: multiconv)

It seems that one has now always to specify explicitly where rack is?

root@6da6703ce11a:/workspace/py-smtag# smtag-convert2th -c 190414 -f 5X_L1200_artice_embeddings -X5 -L1200 -E ".//sd-panel"                                     
ERROR:root:                                                                                                                                                     
            #######################################################################################################                                             
            #                                                                                                                                                   
            ERROR: PRODUCTION DIRECTORY DOES NOT EXIST                                                                                                          

            The specified production directory does not exist:

                /opt/conda/lib/python3.6/site-packages/smtag-2.0.2-py3.6.egg/rack


            In order to configure you production directory use one of this options, sorted by lesser precedence:
            1) Set the environment variable `SMTAG_PRODUCTION_DIRECTORY` (check your .env file)
               Examples:
                SMTAG_PRODUCTION_DIRECTORY='/absolute/path/to/resources_folder' python -m smtag.predict.egine --demo
                SMTAG_PRODUCTION_DIRECTORY='./relative/path/to/resources_folder' python -m smtag.predict.egine --demo
            2) Set the `--production_directory` flag or its abbreviated version `-p`
               Examples:
                python -m smtag.predict.egine --demo --production_directory "/absolute/path/to/resources_folder"
                python -m smtag.predict.egine --demo -p "./relative/path/to/resources_folder"
            3) Set the `smtag.config.prod_dir` programatically
               Examples:
                import smtag
                smtag.config.prod_dir = "/absolute/path/to/resources_folder"
                smtag.config.prod_dir = "./relative/path/to/resources_folder"

            If none of the above options is specified, the default directory is assumed to be:

                /opt/conda/lib/python3.6/site-packages/smtag-2.0.2-py3.6.egg/rack


            #
            #######################################################################################################

while this works

root@6da6703ce11a:/workspace/py-smtag# smtag-convert2th -c 190414 -f 5X_L1200_artice_embeddings -X5 -L1200 -E ".//sd-panel" -p rack

with

root@6da6703ce11a:/workspace/py-smtag# ls -thl
total 572M
drwxr-xr-x 63 root root 4.0K Aug  6 20:39 runs
drwxr-xr-x  2 root root 4.0K Aug  4 05:56 smtag.egg-info
drwxrwxr-x  2 1002 1002 4.0K Aug  4 05:55 docs
drwxrwxr-x  5 1002 1002 4.0K Jul 23 06:17 vsearch
-rw-rw-r--  1 1002 1002 1.2K Jul 23 06:08 Dockerfile
drwxrwxr-x  7 1002 1002 4.0K Jul 23 06:08 smtag
drwxr-xr-x  2 root root 4.0K Jul 21 14:34 __pycache__
-rw-rw-r--  1 1002 1002 2.2K Jul 21 14:13 setup.py
-rwxrwxr-x  1 1002 1002  117 Jul 19 04:31 dck
drwxrwxr-x 10 1002 1002 4.0K Jul 18 20:45 testdir
-rw-r--r--  1 1002 1002 572M Jul 18 20:41 testdir.zip
drwxrwxr-x  2 1002 1002 4.0K Jul 18 17:31 rack
drwxr-xr-x  2 root root 4.0K Jul 17 16:59 src
drwxrwxr-x  8 1002 1002 4.0K Jul 17 11:19 resources
drwxr-xr-x  2 root root 4.0K Jul 17 04:26 dist
drwxr-xr-x  4 root root 4.0K Jul  9 10:25 build
-rw-rw-r--  1 1002 1002 1.9K Jul  8 20:22 docker-compose.yml
drwxrwxr-x  2 1002 1002 4.0K Jun 18 07:49 test
-rw-rw-r--  1 1002 1002  537 Jun 18 07:49 setup.cfg
-rw-rw-r--  1 1002 1002   42 Jun 18 07:49 LICENSE.txt
-rw-rw-r--  1 1002 1002  222 Jun 18 07:49 MANIFEST.in
-rw-rw-r--  1 1002 1002  458 Jun 18 07:49 README.md

[ariera]

hmmm it is working fine for me on multiconv
do you still have the issue?

[ariera]

i think the problem may have been at the moment of running pip install -e /workspace/py-smtag (see the Dockerfile) the rack folder didn't exist at the root level of the project. In other word the /workspace/py-smtag/rack directory didn't exist.

When we run pip install py-smtag, it copies all the files to its internal destination. In this example to /opt/conda/lib/python3.6/site-packages/smtag-2.0.2-py3.6.egg.

All the files? No: only the files declared in the manifest: https://github.com/source-data/py-smtag/blob/master/MANIFEST.in#L10-L11

If you would, for example, delete the rack folder, and then run pip install then the rack folder will not be copied over to /opt/conda/python3.6...

In summary: as it is right now, rack must always exist, even if it is empty.

Could this have been the issue?

hmm, I stil lhave problem locally...
I tried to remove installation first

$ rm -fr .venv

I have a nicely populated rack:

$ ls
Dockerfile              __pycache__             docs                    runs                    smtag.egg-info          vsearch
LICENSE.txt             build                   rack                    setup.cfg               test
MANIFEST.in             dist                    rack.zip                setup.py                testdir
README.md               docker-compose.yml      resources               smtag                   testdir.zip
mac-embo184:py-smtag lemberge$ ls rack
10X_L1200_disease_articke_embeddings_128-5X_L1200_article_embeddings_128_disease_2019-08-25-21-47.zip
5X_L1200_anonym_not_reporter_article_embeddings_128_intervention_assayed_2019-08-22-16-25.zip
5X_L1200_article_embeddings_128_reporter_2019-08-28-00-08_epoch_23_.zip
5X_L1200_article_embeddings_128_small_molecule_geneprod_subcellular_cell_tissue_organism_assay_2019-08-23-17-46.zip
5X_L1200_emboj_2012_no_viz_panel_stop_2019-08-29-08-31.zip
5X_L1200_molecule_anonym_article_embeddings_128_intervention_assayed_2019-08-28-23-33_epoch_51.zip

And re-installed requirements (for my mac with tensorflow 1.14.0, setuptools 41.0.0 needed by tensorboard, otherwise it does not work):

$ python3 -m venv .venv
mac-embo184:py-smtag lemberge$ source .venv/bin/activate
(.venv) mac-embo184:py-smtag lemberge$ pip install -r smtag/requirements.txt
$ python -m setup install

And then it fails :-( What am I doing wrong?

$ smtag-predict --help
ERROR:root:
            #######################################################################################################
            #
            ERROR: PRODUCTION DIRECTORY DOES NOT EXIST

            The specified production directory does not exist:

                /Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/rack

            
            In order to configure you production directory use one of this options, sorted by lesser precedence:
            1) Set the environment variable `SMTAG_PRODUCTION_DIRECTORY` (check your .env file)
               Examples:
                SMTAG_PRODUCTION_DIRECTORY='/absolute/path/to/resources_folder' python -m smtag.predict.egine --demo
                SMTAG_PRODUCTION_DIRECTORY='./relative/path/to/resources_folder' python -m smtag.predict.egine --demo
            2) Set the `--production_directory` flag or its abbreviated version `-p`
               Examples:
                python -m smtag.predict.egine --demo --production_directory "/absolute/path/to/resources_folder"
                python -m smtag.predict.egine --demo -p "./relative/path/to/resources_folder"
            3) Set the `smtag.config.prod_dir` programatically
               Examples:
                import smtag
                smtag.config.prod_dir = "/absolute/path/to/resources_folder"
                smtag.config.prod_dir = "./relative/path/to/resources_folder"

            If none of the above options is specified, the default directory is assumed to be:

                /Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/rack
            

            #
            #######################################################################################################
        
Traceback (most recent call last):
  File "/Users/lemberge/code/py-smtag/.venv/bin/smtag-predict", line 11, in <module>
    load_entry_point('smtag==2.0.2', 'console_scripts', 'smtag-predict')()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/__init__.py", line 15, in <module>
    config = Config()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/common/config/configuration.py", line 84, in __init__
    self.prod_dir = fetch_production_directory()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/common/config/configuration.py", line 94, in prod_dir
    self.__production_directory = validated_production_directory(new_production_directory)
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/common/config/production_directory.py", line 124, in validated_production_directory
    raise ProductionDirectoryDoesNotExistError(path)
smtag.common.config.production_directory.ProductionDirectoryDoesNotExistError

[ariera]

either set SMTAG_PRODUCTION_DIRECTORY or use -p to indicate where the rack folder you want use lives.

In your case i think:

smtag-predict --help -p rack

or

SMTAG_PRODUCTION_DIRECTORY=rack smtag-predict --help

would work

is it possible to make ./rack/ the default rack so that I don't have to always add -p rack? So sorry to annoy with this... Not a priority anyway.