[BUG]TaskGraph UI lacking "Add Nodes" menu item to start working

[complyue]

Describe the bug

The actual RMB click menu appears very different than the screenshot animation, especially there is no [Add Nodes] entry so no way to start working at all.

Steps/Code to reproduce bug

Open notebook https://github.com/NVIDIA/fsi-samples/blob/main/gQuant/plugins/gquant_plugin/notebooks/01_tutorial.ipynb

Try follow the steps demonstrated in the screenshot animation, but right mouse button click gives a menu UI like this:

Expected behavior

To have the same or equivalent UI to accomplish the steps.

Environment overview (please complete the following information)

• Environment location: local
• Method of gQuant install: [QST] Upgrade gquant_plugin's rapids dependency to v21.x? #153 (comment)

Environment details
Please run and paste the output of the /print_env.sh script here, to gather any other relevant environment details

N/A

Additional context

https://github.com/NVIDIA/fsi-samples/blob/main/gQuant/plugins/gquant_plugin/notebooks/cuIndicator/indicator_demo.ipynb would err out like this:

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Input In [10], in <module>
      6 task_list = [task_load_csv_data, task_sort]
      7 task_graph = TaskGraph(task_list)
----> 9 df = task_graph.run(outputs=['sort.out'])[0]
     11 def one_stock(df, stock_id):
     12     return df.query('asset==%s' % stock_id)

File ~/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/taskGraph.py:753, in TaskGraph.run(self, outputs, replace, profile, formated, build)
    751     return result
    752 else:
--> 753     return self._run(outputs=outputs, replace=replace, profile=profile,
    754                      formated=formated, build=build)

File ~/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/taskGraph.py:524, in TaskGraph._run(self, outputs, replace, profile, formated, build)
    521 replace = dict() if replace is None else replace
    523 if build:
--> 524     self.build(replace, profile)
    525 else:
    526     if replace:

File ~/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/taskGraph.py:454, in TaskGraph.build(self, replace, profile)
    452 profile = False if profile is None else profile
    453 # make connection only
--> 454 self._build(replace=replace, profile=profile)
    456 # Columns type checking is done in the :meth:`TaskGraph._run` after the
    457 # outputs are specified and participating tasks are determined.
    458 
    459 # this part is to update each of the node so dynamic inputs can be
    460 # processed
    461 self.breadth_first_update()

File ~/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/taskGraph.py:405, in TaskGraph._build(self, replace, profile)
    403         node = get_node_obj(output_task, tgraph_mixin=True)
    404     else:
--> 405         node = get_node_obj(task, replace.get(task_id), profile,
    406                             tgraph_mixin=True)
    407     self.__node_dict[task_id] = node
    409 # build the graph

File ~/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/config_nodes_modules.py:249, in get_node_obj(task, replace, profile, tgraph_mixin, dask_ray_setup)
    246                 continue
    248 if NodeClass is None:
--> 249     raise Exception("Cannot find the Node Class:" +
    250                     node_type)
    252 if module_dir:
    253     append_path(module_dir)

Exception: Cannot find the Node Class:CsvStockLoader

[avolkov1]

Could you look in the terminal where jupyter lab was launched from and share if you have any errors printed out?

If one of the nodes fails to load, then the "Add Nodes" menu will not show up. Most likely it's a misconfiguration of one of the plugins. We should enhance Greenflow to skip faulting nodes and/or plugins, and print out exceptions explaining the cause. But currently, you have to look in the terminal for the errors to find the faulty node and/or plugin.

In the error you shared Exception: Cannot find the Node Class:CsvStockLoader it looks like maybe you don't have the gquant plugin installed. Install either from source code or from pip.

cd gQuant/plugins/gquant_plugin
pip install .

or PyPi:

pip install greenflow-gquant-plugin

If you cd into the "gQuant/plugins/gquant_plugin/notebooks" directory launch jupyter lab in that directory.

If you cd into the "gQuant/plugins/gquant_plugin" and launch jupyter lab from this top level directory there's a "greenflowrc" file that has:

[ModuleFiles]
my_node= %(MODULEPATH)s/my_node.py

You can just mv/rm that greenflowrc.

Alternatively start jupyter lab with a MODULEPATH environment. Example:

# corresponds to: "gQuant/plugins/gquant_plugin/modules"
export MODULEPATH=${PWD}/modules
jupyter lab --ip=0.0.0.0 # etc...

The UI will be slightly different from the animation, since the animations were made a while ago. But it will be similar enough to follow along.

[complyue]

I refined my start dir/cmdl a bit according to your suggestion, and here is the background terminal log:

(fsi) dw@cnode:~/nvidia/fsi-samples/gQuant/plugins/gquant_plugin/notebooks$ MODULEPATH=/workspace/nvidia/fsi-samples/gQuant/plugins/gquant_plugin/modules/modules jupyter-lab --allow-root --ip=0.0.0.0 --no-browser --NotebookApp.token=''
NumExpr defaulting to 8 threads.
[I 2022-01-19 15:28:01.986 ServerApp] greenflowlab | extension was successfully linked.
[I 2022-01-19 15:28:01.986 ServerApp] jupyter_server_proxy | extension was successfully linked.
[W 2022-01-19 15:28:01.990 LabApp] 'token' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2022-01-19 15:28:01.995 ServerApp] jupyterlab | extension was successfully linked.
[I 2022-01-19 15:28:02.012 ServerApp] nbclassic | extension was successfully linked.
[W 2022-01-19 15:28:02.075 ServerApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2022-01-19 15:28:02.089 ServerApp] nbclassic | extension was successfully loaded.
[I 2022-01-19 15:28:02.089 ServerApp] Registered greenflowLab extension at URL path /greenflowlab
[I 2022-01-19 15:28:02.089 ServerApp] greenflowlab | extension was successfully loaded.
[I 2022-01-19 15:28:02.117 ServerApp] jupyter_server_proxy | extension was successfully loaded.
[I 2022-01-19 15:28:02.118 LabApp] JupyterLab extension loaded from /workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/jupyterlab
[I 2022-01-19 15:28:02.118 LabApp] JupyterLab application directory is /workspace/anaconda3/envs/fsi/share/jupyter/lab
[I 2022-01-19 15:28:02.122 ServerApp] jupyterlab | extension was successfully loaded.
[I 2022-01-19 15:28:02.122 ServerApp] Serving notebooks from local directory: /workspace/nvidia/fsi-samples/gQuant/plugins/gquant_plugin/notebooks
[I 2022-01-19 15:28:02.122 ServerApp] Jupyter Server 1.13.2 is running at:
[I 2022-01-19 15:28:02.123 ServerApp] http://cnode:8888/lab
[I 2022-01-19 15:28:02.123 ServerApp]  or http://127.0.0.1:8888/lab
[I 2022-01-19 15:28:02.123 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2022-01-19 15:28:15.515 LabApp] Build is up to date

GREENFLOW_CONFIG NOT SET. SETTING TO: /workspace/nvidia/fsi-samples/gQuant/plugins/gquant_plugin/notebooks/greenflowrc
ServerApp - ERROR - Uncaught exception GET /greenflowlab/all_nodes?1642577295274 (192.168.14.89)
HTTPServerRequest(protocol='http', host='192.168.13.120:8888', method='GET', uri='/greenflowlab/all_nodes?1642577295274', version='HTTP/1.1', remote_ip='192.168.14.89')
Traceback (most recent call last):
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/tornado/web.py", line 3173, in wrapper
    return method(self, *args, **kwargs)
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflowlab/handlers.py", line 115, in get
    result = add_nodes()
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflowlab/server_utils.py", line 353, in add_nodes
    nodeObj = init_get_labnode_obj(node_inst, False)
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflowlab/server_utils.py", line 135, in init_get_labnode_obj
    node.update()
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/_node_flow.py", line 146, in update
    NodeTaskGraphExtensionMixin.cache_update_result(self)
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/_node_taskgraph_extension_mixin.py", line 33, in cache_update_result
    self.meta_data_cache = self.meta_setup()
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow/dataframe_flow/_node_flow.py", line 176, in meta_setup
    meta = icls.meta_setup(self)
  File "/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/greenflow_gquant_plugin/ml/gridRandomSearchNode.py", line 635, in meta_setup
    from ray.tune import Analysis
ImportError: cannot import name 'Analysis' from 'ray.tune' (/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/ray/tune/__init__.py)
[W 2022-01-19 15:28:15.831 ServerApp] Unhandled error
[E 2022-01-19 15:28:15.832 ServerApp] {
      "Host": "192.168.13.120:8888",
      "Connection": "keep-alive",
      "X-Xsrftoken": "2|035ad2e6|ce58f34da9585e9e3089b2e87121921a|1642412463",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36",
      "Content-Type": "application/json",
      "Accept": "*/*",
      "Referer": "http://192.168.13.120:8888/lab",
      "Accept-Encoding": "gzip, deflate",
      "Accept-Language": "zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7",
      "Cookie": "_xsrf=2|035ad2e6|ce58f34da9585e9e3089b2e87121921a|1642412463"
    }
[E 2022-01-19 15:28:15.832 ServerApp] 500 GET /greenflowlab/all_nodes?1642577295274 (192.168.14.89) 315.69ms referer=http://192.168.13.120:8888/lab

Seems the direct failure is ImportError: cannot import name 'Analysis' from 'ray.tune' (/workspace/anaconda3/envs/fsi/lib/python3.8/site-packages/ray/tune/__init__.py) , I guess it's due to dependency version mismatch?

I created the fsi conda env with just python=3.8 and followed #153 (comment) to further populate it like:

export RAPIDS_VERSION=21.06
export RAPIDS_CUDA_VER=11.2

conda install -y mamba -n base -c conda-forge
mamba install -y -c rapidsai -c nvidia -c conda-forge -c defaults \
      rapids=$RAPIDS_VERSION cudatoolkit=$RAPIDS_CUDA_VER python=3.8 && \
    conda clean --all -y

mamba install -y -c conda-forge -c defaults \
      jupyterlab'>=3.0.0' jupyter-packaging'>=0.9.2' jupyterlab-system-monitor \
      nodejs=12.4.0 python-graphviz pydot ruamel.yaml && \
    conda clean --all -y && \
    jlpm cache clean && \
    jupyter lab clean

pip install bqplot==0.12.21 && \
    jlpm cache clean && \
    jupyter lab clean

# or install from PyPi https://pypi.org/project/greenflowlab/
pip install greenflowlab

# greenflowlab will install greenflow as well.

# plugins are here https://github.com/NVIDIA/fsi-samples/tree/main/gQuant/plugins
# cd to each directory and install
cd gQuant/plugins/gquant_plugin
pip install .

I installed greenflowlab/greenflow from pip, but gquant_plugin from the locally checkout repo, not quite sure what combination should be more desirable?

[complyue]

Update:

greenflow_gquant_plugin doesn't state its dependency version for ray, but as https://github.com/NVIDIA/fsi-samples/commits/main/gQuant/plugins/gquant_plugin/greenflow_gquant_plugin/ml/gridRandomSearchNode.py is lastly updated on May 6, 2021, I tried pip install -Iv ray==1.3.0 which was released on Apr 23, 2021.

Though pip would wrongly upgrade numpy to 1.22.1 causing numba to fail, I managed to fool conda/mamba to downgrade numpy with mamba install 'numpy=1.21' followed by mamba install 'numpy<1.21'.

Now I have good looking [Add Nodes] UI, but indicator_demo.ipynb still suffers the Exception: Cannot find the Node Class:CsvStockLoader, with no error log in the background terminal window.

[complyue]

A lucky guess took me a step forward w.r.t. indicator_demo.ipynb, if I change:

task_load_csv_data = {
    TaskSpecSchema.task_id: "load_csv_data",
    TaskSpecSchema.node_type: "CsvStockLoader",
    TaskSpecSchema.conf: {"file": "../data/stock_price_hist.csv.gz"},
    TaskSpecSchema.inputs: {}
}

To:

task_load_csv_data = {
    TaskSpecSchema.task_id: "load_csv_data",
    TaskSpecSchema.node_type: "CsvStockLoader",
    TaskSpecSchema.conf: {"file": "../data/stock_price_hist.csv.gz"},
    TaskSpecSchema.inputs: {},
    TaskSpecSchema.module: 'greenflow_gquant_plugin.dataloader',
}

Then it'll fail with Exception: Cannot find the Node Class:SortNode instead of Exception: Cannot find the Node Class:CsvStockLoader.

So is this the way codeful graph nodes are supposed to be written? Should I report a bug against indicator_demo.ipynb and fix it somehow?

[avolkov1]

Update:

greenflow_gquant_plugin doesn't state its dependency version for ray, but as https://github.com/NVIDIA/fsi-samples/commits/main/gQuant/plugins/gquant_plugin/greenflow_gquant_plugin/ml/gridRandomSearchNode.py is lastly updated on May 6, 2021, I tried pip install -Iv ray==1.3.0 which was released on Apr 23, 2021.

Though pip would wrongly upgrade numpy to 1.22.1 causing numba to fail, I managed to fool conda/mamba to downgrade numpy with mamba install 'numpy=1.21' followed by mamba install 'numpy<1.21'.

Now I have good looking [Add Nodes] UI, but indicator_demo.ipynb still suffers the Exception: Cannot find the Node Class:CsvStockLoader, with no error log in the background terminal window.

So looks like the ray API has changed. We tested in Summer June 2021 (hence rapids 21.06). There is a dependency for gquant plugin on ray:

fsi-samples/gQuant/plugins/gquant_plugin/setup.py

Line 7 in 6ee3ba1

"greenflow", "bqplot", "tables", "ray[tune]", "matplotlib", "ray[default]",

Reinstall ray in your python environment like this:

pip install ray[default]==1.6.0 ray[tune]==1.6.0

Let's focus on "01_tutorial.ipynb" and I can help out with "indicator_demo.ipynb" after getting greenflow running.

[complyue]

Let's focus on "01_tutorial.ipynb"

Sure, the UI works for me now w.r.t. notebooks.

Shall we close this issue now, or leave it open to track an explicitly version requirement of ray by gquant_plugin package?

pip install ray[default]==1.6.0 ray[tune]==1.6.0 is a working workaround at least.

[avolkov1]

Sure, the UI works for me now w.r.t. notebooks.
Shall we close this issue now, or leave it open to track an explicitly version requirement of ray by gquant_plugin package?
pip install ray[default]==1.6.0 ray[tune]==1.6.0 is a working workaround at least.

Good, I'm happy the UI works and you're able to see the plugins in the "Add Nodes" menu.

We can leave this issue open. Once I get around to it I'll update the setup in gquant plugin and pin the ray version. Then I'll close this issue. Thanks for uncovering it.