Skip to content

The code for method-level-summary with SparseCoder breaks python packages #1

@MariaDziuba

Description

@MariaDziuba

Script that causes the problem: $(PROJECT_ROOT)/Method-Level-Summary/SparseCoder/run.sh
Dockerfile:

FROM pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime
WORKDIR /app
COPY ./requirements.txt .
RUN apt update && \
    apt-get install git -y && \
    pip install -r requirements.txt

##############################################################################
# Installation/Basic Utilities
##############################################################################
RUN apt-get update && \
        apt-get install -y --no-install-recommends \
        software-properties-common build-essential autotools-dev \
        nfs-common pdsh \
        cmake g++ gcc \
        curl wget vim tmux emacs less unzip screen nano git \
        htop iftop iotop ca-certificates openssh-client openssh-server \
        rsync iputils-ping net-tools sudo \
        llvm-9-dev

requirements.txt

aiohttp==3.8.4
aiosignal==1.3.1
async-timeout==4.0.2
attrs==22.2.0
beautifulsoup4==4.11.2
bs4==0.0.1
certifi==2022.12.7
charset-normalizer==3.0.1
clearml==1.9.2
filelock==3.9.0
frozenlist==1.3.3
fsspec==2023.1.0
furl==2.1.3
huggingface-hub==0.12.1
idna==3.4
joblib==1.2.0
jsonschema==4.17.3
lightning-utilities==0.7.1
multidict==6.0.4
numpy==1.24.2
orderedmultidict==1.0.1
packaging==23.0
pandas==1.5.3
pathlib2==2.3.7.post1
Pillow==9.4.0
psutil==5.9.4
PyJWT==2.4.0
pyparsing==3.0.9
pyrsistent==0.19.3
python-dateutil==2.8.2
pytorch-lightning==1.9.3
pytz==2022.7.1
PyYAML==6.0
regex==2022.10.31
requests==2.28.2
scikit-learn==1.2.1
scipy==1.10.1
six==1.16.0
soupsieve==2.4
threadpoolctl==3.1.0
tokenizers==0.13.2
torch==2.0.0
torchmetrics==0.11.3
tqdm==4.64.1
transformers==4.28.1
typing_extensions==4.5.0
urllib3==1.26.14
yarl==1.8.2
boto3==1.26.94
datasketch
typer
fairscale
nltk
openai
peft==0.2.0
deepspeed==0.8.3
pygments
datasets==2.11.0
bitsandbytes==0.38.1
sentencepiece==0.1.97
notebook==6.5.4
jupyterlab==3.6.3

Error:

Traceback (most recent call last):
  File "/home/modzyuba1/SparseCoder/Method-Level-Summary/SparseCoder/run.py", line 44, in <module>
    from lsg_converter import LSGConverter
  File "/opt/conda/lib/python3.10/site-packages/lsg_converter/__init__.py", line 1, in <module>
    from .converter import LSGConverter
  File "/opt/conda/lib/python3.10/site-packages/lsg_converter/converter.py", line 6, in <module>
    from .bart.convert_bart_checkpoint import *
  File "/opt/conda/lib/python3.10/site-packages/lsg_converter/bart/convert_bart_checkpoint.py", line 1, in <module>
    from .modeling_lsg_bart import *
  File "/opt/conda/lib/python3.10/site-packages/lsg_converter/bart/modeling_lsg_bart.py", line 4, in <module>
    from transformers.modeling_attn_mask_utils import _prepare_4d_attention_mask, _prepare_4d_causal_attention_mask
ModuleNotFoundError: No module named 'transformers.modeling_attn_mask_utils'

OS version: Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-144-generic x86_64)

Steps to reproduce the error:

  1. Build an image, run container
  2. Pip install transformers ==4.37.1
  3. Then run Python in interactive mode:
python
>>> import transformers
>>> transformers.__version__ 
>>> 4.37.1
  1. Run $(PROJECT_ROOT)/Method-Level-Summary/SparseCoder/run.sh in the docker container and get the error
  2. Do step 3 again
    >>> 4.32.0
  3. Do step 2 again
    transformers==4.37.1 successfully installed
  4. Do step 3 again
    >>> 4.32.0

Expected behavior:
This code should work, because missing modules are presented in transformers 4.37.1. https://github.com/huggingface/transformers/blob/v4.37.1/src/transformers/modeling_attn_mask_utils.py
The installed version of transformers is 4.37.1. After your code runs, Python packages become broken.
The transformers version does not change after the second pip install, but should

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions