issue with parallel processing

[bahugunaashish]

I am getting the following error while running my model with parallel processing.
"
Traceback (most recent call last):
File "/home/abahugu/.conda/envs/py312/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
file = self._cache[self._key]
~~~~~~~~~~~^^^^^^^^^^^
File "/home/abahugu/.conda/envs/py312/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in getitem
file = self._cache[self._key]
~~~~~~~~~~~^^^^^^^^^^^
File "/home/abahugu/.conda/envs/py312/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in getitem
value = self._cache[key]
~~~~~~~~~~~^^^^^
value = self._cache[key]
~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/abahugu/OpenSees/_OPSTOOL_ODB/RespStepData-4.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'e8a5b23a-c486-4f8e-a61d-10b094d9c569']

During handling of the above exception, another exception occurred:
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/abahugu/OpenSees/_OPSTOOL_ODB/RespStepData-4.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'ba649dc1-31cc-4e30-a1ff-d8a6a89e1679']
"
could you please help me with this issue

[Vachan-Vanian]

Your code is trying to open or access a netCDF file (/home/abahugu/OpenSees/_OPSTOOL_ODB/RespStepData-4.nc) in append mode ('a') and there appears to be a conflict in the cache with different UUID identifiers ('e8a5b23a-c486-4f8e-a61d-10b094d9c569' and 'ba649dc1-31cc-4e30-a1ff-d8a6a89e1679')

To fix this issue, you could try:

• Use a file lock mechanism when accessing the files
• Open files in read-only mode ('r') when possible
• Implement a retry mechanism with exponential backoff
• Make sure you're properly closing files after use
• etc

But further information is needed to assess your problem.

Could you share more details about how you're implementing the parallel processing (multiprocessing, threading, or distributed computing framework)?

[bahugunaashish]

@Vachan-Vanian thank you for the reply. I am using the auto-partition option in openseespy. I am not assigning any CPU tasks manually.
following is my code snippet where I am utilizing the opstool module.

"
odb = opst.post.CreateODB(odb_tag = 4)
while (op.nodeDisp(nodetag, dof)< maxDisp):

ok = op.analyze(1)

if ok != 0:
  
print('trying again with load inc : 0.01')
  
op.integrator('LoadControl', 1e-2)
  
ok = op.analyze(1)

if ok != 0:
  op.integrator('LoadControl', 1e-3)
  ok = op.analyze(1)
  

if ok != 0:
  op.integrator('LoadControl', 1e-3)
  ok = op.analyze(1)

odb.fetch_response_step()

odb.save_response()"

[Vachan-Vanian]

@bahugunaashish the code snipped you provided is not well formatted. I guess this is what look like?

odb = opst.post.CreateODB(odb_tag = 4)
while (op.nodeDisp(nodetag, dof)< maxDisp):
    
    ok = op.analyze(1)
    
    if ok != 0:
        print('trying again with load inc : 0.01')
        op.integrator('LoadControl', 1e-2)
        ok = op.analyze(1)
    
    if ok != 0:
        op.integrator('LoadControl', 1e-3)
        ok = op.analyze(1)
    
    if ok != 0:
        op.integrator('LoadControl', 1e-3)
        ok = op.analyze(1)
    
    odb.fetch_response_step()
    
    odb.save_response()

I have 2 observations.

1. this part is repeated

    if ok != 0:
        op.integrator('LoadControl', 1e-3)
        ok = op.analyze(1)

2. You are calling odb.save_response() all the time inside while. It should be called once after while loop

    odb.save_response()

The correct code may be

odb = opst.post.CreateODB(odb_tag = 4)
while (op.nodeDisp(nodetag, dof)< maxDisp):
    
    ok = op.analyze(1)
    
    if ok != 0:
        print('trying again with load inc : 0.01')
        op.integrator('LoadControl', 1e-2)
        ok = op.analyze(1)
    
    if ok != 0:
        op.integrator('LoadControl', 1e-3)
        ok = op.analyze(1)
    
    odb.fetch_response_step()
    
odb.save_response()

P.S. what you mean by "auto-partition option in openseespy" ?
there is partition command https://openseespydoc.readthedocs.io/en/latest/src/partition.html

[bahugunaashish]

@Vachan-Vanian, thanks, I appreciate your help; I apologize for the inconvenience. this is what my code looks like;

auto-partition means I am not using any separate function to assign the task to cpus. I am just using op. partition() command at the end of defining my model to partition.

Thanks,
Ashish

[yexiang92]

@bahugunaashish Hi, this seems to be an exception in xarray's file management when parallelizing. Can you explain how you parallelize it?

[bahugunaashish]

Hey @yexiang1992, thank you for developing this cool post and pre-processing module for Openseespy. If I understand your question correctly, you're asking how I parallelize the opstool in my script; I didn't, I am just calling it in the loop as we do for sequential analysis. and if you are asking how I parallelize my model I am just using op.partition command at the end of model definition it works fine with opensees recorder command. If it's the first one, do I need to follow a different procedure to call the opstool module function so it will work with parallel processing?

Thanks,
Ashish

[yexiang92]

@bahugunaashish You're welcome. I need a complete minimized example to reproduce the error, can you provide it?

[bahugunaashish]

@yexiang1992, here is the example. I have used one of your models, the "Pier" 3D model, for this purpose. please find the attached file, and I have added a cluster log file for your reference. Thank you for helping me.

slurm-2510415.txt
testpy.txt

Thanks,
Ashish

[yexiang92]

@yexiang1992, here is the example. I have used one of your models, the "Pier" 3D model, for this purpose. please find the attached file, and I have added a cluster log file for your reference. Thank you for helping me.

slurm-2510415.txt testpy.txt

Thanks, Ashish

It seems that you use a parallel version of OpenSeesPy, if so you should use different odb_tag for different processors. Try it.

[bahugunaashish]

@yexiang1992 thank you for the suggestion. Could you explain how can I use different tags for different processors?

[yexiang92]

@bahugunaashish

If you are simply parallelizing several independent models, I recommend using the joblib module.

One possible way to write it:

`
from joblib import Parallel, delayed

def model(i):
# something
opst.createODB(odb_tag=i+1)

Parallel(n_jobs=-1)(delayed(model)(i) for i in range(10))
`

Embarrassingly parallel for loop

[bahugunaashish]

@yexiang1992, Thank you for the suggestion and clarification

[bahugunaashish]

Hey @yexiang1992! Can we record the response of the model at a specific time step with this module? I have a large model that has around 0.5 million elements. I have limited memory in my system, if somehow I can reduce the size of the response file.

Thanks,
Ashish

[yexiang92]

Hi, @Vachan-Vanian, you have many ways to reduce the size, for example:

for i in range(N):
if i % 10 == 0: # or other n
fetch_response_step()

Also, you can save only specific element tags, see the parameters of CreateODB for details.

[bahugunaashish]

@yexiang1992, Thanks for the suggestion.