Unable to convert synergi to opendss in Ubuntu 22.04

[AAndersn]

Unable to convert from synergi to opendss using ditto-cli or python library.

Machine: Ubuntu 22.04

Ditto Branch: tried with both fuse_wenbo and synergi_read_fixes

Models: PNNL Campus and Nantucket (private models)

Error message:

/home/shared_user/ditto/ditto/readers/synergi/mdbtools/bin/mdb-tables: error while loading shared libraries: libmdb.so.2: cannot open shared object file: No such file or directory
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[2], line 1
----> 1 test_Synergi_to_opendss()

Cell In[1], line 36, in test_Synergi_to_opendss()
     34 m = Store()
     35 r = Reader(data_folder_path=os.path.join(current_directory, './nantucket',model))
---> 36 r.parse(m)
     37 #TODO: Log properly
     38 # print('>Synergi model {model} read...'.format(model=model))
     39 output_path = tempfile.TemporaryDirectory()

File ~/ditto/ditto/readers/synergi/read.py:130, in Reader.parse(self, model)
    126 if self.ware_house_input_file is not None:
    127     self.ware_house_input_file = os.path.join(
    128         os.path.dirname(self.input_file), self.ware_house_input_file
    129     )
--> 130     self.SynergiData = DbParser(
    131         self.input_file, warehouse=self.ware_house_input_file
    132     )
    133 else:
    134     self.SynergiData = DbParser(self.input_file)

File ~/ditto/ditto/readers/synergi/db_parser.py:25, in DbParser.__init__(self, input_file, **kwargs)
     22 if "warehouse" in kwargs:
     23     self.paths["warehouse"] = kwargs["warehouse"]
---> 25 self.ParseSynergiDatabase()

File ~/ditto/ditto/readers/synergi/db_parser.py:32, in DbParser.ParseSynergiDatabase(self)
     28 """
     29 Use Pandas Access to convert the MDB tables to Pandas DataFrames.
     30 """
     31 print("Opening synergie database - ", self.paths["Synergi File"])
---> 32 table_list = mdb.list_tables(self.paths["Synergi File"])
     34 table_list_warehouse = []
     35 if "warehouse" in self.paths:

File ~/ditto/ditto/readers/synergi/pandas_access.py:43, in list_tables(rdb_file, encoding)
     39     tables = subprocess.check_output(
     40         [os.path.join(path_to_mdbtools_win, "mdb-tables.exe"), rdb_file]
     41     ).decode(encoding)
     42 else:
---> 43     tables = subprocess.check_output(
     44         [os.path.join(path_to_mdbtools, "mdb-tables"), rdb_file]
     45     ).decode(encoding)
     46 return tables.strip().split(" ")

File /usr/lib/python3.10/subprocess.py:421, in check_output(timeout, *popenargs, **kwargs)
    418         empty = b''
    419     kwargs['input'] = empty
--> 421 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    422            **kwargs).stdout

File /usr/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    524     retcode = process.poll()
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['/home/shared_user/ditto/ditto/readers/synergi/mdbtools/bin/mdb-tables', 'input.mdb']' returned non-zero exit status 127.

Have attempted to force installation of libmdb2 but library no longer exists. Options through apt-get are libdmb3 and mdbtools

[daniel-thom]

I suspect that the root cause of your problem is that ditto is downloading an mdbtools package that is incompatible with your Linux system. Would you be willing to try a workaround? You may be able to install mdbtools manually and then change the ditto source code to use those binaries instead of the ones downloaded by ditto.

$ apt install mdbtools

Open ~/ditto/ditto/readers/synergi/pandas_access.py. Replace every use of path_to_mdbtools with only the executable. All executables should be in your system path. For example, line 43 should be changed

from

        tables = subprocess.check_output(
            [os.path.join(path_to_mdbtools, "mdb-tables"), rdb_file]
        ).decode(encoding)

to

        tables = subprocess.check_output(
            ["mdb-tables", rdb_file]
        ).decode(encoding)

Note that you may be the first person to use this version of mdbtools with ditto, and there could be unexpected results.

[singha42]

The solution worked, not getting the libmdb errors anymore. Now, ditto is looking for warehouse.mdb which we do not have.

[nadiavp]

The solution worked, not getting the libmdb errors anymore. Now, ditto is looking for warehouse.mdb which we do not have.

The warehouse.mdb is a file which has all the component specs that is referenced by the other mdb files. It may be named something different in your setup, but should contain tables starting with the name Dev. For example DevConductors would list all the conductor types used in the model and their specifications.

[daniel-thom]

This example in the code should fix the issue once you identify your warehouse file.

    """
    Synergi Reader class.

    **Usage:**
    - With only one MDB database holding everything:
        >>> r = Reader(input_file="path_to_your_mdb_file")
        >>> r.parse(m)

    - With an additional MDB database for the Warehouse:
        >>> r = Reader(input_file="path_to_your_mdb_file", warehouse="path_to_your_warehouse_mdb_file")
        >>> r.parse(m)

[singha42]

Is there any synergi test system that could be used to test the conversion? I only have one mdb file that all the tables regarding feeder information.

[daniel-thom]

I don’t see any synergi test systems in the repo.

Are you still getting an error for warehouse.mdb? I think there is another error path. Are you creating a Reader as in the example above or are you using the ditto CLI command for convert? If you are using the CLI then the problem is likely this line: https://github.com/NREL/ditto/blob/master/ditto/converter.py#L150. It will force the reader to look for warehouse.db when that file is not needed.

[singha42]

Thanks Dan. I tried commenting that line but I see in
https://github.com/NREL/ditto/blob/41b93f954af5836cbe5986add0c104b19dc22fde/ditto/readers/synergi/db_parser.py#L36
warehouse is used everywhere and ditto-cli still throws this error:

shared_user@gridatlas-pnnl-internal:~$ ditto-cli convert --from synergi --input /home/shared_user/grid_atlas_models/input_models/synergi/pnnlcampus/FeederModels.mdb --to opendss --output ./
Log file currently not supported, please contact the developers for information on how to generate log files
Opening synergie database - /home/shared_user/grid_atlas_models/input_models/synergi/pnnlcampus/FeederModels.mdb
Opening warehouse database - /home/shared_user/grid_atlas_models/input_models/synergi/pnnlcampus/warehouse.mdb
File not found
Couldn't open database.
Traceback (most recent call last):
File "/home/shared_user/.local/bin/ditto-cli", line 33, in
sys.exit(load_entry_point('ditto.py', 'console_scripts', 'ditto-cli')())
File "/home/shared_user/.local/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/shared_user/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/shared_user/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/shared_user/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/shared_user/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/shared_user/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/shared_user/ditto/ditto/cli.py", line 183, in convert
).convert()
File "/home/shared_user/ditto/ditto/converter.py", line 245, in convert
self.reader.parse(self.m)
File "/home/shared_user/ditto/ditto/readers/synergi/read.py", line 130, in parse
self.SynergiData = DbParser(
File "/home/shared_user/ditto/ditto/readers/synergi/db_parser.py", line 25, in init
self.ParseSynergiDatabase()
File "/home/shared_user/ditto/ditto/readers/synergi/db_parser.py", line 37, in ParseSynergiDatabase
table_list_warehouse = mdb.list_tables(self.paths["warehouse"])
File "/home/shared_user/ditto/ditto/readers/synergi/pandas_access.py", line 43, in list_tables
tables = subprocess.check_output(
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mdb-tables', '/home/shared_user/grid_atlas_models/input_models/synergi/pnnlcampus/warehouse.mdb']' returned non-zero exit status 1.

[daniel-thom]

Yeah, there are more bugs here. The warehouse file needs to exist.

[wenbowangnrel]

You can use synergi software to load in your single mbd file, then save to warehouse.mdb. Then use the newly saved file in the conversion process.