xsd2py failing

[thomas-mangin]

Hello Felix,

I never had to use xsd/wsdl/soap before but I am trying to find a library which can help me implement correctly a tr-069 server (instead of just generating/parsing the XML 'by hand') like many available open source projects.

soapfish seems to be a good contender for the job but before I can even get started I have this issue.
Any help would be welcomed ! Thank you.

# xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/usr/local/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish', 'console_scripts', 'xsd2py')()
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 171, in main
    code = generate_code_from_xsd(xmlelement, encoding='utf-8')
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 69, in generate_code_from_xsd
    standalone=True)
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 151, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/thomas/source/tr-69/soapfish/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_files, schema.targetNamespace, cwd, base_path) }}
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 48, in resolve_import
    location = os.path.relpath(location, base_path)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 430, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 360, in abspath
    if not isabs(path):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 54, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'

[ngnpope]

I have been working on a lot of fixes and this one might be solved in my improvements branch by the commit 828e4a5. I still have a few more things to finish before I submit a pull request.

According to the specification xsd:import[schemaLocation] and xsd:include[schemaLocation] are optional. Please try the branch to see if you have any success. There are also some other issues with generation of names for schemas based on location that I am currently attempting to fix.

[FelixSchwarz]

@thomas-mangin Can you try Nick's branch as suggest in his comment above?

[thomas-mangin]

I was pretty sure to run this branch but to make absolutely sure, I done this ( OSX )

# git branch -v
* improvements 58e7fc3 misc: drop support for django<1.6.0
  master       2407838 merge element reordering for correct XSD restriction/extension rendering (by Iuri de Silvio)

# git log | head -8
commit 58e7fc39e961ff7ccb644e36f5e3024345ba6eb8
Author: Nick Pope <nick.pope@flightdataservices.com>
Date:   Mon Mar 21 16:32:36 2016 +0000

    misc: drop support for django<1.6.0

    License: BSD

# pip uninstall soapfish
Uninstalling soapfish-0.6.0.dev0:
  /usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg
Proceed (y/n)? y
  Successfully uninstalled soapfish-0.6.0.dev0

# cd  /usr/local/bin
# rm xsd2py wsdl2py py2xsd py2wsdl
# cd -

# python setup.py install
[...]
Using /usr/local/lib/python2.7/site-packages
Finished processing dependencies for soapfish==0.6.0.dev0

# find /usr/local/lib/python* | grep soapfish | cut -f1-7 -d/ | uniq
/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg

# find /usr/local/opt/python* | grep soapfish

# which xsd2py
/usr/local/bin/xsd2py

# cat /usr/local/bin/xsd2py
#!/usr/local/opt/python/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'soapfish==0.6.0.dev0','console_scripts','xsd2py'
__requires__ = 'soapfish==0.6.0.dev0'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
    )

/usr/local/bin/xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/usr/local/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 171, in main
    code = generate_code_from_xsd(xmlelement, encoding='utf-8')
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 69, in generate_code_from_xsd
    standalone=True)
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 151, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_files, schema.targetNamespace, cwd, base_path) }}
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 48, in resolve_import
    location = os.path.relpath(location, base_path)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 430, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 360, in abspath
    if not isabs(path):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 54, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'

So it seems the bug is still present in this branch -- sorry 😢

[thomas-mangin]

And sorry - I should have said I was running Nick's branch - but looking at the recent tickets it looked like the code to run to try parse complex XSDs.

[FelixSchwarz]

Thank you for your quick response. Unfortunately I'm short on time (easter around the corner) so if you need something faster than 2 weeks you need either to debug this yourself or look at another library...

attached reproducer file so it can't go away: cwmp14.txt

[thomas-mangin]

Being an open source developer too, I can appreciate your position. As it seems your library is the right fit for us, you will most likely see me pop up back here in a few weeks - I hope you won't mind the pestering if your life an work got in the way of happy hacking 😄

Thank you for your help so far.

[FelixSchwarz]

Looking forward to your questions/contributions :-)
Btw: An open source(?) TR069 project sounds interesting. Feel free to include pointers once there is something to see.

[iurisilvio]

It is related to my changes to support relative paths and these imports.

  <xs:import namespace="http://schemas.xmlsoap.org/soap/envelope/" schemaLocation="http://schemas.xmlsoap.org/soap/envelope/"/>
  <xs:import namespace="http://schemas.xmlsoap.org/soap/encoding/" schemaLocation="http://schemas.xmlsoap.org/soap/encoding/"/>

When I had this problem, I just removed these imports and it worked. Of course it is not the fix, just a quick hack.

I'll try to fix it soon.

[ngnpope]

@iurisilvio I have further changes I was working on related to the relative path issues so I'll look at your changes and compare to mine. I should be able to push further updates later today.

[FelixSchwarz]

Based on what @pope1ni said is it ok if I wait a bit with merging?

@iurisilvio: thank you very much for your quick fix, much appreciated.