Procedure to regenerate pycbf?

[ndevenish]

I'm having trouble working out how to regenerate pycbf on the new release. it looks like the linux.sh file is maybe out of date, and it looks to me as though the files on release don't cleanly regenerate or even compile in latex, which is a prerequisite for the nuweb workflow.

Rough list of the most important Issues I've encountered:

• The pycbf/CBFlib.txt file is (six years!) out of date:

  $ make $(pwd)/pycbf/CBFlib.txt
  make: '/home/cbfbuilder/cbflib/pycbf/CBFlib.txt' is up to date.
  $ touch doc/CBFlib.html
  $ make $(pwd)/pycbf/CBFlib.txt
  links -dump /home/cbfbuilder/cbflib/doc/CBFlib.html > /home/cbfbuilder/cbflib/pycbf/CBFlib.txt
  

• Latex errors when attempting to regenerate the python bindings. Note that this isn't done in the Makefile on any normal build; not in make all, and not in make tests (which only builds _pycbf.so). This issue can be demonstrated by simply asking for pycbf.py (or pycbf.pdf, which is the actual rule for trying to regenerate the sources):

  $ make $(pwd)/pycbf/pycbf.py
  (cd /home/cbfbuilder/cbflib/pycbf; \
  
  ....
  
  (./pycbf.toc
  ! Missing $ inserted.
  <inserted text>
                  $
  l.16 ...th write logic added}{17}{subsection.10.4}
  
  ?
  

• Generally the Makefile doesn't seem to build nuweb things properly e.g. the rule for make_pycbf.py is plain incorrect:

  $(PY3CBF)/make_pycbf.py: $(NUWEB_DEP) $(NUWEB_DEP2) $(PY3CBF)/make_pycbf.w
      (cd $(PY3CBF); $(NUWEB) make_pycbf.w)

  This runs nuweb only - the only output from this command is a new make_pycbf.tex. This latex script also seems to not compile (and is not checked in), so I am unable to confirm if the checked in code is an up-to-date regeneration.

Without being able to compile the nuweb sources, I can't regenerate anything that depends on them e.g. everything. So what am I doing wrong, how do I actually generate these?

Finally... I notice that you've kept the whole nuweb lifecycle in this release. I'd urge reconsideration for future releases, since it looks as though it's not being developed in a literate programming way any more, and looks as though it barely was - almost the entirety of most .w files appear to be plain sections with the pasted code, beyond the initial efforts at documentation in pycbf.w. It would make the build process considerably simpler, make it so that commit changes don't have the same change in three different files all over the place (.w, .tex, and the generated output), and eliminate the error of "forgetting to regenerate after changes" (see e.g. much of the noise on the now out-of-date #19, which hadn't been correctly regenerated in a while).

[yayahjb]

Dear Nicholas, Thank you for clarifying the issues with pycbf in cbflib. I apologize, but there will have to be a delay until late September before I can rework that section of the build myself. I have to deal with some medical issues at the same time I need to finish a proposal at the same time that I am moving apartments, so I am out of bandwidth for a while. I am copying this to Aaron and Billy to see if they might have some time to help you before I am available again. If one of you younger people has not redone that ancient section of the build before then, I hope to start a clean rework from scratch of the python parts of cbflib about a month from now, i.e. approximately 19 September. It will make a nice background activity to listening in on NOBUGS Regards, Herbert

…

On Sat, Aug 20, 2022 at 6:29 AM Nicholas Devenish ***@***.***> wrote: I'm having trouble working out how to regenerate pycbf on the new release. it looks like the linux.sh file is maybe out of date, and it looks to me as though the files on release don't cleanly regenerate or even compile in latex, which is a prerequisite for the nuweb workflow. Rough list of the most important Issues I've encountered: - The pycbf/CBFlib.txt file is (six years!) out of date: $ make $(pwd)/pycbf/CBFlib.txt make: '/home/cbfbuilder/cbflib/pycbf/CBFlib.txt' is up to date. $ touch doc/CBFlib.html $ make $(pwd)/pycbf/CBFlib.txt links -dump /home/cbfbuilder/cbflib/doc/CBFlib.html > /home/cbfbuilder/cbflib/pycbf/CBFlib.txt - Latex errors when attempting to regenerate the python bindings. Note that this *isn't* done in the Makefile on any normal build; not in make all, and not in make tests (which only builds _pycbf.so). This issue can be demonstrated by simply asking for pycbf.py (or pycbf.pdf, which is the actual rule for trying to regenerate the sources): $ make $(pwd)/pycbf/pycbf.py (cd /home/cbfbuilder/cbflib/pycbf; \ .... (./pycbf.toc ! Missing $ inserted. <inserted text> $ l.16 ...th write logic added}{17}{subsection.10.4} ? - Generally the Makefile doesn't seem to build nuweb things properly e.g. the rule for make_pycbf.py is plain incorrect: $(PY3CBF)/make_pycbf.py: $(NUWEB_DEP) $(NUWEB_DEP2) $(PY3CBF)/make_pycbf.w (cd $(PY3CBF); $(NUWEB) make_pycbf.w) This runs nuweb only - the only output from this command is a new make_pycbf.tex. This latex script also seems to not compile (and is not checked in), so I am unable to confirm if the checked in code is an up-to-date regeneration. Without being able to compile the nuweb sources, I can't regenerate anything that depends on them e.g. everything. So what am I doing wrong, how do I actually generate these? Finally... I notice that you've kept the whole nuweb lifecycle in this release. I'd urge reconsideration for future releases, since it looks as though it's not being developed in a literate programming way any more, and looks as though it barely was - almost the entirety of most .w files appear to be plain sections with the pasted code, beyond the initial efforts at documentation in pycbf.w. It would make the build process *considerably* simpler, make it so that commit changes don't have the same change in three different files all over the place (.w, .tex, and the generated output), and eliminate the error of "forgetting to regenerate after changes" (see e.g. much of the noise on the now out-of-date #19 <#19>, which hadn't been correctly regenerated in a while). — Reply to this email directly, view it on GitHub <#59>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB6EAI52N2NV25PKL6HLNLV2CXRBANCNFSM57DAKTMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ndevenish]

No problem, there isn't any urgency, I was just intending to integrate the 0.9.7 changes into https://github.com/dials/pycbf. Obviously that's not possible with the current state of pycbf. Hope everything goes well.

Nick

[hattne]

I suspect some of these issues have been fixed elsewhere, a couple of others should be addressed in #107.

• pycbf.pdf can be generated from pycbf.tex (which is in turn generated from pycbf.w), but nothing depends on it—pycbf.pdf will not be created with either make all or make tests.
• Python bindings are regenerated if necessary, in particular make $(pwd)/pycbf/pycbf.py recreates pycbf.py. Other than redundancies in the dependency chain, I don't see what could have been wrong with it before. Maybe @ndevenish did the tests before the @o command in make_pycbf.w gained the -i and -t options in Cbflib 0.9.7 devel #41?
• As noticed, pycbf.py is not recreated with make all, because all does not depend on the Python module, but tests does. I think that's a relatively serious shortcoming.
• The .tex file created by make $(pwd)/pycbf/make_pycbf.py is included by nuweb -o pycbf; I suspect it was never intended to be compiled on its own.

Even with #107 issues remain: it seems stuff is unnecessarily regenerated and the targets, dependencies could be cleaned up, and py3setup.py is invoked from more than one rule. I'd like to take a look that in a subsequent pull request, with the aim of having this issue closed.