Add python LSP wrapper script for stdio

[dannyZyg]

Provides a python script to support lsp clients which can't use UDP but can use stdio (e.g. neovim) - as described in this issue #9

The script launches sclang in the background and while communicating with the language server quark over UDP, relays this IO over stdio to the client.

Special thanks to @themissingcow for getting this working properly!

Please note a few changes to the sc files. Let us know if these are OK.

Note: this has only been tested with MacOS so far.

[davidgranstrom]

Nice work! Got the LSP running now in nvim with some minor adjustments to the scnvim Document implementation davidgranstrom/scnvim#252

---

When following the installation instructions for the python stdio wrapper (User installation section) I got the following error

Click to expand

❯ python3 -m pip install .

/error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:

    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with

    brew install pipx

    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.

    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.

    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I'm not a python dev, so maybe my environment is not properly configured. But following the advice from the error message I installed pipx, and then did pipx install . in the root of the sc_language_server directory which worked. Maybe this info could be helpful to anyone else who runs in to the same issue, or even added to the README in this PR?

[dannyZyg]

Hey @davidgranstrom thanks so much for trying it out and for your notes. Python installs/environments are a bit of a minefield so this is a great find.

I've updated the readme to suggest using pipx if the global install fails, with some basic instructions too.

Thanks also for the scnvim fix! Please let me know if you find anything else!

[sadguitarius]

Hi @dannyZyg, I'm trying this out on Windows and I'm stuck on the calls to the fcntl module, which is not included on Windows. Do you know of an alternative way to handle this that's platform independent? I know some Python but I haven't done any asyncio stuff before. Thanks!

[dannyZyg]

Hi @dannyZyg, I'm trying this out on Windows and I'm stuck on the calls to the fcntl module, which is not included on Windows. Do you know of an alternative way to handle this that's platform independent? I know some Python but I haven't done any asyncio stuff before. Thanks!

Hey @sadguitarius yep you are right! Looks like it will need a bit of finessing to handle windows too, mainly in the stdin thread (i hope). I can try and take a look at it at some point, but don't have any way of testing it outside of setting up a VM.

[elgiano]

Heeey thank you all, I managed to get sclang lsp in nvim and I couldn't be happier (long time desired :D)

Apart from some changes to LanguageServer.quark itself, I also made some changes to @dannyZyg's python script, which are not really required but I think they would be good to have:

• add linux default sclang path: sclang
• don't require a libraryConfig: sclang can figure out a default one itself
• add support for extra args to be passed to sclang, this way one can run sclang on a different udp port, useful in case one runs two instances, one only for LSP and the other needing port 57120. This way the lsp instance doesn't block sclang default langPort
• don't check for sclang existence, and instead let subprocess fail if FileNotFound. This allows to have the linux default to "sclang" without specifying path. I changed the error message from FileNotFoundException to be the same as @dannyZyg RuntimeError
• minor changes to argparse

Here is the diff:

203,206c203,204
<         "darwin": {
<             "sclang": "/Applications/SuperCollider.app/Contents/MacOS/sclang",
<             "config": "~/Library/Application Support/SuperCollider/sclang_conf.yaml",
<         }
---
>         "darwin": "/Applications/SuperCollider.app/Contents/MacOS/sclang",
>         "linux": "sclang"
209,211c207
<     sys_defaults = defaults.get(sys.platform, {})
<     sclang_path = sys_defaults.get("sclang", "")
<     config_path = sys_defaults.get("config", "")
---
>     sclang_path = defaults.get(sys.platform, "")
232d227
<         config_path: str | None,
246d240
<         self.config_path = config_path or self.config_path
249c243
<     async def start(self) -> int:
---
>     async def start(self, extra_args: [str] = []) -> int:
259,261d252
<         if not os.path.exists(self.sclang_path):
<             raise RuntimeError(f"The specified sclang path does not exist: {self.sclang_path}")
< 
272,276c263
<         if self.config_path:
<             config = os.path.expanduser(os.path.expandvars(self.config_path))
<             if not os.path.exists(config):
<                 raise RuntimeError(f"The specified config file does not exist: '{config}'")
<             command.extend(["-l", config])
---
>         command.extend(extra_args)
280,287c267,278
<         self.__subprocess = await asyncio.create_subprocess_exec(
<             *command,
<             stdout=asyncio.subprocess.PIPE,
<             stderr=asyncio.subprocess.PIPE,
<             # stdin must be set to PIPE so stdin flows to the main program
<             stdin=asyncio.subprocess.PIPE,
<             env={**my_env, **additional_vars},
<         )
---
>         try:
>             self.__subprocess = await asyncio.create_subprocess_exec(
>                 *command,
>                 stdout=asyncio.subprocess.PIPE,
>                 stderr=asyncio.subprocess.PIPE,
>                 # stdin must be set to PIPE so stdin flows to the main program
>                 stdin=asyncio.subprocess.PIPE,
>                 env={**my_env, **additional_vars},
>             )
>         except FileNotFoundError as e:
>             e.strerror = ("The specified sclang path does not exist")
>             raise
377c368
<         prog="sclsp_runner",
---
>         formatter_class=argparse.RawDescriptionHelpFormatter,
378a370,373
>         epilog='''
> example with extra sclang args (custom langPort and libraryConfig):
>   %(prog)s --sclang-path /path/to/sclang -v --log-file /path/to/logfile -- -u 57300 -l custom_sclang_conf.yaml
> '''
380a376,377
>     print_default = '(default: %(default)s)'
> 
384a382
>         help=print_default
386,390c384
<     parser.add_argument(
<         "--config-path",
<         required=not sc_runner.config_path,
<         default=sc_runner.config_path,
<     )
---
>     parser.add_argument("--ide-name", default=sc_runner.ide_name, help=print_default)
393d386
<     parser.add_argument("--ide-name", default=sc_runner.ide_name)
395a389,390
>     parser.add_argument('extra_sclang_args', nargs="*",
>                         help="cli arguments for sclang (see example below)")
433d427
<         config_path=args.config_path,
449c443
<     sys.exit(asyncio.run(sc_runner.start()))
---
>     sys.exit(asyncio.run(sc_runner.start(args.extra_sclang_args)))

[dannyZyg]

Hey @elgiano thanks for these suggested changes, i think they are good! I've made them to my branch now. Let me know if any issues.

Good idea changing the default port...i think most of the time people using this would be running more than one instance of sclang (at least I am!).

[sadguitarius]

Hi @dannyZyg, would something along the lines of this work for cross-platform support? I'm still trying to grok how Python subprocesses and async stuff work. I'm happy to give this a shot, but if you get the chance to take a look at the link, could you let me know if this looks like a possible solution? Thanks!

[sadguitarius]

also some good stuff here

[ranjithshegde]

@dannyZyg Thank you for this wonderful work! Got it fired up on my local yesterday. Such a relief to have it just work.

Sorry for commig here with suggestions and not PRs, but I am wondering if dependency on external like python absolutely necessary, when ScLang can do UDP, perhaps somewhere within this quark, or just using lua via neovim's implementation of libuv? Many here seem to be not as familiar with python and also seems (at least to me personally) like another layer of complexity.

Havent looked into it the complexity or the feasibility of either yet. But if there is no rush to merge, in about 2 weeks, I could take a crack at either extending the ScLang part of the quark or as an add-on to @davidgranstrom's excellent scnvim plugin?

[salkin-mada]

A fully working one stop setup extension to scnvim would be wonderful

[dannyZyg]

Thanks for everyone's input! I don't have any time to work on this at the moment. It is also unclear if this is the desired solution?

While it could be nice to have this in lua, or even built into scnvim, there is some flexibility by having this as a standalone piece.

• a language server needs to be a separate process anyway which neovim manages when required by a buffer (as far as I understand).
• combining this into scnvim excludes other editors from using this if they also require stdio vs UDP.
• Is it such a big issue if this is a python script? I would think most systems would have a python install. There are no other dependencies, so installation and windows support are the only issues that would need to be addressed. Plenty of other language servers use python, and once this is complete it could be managed through tools like Mason to make installation easier.

If someone wants to have a crack at that though please do, and feel free to use anything here if needed.

[sadguitarius]

My (possibly useless) input: off the top of my head, at least Helix would be an editor that would benefit from a Python or other solution that doesn't involve Lua. I think I'm the odd one out in terms of the current Python script not being workable because I'm not on a POSIX machine, but it sounds like there are some alternative async methods that could work with Python. I think there are also some thin LSP wrappers written in NodeJS as well, which is a platform I know next to nothing about, but I'm sure someone here does. Might that be a smoother solution?

[dannyZyg]

I think I'm the odd one out in terms of the current Python script not being workable because I'm not on a POSIX machine, but it sounds like there are some alternative async methods that could work with Python.

There are definitely changes that could be made to the script to get it working with windows too! I'm happy to try and figure that out at some point when I have some time, but as mentioned, I don't have windows to be able to test it out. I'm also not sure yet what the consensus is on this whole approach anyway.

[ranjithshegde]

@sadguitarius agreed, an editor agnostic approach should be the way to go! Not lua then I guess

@dannyZyg I wasnt trying to discourage usage of python. Because this quark allows sclang -i -vscode, I was thinking why not either make that the standalone LSP for ALL, or just extend scnvim. The sole reasoning being supercollider+quarks+wrapper+scnvim+nvim_local_lsp_settings just felt like a lengthy approach if alternatives exist.
I also see what you mean about a completely standalone approach that is plugin/editor agnostic.

Would everyone be open to exteding the quark iself to choose between stdio or udp? I am also far too occupied to be able to take a crack at it reasolably soon, so, I fully support the python idea, and also of course, appreciate your effort into this!!

[dannyZyg]

The sole reasoning being supercollider+quarks+wrapper+scnvim+nvim_local_lsp_settings just felt like a lengthy approach if alternatives exist.

@ranjithshegde I agree that it is a lot to setup and wrap your head around, and is not ideal. The wrapper does add one more layer of complexity, but I think the other layers would still exist in most scenarios.

Would everyone be open to exteding the quark iself to choose between stdio or udp? I am also far too occupied to be able to take a crack at it reasolably soon, so, I fully support the python idea, and also of course, appreciate your effort into this!!

I know very little about the sclang/quark side of this stuff, but I think extending the quark itself to support stdio as well as UDP is a cool idea. @scztt , was there a reason why this was not considered? Are there any barriers to someone doing this?

(considering sclang already outputs on stdio, we could redirect that to a log file or something? This is one reason not to do it I suppose)

[themissingcow]

I know very little about the sclang/quark side of this stuff, but I think extending the quark itself to support stdio as well as UDP is a cool idea. @scztt , was there a reason why this was not considered? Are there any barriers to someone doing this?

I think #9 (comment) was the reason the wrapper was initially suggested.

[ranjithshegde]

Given all the details, I suppose this is the best way forward.
Nvim does have tcp support, I suppose there might be udp support in core one day, who knows.
But this works great!

[scztt]

I know very little about the sclang/quark side of this stuff, but I think extending the quark itself to support stdio as well as UDP is a cool idea. @scztt , was there a reason why this was not considered? Are there any barriers to someone doing this?
(considering sclang already outputs on stdio, we could redirect that to a log file or something? This is one reason not to do it I suppose)

Yes, the goal is to support stdio eventually, the UDP solution is a temporary hack. What's required here is essentially refactoring every print statement / stdio interaction in sclang to repoint in a way that they can be overridden in the LSP case. This also requires re-routing stdio streams from external processes that are launched from sclang (e.g. the server), which are currently just passed through stdio. In general, this is probably a more general refactor where we just handle routing text streams better... I don't think these changes are very tricky, but it touches a lot of code and would take some care to do right.

I WOULD say: a proper stdio fix is PROBABLY only like 5 days if someone had a clear picture of the work required - and maybe even less, a day or two. So, if building a python solution is going to turn into a loooong arc project, it might be better to look at the stdio solution. I'd love to do this, but I have a lot of stabilization work to do on the LSP/vscode plugin still, and at least for myself I think I should prioritize that for the time being.

If anyone wants to try to tackle the stdio problem, I'd be happy to provide a pretty detailed description of what changes would need to happen!

[sadguitarius]

Thank you for the input @scztt! I'd be more than happy to take a look at this. Feel free to DM or post here with additional details about the stdio requirements.

[dannyZyg]

Hey @sadguitarius and @scztt, just checking in on the sclang stdio fix approach. @sadguitarius did you have a go at this? Is any of the approach documented anywhere? I'd be up for having a crack at it if no one else has yet - it would nice to have LSP support built in by default, at least in the sense of being able to add this quark and get it set up without fuss (on any platform too).

[sadguitarius]

Nothing's happened on my end yet. I just moved, so I'm finally getting my dev setup back in order. In the meantime, I also noticed this and it made me wonder about the possibility of a nvim plugin that would do the heavy lifting of communicating with the LSP process over OSC. Does that even make sense?

[dannyZyg]

While I love OSC and that plugin looks cool, I think it's best to stick with stdio for the language server since it is officially supported. I feel like we might just end up writing another wrapper otherwise. Plus, using stdio means the lang server can be used with other editors besides neovim.

It might take me a while to wrap my head around all of it but I'm happy to take a look at the stdio work Scott is describing.

[sadguitarius]

That makes sense. I don't think my knowledge of sclang internals is at the point where I'd be comfortable taking the lead on this without some clear direction about where to look for things. Happy to help in whatever way I can though and to test out cross-platform things.

[sadguitarius]

Just a quick note on my (lack of) progress. I was able to get LSP communication sort of working on Windows using a slightly modified version of @davidgranstrom's js script and modified Document.sc above, which I'd missed earlier. I do see the requests and responses in the log, but the language server doesn't appear to be fully communicating, i.e. I get a few laggy hover popups and I can see that document changes are registered, but I don't see any completion results. I think I have it set up right, but I may have missed a step somewhere. Also I'm wondering if Windows Defender is causing some delays and I may have to add some exclusions to its process.

I did begin to look into modifying how sclang handles print statements and added a command line --lsp option, but what I'm realizing is that the amount of trickery necessary to suppress prints from both the language process and withing sclang itself becomes spaghetti pretty quickly. It's definitely possible, but I think some design decisions ought to be made so it remains maintainable.

Also wanted to note here that at least in the case of Neovim, the number of installable language servers using the standard Mason tools etc. appear to be overwhelmingly in the form of Node packages, so having the official Neovim solution being a Node stdio bridge to UDP messaging is maybe not the end of the world, and it's a hell of a lot easier to implement. Thoughts?

[dannyZyg]

Yes, the goal is to support stdio eventually, the UDP solution is a temporary hack. What's required here is essentially refactoring every print statement / stdio interaction in sclang to repoint in a way that they can be overridden in the LSP case. This also requires re-routing stdio streams from external processes that are launched from sclang (e.g. the server), which are currently just passed through stdio. In general, this is probably a more general refactor where we just handle routing text streams better... I don't think these changes are very tricky, but it touches a lot of code and would take some care to do right.

I WOULD say: a proper stdio fix is PROBABLY only like 5 days if someone had a clear picture of the work required - and maybe even less, a day or two. So, if building a python solution is going to turn into a loooong arc project, it might be better to look at the stdio solution. I'd love to do this, but I have a lot of stabilization work to do on the LSP/vscode plugin still, and at least for myself I think I should prioritize that for the time being.

If anyone wants to try to tackle the stdio problem, I'd be happy to provide a pretty detailed description of what changes would need to happen!

Hey @scztt I'd be quite interested in having a crack at this and now have some time that I could put into it. Would it be possible for you to provide any more details on the required changes? I would be more keen to work on a longterm solution built into SC rather than this wrapper. Let me know if you still think this is viable!