tiny-lsp-client

tiny-lsp-client is an LSP client for Emacs implemented in Rust. I call it tiny because it aims to only support a tiny feature set and because it aims to have a tiny footprint, especially on memory and garbage collection. I created it for fun and because I was curious of how this would turn out. tiny-lsp-client has many limitations, such as only supporting stdio LSP servers and not providing that much information in completion candidates. However, it works, and I use it as my daily driver.

Usage

Installation

Prerequisite: rust and cargo need to be installed.

1. Clone this git repository and add it to your load-path.
2. Navigate to the cloned repo and run cargo build --release
3. (require 'tiny-lsp-client)

Features and configuration

The table below shows the features tiny-lsp-client supports, the corresponding LSP method used to implement it, and the LISP function from tiny-lsp-client to use.

Feature	LSP method	tiny-lsp-client function
Jump to definition	textDocument/definition	#'tlc-xref-backend
Code completion	textDocument/completion	#'tlc-completion-at-point
Hover information	textDocument/hover	#'tlc-eldoc-function

Add tlc-mode to the mode hooks you want, for example:

(add-hook 'c++-mode-hook 'tlc-mode)
(add-hook 'rust-mode-hook 'tlc-mode)

tlc-mode itself doesn’t provide any features (it only provides document synchronization with the LSP server). This is to be as customizable and unobtrusive as possible. Instead, you must yourself enable the features you want, in the way you want.

To enable all currently supported features whenever tlc-mode is enabled, you can do the following:

(defun my-tlc-features ()
  (add-hook 'xref-backend-functions 'tlc-xref-backend nil t)
  (add-hook 'completion-at-point-functions 'tlc-completion-at-point nil t)
  (eldoc-mode 1)
  (add-hook 'eldoc-documentation-functions #'tlc-eldoc-function nil t))

(add-hook 'tlc-mode-hook 'my-tlc-features)

If you use Corfu maybe you want to combine tlc-completion-at-point with some other capfs in cape-capf-super. Or maybe you only want xref for some major modes and capf for some other modes. For this reason, tiny-lsp-client only provides plain, standardized tlc-xref-backend and tlc-completion-at-point functions, and leaves it to you how and when to use them.

For more configuration, refer to the defcustom options and their documentation in tiny-lsp-client.el. One defcustom worthy of extra highlight is tlc-server-cmds. tlc-server-cmds defines which server command to use for various major modes, and it needs customization if your mode and server aren’t already in the default list. Another note is that some of the defcustom change variables in Rust, and must therefore be set using customize-set-variable and not setq.

The “control room”

If there are issues, or you want to check how tiny-lsp-client and the LSP servers behave under the hood, there are some functions to help with that.

Function	Description
tlc-open-log-file	Open tiny-lsp-client’s log file
tlc-info	Show information about running servers
tlc-stop-server	Stop an LSP server
tlc-restart-server	Restart the LSP server of the current root path

What’s logged and where the log is stored is configured through defcustom variables. There’s only one log file. When tiny-lsp-client is writing the first log entry, the existing file is renamed to contain .old as suffix, and if that file already exists, it’s deleted. If you run multiple emacs instances you might want to use different tlc-log-file values.

Design documentation

The diagram below shows the most important aspects of the architecture. I got a lot of inspiration from lspce, lsp-bridge and emacs-lsp-booster when deciding on this architecture.

Starting from the right in the diagram: for each project active, there’s one LSP server running. They run as separate child processes. Each LSP server is managed by a Server object, defined in server.rs. Server has one thread for handling stdin, one thread for handling stdout and two threads in series for handling stderr. The stdin thread encodes Rust objects representing LSP JSON and the stdout thread decodes LSP JSON into Rust objects. In general, LSP JSON contains more fields than what tiny-lsp-client makes use of, so the Rust objects are more slim than the full JSON. The stdin thread reads the channel from lib.rs and encodes and sends the JSON to the LSP server as soon as something is available. The stdout thread reads from the LSP server and decodes. If the message is relevant (e.g. "textDocument/definition" response), it’s sent over the channel to lib.rs. If not relevant (e.g. "textDocument/publishDiagnostics") it’s dropped.

lib.rs is the interface between emacs and Rust. To keep the burden on emacs as low as possible, the interface consists of function calls with lisp objects as parameters and return values, with the bare minimum of data. For example, when sending a "textDocument/findDefinition" request, these are the lisp, Rust and JSON values:

(tlc--rust-send-request
  "/some/path/file.rs"
  "textDocument/definition"
  `(,file-path 4 10))

send_request(
    "textDocument/definition".to_string(),
    Message::Request(
        RequestParams::DefinitionParams(DefinitionParams {
            text_document: TextDocumentIdentifier { uri: "/some/path/file.rs" },
            position: Position {
                character: 4,
                line: 10,
            },
        }),
    )
)

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "textDocument/definition",
  "params": {
    "textDocument": {
      "uri": "file:///some/path/file.rs"
    },
    "position": {
      "line": 4,
      "character": 10
    }
  }
}

There is one Server object per project, and lib.rs stores a map of (root path, Server). All functions for sending and receiving LSP JSON messages have root path as one of the parameters.

lib.rs’s sending of requests and notifications is asynchronous and is done by creating a Rust object representing the JSON and then putting it on the channel to the correct Server.

lib.rs receives JSON by doing a non-blocking receive call on the channel. If there is a message, it’s transformed into a lisp object and returned to emacs. If there’s no message, no-response is returned. tiny-lsp-client.el waits for a response with a busy-wait loop using sit-for.

tiny-lsp-client.el is where tiny-lsp-client is integrated into. This is where tlc-mode, the minor mode for tiny-lsp-client is implemented, as well as the xref backend and completion-at-point-functions.

Conversion between LISP and Rust values, and error handling

When calling a LISP function implemented in Rust (in lib.rs), the arguments need to be converted from LISP to Rust, and the return value needs to be converted from Rust to LISP. emacs.rs provides the traits FromLisp and IntoLisp for this. These traits have been implemented for common types, such as String, Vec and integers. emacs.rs also provides lisp_function_in_rust(), a wrapper to which the user (lib.rs) passes a safe closure that takes a FromLisp argument and returns an IntoLisp value. In other words, the wrapper shields unsafe and LISP related things from the caller.

In the conversion between LISP and Rust, there’s a need for error handling for these reasons:

1. The called LISP function might return an error, e.g., wrong types.
2. The Rust value is a tuple of arity 3, but the passed value has arity 2.

For these reasons, FromLisp and IntoLisp return pub type LispResult<T> = Result<T, String>;. If case 1 happens, emacs::handle_non_local_exit() automatically returns an Err value with a reason. If case 2 happens, the implementation if FromLisp or IntoLisp must return an Err value themselves. As soon as case 1 has happened, according to emacs documentation, the API functions no longer return meaningful values. So in the Rust code, if Err has happened, it’s propaged up all the way the the Rust function returns and throws a LISP level error.

Testing

Tests are located in test/. Change directory to test/ and use run_tests.sh:

• run_tests.sh to run all tests
• run_tests.sh mode-test.el to run all tests in the file mode-test.el
• run_tests.sh mode-test.el start-server-hooks-test to run the test start-server-hooks-test in mode-test.el

Log files are stored in test/logs.

Since the test cases use real LSP servers (clangd, jdtls etc), you need to have those programs installed. As an alternative, see Docker container.

Docker container

Since there are many test dependencies, a Docker container exists to make it easier to run the tests. Inside the test/ directory, run docker compose up --build to start the test container. In another terminal, run docker exec -it tiny-lsp-client-test bash to attach to the container. Now you can use run_tests.sh as above and with all test dependencies available.

The entire repository is mounted in the container. So if you change source code and/or tests, there’s no need to restart or rebuild the container. The source code is not copied to the container.

If you run clangd tests in Docker and bare-metal you might need to delete CMakeCache.txt and other generated files.

Test strategy

There are 4 types of test suites:

1. *-test.el
2. mode-test.el
3. lisp-bindings-test.el
4. Rust unit tests

*-test.el

For each language server inside tlc-server-cmds there’s one corresponding test file, e.g., clangd-test.el for clangd. Those language specific test files aim to test each LSP message type towards each language server. This typically means a few test cases that open a file, edit it, jump to defintion, and use completion. The idea is for the tests to act as a compatibility test towards the LSP server.

mode-test.el

mode-test.el has tests that are language-agnostic and are more related to tlc-mode and the emacs lisp layer. So for example testing that start-server-hooks are run then tlc-mode is enabled, that unicode works, and more thorough xref and capf tests with edge cases. One could argue some of these things should be tested for each LSP, but hopefully a few basic tests per LSP is enough.

lisp-bindings-test.el

lisp-bindings-test.el tests the lisp bindings provided by lib.rs directly without involving tiny=lsp-client.el. This suite was mainly useful when creating the basic of tiny-lsp-client, but is also useful now when something basic in the lisp-Rust interface breaks. There’s no plan to add more tests to it as more LSP messages are supported.

Also, lisp-bindings-test.el is the only suite run with cargo build --release and with garbage collection triggered.

Rust unit tests

Some of the Rust layer is tested in unit tests, that can be run with cargo test. Of special interest is server/tests.rs, which uses the Rust layer direcly towards an LSP server, without involving any lisp. This was mainly useful when creating the basic of tiny-lsp-client, but is also useful now when something basic breaks. There’s no plan to add more tests to it as more LSP messages are supported. However, other unit tests might be added.

Garbage collection test

One of the main goals of tiny-lsp-client is to cause as few garbage collections as possible. To test this, I ran two tests, an “xref” test and an “edit” test, and count how many garbage collections are caused by tiny-lsp-client, eglot, and if no LSP client is enabled.

The “xref” test iterates through all points in a short C++ file and triggers xref’s goto defintion. The “edit” test inserts one character at a time to a short C++ file. For both tests, I run them in a loop 100 times. See test/performance-test.el for all details.

• For “xref”, eglot causes 2.5 times as many garbage collections as tiny-lsp-client
• For “edit”, eglot causes 16 times as many garbage collections as tiny-lsp-client

These numbers need to be taken with a grain of salt:

• The tests don’t represent typical workloads.
• I disabled eglot’s accumulation of changes to make it clearer when eglot sends changes to the LSP sever.
• When having eglot send changes less frequently, in total there were fewer garbage collections than if no LSP client at all! Clearly something is fishy.
• I have set gc-cons-threshold to a very low value of 8000, which might have unintended consequences.

I also want to emphasize that eglot is in no means “bad” because of making more garbage collections. tiny-lsp-client causes fewer garbage collections for these reasons:

• eglot creates LISP objects for the full JSON sent to/received from the LSP server whereas tiny-lsp-client only creates very small LISP objects. See Design documentation for more details.
  • I haven’t figured out if eglot in emacs 30 encodes/decodes JSON in LISP or C. But even if it happens in C, LISP objects for the full JSON are still created, which causes garbage collection load.
• eglot parses more LSP messages, e.g., textDocument/publishDiagnostics, which can be numerous and big. tiny-lsp-client drops those messages before they reach LISP. This limited feature set of tiny-lsp-client is by design.

Copyright, license and acknowledgments

The license for this repository is GPL v3, as specified by the separate LICENSE file.

In general I have learned a lot and gotten many ideas and code snippets from some other LSP projects: lsp-mode, eglot, lspce, lsp-bridge and emacs-lsp-booster. A big thank you to them. I have marked in the code what I have copied from them (happily under GPL!).

No copyright infringement intended. If you see an issue, I’m more than happy to fix it.

Backlog

• New functionality
  • Higher priority
    • Rename
    • Signature help in eldoc
  • Lower priority
    • For e.g. jdtls and hls make it possible to jump to library files
    • Multiple servers per buffer, could be relevant for HTML, CSS and Javascript
      • But I can’t do this until I know how I want to use it
      • Could be that tlc-server-cmds should be removed, and general hook per major/minor mode that starts a server
• Bugs
  • Higher priority
  • Lower priority
  • Hard to reproduce
    • when doing completions in c++ test file, clangd complained about non opened document
    • once, when jump to defintion while starting rust analyzer, seemed to get stuck in infinite loop with 0 wait between try recv response. Lot’s of RAM was being consumed. But it might be fixed once a request timeout is being used.
  • Under observation
    • sometimes core dump when doing async completion
    • Sometimes duplicate didOpen/didClose
    • At stop, sometimes get duplicate didOpen due to mode and server out of sync
• Improvements of existing code/functionality
  • Higher priority
    • Error cases
      • Handle, test and understand error responses from LSP servers better
      • Test and understand all cases of tlc--wait-for-response better
      • Test Untyped and unsupported respones. Check log file.
    • Keep track of server capabilities
    • log tests
      • Unit tests if possible
      • test what happens if tlc-log-file is never set
  • Lower priority
    • Generalized FromLisp for union types. Maybe it’s possible to check if the lisp value is of correct type. Maybe a can_from_lisp function on the FromLisp trait? Maybe emacs-module-rs does it in a smart way?
    • Understand how LSP processes are killed even if not same process group. It happens even when wrong process id in initialize message.
    • Prevent crashes in rust code due to bad user settings, e.g. empty server cmd.
    • Remove duplicate completions in Rust code
    • Support utf-32 line pos, that eglot has as preffered, since it has better performance than utf-16.
    • consider if utf-16 actions need to be taken for xref, like eglot seems to do
    • in general, ensure no bad args sent to rust. e.g. stop-server with “path” because doesn’t start with /
    • consider caching didChange like eglot
    • Clean up server.rs by having smaller functions (like read_header) and having a flatter sructure since break can return early.
    • tests
      • Use clangd instead of rust-analyzer inside rust unit tests
      • More unit tests
      • test error during xref or capf
      • Some tests are unstable, especially when running all
    • get inspiration from emacs-module-rs and generalize lisp<->rust conversion to encapsulate unsafe code better
      • Also prevents endless loop when trying to convert lisp value to wrong type
    • understands bounds and symbol better for capf
    • understand commit chars, range, etc from LSP better. Check what lsp-mode and eglot do
    • completion ideas
      • if interrupted, send dabbrev
      • if interrupted, keep calculating and use that as next last result. So like the “cached async” but it keeps on refreashing and doesn’t do it just once in the beginning.
      • for async, measure how much time it takes to do everything except while-no-input. And with. And try to skip it for debug purposes
      • Understand how while-no-input, sit-for, and sleep-for interact.
      • Understand how company completes not just from prefix when used with LSP
      • Understand how lsp-mode and eglot handle async and cached completions
  • Under observation
    • Test with real-world usage to see how smooth completion is, and make it smoother

notes

rust-analzyer completion:
{
  "additionalTextEdits": [],
  "deprecated": false,
  "filterText": "S",
  "kind": 25,
  "label": "S",
  "sortText": "7fffffff",
  "textEdit": {
    "newText": "S",
    "range": {
      "end": {
        "character": 6,
        "line": 493
      },
      "start": {
        "character": 4,
        "line": 493
      }
    }
  }
}
clangd completion:
{
  "detail": "long",
  "filterText": "my_function4",
  "insertText": "my_function4",
  "insertTextFormat": 1,
  "kind": 3,
  "label": " my_function4()",
  "score": 1.0087924003601074,
  "sortText": "407edfe4my_function4",
  "textEdit": {
    "newText": "my_function4",
    "range": {
      "end": {
        "character": 6,
        "line": 25
      },
      "start": {
        "character": 4,
        "line": 25
      }
    }
  }
}
erlang_ls completion:
{
  "kind": 14,
  "label": "when"
},
{
  "kind": 14,
  "label": "xor"
},
{
  "data": {},
  "insertText": "binary_to_atom",
  "insertTextFormat": 1,
  "kind": 3,
  "label": "binary_to_atom/1"
},
{
  "data": {},
  "insertText": "binary_to_existing_atom",
  "insertTextFormat": 1,
  "kind": 3,
  "label": "binary_to_existing_atom/1"
},

It seems like company calls the capf function for every keystroke, and thus triggering a request towards the LSP. But built-in capf only does it once. So built-in assumes the retrived once are always valid kind of. Maybe performance impact. Can consider optimizations.

URI

From clangd to client "uri": "file:///usr/include/c%2B%2B/15/iostream" So in messages, need to decode by changing percentage, and encode into json with percentages.

Performance

1. Operations inside send/recv threads are essentially free. No GC and no blocking for user. So JSON encode/decode is done there.
2. Operations inside lib.rs are cheap. No GC (except for lisp) and rust is faster than lisp (probably for native compiled lisp too, but would be interesting to compare). However, the user needs to wait.
3. Operations inside tiny-lsp-client.el and other lisp code are expensive.

So prioritize to put operations in 1, and then 2, and only 3 if needed. capf filtering has to be done at 3, and this is where lsp-bridge can avoid big costs. Maybe I can call a rust function to filter? Maybe filtering 50K isn’t expensive (that’s normal work for counsel “rg –files” and also see this SO QA: https://emacs.stackexchange.com/questions/15276/how-do-i-write-a-simple-completion-at-point-functions-function)