Tracking issue: Major version bumps

[DanielVoogsgerd]

We have a lot of dependencies, we also have a lot of dependencies which still need some major upgrades.

Here is a tracking issue to keep track of the upcoming major version updates.

I hope that this could lower our total dependency count a lot and therefore lower the compile times as well.

The relevant dependencies in decreasing order of usage:

• reqwest

• tonic

• rand

• base64

• env_logger

• prost

• bollard

• hyper

• x509-parser

• async-compression

• thiserror

• rustyline-derive

• rustyline

• openapiv3

• getrandom

• dashmap

• dirs

• graphql_client

• rustls

• tokio-rustls

• rustls-pemfile

• nom_locate

• nom

• scylla

Dependencies with overlapping functionality:

• time vs chrono
• base64 vs base64ct

Using cargo upgrade --dry-run --compatible ignore --incompatible allow we find what needs to be done.

name              old req compatible latest  new req
====              ======= ========== ======  =======
async-compression 0.3.15  0.3.15     0.4.20  0.4.20
env_logger        0.10.0  0.10.2     0.11.7  0.11.7
prost             0.12.0  0.12.6     0.13.5  0.13.5
rand              0.8.5   0.8.5      0.9.0   0.9.0
reqwest           0.11.27 0.11.27    0.12.12 0.12.12
scylla            0.12.0  0.12.0     1.0.0   1.0.0
    Checking brane-ast's dependencies
name old req compatible latest new req
==== ======= ========== ====== =======
rand 0.8.5   0.8.5      0.9.0  0.9.0
    Checking brane-cc's dependencies
    Checking brane-cfg's dependencies
name           old req compatible latest  new req
====           ======= ========== ======  =======
rustls         0.21.6  0.21.12    0.23.23 0.23.23
rustls-pemfile 1.0.1   1.0.4      2.2.0   2.2.0
x509-parser    0.15.0  0.15.1     0.17.0  0.17.0
    Checking brane-cli's dependencies
name             old req compatible latest  new req
====             ======= ========== ======  =======
base64           0.21.0  0.21.7     0.22.1  0.22.1
bollard          0.14.0  0.14.0     0.18.1  0.18.1
dirs             5.0.1   5.0.1      6.0.0   6.0.0
graphql_client   0.13.0  0.13.0     0.14.0  0.14.0
hyper            0.14.29 0.14.32    1.6.0   1.6.0
openapiv3        0.5.0   0.5.0      2.0.0   2.0.0
rand             0.8.5   0.8.5      0.9.0   0.9.0
reqwest          0.11.27 0.11.27    0.12.12 0.12.12
rustls           0.21.6  0.21.12    0.23.23 0.23.23
rustyline        11.0.0  11.0.0     15.0.0  15.0.0
rustyline-derive 0.8.0   0.8.0      0.11.0  0.11.0
tonic            0.11.0  0.11.0     0.12.3  0.12.3
x509-parser      0.15.0  0.15.1     0.17.0  0.17.0
    Checking brane-cli-c's dependencies
    Checking brane-ctl's dependencies
name    old req compatible latest  new req
====    ======= ========== ======  =======
bollard 0.14.0  0.14.0     0.18.1  0.18.1
dirs    5.0.1   5.0.1      6.0.0   6.0.0
rand    0.8.5   0.8.5      0.9.0   0.9.0
reqwest 0.11.27 0.11.27    0.12.12 0.12.12
    Checking brane-drv's dependencies
name       old req compatible latest  new req
====       ======= ========== ======  =======
dashmap    5.4.0   5.5.3      6.1.0   6.1.0
env_logger 0.10.0  0.10.2     0.11.7  0.11.7
prost      0.12.0  0.12.6     0.13.5  0.13.5
reqwest    0.11.27 0.11.27    0.12.12 0.12.12
tonic      0.11.0  0.11.0     0.12.3  0.12.3
    Checking brane-dsl's dependencies
name       old req compatible latest new req
====       ======= ========== ====== =======
nom        7.1.0   7.1.3      8.0.0  8.0.0
nom_locate 4.1.0   4.2.0      5.0.0  5.0.0
rand       0.8.5   0.8.5      0.9.0  0.9.0
thiserror  1.0.40  1.0.69     2.0.12 2.0.12
    Checking brane-exe's dependencies
name   old req compatible latest new req
====   ======= ========== ====== =======
base64 0.13.0  0.13.1     0.22.1 0.22.1
    Checking brane-job's dependencies
name       old req compatible latest  new req
====       ======= ========== ======  =======
base64     0.21.0  0.21.7     0.22.1  0.22.1
bollard    0.14.0  0.14.0     0.18.1  0.18.1
env_logger 0.10.0  0.10.2     0.11.7  0.11.7
hyper      0.14.29 0.14.32    1.6.0   1.6.0
reqwest    0.11.27 0.11.27    0.12.12 0.12.12
tonic      0.11.0  0.11.0     0.12.3  0.12.3
    Checking brane-let's dependencies
name       old req compatible latest  new req
====       ======= ========== ======  =======
base64     0.13.0  0.13.1     0.22.1  0.22.1
env_logger 0.10.0  0.10.2     0.11.7  0.11.7
reqwest    0.11.27 0.11.27    0.12.12 0.12.12
tonic      0.11.0  0.11.0     0.12.3  0.12.3
    Checking brane-plr's dependencies
name    old req compatible latest  new req
====    ======= ========== ======  =======
rand    0.8.5   0.8.5      0.9.0   0.9.0
reqwest 0.11.27 0.11.27    0.12.12 0.12.12
tonic   0.11.0  0.11.0     0.12.3  0.12.3
    Checking brane-prx's dependencies
name         old req compatible latest  new req
====         ======= ========== ======  =======
env_logger   0.10.0  0.10.2     0.11.7  0.11.7
reqwest      0.11.27 0.11.27    0.12.12 0.12.12
rustls       0.21.6  0.21.12    0.23.23 0.23.23
tokio-rustls 0.24.0  0.24.1     0.26.2  0.26.2
tonic        0.11.0  0.11.0     0.12.3  0.12.3
    Checking brane-reg's dependencies
name         old req compatible latest  new req
====         ======= ========== ======  =======
base64       0.21.0  0.21.7     0.22.1  0.22.1
env_logger   0.10.0  0.10.2     0.11.7  0.11.7
reqwest      0.11.27 0.11.27    0.12.12 0.12.12
rustls       0.21.6  0.21.12    0.23.23 0.23.23
tokio-rustls 0.24.0  0.24.1     0.26.2  0.26.2
    Checking brane-shr's dependencies
name              old req compatible latest  new req
====              ======= ========== ======  =======
async-compression 0.3.15  0.3.15     0.4.20  0.4.20
reqwest           0.11.27 0.11.27    0.12.12 0.12.12
getrandom         0.2.8   0.2.15     0.3.1   0.3.1
    Checking brane-tsk's dependencies
name           old req compatible latest  new req
====           ======= ========== ======  =======
base64         0.21.0  0.21.7     0.22.1  0.22.1
bollard        0.14.0  0.14.0     0.18.1  0.18.1
graphql_client 0.13.0  0.13.0     0.14.0  0.14.0
hyper          0.14.29 0.14.32    1.6.0   1.6.0
prost          0.12.0  0.12.6     0.13.5  0.13.5
rand           0.8.5   0.8.5      0.9.0   0.9.0
reqwest        0.11.27 0.11.27    0.12.12 0.12.12
tonic          0.11.0  0.11.0     0.12.3  0.12.3
    Checking overview's dependencies
    Checking specifications's dependencies
name    old req compatible latest  new req
====    ======= ========== ======  =======
base64  0.21.0  0.21.7     0.22.1  0.22.1
prost   0.12.0  0.12.6     0.13.5  0.13.5
reqwest 0.11.27 0.11.27    0.12.12 0.12.12
tonic   0.11.0  0.11.0     0.12.3  0.12.3

I am going to do this on a dependency basis, not on a workspace member basis, as I suspect the fixes are probably pretty repetitive.

[Lut99]

Ah good! Yeah this is neat.

I remember you talking a while ago about moving some deps to the [workspace.dependencies]-level. I can’t recall the conclusion though, but it seems relevant to make this better maintainable!

[Lut99]

On that same note, is there a way of checking deps are actually used? I want to say I’ve been very strict with that over the years, but there’s every possibility some of the larger crates are littered with graveyard deps in the first place…

And then finally, there’s the matter of crate consistency. I think in some cases I might have switched crates halfway through - the logging thing is a good example. Might be worth to also sit down one day and try to homogenize that (e.g., make them all use the tracing thing or at least envlogger/humanlog, whichever you prefer)

[DanielVoogsgerd]

I remember you talking a while ago about moving some deps to the [workspace.dependencies]-level. I can’t recall the conclusion though, but it seems relevant to make this better maintainable!

We can, not sure if we should, at least not for all packages. They are a bit more of a pain to work with than the normal way of declaring packages, but like you said, they do make it easier to be consistent on what version is actually used. Maybe it can be nice for packages that we use a lot throughout the workspace, but doing it for every dependency might be more hassle than it's worth.

Edit: A small additional point against setting all packages via workspace dependencies would be that you would unnecessarily increase the minimal version, but I think this is so minor that it should not influence the decision either way. Just for completeness.

The main pain arises from Dependabot that is just broken for Cargo. A solution like this would have my preference, it could just open a PR with the updated manifest, and if CI fails it is a clear indication that more work is necessary for that PR to be merged.

On that same note, is there a way of checking deps are actually used?

There are 3 ways of doing it that I know of. In ascending order of sophistication.

• greping for usage in the crate. Either the name of the package is qualified, or there must be a use statement.
• RUSTFLAGS=-Wunused-crate-dependencies warns during compilation if a dependency is unused, but it needs post processing in crates with multiple binaries (maybe libraries as well, not sure). In the case of two binaries A and B it will first compile A and then B, so lets take two dependencies A-dep and B-dep that is used in their respective binary, then during As compilation it will complain that B is unused and vice versa. So you would have to collect the warnings and take the intersection to find all the unused dependencies
• cargo-udeps, nice little crate that does a best effort, but it does not find all issues.

I want to say I’ve been very strict with that over the years, but there’s every possibility some of the larger crates are littered with graveyard deps in the first place…

Reasonably, I found some using udeps in #111, and occasionally I encounter one during some maintenance work

And then finally, there’s the matter of crate consistency. I think in some cases I might have switched crates halfway through - the logging thing is a good example. Might be worth to also sit down one day and try to homogenize that (e.g., make them all use the tracing thing or at least envlogger/humanlog, whichever you prefer)

Yeah, this is the biggest pain, because there exists no algorithm for such determination. We could sit down, or if you prefer, feel free to look through the manifests and reason one by one whether we have found a better alternative by now, I wish I could do it for you, but I haven't made the decision to switch so besides some obvious cases it is hard for me to determine if you used crates with enough overlap to reduce them to one.

[Lut99]

The main pain arises from Dependabot that is just broken for Cargo. A solution like this would have my preference, it could just open a PR with the updated manifest, and if CI fails it is a clear indication that more work is necessary for that PR to be merged.

Ow, as in, for this workspace dependency thing you mean? Yes that'd be annoying. And you're right that only relatively new versions of Cargo support this feature.

Still, for some deps (e.g., bollard), which are used everywhere, might be worth it. But you're better at this than I am xD so maybe not!

Reasonably, I found some using udeps in #111, and occasionally I encounter one during some maintenance work

Oh, I forgot about that one! Well then probably we have the most of them.

Yeah, this is the biggest pain, because there exists no algorithm for such determination. We could sit down, or if you prefer, feel free to look through the manifests and reason one by one whether we have found a better alternative by now, I wish I could do it for you, but I haven't made the decision to switch so besides some obvious cases it is hard for me to determine if you used crates with enough overlap to reduce them to one.

Yeah of course not... Well, I'll keep an eye out for any I notice. And feel free to ask me of course if you find any suspects :)

Nice that the dependency list is growing cleaner and cleaner though! 👍

[DanielVoogsgerd]

Ow, as in, for this workspace dependency thing you mean? Yes that'd be annoying. And you're right that only relatively new versions of Cargo support this feature.

No, just dependency management in general. The PRs I am creating are very trivial in basis. When a new major version is released, bump the manifest version to the lowest minor-patch version of the new major version. Most often the incompatibility is some other part of the API that you aren't even touching so nothing else needs to be done, but if Dependabot already does that manifest part and some problem occurs, it is really neat to add another commit to the branch in which you solve the issues in the usage. This would normally be the workflow I would recommend.

However, Dependabot is quite broken when it comes to Cargo, requiring us limit updates to minor/patch versions and setting lockfile-only for them.

Relevant issues:

• Non-overlapping Rules
• increase if necessary
• Incorrect leading zero behavior

In conclusion, this is just some manual labour for the foreseeable future. (Or I might look into alternative like RenovateBot

Still, for some deps (e.g., bollard), which are used everywhere, might be worth it. But you're better at this than I am xD so maybe not!

Yeah, I think you are right, but more in a see as we go approach. It is very important if we have workspace members that must have the same version of a package in order to work correctly. This is mainly the case when a dependencies type leaks through the API. This is very common with transparent error types. (This also means any major update of a dependency is a breaking change, btw).

Yeah of course not... Well, I'll keep an eye out for any I notice. And feel free to ask me of course if you find any suspects :)

Will do

[Lut99]

However, Dependabot is quite broken when it comes to Cargo, requiring us limit updates to minor/patch versions and setting lockfile-only for them.

That's... disappointing! Hmm. Maybe check RenovateBot at some point in the future (when we're dead).

Yeah, I think you are right, but more in a see as we go approach. It is very important if we have workspace members that must have the same version of a package in order to work correctly. This is mainly the case when a dependencies type leaks through the API. This is very common with transparent error types. (This also means any major update of a dependency is a breaking change, btw).

Fair point! Then it's extra nice to have them synced, yes.

[Lut99]

I can't remember where we talked about it (🌚) but I agree that the base64 is still worth it keep around, then.

RE the time VS chrono: are we depending on time somewhere explicitly? I remember that it's definitely a transitive dependency (maybe even through chrono) and that this sometimes results in RUSTSEC errors (it depends on libc's notion of time which is a bit... dated xD). So I don't mind moving time to chrono at all where possible.