While I did not have much time for Rust during the following months anyway, I have applied for the Sovereign Tech Fellowship in the meantime, and I was very lucky to actually get into this year’s cohort of funded open source maintainers! And what is even better is that another Rust maintainer, Denis Cornehl, who maintains docs.rs, got into the same cohort too.
My funding started in June, and should last for a year, which is awesome, because now I can again fully focus on working on Rust. I thought that I would post periodic (bi-monthly) updates about the upstream Rust work that I do, similar to the one I shared earlier this year. Every two months, I’ll try to pick a few highlights, summarize the rest of interesting stuff that I worked on, and also provide contribution statistics and a raw list of opened PRs.
This post details my open source Rust work done in June and July 2026.
After a relatively long time, I spent some effort on optimizing Rust compile times, which was fun, as this is one of my favourite activities. I approached it from a couple of angles.
During my bors talk at RustWeek 2026, I noted that the compile times of tests in bors are pretty terrible. Recently, I finally spent some time analyzing the causes of it. I thought that I would be working on removing bottlenecks from the compiler (and I still plan to do that!), but by profiling, I actually found some bottlenecks in Cargo, so I took a detour and spent a few days to try to make it faster.
I sent a couple of pull requests to Cargo that removed unnecessary cloning (#17167, #17176, #17178), switched to a better data structure (#17180) or pre-allocated memory (#17177). With the exception of #17167, the perf. effect of those PRs was very small.
What helped much more was replacing the hashing algorithm used by hashmaps and sets in Cargo. Most of its associative data structures were using the default hasher from the standard library, which is quite slow. I switched all such data structures to use FxHash, the hashing algorithm used by the Rust compiler, in #17169. This resulted in ~8-10% wall-time improvement for no-op Cargo invocations on Zed, which was quite cool.
Finally, as discussed later in this post, I also configured Rust’s CI to optimize Cargo with PGO in #159149.
The changes I made were mostly only micro-optimizations, rather than algorithmic changes, which typically means that you won’t get that much speedup. That being said, the “final” results are still quite nice. I took nightly Cargo from 1st of July, which is a day before I started working on the optimizations, and compared it with
5a8cd237d4fad99d862aed51fab8cb5345c970131, which is the merge commit of the PR that added PGO to Cargo. I am not aware of other performance work happening in Cargo in the meantime and the Cargo submodule was since synced to rust-lang/rust, so this benchmark should mostly correspond to the improvements that I made.
I tested a no-op (also called “fresh”, meaning that all sources were up-to-date and the Rust compiler was not invoked) cargo test --no-run execution on bors:
/pr/pe/ru/bors [benchmark]$ hyperfine "cargo +nightly-2026-07-01 test --no-run" "cargo +5a8cd237d4fad99d862aed51fab8cb5345c97013 test --no-run" --runs 50
Benchmark 1: cargo +nightly-2026-07-01 test --no-run
Time (mean ± σ): 227.6 ms ± 3.8 ms [User: 141.2 ms, System: 90.3 ms]
Range (min … max): 222.1 ms … 239.2 ms 50 runs
Benchmark 2: cargo +5a8cd237d4fad99d862aed51fab8cb5345c97013 test --no-run
Time (mean ± σ): 184.3 ms ± 6.6 ms [User: 104.2 ms, System: 84.3 ms]
Range (min … max): 178.4 ms … 216.9 ms 50 runs
Summary
cargo +5a8cd237d4fad99d862aed51fab8cb5345c97013 test --no-run ran
1.24 ± 0.05 times faster than cargo +nightly-2026-07-01 test --no-run
I consistently saw a 20% wall-time improvement on bors. On Zed, I measured a similar 18% wall-time win (fresh cargo check time went from 1100ms to 930ms).
Since I was using the “Performance Optimizer Observation Platform” (poop) a lot while benchmarking Cargo during my optimization experiments, here are its results on bors:
poop "cargo +nightly-2026-07-01 test --no-run" "cargo +5a8cd237d4fad99d862aed51fab8cb5345c97013 test --no-run"
Benchmark 1 (22 runs): cargo +nightly-2026-07-01 test --no-run
measurement mean ± σ min … max outliers delta
wall_time 228ms ± 12.8ms 219ms … 271ms 3 (14%) 0%
peak_rss 106MB ± 402KB 106MB … 107MB 0 ( 0%) 0%
cpu_cycles 559M ± 22.0M 540M … 636M 2 ( 9%) 0%
instructions 1.01G ± 6.23M 1.01G … 1.04G 1 ( 5%) 0%
cache_references 35.3M ± 461K 34.6M … 37.1M 2 ( 9%) 0%
cache_misses 9.29M ± 112K 9.08M … 9.64M 2 ( 9%) 0%
branch_misses 4.37M ± 55.4K 4.32M … 4.58M 1 ( 5%) 0%
Benchmark 2 (28 runs): cargo +5a8cd237d4fad99d862aed51fab8cb5345c97013 test --no-run
measurement mean ± σ min … max outliers delta
wall_time 183ms ± 6.30ms 179ms … 214ms 1 ( 4%) ⚡- 19.7% ± 2.5%
peak_rss 90.6MB ± 167KB 90.2MB … 90.9MB 1 ( 4%) ⚡- 14.7% ± 0.2%
cpu_cycles 415M ± 8.49M 408M … 446M 2 ( 7%) ⚡- 25.7% ± 1.6%
instructions 736M ± 5.41M 735M … 764M 1 ( 4%) ⚡- 27.4% ± 0.3%
cache_references 26.1M ± 489K 25.5M … 28.2M 1 ( 4%) ⚡- 26.1% ± 0.8%
cache_misses 5.98M ± 82.0K 5.84M … 6.29M 1 ( 4%) ⚡- 35.6% ± 0.6%
branch_misses 3.47M ± 29.5K 3.43M … 3.58M 1 ( 4%) ⚡- 20.7% ± 0.6%
Around a quarter less executed instructions. Not bad for a few days of work! That being said, those were the low-hanging fruits, and I expect that further improvements using a similar approach would be difficult. Larger wins could be gained by parallelization or algorithmic changes, e.g. related to caching. And of course, those wins were only on no-op Cargo builds. In situations where the Rust compiler has to be invoked, it will typically dwarf any overhead coming from the execution of Cargo itself. But not always, and every win helps, so I’m nevertheless happy with the result.
Speaking of parallelization, it would help if Cargo was loading TOML manifests in parallel, as this takes a considerable chunk of its no-op execution time. Possible approaches to achieve this are described in this Cargo issue. I spent a few days trying to perform this parallelization, but I don’t have much to show for it. I ran into several problems related to how the Cargo codebase is currently structured (some things are not 'static or Send/Sync, which makes parallelization annoying, and Cargo doesn’t use tokio, so running async functions concurrently or in parallel is tricky).
It would probably require larger refactorings to Cargo to make this parallelization feasible. In the end, I managed to implement a very hacky version of a parallel worker queue, which provided a 15% wall-time win on Zed with 8 threads (on top of the previous optimizations, except for PGO). But it actually involves unsafe to deal around one variable not being 'static, and it is nowhere near being production ready. Maybe I will come back to this sometime later. If this sounds interesting to you, feel free to pick up this work (though I would suggest talking to the Cargo team on Zulip first).
I would like to thank the Cargo team, especially Ed Page and Weihang Lo, for their quick reviews and fruitful discussions.
#[sqlx::test]This is also related to the build times of bors. I already blogged about this in detail recently, so I will keep this section short. Bors uses hundreds of #[sqlx::test] macro attributes, and it turns out that each such attribute was embedding all database migrations into the source code. There is a way to avoid it (described in the linked blog post), which I applied to bors. And then I tried to document this and make it easier to fix directly in sqlx. This work is still ongoing.
A few years ago, I spent a lot of time improving the way we use PGO and BOLT to optimize the Rust compiler and LLVM. Recently, I thought that it might be worth it to optimize also other parts of the toolchain (such as rustdoc, Cargo or Clippy) with PGO too.
First, I modified bootstrap2 in #158912 to allow specifying PGO profiles in a granular way, for individual tools and components. Then I used this new machinery to PGO optimize:
doc builds.bors by ~5%3.Clippy builds.
While I was working on this, the newly introduced additional PGO training made our Linux CI job that produces these optimized artifacts (dist-x86_64-linux) too slow, so I took another look at optimizing it. By adding some parallelization in #159524, I made the job ~20 minutes faster, which more or less balanced out the additional time required to perform more PGO.
Perhaps more importantly, @Mark-Simulacrum started looking into throwing more hardware at this slow job, so in the near future, it might actually get much faster than it is today! That would be really cool, as it would allow us to get performance results on PRs very quickly.
The Cargo team is planning to stabilize a new layout of the build directory, which would unlock a lot of cool improvements. In fact, the Cargo team already stabilized this recently, but it since had to be reverted; I’ll describe why below.
The new build dir layout has one property that is a bit annoying. Before, Cargo used to put all artifacts of dependencies (mainly .rlib and .rmeta files) into a single directory, and then passed a single flag to rustc (-L <dir>), so that the compiler could look them up in this directory. With the new build dir layout, Cargo instead creates a separate directory per dependency, and then passes each directory to rustc with a separate -L argument. This might not sound so bad, until you get to projects like Zed, which have over two thousands dependencies.
There are two issues with this. The first one is operating system limits on the length of command-line arguments. Passing thousand of CLI paths, where each path can have 100 bytes or more, quickly adds up. Of course, this was a problem long before the new build dir layout, especially on Windows. Cargo can already work around this using argfiles; if an invocation of rustc fails because of the command-line length limit, Cargo will instead write the arguments into an (arg)file, and then pass that file to rustc using the @argfile syntax. This sort of works, although it is not optimal for performance, because Cargo has to start rustc twice, and also write a file to disk for each process invocation. Another issue is that some tools do not handle argfiles when dealing with Rust compilation, which means that they might stop working with the new build dir layout. This was the case for sccache, because it did not support argfiles. Luckily, that is being fixed now.
So what is the problem, when we have the argfile workaround? Well, shortly after stabilization, I tried the new build dir layout on Zed, and it failed. It turns out that not just the command-line length is a problem, but also environment variable length. Recently, Cargo started passing many more paths than before to the LD_LIBRARY_PATH variable4. On Zed, this resulted in this environment variable having over 150 KiB (!) of data. Unsurprisingly, Linux didn’t like that.
The second issue is the performance of dependency lookups in the Rust compiler. Historically, the compiler was optimized precisely for the case of being passed a single -L argument, which was supposed to contain all the necessary dependencies. The compiler thus does a lot of preprocessing to optimize lookups of dependencies in a single large dependency. With the new build dir layout, this is completely opposite, because the compiler receives hundreds or even thousands of -L arguments, but each such directory typically contains only a single file.
This means that the previous optimizations can actually be harmful to performance, and also result in quadratic behavior. On our artificial large-workspace benchmark, which contains hundreds of crates that depend on one another, the old dir layout resulted in ~4 thousand dependency queries in the compiler, while the new dir layout resulted in ~2 million queries. In absolute terms, it was only a ~20ms difference, but it is still not ideal to do so much unneeded work.
This was interesting to me, because over the past year I did some work on optimizing search path lookups (#145408, #153131). So I rewrote the dependency lookup in a way that would be independent on the number of -L paths passed to the compiler, and thus be fast with both the old and the new build dir layout. This work was done in #158823. Enabling the new build dir layout made the large-workspace benchmark ~twice slower, and merging this PR made it ~2x faster, so it more or less balanced itself out.
@ranger-ross, who implemented the new build dir layout and is pushing its stabilization (and who recently became a member of the Cargo team :tada:) already implemented several optimizations to avoid passing unnecessary paths to rustc (#17168, #17191, #17236). I also tried to help slightly with this in #17174, which avoids passing unnecessary lint flags when compiling dependencies (for which most lints do not report warnings anyway).
While that helps, the fact remains that Cargo still passes a lot of paths to rustc with the new build dir layout. I suppose that we will have to see whether this will be an issue in practice or not. Having the dependencies exist in separate directories is core to the new build dir layout, but maybe we could do some workarounds to merge both approaches. For example, Cargo could create a single temporary directory with symlinks pointing to the individual dependency files. But it is hard to tell how fast that would be.
Anyway, after these optimizations landed, the new build dir layout was re-enabled, and it is now used by default in nightly Cargo. If you have use-cases that are broken by the new build dir layout, please report them here. I hope that the new build dir layout will work fine on nightly :crossed_fingers:.
bootstrap build system has been broken for years. I fixed that in #158169, and re-enabled debuginfo compression for the little debuginfo that we ship in our distributed Rust artifacts.
zlib, especially with the pure-Rust miniz_oxide implementation used by the standard library. I tried to replace the implementation with the much faster zlib-rs crate, but it turns out that it didn’t help at all here :( I even tried switching to zstd, but with the pure-Rust ruzstd crate, it was even slower than zlib. Some issues are just annoying like that.cargo-semver-checks into Rust’s CI, so that we can check whether the standard library hasn’t introduced a semver breaking change by accident (#159188, #159671, #160253).-Zembed-metadata=no in Cargo in #17149, #17266 and #17267. You can read more about this compiler/Cargo flag in my previous blog post.
@Mark-Simulacrum did some experiments with using EC2 runners for some of our CI jobs, instead of using CodeBuild. It looks like it could speed up the jobs we most care about by 2x while making them cheaper, by using Zen 5 machines for them. It will also require some tooling implemented somewhere in our infra though. We decided to test this in bors; it is now ready for experimentation, after some bors implementation work (#786, #791, #792, #794, #799, #800, #802).rust-lang repositories.@JonathanBrouwer to the triage rotation, to bolster our ranks.rust-lang/rust repository.Below you can find some statistics and a list of PRs that I opened during June and July 2026.
rust-lang GitHub organization.Symbol functions (merged)x perf compare (merged)cli.rs (merged)distcheck when running on CI (open)dist jobs (merged)test-dashboard (merged)rustc-hash (merged)rustc-perf to main (merged)maintainers-in-residence marker team (merged)main (merged)rustc-perf PR note when marking PRs as rollup=never (merged)frontend_threads to Threads (merged)doc-json (merged)doc-json filter to the compare page (merged)-Zno-embed-metadata to -Zembed-metadata=no (merged)ManifestErrorContext (merged)--cap-lints=allow is set (merged)-Zembed-metadata value from the config (merged)-Zembed-metadata=no by default on nightly Cargo (open)josh binaries into a local directory (merged)#[sqlx::test]s can slow down rebuild time (open)needs-fcp label when an FCP is started (merged)Overall, these two months felt really good. Even though I was very busy in my personal life, and I wasn’t working much during the first two weeks of June due to preparing for RustMeet, I feel like I managed to do a lot of work on Rust, which always fills me with joy. As always, none of this would be possible without the help of many other people from the Rust Project, who collaborated with me, discussed various things with me, and reviewed my code. Thank you very much!
I would also like to thank the Sovereign Tech Agency for funding my open source maintenance work! I am really grateful for this opportunity.
Recently, I also enrolled into GitHub Sponsors. I did not advertise it much, because I currently have funding for my open source work through the Sovereign Tech Fellowship. However, that is for one year (and it also isn’t full-time), and it is hard to say whether I will be able to find the next source of funding after that. So it is good to have at least some backup. If you’d like to support my open source Rust work, I would really appreciate it! You can find more ways of supporting Rust Project maintainers here.
By the way, since some people keep asking me, unless otherwise noted, I usually do not use LLMs for my Rust work. In the past two months, there was one pull request where I performed a command-line argument refactoring that was very mechanical and boilerplate-y (moving from getopts to clap) with the assistance of an LLM.
If you have any questions regarding my upstream Rust work, feel free to ask on Reddit.
Any merged commit in rust-lang/rust can be directly installed using rustup-toolchain-install-master. It’s a super useful tool! ↩
The build system of the Rust toolchain. ↩
We cannot really see the effects of Cargo performance improvements in rustc-perf. ↩
The name of this environment variable, used for looking up paths for the dynamic linker, is different on other operating systems. ↩
You might find this post especially useful if you have a project with many #[sqlx::test] tests.
One of the upstream Rust projects that I worked on during the past few years was the rewrite of bors,
the merge queue bot we use to merge all rust-lang/rust PRs. If you are interested in learning more about this bot, check out my talk from RustWeek 2026.
I’m quite proud of the integration test suite of bors, which I spent a lot of effort on, and thanks
to which the bot has been working pretty much flawlessly since we deployed it to production in January 2026
(despite GitHub lately often having… troubles).
One thing that I’m not very happy about though is the incremental rebuild time of bors, and in particular its
test suite. It takes a long time (~8-10s) to rebuild the tests after each change on my laptop,
which is quite bad for productivity.
Recently I finally found some time to profile1 its build time, and learned that it is caused by a combination of several factors:
bors tests
quite often.rustc takes a long time to load and persist the incremental session. I plan to take a look into this.lld a whole second (!)
to link the tests. With wild, it’s just ~200ms.sqlx::tests that I am using heavily in bors take a long time to compile. This is what I will focus on
in this post.As a frame of reference, for my benchmark I was using touch <test-file> && time cargo test --no-run.
Even after a no-op change, it took ~7.5 seconds to recompile the tests, which is super slow.
Of course, it is very well known that sqlx’s proc macros can slow down compilation times,
because of all the crimes interesting things that they do2. However, the case that I encountered
might not be so obvious. In my case, sqlx did not actually even connect to a database! Because I’m compiling with SQLX_OFFLINE=1, unless I work directly on SQL queries. And yes, I am setting
opt-level = 3 for the sqlx-macros crate, as recommended by the sqlx documentation.
So what is happening here? To find out, it is important to understand what is happening when you have a test like this:
#[sqlx::test]
async fn test_foo(pool: sqlx::PgPool) {}
The #[sqlx::test] attribute is super useful, because it creates a new database before the execution
of the test, runs migrations on it, and then gives you a database connection pool, so that you can
run your tests against an actual database, and not against a mocked HashMap3.
Wait, did I say migrations? Hmm, where does it find them? Well, from disk, of course! Each usage of
#[sqlx::test] will gather all migrations from a directory on disk, and then read, parse, validate
and hash each migration. Perhaps counter-intuitively, this part is not that slow! Turns
out that Rust is actually quite fast (who knew, right??), and if you do not have gigabytes of migrations, I/O is probably also not a problem4.
What is worse is the generated output of those macros. For each such test, the macro will generate a complete list of migrations, including their text content and a checksum in the form of a byte array, in the Rust source code as a constant. So if you expand the macro, before each test you’ll find something like this:
args.migrator(&::sqlx::migrate::Migrator {
migrations: ::std::borrow::Cow::Borrowed(&[
::sqlx::migrate::Migration {
version: 20240517094752i64,
description: ::std::borrow::Cow::Borrowed("create build"),
migration_type: ::sqlx::migrate::MigrationType::ReversibleUp,
sql: ::std::borrow::Cow::Borrowed("CREATE TABLE <skipped>)"),
no_tx: false,
checksum: ::std::borrow::Cow::Borrowed(&[193u8, 202u8, <skipped>]),
},
::sqlx::migrate::Migration {
<skipped>
},
<skipped>
]),
<skipped>
});
The example above is shortened, and it skips a lot of stuff. The actual generated code will be much longer, and of course it scales with the number (and content) of your migrations.
Now, if you have something like this in your source code once, that’s not so bad. However, in bors, there are ~350 sqlx tests and 30 migrations. And at that point, it starts to add up rather quickly.
To test my hypothesis that migrations might be causing some of the build slowness, I tested what would happen
if I had only one migration by deleting the rest of them. And sure enough, the rebuild time immediately went from ~7.5s to ~5s! What is perhaps even more telling is that the size of the output of cargo expand --lib --tests went from 32 MiB (!) with 30 migrations to “only” 6 MiB with a single migration. Compiling an additional 26 MiB of Rust code sure isn’t for free.
It wasn’t just about the compilation time of the generated code though. In the profiles, it looked like converting all the migration description data to tokens using the quote crate during the proc macro execution also takes a non-trivial amount of time.
This behavior can be pretty inconspicious, because at the start of the project, the rebuilds were fast (or at least, faster). But then, with each added test, and each added migration, the rebuild time slowly increases, so it creeps up on you.
I tried if the experimental proc macro caching feature that I landed in the compiler last year might help, but it didn’t. Probably the time needed to compile the generated code dwarfs the time to run the proc macro itself, so it does not help if the proc macro itself is cached.
Nevermind: #[sqlx::test] is an attribute, not a derive macro, so this flag doesn’t apply here.
Thanks to @futile for the correction.
First, I started thinking about reducing the size of the generated code, e.g. by representing the checksum byte array in a more compact form. While this would likely help, I realized that as long as we generate code for all migrations next to each test, there will still be too much code.
Then I tried to patch sqlx to move the loading of the migrations to (test) runtime from compilation
time, to get rid of all the inlined migrations. This actually had the desired effect! The rebuild time went down to ~5s, and (at least in the case of bors), the cargo expand output was reduced to ~6 MiB, all while the execution time of the tests wasn’t affected in a measurable way. The change wasn’t even very complicated, all that’s needed is to generate code that will call the function to load migrations at runtime, rather than calling that function in the proc macro and then embedding the description of all migrations in the generated source code.
So essentially, you go from this (pseudo-code):
fn sqlx_proc_macro() -> TokenStream {
let migrations = generate_migrations();
quote! {
Migrator {
migrations: #migrations
}
}
}
to this:
fn sqlx_proc_macro() -> TokenStream {
quote! {
Migrator {
migrations: ::sqlx::generate_migrations()
}
}
}
However, loading the migrations at runtime might have some disadvantages, e.g. the tests would no longer be self-contained.
I went to the sqlx Discord server and asked if people have any suggestions. My proposal was that sqlx could either load the migrations at runtime (which I described above), or that there could be a shared variable with the migrations that all the tests would reference, to avoid the generated code bloat. Funnily enough, one person responded and told me that the second solution is already implemented (well, sort of). You can actually specify a path to a variable containing the migrations to apply in #[sqlx::test]:
// The macro generates the migrations, we store it in a single variable.
const MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!();
// Each test just references the variable, instead of inlining all the
// migrations next to the test's source code.
#[sqlx::test(migrator = "crate::MIGRATOR")]
async fn test_1(pool: sqlx::PgPool) {}
#[sqlx::test(migrator = "crate::MIGRATOR")]
async fn test_2(pool: sqlx::PgPool) {}
I wasn’t sure if
constorstaticwould be better here, and I didn’t measure any rebuild time differences. Butstaticmakes more sense to me, to ensure that the data exists just once in the final binary.
This solution worked great for bors! After adding the migrator argument to all #[sqlx::test] instances with a bit of Find + Replace magic, the rebuild time went down to ~5s.
While this works, what I don’t like about it is the annoying migrator = "crate::MIGRATOR" attribute
that I have to remember to add to all my tests to avoid the migration code bloat problem. I think that
it would be quite elegant to specify a default value for the migrator argument in the sqlx.toml configuration
file added in the 0.9 release of sqlx, so that the shared variable would be used by default, without having
to think about it.
I opened an issue that proposes this feature. Let’s see what do sqlx maintainers think,
although I would understand if they had some concerns about a feature like this.
In any case, even this manual solution was quite helpful, and it made my test rebuild times a bit faster. There is still a lot of space for improvements, as 5s is still slow, though.
Maybe it would be useful to mention this “footgun” in the #[sqlx::test] documentation. I suggested that here.
I think that this post offers two main takeaways:
#[sqlx::test] tests, and you suffer from long rebuild times,
try to use the migrator = "..." trick to see if that helps.cargo expand can be a relatively good predictor of compilation time :)If you have further suggestions on how to optimize rebuild times when using sqlx, let me know on Reddit.
Using just samply record, nothing fancy this time. ↩
Such as connecting to a running database during compilation. I always get blank stares when I explain this to people new to Rust when I do Rust trainings. ↩
I have a talk and a WIP blog post about similar situations… testing is fun (for some definition of “fun”). ↩
At least if you’re not on Windows and have an SSD. ↩
Because sometimes I see people online who compare the number of CVEs in Rust and C/C++ software, which tends to be accompanied by claims about Rust not being really memory safe or not being worth adopting when CVEs can still exist in it. And sometimes I also observe similar views when I teach Rust to programmers who are used to programming in C or C++.
Now, anyone is, of course, free to do such comparisons, and make their own conclusions based on it. But I think that there is an important difference in how potential vulnerabilities related to memory safety are treated in Rust and C/C++, which might not be obvious at first, especially if you don’t know how Rust works. I’d like to explain that in this post.
But first, I should clarify that it is absolutely possible to cause memory unsafety bugs and undefined
behaviour in Rust. In the vast majority of cases1, the unsafe keyword is required for
this to happen, but anyone who claims that Rust programs cannot experience UB at all is simply incorrect.
It is also perfectly possible to cause general vulnerabilities (meaning those unrelated
to memory unsafety) in Rust. Forgetting to add a check that your admin dashboard is only accessible to admins can happen in any language, after all.
And yet, there is something very different between potential vulnerabilities in Rust and C or C++, which is
related to the core reason of why Rust is actually much more memory safe in practice than C
or C++. I’ll try to demonstrate it on the curl networking library, which is written in C.
(lib)curl is one of the most used and well-maintained open source libraries in the world. Its primary
developer, Daniel Stenberg, is one of the most prolific open source maintainers of our time,
and together with many other people, he has been dilligently improving this library for the past 30 years.
Despite having to deal with a recent avalanche of CVEs found by LLMs,
he and his collaborators are doing a very good job of keeping curl safe from potential exploits and vulnerabilities,
and they take pride in curl being a very robust piece of software.
So, let’s take that to the test, shall we? I opened the documentation
of libcurl and found the first function I saw that accepts an argument, curl_getenv. This is supposed to be a simple function that provides a portable
abstraction for getting the value of an environment variable across different operating systems.
curl is supposed to be safe and robust, so surely this function doesn’t contain any UB or memory unsafety, right?
So what about the following C program?
#include <curl/curl.h>
int main(void) {
curl_getenv(NULL);
}
This 5-line C program is as simple as it gets, it just calls the curl_getenv function with a
NULL pointer argument, and compiles without any warnings. And yet, when you execute it, you (might)
get a segfault, and thus a memory safety bug, and thus a potential vulnerability/exploit:
$ gcc test.c -otest -lcurl -Wall -Wextra
$ ./test
Segmentation fault (core dumped)
Of course, this program is artificially simple, but that’s kind of the point. In practice, situations like this can (and do) easily happen in larger programs by accident all the time.
Huh. So maybe curl isn’t so safe after all? Should I go and report this as a vulnerability in curl?!
No, of course not. That would be stupid. I know that, you know that. But how do we actually know it? That’s the interesting part.
Consider a very similar program that would call the function like this: curl_getenv("FOO"). What if
that program would still segfault, and thus contain a potential vulnerability? I am sure that the curl
maintainers would like to know about that happening, and would consider it to be a pretty big issue if I reported it! At the same time, I’m sure that they would (rightfully) tell me off if I reported the first program as a vulnerability in curl. Yet those two programs differ only by so little.
So, what gives? Well, in practice, UB like the one in my original example is said to be caused by “wrong usage”2, and it is not considered to be an issue in the library or API that I am using, but in my (application) code. This is done mostly for the following two reasons:
In C, it is often not possible to specify the contract (invariants, preconditions, postconditions, etc.) of APIs precisely3 due to its limited type system, and library authors often don’t bother describing all possible kinds of wrong usage, as it would not be practical.
Indeed, the documentation of curl_getenv does not say that calling it with NULL is forbidden and might lead to a segfault! The authors thus assume that you will use the library “correctly” (whatever that means), and if you don’t, then any caused vulnerabilities are your fault.
The fact that it is so simple to trigger UB by accident in C or C++ means that if we reported all the potential possibilities of causing a vulnerability, such as the one in my example program, most C or C++ libraries would be flooded by millions of CVEs. It wouldn’t make sense to do that, because there would be five different ways of potentially causing a vulnerability in every function call.
And thus, in C and C++, we usually do not consider similar situations to warrant a CVE in the used library. In other words, we create CVEs for specific misuses of a library, not for the existence of a library API that can be misused.
So, what is the crucial difference between how the situation above would be treated in C or C++ and
in Rust? hyper is likely the most popular networking/HTTP library in Rust, spiritually
similar to libcurl in C. Imagine that hyper would have a similar simple function that would take
an argument, I would write a Rust program like this:
fn main() {
hyper::foo(None);
}
Then I’d hit cargo run, and the program would segfault. Would that be a CVE in hyper? Yes, absolutely4!
The program does not contain any unsafe blocks, so if a memory bug occurs, it had to be caused by the hyper library having a soundness bug.
The difference is that in Rust, when it is in any conceivable way possible to use a library such
that a memory bug occurs, without using unsafe in the user code, it is always a bug in the library,
not in the user code. That is why we call such APIs unsound, or say that they have a soundness hole, because there is
a way to use them wrong (w.r.t. memory safety) in safe Rust.
In other words, we create CVEs when it is possible to use a safe library API in a way that might cause a memory bug, even if we haven’t (yet) found any program in the wild that would actually do so. This means that some of the CVEs reported in Rust are much more “strict” than the ones in C or C++, which some people don’t find “fair”.
If we applied the same logic to C, then curl_getenv should be flagged as a CVE in curl, because it is possible to use it in a way that causes a memory bug. But of course, this doesn’t really make sense in C, because there is no concept of safe and unsafe C (or rather, all C code is implicitly unsafe), which is why I said earlier that reporting this CVE would be stupid.
The answer to the question “do I use this function correctly” (with respect to memory safety issues, not logic bugs), which is often difficult to figure out in C or C++, is very simple in Rust:
unsafe, then the answer is simply YES. It is impossible to use it incorrectly.unsafe, I must mark the call with an unsafe block, which makes it immediately obvious during code review and in the codebase that this place is potentially dangerous. In this (usually very rare)
case, we revert back to the level of C or C++.The first part of the answer is what enables Rust’s memory safety to scale in practice. If you do not use unsafe in your code, which is not needed in the vast majority of situations (unless you are writing something like an operating system or a lock-free data structure), and don’t encounter a compiler bug, you know that any potential causes of memory unsafety are not your fault. If a library does not expose any unsafe interface, you simply cannot use it in a way that would cause memory bugs, unless the library uses unsafe internally and has a bug. But if that happens, the bug is fixed within that library, and all its users are then again automatically safe from memory bugs.
This is the difference between Rust and C or C++. Even though the developers of curl are doing awesome work to build a perfectly safe and robust C library, the millions of other C programs that use it can still very easily introduce memory unsafety just by “holding it wrong”, without the curl developers having any way of preventing that.
I used curl as an example, but the same holds for pretty much any C or C++ library, including
the standard libraries of those two languages (and also other memory unsafe languages in general).
Originally I wanted to show more examples, but in the end I realized that they are all the same, so I stayed with the single curl function, because I think it demonstrates the difference well.
What I described in this blog post is not really ground-breaking in any way, and I think that it is kind of universally understood by most people who know how Rust works. But I also don’t remember seeing any blog post about this, and I was repeatedly explaining it to some people, so I wanted to write down some thoughts about it, so that I can simply link to them the next time this debate happens again.
I hope that what I described above shows that comparing raw numbers of CVEs per line of code of Rust and C or C++ is all kinds of misleading, and we should take that into account when comparing the memory safety of Rust and other systems programming languages.
If you have a different idea on how CVEs (should) work in Rust or C/C++, let me know on Reddit.
Essentially only except for compiler bugs. ↩
Someone might even say “skill issue” :laughing: ↩
In C++, the type system offers many more amenities for describing invariants, yet it still also offers many other ways to sneakily introduce UB by accident, so the end result is similar. ↩
Again, barring any compiler bugs or hardware issues with my RAM module :) ↩
First, some statistics. The GitHub GraphQL API claims that:
1497 PRs in 2025 (+98% from 755 opened PRs in 2024).
1160 (77.49%) out of those were relevant to the rust-lang organization or other upstream Rust work1. I only consider these in this post.50 different upstream-Rust-related repositories in the past year.976 PRs in 2025 (+131% from 421 reviewed PRs in 2024).
753 (77.15%) out of those were relevant to upstream Rust2.I used this script to calculate these statistics.
When I saw these numbers, I was quite surprised. It felt like too much, and I had to go double-check the script I used to compute it. But after examining the list of my opened PRs, and thinking a bit more about my Rust contributions, it started making sense. The high number of PRs that I open(ed) stems from the fact that the kind of work I usually do in Rust is quite maintenance-heavy3. As a member of the Rust Infrastructure team, I do many small and drive-by contributions to various repositories to fix CI, update configs or docs, set up some tooling, etc. I definitely couldn’t open a thousand PRs in a year if they were all implementations of new compiler/languages features or something like that4.
A lot of that work can be quite unglamorous, like doing house chores. There are probably less than 50 pull requests from the past year that I could show to someone and say “here, I did something cool”. The rest is mostly work that “had to be done”. And there’s a lot of that work! Even though 1160 PRs might sound like a lot, it is just a drop in the ocean in the
upstream Rust world. In 2025, just the rust-lang/rust repository
received 10483 (!) pull requests5. I personally opened only 307 out of those PRs. That shows just
how much work goes into maintaining Rust (and note that this is just a single repository, although the most important and extensive one that we have). I try to help this effort by often (but not always) prioritizing unblocking other Rust Project contributors, rather than pushing my own work forward, as that tends to have a larger effect down the line.
It should also be noted that opening pull requests is just a fraction of the work that a typical open-source contributor/maintainer does. There are certainly many other activities that I do, such as code reviews, writing and reading design docs, discussing various ideas with other people on Zulip and GitHub, reading and writing blog posts, triaging issues, governing and making decisions, listening to podcasts to find inspiration, attending and talking at conferences, watching recordings of conference talks that I didn’t see live or thinking about compiler optimizations while walking my dog :) Open-source is about communication as much as it is about writing code (and even that is a form of communication). So quantifying work based on just the number of opened pull requests does not paint the whole picture.
It can be quite hard to demonstrate the value of work like this, especially to companies. That is something that we are trying to improve with the Rust Foundation Maintainer Fund (and other similar efforts). Maintaining Rust so that it keeps working as well today as it did yesterday is a lot of work. And moving it forward by adding new features is even harder6. That is why I think that it is crucial to support contributors that maintain and develop Rust. Without them, Rust could not thrive and evolve.
While I am currently one of the lucky people who are funded for their upstream Rust work7, many Rust contributors do not have access to stable funding. While the Rust Foundation (together with the Rust Project) is currently trying to figure out how we can build a sustainable mechanism for funding full-time Rust maintainers, there are also more targeted ways to help fund individual contributors “in the small”. If you are feeling generous, consider sponsoring individuals who make Rust better on a daily basis.
Here is a short summary of a few things from 2025 that stand out (in no particular order). In 2025, I:
rust-lang-ci/rust repository (finally!), syncing crates.io crate owners in the team repo, moving the Rust web to a static website and adding some cool things to it, and implementing many improvements to our bots (mostly triagebot and bors).cargo build --timings output (not stabilized yet).#[derive(From)] feature), got it accepted :tada: and implemented it. Although it will need to be landed over an edition due to some backwards incompatible name resolution issues, so it might take some time before it gets stabilized.It goes without saying that everything mentioned above was team work that couldn’t be done without the help of the awesome people who are Rust :heart:.
And here is the promised list of the Rust PRs that I opened in 2025. They are categorized by repository (repos are ordered by PR count in descending order) and within each repo they are ordered by creation time in ascending order. Unless otherwise noted, the pull request was merged, although I also kept the opened and closed ones, just to show the open/close/merge ratio.
Below the list you can find a few more thoughts on my 2025 Rust contributions and the conclusion.
x perf directly in bootstrapci.py script in Rustllvm.ccache to build.ccacherun-make-support-Zembed-metadata to allow omitting full metadata from rlibs and dylibsdist::Gcc build step-znostart-stop-gc usage with LLD20RUN_CHECK_WITH_PARALLEL_QUERIESCI_JOB_DOC_URL to DockerDIST_TRY_BUILD for try jobs that were not selected explicitlyrun-make-supportuse-lldcitool to reduce compilation time (closed)opt-dist requires metrics to be enabledrust-cache to speed-up citool compilationopt-dist CLI flag and make it work without LLDdist-x86_64-linux CI job to GitHub temporarilytry-perf branchstdarch into a Josh subtreecreate-docs-artifacts.sh with new bors--json=timings)mingw-check-tidy jobmingw-check-tidy on auto buildscalculate_matrix job on master to cache citool buildsdist-alt jobscitool tests on the auto branchCargo.lock to CI-rustc allowed list for non-CI env”rust.incremental when using download CI rustc-nopt CI jobs--build into account in bootstrapStepMetadata to describe steps--cap-lints (closed)stdarch bootstrap smoke test (open)test_string_interning in bootstrapstd_detect into stdlibtidyToolTarget and port RemoteTestServer and WasmComponentLd to it (closed)doc::Std stepToolTarget to bootstrap--skip-std-check-if-no-download-rustccheck for compiletest and RA using tool macrowasi_sdk_path instead of reading environment variable in bootstrapx test pipelineFn bound when calling a generic parameter (open)rustc_private tools-Zself-profile-events=argsx test on CI-Zprint-mono-items#[derive(From)]stage parameter when setting up stdlib Cargobuild-dir in bootstrappr-check-2 CI jobcompiler_for in bootstrapONLY_HOSTS in bootstrapx check compiler-Zno-embed-metadata in the standard library (open)copy_src_dirs in dry runcopy_src_dirs in bootstrapAssemble stepPartialOrd/Ord from bootstrapFrom derive macro from preludex run mirix test distcheck self-containedriscv64-gc-unknown-linux-musl from Tier 2 with Host tools to Tier 2PartialEq for InvisibleOriginrustc-dev in distcheckToolRustc to ToolRustcPrivategcc.download-ci-gcc to truelibgccjit symlink when we build GCC locallydownload-ci-llvmx <cmd> <path> from a different directoryrustdoc-json-typesx install Cranelift and LLVM bitcode linkerrust.use-lld to rust.bootstrap-override-lldcompiletest-use-stage0-libtest from CIdownload-rustc if LTO is configuredbors environment also to the outcome jobubuntu:24.04 on ghcrx86_64-gnu-miri jobopt-dist (open)eza 0.21.2 benchmarkcranelift-codegen-0.82.1 benchmarkdiesel-1.4.8 benchmarkhyper to 1.6.0clap-3.1.6 benchmarkexa-0.10.1 benchmarkregex-automata-0.4.8 benchmarkhyper-0.14.18image-0.24.1libc-0.2.124regex-1.5.5ripgrep-13.0.0ripgrep-13.0.0-tinydiesel-2.2.10 -new-solver benchmark (closed)nalgebra-0.33.0-new-solver benchmarksyn-2.0.101-new-solverserde-1.0.219-new-solverbitmaps-3.2.1-new-solverhtml5ever-0.31.0-new-solver--exact-match CLI argument to allow exact matching of benchmarksserde-1.0.219-threads4 benchmark for the parallel frontendperf-config.json is not foundlarge-workspace stress test benchmark[workspace] in compile benchmarkstriagebot benchmark (closed)crates.io benchmark (closed)--exact-match instead of --include on the websiteDocJson profile for benchmarking JSON rustdoc outputcollectorcargo stable benchmarkstyle-servo stable benchmarkinclude-blob secondary benchmarkinclude-blobcargo compilable with modern rustc and re-enable stable benchmarks./target/release example commands to cargo run --releaserust directory does not existeprintln profiler invocation failshtml5ever compilable on betamaster-old branch in rustc-dev-guidementorship-programs team--repo argument in dry-run workflowallowed-merge-teamssync-team code for performing dry runsteam to clapsync-team from team directlygsoc-contributors teamgsoc-contributorsmentors teamgsoc-contributors as a subteam of launching-padrust-timer merge bot and try-perf/perf-tmp branch protectionsa-mir-formalityexpect-testexpect-test teamvscode-themes under automation and archive itjosh-sync repositoryjosh-syncinfra-admins to the bors repopromote-releasemainmainmainrust-analyzer/countme repositoryrust-lang/rustrustfmtt-website Zulip stream to t-website website entryT-website zulip group for the website teamannotate-snippets cratemeasureme cratesthorincc-rsannotate-snippetsgenerate-tokens reusable actionfunding repositoryCRATES_IO_USERNAME environment variable in dry-runar_archive_writercrates_io_og_imagegll repositorysqlx-cli in CITreeState in the DBsqlx::Encode implementation for GithubRepoNameDEFAULT clause and add a test for itIF NOT EXISTS to repository creation migrationhelp and info command output nicer@bors tryTestSyncMarker*) in the command parser@bors try commands to replace homu for try builds@bors infoTest timed out commentbors build finished failed jobsenqueue_prs_on_pr_openedmin_ci_timestalled to failedtest job to succeed before starting deploy job@rustbot assign and r? behavior on PRs (closed)IssueData to the DB when it is unchangedassign.review_prefswhoami Zulip commandlookup [github|zulip] <username> command for determining the GitHub/Zulip username mappinglookup zuliplookup githubclapwork showwork showteam-stats Zulip commanddocs-update and ping-goals also through DMsDebug impl@rustbot reroll commandteam@bors rollup=never to rustc-push PR body@bors rollup=never to rustc-push PR body”GITHUB_TOKEN to Zulip CI steprustc-josh-syncrustc-josh-syncjosh-sync@bors try jobscg_gcc invocationAsync Rust is about concurrency, not (just) performance blog postTwo ways of interpreting visibility in Rust blog post (closed)Evolution of Rust compiler errors blog postDisable debuginfo to improve Rust compile times blog postWhy doesn't Rust care more about compiler performance? blog post (closed)Combining struct literal syntax with read-only field access blog postAdding #[derive(From)] to Rust blog postRust compiler performance survey 2025 results blog postReducing binary size of (Rust) programs with debuginfo blog postVariadic Generics Micro Survey official Inside Rust blog postLaunching the 2025 State of Rust Survey Rust blog postGoogle Summer of Code 2025 results blog postMaking it easier to sponsor Rust contributors blog postInvestigating and fixing a nasty clone bug blog post/logosrustc-josh-sync and add a few fixesrust-version and mention this repo in merge commit messagegh PR creation prompt more accuratepull--allow-noop was passed--allow-noop@rustbot rerollproc-macro-src CI jobstd_detect dev dependency in core_archmulti-user@bors prefix for new bors (open)-Zembed-metadata--json=timings behing -Zsection-timings--timings pipeline graphmaster branch instead of stable in CIid-token permissionstdarch test suitetoml to 0.9master to mainrustc-josh-syncno-merges PR titlejosh-sync instead of miri-script for Josh synchronizationrustc-josh-sync merge commit message for pull PR descriptionmaster to mainrust-lang/rust instead of rust-lang-ci/rust for unrolled rollup build commit URLsrustup.sh to use rustc-josh-sync (open)jemalloc-sys finishesstd::thread::available_parallelism for determining the default number of jobs-znostart-stop-gc linker flagfill_tododerive(From) for single-field structsnative-c crateLooking back at the above list, I think I managed to do a lot of work. Even though after finally finishing my PhD I had a lot of energy and time to work on Rust at the beginning of the year, due to some new family responsibilities, teaching at a university, and doing a lot of “non-technical” work (like surveys, mentoring or governance), I felt a bit unproductive in the second half of the year. At least in terms of actual coding, which is still something that I enjoy the most and that “feels” the best, even though other kinds of work are also important.
In fact, my personal Rust TODO list grew (not shrank) in 2025, which is a bit depressing. A large part of that was likely caused by me trying to finish initiatives already started year(s) ago, which I didn’t have the bandwidth to drive forward during my PhD studies, and so new ideas had to wait.
It is also clear that I maintain perhaps definitely too many projects and initiatives. I think that is a good
thing for the Rust Project, because I’m able to help maintain some things that would probably otherwise
be unmaintained or stuck. However, I’m not sure if it’s such a good thing for myself. Even though I think
that I’m relatively good at handling many initiatives at once9 and I’m used to context-switching many times per day hour, that doesn’t mean that I wouldn’t be (or at least feel) more productive and less stressed if I was working on a smaller set of projects at once. It is my short-to-medium-term goal to reduce
the number of Rust projects that I contribute to, so that I can focus on things that I find the most
impactful and most enjoyable for me to work on, such as compiler performance, which I didn’t get to
at all for the past few months. The hard part, as always, is to find people that would continue to
pass the torch so that things I “leave behind” don’t get stuck again.
One area where I’m thinking of scaling down my contributions is the Rust survey team. I mostly only joined it because it seemed like no one else would do it at the time. And even though I created some scripts to partly automate survey analysis, doing surveys doesn’t exactly spark joy to me, so I would like to pass the torch to others. But it’s tricky…
I hope you found this post interesting. I’m thinking of making this a regular yearly “look back” post (for as long as I’m able to work on Rust), so maybe see you here in a year :)
And in case anyone is wondering, none of the PRs listed above were vibe-coded. I don’t use AI much for normal development, although sometimes it is quite useful for creating one-off scripts, such as those I used for gathering my GitHub statistics.
If you found some other interesting new year open-source data visualizations, let me know on Reddit.
It also includes a handful of contributions to the “Rust ecosystem”, i.e. Rust crates on crates.io that are outside the rust-lang organization. But there are not many of those, and I did not include PRs to my own crates, such as delegate or cargo-pgo. ↩
It’s a bit funny how similar is the ratio of upstream Rust work in both my opened and reviewed PRs. ↩
I plan to blog Soon™ about my definition of software maintenance, either on this blog or on the official Rust blog. Stay tuned. ↩
The first one was #134988. On average, it is more than 28 PRs opened per day. ↩
I like to say that if you gave me a hundred full-time engineers, I would find all of them something to work on in the Rust toolchain immediately. ↩
On average, I spend around 70% of my work time on Rust, and I am funded for all that. ↩
Which is sadly kind of stuck on finding more people that understand this part of the compiler (or on becoming such a person myself). ↩
In reality, I probably suck at it the same as everyone else does, but I don’t know how to operate in any other way, so I don’t have a baseline to compare to. I also kind of suck at working on a single thing for a long time, which is probably why I cling to context switching so much, almost as a form of procrastination. ↩
I’m currently trying to finally bring the bors GitHub merge bot to production, so I spent many days in the past weeks fixing various possible race conditions, making its implementation more robust and improving its test suite.
The bors test suite is mostly integration-based; it has very few what you could call “unit tests”. In most tests, we spawn the whole bors web application, connect it to a real running instance of Postgres, apply database migrations, send HTTP requests that emulate GitHub webhooks to it, and let it communicate over the network with a fake HTTP server that emulates GitHub endpoints1. The mocked HTTP GitHub endpoints are implemented using the cool wiremock crate. A typical bors test looks something like this:
#[sqlx::test]
async fn try_build_failed_modify_labels(pool: sqlx::PgPool) {
let gh = GitHub::default().with_default_config(
r#"
[labels]
try_failed = ["+foo", "+bar", "-baz"]
"#,
);
run_test((pool, gh), async |ctx: &mut BorsTester| {
ctx.post_comment("@bors try").await?;
insta::assert_snapshot!(ctx.get_next_comment_text(()).await?, @r"
:hourglass: Trying commit pr-1-sha with merge merge-0-pr-1…
To cancel the try build, run the command `@bors try cancel`.
");
ctx.pr(()).await.expect_added_labels(&[]);
ctx.workflow_full_failure(ctx.try_workflow()).await?;
ctx.expect_comments((), 1).await;
ctx.pr(())
.await
.expect_added_labels(&["foo", "bar"])
.expect_removed_labels(&["baz"]);
Ok(())
})
.await;
}
While doing a particularly big refactoring, a few tests started failing in a very weird way, by panicking in one of the mocked GitHub endpoints. It was a PATCH endpoint that was trying to deserialize a JSON payload from the request body, but the body was empty, so the deserialization of course failed. That was very weird, because bors only sends a PATCH request at one place in its codebase, and there it does fill the request body. Also, the body was empty only sometimes, in some situations it was normally present. That’s usually the sign of a wonderful bughunt starting.
The receiving endpoint mocked with wiremock contained this deserialization code:
#[derive(serde::Deserialize)]
struct SetRefRequest {
sha: String,
}
let data: SetRefRequest = req.body_json().unwrap(); // The test panicked here
bors uses the octocrab crate for communicating with GitHub. The code that is used to send the PATCH request looked like this:
let res = repo
.client()
._patch(
url.as_str(),
Some(&serde_json::json!({
"sha": sha.as_ref(),
"force": matches!(force, ForcePush::Yes)
})),
)
.await?;
Well, clearly I was sending a body on the sender side (a JSON object with two fields, sha and force), and I was trying to deserialize a request type with the correct shape (the sha attribute) on the receiving side. But in some tests, and only after a few calls of the endpoint, the body was suddenly empty. That was super weird.
My usual motto when dealing with weird bugs is “the problem is in your code, not in a dependency”, which is correct approximately 99% of the time. So I started debugging the failing tests, and the place where the request is sent in bors, but everything looked correct. I always seemed to send the request with a body, but sometimes it arrived without it. So I started suspecting that a problem might be in some dependency. I naturally suspected wiremock first, because it is a much less commonly used crate than octocrab, so I started digging into it. The issue tracker didn’t contain anything that would sound related, so I went back to the good ol’ debugger.
Debugging external Rust crates is not actually that easy. Rust is known for being very abstraction heavy (especially in library code), which can be particularly annoying for step-by-step debugging, and even more when it involves async code. Furthermore, if you want to modify the debugged code (e.g. to add more logging), it is not so easy as e.g. in Python, where I could just go to a virtual environment and modify the source code of the dependency manually. If I do that in Rust, Cargo won’t actually pick up that the source code of a crate in $HOME/.cargo/registry/src/... has been modified (and making modifications here is not a good idea anyway). So I git cloned wiremock and depended on it locally with wiremock = { path = "..." }, so that I could add more logging to it to see what’s going on2.
But again, that did not end up with anything interesting. Once I went up the call stack, I saw that wiremock uses the hyper crate to create its HTTP server. And indeed, the hyper request arrived with an empty body (sometimes). I did not find it very probable that hyper would contain a bug that would just randomly remove request bodies, so I decided to take a look elsewhere.
I still wasn’t prepared to believe that this could be a bug in octocrab, so I pulled up wireshark, to find a definitive proof whether the problem is in the sender or the receiver side. Luckily, the communication was on localhost via HTTP (without TLS), so investigating it with TCP/IP dumps was quite easy, and I finally found something useful. The HTTP stream looked something like this:
PATCH /repos/a/b/git/refs/heads/main HTTP/1.1
x-github-api-version: 2022-11-28
content-type: application/json
authorization: Bearer foo
user-agent: octocrab
host: localhost:5555
content-length: 28
{"sha":"foo", "force":true}
HTTP/1.1 500 Internal Server Error
content-length: 0
PATCH /repos/a/b/git/refs/heads/main HTTP/1.1
x-github-api-version: 2022-11-28
content-type: application/json
authorization: Bearer foo
user-agent: octocrab
host: localhost:5555
HTTP/1.1 500 Internal Server Error
content-length: 0
Notice something strange? The body is missing in the second request! So I finally knew for a fact that the sender caused the issue. So at this point, I had three things in mind:
octocrab.I found it quite hard to believe the first two options (especially since bors contains no unsafe code), so I finally set my sights on octocrab. I went to the place where I was creating the PATCH request and stepped through the internals of octocrab, but the body seemed to be there! Although it was not exactly trivial to check, because HTTP bodies are kind of difficult to inspect; in the general case, they can be streamed, so Rust networking APIs are usually built in a way where you can only consume the body once3.
Since I still couldn’t find anything weird even in octocrab, I went back to the drawing board, and tried to approach the problem from a different viewpoint. After some more thinking, I asked myself: why is the request even being sent multiple times in the first place? Originally, I did not find that weird, because the failing tests were returning a 500 HTTP response from the mocked endpoint, to simulate a problem with GitHub, and bors implements request retrying, so it seemed like it was working as intended. However, after inspecting the test logs4 a bit more, I realized that my own retry code was not actually being triggered! That started sounding really suspicious - why was the request retried then?
After taking another look at the octocrab codebase, I realized that it actually implements its own mechanism for request retries, which I had no idea about. And it is enabled by default! And sure enough, when I disabled the retry mechanism:
let client = Octocrab::builder()
...
.add_retry_config(RetryConfig::None)
.build()
The problem with the empty body was gone! :tada: So I finally fixed the problem on the bors side, and since bors was (unintentionally) doing double retries before, disabling the octocrab retry mechanism actually fixed two bugs at once. I think that this was possibly the first time I found an actual bug in a Rust crate dependency that broke my code in ~7 years of using Rust. Pretty good track record for the Rust ecosystem in my experience so far!
But of course, I couldn’t just stop there. I had to know what is the original cause of the bug, so that I could report it and fix it.
Once I finally knew that the bug was caused by the octocrab retry mechanism, creating a minimum reproducible example was relatively easy:
async fn handler(body: String) -> impl IntoResponse {
eprintln!("Received data: {body}");
StatusCode::INTERNAL_SERVER_ERROR
}
#[tokio::main]
async fn main() {
let port = 5555;
std::thread::spawn(move || {
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap();
runtime.block_on(async move {
let app = Router::new().route("/foo", patch(handler));
let listener = tokio::net::TcpListener::bind(
format!("127.0.0.1:{port}")
)
.await
.unwrap();
axum::serve(listener, app).await.unwrap();
});
});
let client = octocrab::OctocrabBuilder::new()
.personal_token("foo")
// .add_retry_config(RetryConfig::None) // <- this fixes the issue
.base_uri(format!("http://localhost:{port}"))
.unwrap()
.build()
.unwrap();
#[derive(serde::Serialize)]
struct Request {
force: bool,
}
let response = client
._patch(
"/foo",
Some(&Request { force: true }),
)
.await
.unwrap();
eprintln!("Response: {}", response.status());
}
This code printed the following:
Received data: {"force":true}
Received data:
Received data:
Received data:
Response: 500 Internal Server Error
Exactly as in bors. When the request was sent for the first time, the body was correctly sent
over the wire. But when octocrab tried to retry the request, suddenly the body disappeared.
Originally, I did not notice this problem when debugging octocrab, because I sort of stopped too soon,
at a place where the HTTP request was passed from octocrab to a stack of tower middleware.
And at that point the request was still fine! But now it became clear that the problem was somewhere inside
that middleware, which is why it was so hard to find originally.
I wasn’t aware of how retrying works in tower, and if this is a bug in octocrab
or in tower (although now I suspected the former). After a bit of digging, I found that
retrying is indeed implemented in octocrab with a retry Policy, which tells tower how to retry a given HTTP request. The most important part is the clone_request method:
fn clone_request(&mut self, req: &Request<OctoBody>)
-> Option<Request<OctoBody>> {
match self {
RetryConfig::None => None,
_ => {
// `Request` can't be cloned
let mut new_req = Request::builder()
.uri(req.uri())
.method(req.method())
.version(req.version());
for (name, value) in req.headers() {
new_req = new_req.header(name, value);
}
let body = req.body().clone();
let new_req = new_req.body(body).expect(
"This should never panic,
as we are cloning a components from existing request",
);
Some(new_req)
}
}
}
Notice how the code says “Request can’t be cloned”, yet it clones the request body like it’s not a big deal :thinking: That looked suspicious, so I looked at how the request body type looks like, at which point the bug became obvious:
#[derive(Debug)]
pub struct OctoBody(Arc<RwLock<BoxBody>>);
Here is the Clone impl, if you haven’t noticed the issue yet:
impl Clone for OctoBody {
fn clone(&self) -> Self {
OctoBody(Arc::clone(&self.0))
}
}
Yes, cloning the body just performs a shallow clone of an Arc, instead of performing a deep copy of
the whole body (!). And since the body is consumed when the request is sent for the first time, the code
simply started sending an empty body when the request was retried.
This is where the relation to the ergonomic cloning initiative comes in. If we had a clearer distinction
between performing a deep copy vs just cloning a (reference-counted) Handle to something, it would likely be much more
obvious that there is a problem with the .clone() call.
It still seemed suspicious why this was a silent bug though. Shouldn’t it panic somewhere if we try to reuse the same body again? :thinking: If we take a look inside the Body trait implementation of the OctoBody, we see that in the poll_frame method, it locks the RwLock and forwards the poll_frame call into its inner boxed body that it abstracts over:
impl http_body::Body for OctoBody {
type Data = Bytes;
type Error = crate::Error;
#[inline]
fn poll_frame(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Option<Result<Frame<Self::Data>, Self::Error>>> {
let inner = Pin::into_inner(self);
let mut boxed_body = inner.0.write()
.expect("RwLock write lock failed");
Pin::new(&mut *boxed_body).poll_frame(cx) // <-- poll forwarding
}
...
}
That sounds like it could cause an issue if the body is being polled even after it has been “consumed” while sending the request for the first time. You might think that if OctoBody stored an RwLock<Option<...>>, which would be replaced by None once the body is consumed, it would make this into a noticeable panic. But OctoBody actually already mostly does that! It usually stores a http_body_util::Full struct as its boxed Body implementation, and that does exactly that. So why haven’t the code panicked, instead of sending an empty body in a retry?
Well, as it turns out, the poll_frame method is actually not ever called again once the request is consumed for the first time. When hyper polls the body, it first uses the Body::is_end_stream method to check if the body is at its end, and if it is, it does not ever poll it again. And the http_body_util::Full implementation of course returns true from this method once its inner body has been consumed. So when the body is consumed for the first time, it will start returning true from is_end_stream, and when we then try to reuse the same body in another request, hyper will just skip reading the body contents and send an empty body instead. Kind of footgunny if bodies are ever reused!
It is interesting to note that this could have caused an issue only because the Arc was combined
with interior mutability (RwLock). If there was no interior mutability, performing a shallow
(instead of a deep) clone would likely result at most in a performance degradation, but not a logical
bug. Interior mutability is useful, but it also relaxes some of the benefits we are normally
used to when using Rust!
It was also surprising to me that this bug has existed in the octocrab crate for more than two years,
but it does not seem like anyone noticed it. I suppose that sending a request to the GitHub API that
both has a body (so it is not a simple GET request) and that gets retried (so GitHub returned e.g. a 500 error
the first time the request was sent) is just not that common. It seems like my heavy usage of HTTP integration
testing in bors is uncovering all sorts of interesting issues :laughing:
The bug might look obvious in retrospect, but it took me several hours to debug it, and I was getting pretty desperate after that time. I’m certainly not used to spending so much time debugging a single issue when working with Rust! In hindsight, I should have tried to send the PATCH request manually, without octocrab. That would give me a hint much sooner that the problem is indeed within octocrab. I guess avoided it because of my usual thought of “the bug has to be in my code”. Well, the next time, I will probably start suspecting dependencies a bit sooner :sweat_smile:.
After finally understanding what was going on, I reported the issue in the octocrab issue tracker and went to sleep. It was a long day :zzz:.
After I rested for a bit and regained enough motivation to take a look at octocrab again a few days later, I took a
shot at fixing the actual issue. The root of the issue was in performing a shallow copy, instead of
copying the whole body contents. That is actually easier said than done, as HTTP request bodies are not exactly
known to be easy to clone, as they could be streamed. However, since octocrab does not necessarily has
to be the most blazingly fast :rocket: zero-copy :fire: library under the sun, I figured that a bit of
copying in the common case won’t hurt that much.
What I did was to add a separate reference-counted copy of the request body, using the venerable bytes crate:
#[derive(Debug)]
pub struct OctoBody {
body: Arc<RwLock<BoxBody>>,
// Copy of the whole body, used for retrying requests
buffered: Option<Bytes>,
}
The buffered field is an Option, because in the most general case, the body might be streamed, in which case I can’t easily turn it into Bytes outside of an async context. However, at that case, we will now at least no longer send an empty body by mistake, but rather do not perform a retry at all.
In cases where the body is available in a buffered way (which I assume will be in 99% of cases when using octocrab), we copy the body contents into Bytes when wrapping it into OctoBody:
impl From<Vec<u8>> for OctoBody {
fn from(buf: Vec<u8>) -> Self {
let buffered: Bytes = Bytes::from(buf.clone());
Self::create(http_body_util::Full::from(buf), Some(buffered))
}
}
This pessimistically clones the body contents even if we never actually do a retry, but at least it makes the code work as it should. It might be better to only perform this copy if the retry feature of octocrab is enabled. Could be an interesting follow-up pull request, if you’re interested!
The final piece of the puzzle was adding a method for deep copying the body, which I called try_clone:
/// Try to perform a deep clone of this body
pub fn try_clone(&self) -> Option<Self> {
self.buffered.as_ref().map(|buffered| {
Self::create(
http_body_util::Full::from(buffered.clone()),
Some(buffered.clone()),
)
})
}
and then using this method in the clone_request method:
fn clone_request(&mut self, req: &Request<OctoBody>)
-> Option<Request<OctoBody>> {
match self {
RetryConfig::None => None,
_ => {
let body = req.body().try_clone()?;
...
}
Notice how try_clone returns an Option. If the request body cannot be cloned, the clone_request
method will also return None, and the request simply will not be retried. Which is much better than
sending an invalid request!
My fix was sent in this pull request, which was
merged soon after, and included in the 0.49.1 octocrab release. If you use octocrab and do not disable the automatic retry mechanism, you might want to update to it.
While writing this blog post, I wondered if LLMs could actually find this bug sooner, and save me the hours of debugging pain. I’m not a very skilled vibe-coder, but I do use Claude sometimes, so out of curiosity, I asked it to locate the bug.
First, I showed it the minimal reproducible example from my octocrab issue. It found out that the problem is with retries pretty much immediately, although with this example that was already quite obvious. After asking it to dig deeper, it also found out the exact cause of the underlying clone bug directly in octocrab. That was relatively impressive, although it also pointed me to the patched 0.49.1 release, and cited text that was suspiciously similar to my original issue and my PR. So it clearly saw my fix :laughing: And thus it’s unclear whether it would be able to find the issue even before I actually fixed it. Well, too late to test that now.
In a more realistic scenario, I reverted bors to a commit right before I disabled the retries, and asked Claude to try to find the issue there. At first, it produced some nonsense suggestions, such as claiming that I send an empty string for the sha field (which is quite different from sending an empty request body…). Then it kind of forgot that I’m using Rust and not C++ and claimed that the issue is a “subtle temporary lifetime problem with async/await”:
I think I found the issue! It's a subtle temporary lifetime problem
with async/await in Rust.
Look at this code in src/github/api/operations.rs:172-181:
let res = repo
.client()
._patch(
url.as_str(),
Some(&serde_json::json!({
"sha": sha.as_ref(),
"force": matches!(force, ForcePush::Yes)
})),
)
.await?;
The problem is that serde_json::json!(...) creates a temporary value,
and you're passing a reference to it (&serde_json::json!(...)).
Even though octocrab's build_request is synchronous and serializes
the body before the .await, there can be subtle issues with temporary lifetime
extension in async contexts, especially across different compiler versions
or optimization levels.
Here's what might be happening:
1. The json! macro creates a temporary serde_json::Value
2. A reference to this temporary is passed to _patch
3. Due to the async context, there's a possibility the temporary
gets dropped before serialization completes
4. This results in either garbage data or an empty body being sent
The Fix
Store the JSON value in a variable first to ensure it lives long enough:
let body = serde_json::json!({
"sha": sha.as_ref(),
"force": matches!(force, ForcePush::Yes)
});
let res = repo
.client()
._patch(url.as_str(), Some(&body))
.await?;
After I patiently explained that this is not the kind of issue that can just happen (without unsafe) in Rust, and hinted that the problem might also be in a dependency, it finally started looking at octocrab, and actually did find the shallow clone problem (although it is again unclear whether that would happen without it knowing about my fix):
Ah! I found something suspicious. Look at the OctoBody implementation -
it uses Arc<RwLock<BoxBody>> and implements Clone by cloning the Arc:
impl Clone for OctoBody {
fn clone(&self) -> Self {
OctoBody(Arc::clone(&self.0))
}
}
This means when the body is cloned, both clones share the same underlying
body stream! If one clone consumes the body (by polling it), the other clone
will have an empty body!
However, even though it found the actual bug, it then started to lose confidence that this bug is the culprit, in a very funny way. It examined my Cargo.toml:
Wait, I don't see the "retry" feature enabled for octocrab in
bors's Cargo.toml. Let me check if bors has its own retry logic
that might be cloning requests:
It failed to realize that retry is a default feature of octocrab, so even if I do not enable it myself, it was actually enabled by default. This was enough for it to stop suspecting octocrab, and go on a hunt to find (hopefully fictitious) retry bugs in the logic of bors. At that point I lost my patience with it and stopped it.
I think that this shows both the advantages and disadvantages of debugging using LLMs. They can sometimes find even pretty tricky issues, but they can also get sidetracked, at which point they need someone else (either a human that actually understands what is going on, or another LLM :) ) to guide them. I can imagine someone less knowledgeable about Rust wasting quite some time debugging imaginary issues in bors found by Claude, in an attempt to locate what might be causing the empty request bodies.
Anyway, next time I start a similar bughunt, I will probably just start Claude in a terminal and let it run concurrently. Then we can see who can get to the core of the bug sooner :)
I’m really glad that I managed to find the underlying issue, as it was a satisfying conclusion to an otherwise pretty maddening debugging experience. And in the end, it helped fix other pre-existing bugs in bors and its test suite and also fix a nasty bug in a popular Rust crate at the same time. So, a win-win?
By the way, this post is not taking a shot at octocrab in any way. It’s a very useful library that
saved me a lot of time over the years! Kudos to XAMPPRocky for creating
it and for merging my bugfix PR and releasing a new version so promptly.
If you have similar Rust bug stories, perhaps related to wrong usage of clone, I’d be glad to hear them on Reddit.
I recently had a talk about my experiences with Rust testing, inspired primarily by the bors test suite. Since the recording will probably not be available online, I plan to turn that talk into a blog post Soon™. Stay tuned. ↩
If it was a transitive dependency, I could use the [patch] section in Cargo.toml. ↩
This is actually kind of foreshadowing the cause of the bug :) ↩
Using the traced-test crate. ↩
Perhaps contrary to popular belief, Rust binaries can actually be quite small if you spend some effort on it. However, once you start including debug information (debuginfo) in them, the binary sizes typically start increasing at an alarming rate.
If you wonder why we should care about binary size when debuginfo is enabled, as that’s typically used for debug builds, it is also commonly used for distributing release/production binaries that include at least debug line tables, which are needed to produce quality backtraces.
In this post, I’ll shortly describe why debuginfo takes so much space in Rust programs and how we can optimize it. Note that I will only talk about the DWARF debuginfo format and Linux ELF binaries. Furthermore, I will only be considering debuginfo embedded directly in the binary. If you distribute “split” debuginfo in a separate file, you might not care about its size so much, but that approach has other trade-offs, and I won’t consider it in this post.
Apart from (DWARF) debuginfo generally taking quite a lot of space on its own, there is a less obvious reason why debuginfo in binaries can be so large, and that is because linkers typically do not “tree-shake” unused debuginfo (unlike unused code, which gets removed). I only found out about this recently, which was the motivation for writing this post.
Let’s try to demonstrate that with a simple example. Consider the following crate (let’s call it dep), which has a single function foo:
pub fn foo() { println!("foo"); }
and another crate called mycrate that depends on dep and calls its foo function:
fn main() {
dep::foo();
}
When compiled with Rust 1.90.0 on x64 Linux in --release mode, the resulting mycrate binary has exactly 449328 bytes.
If I add another function bar to the dep crate, without changing anything else:
pub fn foo() { println!("foo"); }
pub fn bar() { println!("bar"); }
and recompile, the resulting binary still has the exact same size as before. And that is what I would expect, because the bar function is not called anywhere, so it should not take space in my binary.
However, the situation changes if I add debuginfo. If I compile the main crate with two versions of dep (just foo vs foo + bar) in release mode with debug = "line-tables-only", the size of the binary jumps from 3 874 080 bytes to 3 874 232 bytes! So even if the bar function is not used anywhere, its debuginfo is still taking space in my binary :angry: This is not so bad for a single function, but if you consider the amount of code that Rust programs typically depend on, and how much of that code is actually unused, it starts to quickly add up.
This also explains why the binary size suddenly jumped to almost 4 MiB with debuginfo enabled; that is the debuginfo of the standard library being included in it. Even though a lot of it is actually unused/dead code (because my program doesn’t use these parts of the stdlib), its whole debuginfo is still included. Even though the stdlib is a bit special in this regard, because it is precompiled, the same would actually happen with a “normal” crate dependency that you build from scratch.
I’m not really an expert on linkers, but from what I understand, when linkers see debuginfo entries that are unused, they do not remove them, but merely mark them with special “tombstone” values. So they effectively become just dead weight that takes space in the binary, but don’t do anything. At least this is what MaskRay (who is probably one of the most knowledgeable persons about linkers) says. Apparently, removing these debuginfo sections in linkers could be very costly in terms of performance, due to DWARF’s tree representation. I found this forum post where someone claims that they tried to implement it in the LLD linker, and that made it seven times slower. I’m somewhat dubious that this is the best that we could achieve, but for now it seems that linkers simply do not remove unused debuginfo entries and we have to live with that. Maybe a sufficiently smart linker will be able to do it one day :)
Of course, this problem is not unique to Rust, as it also happens for other “natively compiled” languages, such as C or C++. However, for Rust I think that the issue is typically worse than for C/C++, because Rust programs tend to both have a lot of dependencies, and crucially if you compile them with debuginfo, by default you will be building debuginfo for every. single. dependency. that you have, even though most of that debuginfo will likely never be needed. I think that this is not so prevalent for C or C++, because there people often use prebuilt libraries for which they either might not have debuginfo available, or it is fetched through some external mechanism (e.g. -dbg packages from the distro package repository) and is thus not embedded directly into the final binary.
As usually, I’ll use hyperqueue as a test subject for demonstrating how we can reduce the binary size when debuginfo is included. I tried it in two configurations, the first is simply compiling with --release, while the second is compiling with --profile dist, which in HQ’s case enables more optimizations (1 CGU, thin LTO, etc.). This is the actual configuration that we use for distributing hyperqueue to end-users. Both configurations include line-tables-only debuginfo.
One approach that can be used to reduce debuginfo binary size is to perform garbage collection. That finds the unused placeholder (“tombstone”) values and gets rid of them. There are probably more ways of doing that, but I found that the llvm-dwarfutil1 tool can do it, so I tried it:
$ llvm-dwarfutil hq hq-gc
Here are the results:
| Version | release profile |
dist profile |
|---|---|---|
| Original | 70 924 912 (100%) |
50 832 304 (100%) |
| GC | 59 046 440 (83%) |
53 162 184 (104%) |
For the release profile, it resulted in a nice 17% binary improvement, not bad! However, for the more optimized (and smaller) dist profile, the resulting binary is actually larger. So it seems that sometimes it doesn’t really help.
However, when it does help, this approach doesn’t really have any disadvantages, as far as I am aware (but if you know of any, I’d be glad to learn!), so it still seems quite useful. It did print a warning about not supporting the .debug_gdb_scripts section, and some other warnings, but the resulting binary seems to work and produce correct backtraces. The garbage collection took under two seconds.
When I tried to apply the garbage collection to a hello world Rust application compiled in --release with debuginfo, it reduces the size from the original ~4 MiB down to ~2 MiB. Not bad at all! Seems like it is able to remove a lot of the unused debuginfo from the Rust standard library.
The second approach, which we actually use on our CI, and which is probably more widely known, is using debuginfo compression. This can be done using the venerable objcopy tool:
$ objcopy --compress-debug-sections=zlib hq hq-zlib
I tried both zlib compression (which took ~1s to compress) and zstd (which took ~0.1s to compress). Here are the results:
| Version | release profile |
dist profile |
|---|---|---|
| Original | 70 924 912 (100%) |
50 832 304 (100%) |
zlib |
24 775 848 (35%) |
18 751 280 (37%) |
zstd |
23 413 992 (33%) |
17 472 960 (34%) |
The results are pretty great! Going from ~50 MiB down to ~18 MiB is really impressive.
I suppose that the disadvantage of this approach is the risk that some tools that work with debuginfo (such as debuggers, profilers or symbolizers) won’t be able to understand the compressed debuginfo sections or won’t be able to decompress them. This is especially problematic for zstd, which is less supported “in the wild”. Even here, when I ran the zstd compressed binary and it panicked, the stacktrace no longer contained proper line numbers (I guess that gimli, which is included in the stdlib and AFAIK deals with decoding the backtraces, can’t deal with zstd?). That is why we use zlib2 for our distributed hyperqueue artifacts, even though it produces slightly larger binaries than zstd.
EDIT: thecakeisalie16 showed on Reddit that you can also perform debuginfo compression directly using the linker with RUSTFLAGS="-Clink-arg=-Wl,--compress-debug-sections=zlib". Didn’t know about that, very cool!
Interestingly, these approaches can be combined to produce even smaller binaries, by first running gargage collection and then compression (the other direction doesn’t really make sense). And funnily enough, compressing the garbage-collected dist binary, which is actually larger than the original binary, produces a smaller file than compressing the original dist binary!
Here are the final results with all the approaches I described above:
| Version | release profile |
dist profile |
|---|---|---|
| Original | 70 924 912 (100%) |
50 832 304 (100%) |
| GC | 59 046 440 (83%) |
53 162 184 (105%) |
zlib |
24 775 848 (35%) |
18 751 280 (37%) |
GC + zlib |
20 457 568 (29%) |
17 318 760 (34%) |
zstd |
23 413 992 (33%) |
17 472 960 (34%) |
GC + zstd |
19 777 752 (28%) |
16 492 768 (32%) |
You can find the script that I used to generate this table here, in case you wanted to perform a similar experiment on your own binaries.
On hyperqueue’s CI, we currently only compress the debuginfo using objcopy, but based on these results, we might want to also use the debuginfo GC in the future (although getting ahold of llvm-dwarfutil on CI might be a bit annoying).
A natural question to ask is whether the Rust compiler should do any of the above by default. Last year, I made --release Rust binaries smaller by default by stripping the standard library debuginfo from them when it wasn’t required. I think that was a pretty successful initiative, and it is relatively similar to what I described in this post (the compiler invokes a strip binary on the binary to remove the debuginfo, while here we invoke llvm-dwarfutil/objcopy), so it might not be too far-fetched?
That being said, I think that compressing debuginfo goes too far in what I would expect that a compiler invocation would do for me, and has the trade-offs mentioned above. Maybe GC would be a better fit (in fact I expected that the linker already does it for me), but the linkers that we currently use don’t seem to have this ability, and doing it post-compilation using e.g. llvm-dwarfutil might be unnecessarily slow? Testing this on some ludicrously large binary would probably be needed to get a better intuition about trade-offs involved in this.
Of course, if we ever wanted to do this, we would have to decide whether to do it by default (which would likely increase compile times) or whether we want to make it configurable. But in general, it probably seems too specialized to me to be in rustc. Maybe we could at least document it e.g. in the Cargo book or in min-sized-rust.
Anyway, I wanted to share this post to increase the awareness of compressing and garbage collecting debuginfo in (Rust) binaries, so that we can all ship less data to our users and avoid needless waste. Hopefully you found it useful.
I’m definitely not an expert in this area though, so if you have any suggestions/corrections/tips, I’d be glad to hear from you on Reddit.
]]>#[derive(From)] can be used in nightly now. See here on how to use it.
You can follow this tracking issue for more updates.
I use the newtype pattern in Rust a lot,
and I would like to make certain use-cases around it (even) simpler to express. A typical thing to
do with newtypes is that you implement a bunch of standard traits for them, which will delegate
to the inner field. The easiest way of achieveing that is of course with a #[derive(...)]:
#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Default)]
struct WorkerId(u32);
However, this approach cannot be currently used with all “built-in” traits from the standard library.
One of those traits is the From trait,
which is used to convert between values of two types. Even though sometimes it might not be a good idea
to implement From when using a newtype (more on that later), in other cases it’s quite useful,
as it makes it easier to go from a value of the inner field to a value of the newtype (especially
in generic contexts). A manual implementation of the trait will usually look something like this:
impl From<u32> for WorkerId {
fn from(value: u32) -> Self {
Self(value)
}
}
This code is trivial to implement, of course, and there’s not much space for bugs here, but it’s also pure boilerplate, and especially if you have a lot of newtypes in your crate, it just seems wasteful, as it’s clear that the compiler should be able to generate this implementation easily.
In fact, writing this impl over and over again usually annoys me enough that I end up writing a
macro to generate newtypes, so that I don’t have to copy this (and other similar) code all over the
place. But writing and using such macros has trade-offs (same as using third-party crates for
this, such as derive_more), and
I thought that rustc should just give me the option to #[derive(From)].
After running into this missing feature yet again, I realized that… I could just add it to Rust :laughing: Making language changes can be daunting, but this specific change seemed quite simple, mostly because the behavior of the feature seems obvious. And so I decided to try it and write an RFC. To test the waters, I first drafted a “Pre-RFC” on the IRLO forum. It received a relatively positive response, even from a member of the Language team, which made me confident enough to turn it into an actual RFC, which I did a few weeks later.
I have to say that I was quite pleasantly surprised by the positive response to my RFC. I’m used to reading big controversial RFCs that generate hundreds of comments and many arguments, but here the response was mostly “yep, this sounds like a good idea” (with a few exceptions that I will discuss later). I even received a few suggestions on how this feature could be improved in a future RFC.
The PR itself also received a lot of positive emojis!
At the time of writing of this blog post, it has over one hundred emoji reactions, which actually
makes it into the top ~60 most “liked” RFCs in the repository, even though it’s very recent. Emojis
are of course not a good way to decide which RFCs should be accepted, but it is a signal that many people seem to share the same annoyance of lacking an easy way to derive From impls.
I think that one of the strengths of this RFC is that it is arguably not even a new feature, per se.
It is one of those things that “just” fill a gap in Rust; it allows combining two features that were working
separately before (the From trait and deriving standard library traits), but couldn’t be used together.
One thing that was a bit unclear with the RFC though was determining which Rust team(s) should actually be responsible for deciding its fate. The first decision process (FCP) was started for the Language team. Then it was suggested that it should actually be handled solely by the Libs API team. And in the end we did a dual Language + Libs API FCP. This is actually a recurrent problem in the Rust Project (and I’m sure in many/most other organizational structures), where we are relatively often unsure who should actually be responsible for deciding (or doing) something, which can sometimes block progress.
Luckily, that didn’t happen here, and I can happily announce that after a few minor road bumps, the FCP has been finished, and my first ever Rust RFC was accepted :tada:.
Now that I have spoiled the outcome of the RFC, I should probably talk about the actual design of the feature. Even though it sounds simple, there are a few things to consider.
First, let’s make it clear what is the goal of #[derive(From)]. When applied on an ADT, i.e. a struct
or an enum1, it should generate an implementation of the From trait for it. Let’s start
with structs – when you apply #[derive(From)] to a struct, you should get a From impl for the
struct that creates it from a value of its field, like this:
#[derive(From)]
struct Foo(u32);
// generates:
impl From<u32> for Foo {
fn from(value: u32) -> Self {
Self(value)
}
}
If you look at the example above, an obvious problem presents itself: what to do if there is not exactly one field in the struct?
struct Foo(u32, bool);
There are three solutions that come to mind:
From impl from all the values at once, i.e. From<(u32, bool)>From impl#[derive(From)] in this caseI think that option 1) would be quite unintuitive, and it would run into an issue for the most common
use-case where you have a single field - should we generate From<FieldType>, and thus break consistency
with other field counts, or From<(FieldType,)> (i.e. generate the struct from a tuple
of size one), which is quite unergonomic and uncommon? This approach also wouldn’t really work for
structs with named fields.
I left option 2) as a future possibility. One of the suggestions that were proposed was that we could
annotate a single field with an attribute (e.g. #[from]), and then create the rest of the fields
using their Default impl:
#[derive(From)]
struct Foo(#[from] u32, bool);
impl From<u32> for Foo {
fn from(value: u32) -> Self {
Self(value, Default::default())
}
}
I like this option a lot, and I think that we should explore it in the future. A similar
approach is also being used for selecting the default enum variant using the #[default] attribute
when deriving Default, so there is a precedent for doing something like this. However, since I wanted
to keep this RFC minimal in order to improve its chances of being accepted without falling into
deep “design holes”, I didn’t include it in the RFC and left it as a “future improvement”.
And thus I chose to go with option 3), which means that you can currently only use #[derive(From)] on
structs that have exactly one field. I don’t think that it’s a big limitation though, because I expect
that in most cases this derive will be used for newtypes (which typically store exactly one field).
In general, the From trait is most commonly used for converting a single value of one type into a
single value of another type, so it makes sense to have this limitation in the initial design.
In terms of enums, the situation is more complicated. In theory, we could allow the macro to be used
on enums that have exactly one variant with exactly one field, but that doesn’t sound like a very
useful scenario. For simplicity, I thus also decided to forbid using #[derive(From)] on enums,
at least for now.
Apart from this main design question, some concerns were expressed by several people on the RFC.
One noted concern was whether #[derive(From)] struct Foo(u32); should generate From<u32> for Foo,
From<Foo> for u32, or both. To me, the answer is obvious (it should be From<u32> for Foo),
because that is how all the other derive macros work; they always generate an impl for the type on
which the derive macro is applied. I think that this behavior is the most intuitive one, and we should
keep consistency with the previous deriveable macros. This direction also makes the newtype use-case
work, and I consider that to be the most important driving use-case for this feature.
However, it is actually interesting to also consider how we could support the second direction, because
it can also be useful to go from a value of the struct back into the inner field, of course. I omitted
this functionality from my RFC, but one of the proposed solutions was to add something like #[derive(Into)],
which would generate From<StructType> for FieldType. Of course, this naming could be quite confusing,
because we would not actually generate an impl of the Into trait, because that is generated
automatically by a blanket impl in the
standard library. But that is a discussion for another RFC.
This is the usual worry with any (Rust) RFC that adds a new feature – are we sure that we got the right design for it, that it will be extensible in the future, and won’t close doors to a better solution for the same problem?
Since I kept my RFC minimal, this is a valid concern. Maybe we should first figure out how to support
the From impl in the other direction? Or flesh out the #[from] extensions? Some reviewers wanted
to see the “grand picture” of this feature before it will be stabilized (or even before the RFC would
get accepted).
I sympathize with those views, as I am myself also often worried about accepting a feature too soon, when we don’t know all the details yet. However, in this specific case, I am quite confident that the feature that I proposed will be perfectly compatible with any relevant extension, and thus we should land it as soon as possible, without waiting on the more complicated discussions about the possible extensions, to achieve incremental progress.
My confidence comes from the fact that (at least to me), the behavior of #[derive(From)]
is simply obvious, and there is no other way how it could reasonably behave.
If rustc generated anything else than the following for #[derive(From)] struct Foo(u32):
impl From<u32> for Foo {
fn from(value: u32) -> Self {
Self(value)
}
}
Then I think that the feature would be wrong. It might have some other extensions or configuration points, but this core functionality is in my opinion set in stone, and there is no other way how it could work. And thus I wanted to get my minimalistic RFC out the door, so that we can start testing the feature in the wild, and mainly so that it can start providing value to Rust users.
It should be possible to lift the limitations (such as allowing it on structs with multiple fields
and implementing the #[from] attribute), or add related functionality (such as #[derive(Into)])
to this feature in the future in a backwards-compatible way2.
From on newtypes even a good idea?One additional, a more philosophical, concern discussed on my RFC was whether implementing
From for newtypes is even a good idea in the first place, and thus whether we should teach rustc
to do it.
There are two angles to this discussion. The first one is whether newtypes should implement the
From trait or not. To answer that question, I’ll try to examine the various motivations
for using newtypes in a bit more detail.
I stress motivations here so much because newtype is a pattern, and the motivation for using a pattern is one of the things that defines it. You can have two identical pieces of code, where one upholds a certain pattern, and the other doesn’t, which can happen if those pieces of code were created because of a different motivation. That is why there are some patterns in Gang of Four that can have pretty much the same source code representation, but they still form different patterns because they have a different motivation.
The primary motivation for using a newtype, one that is upheld by every usage of a newtype, is to
create a new type in the type system (d’uh). This allows us distinguish between e.g. WorkerId and
TaskId, so that it is harder to accidentally combine values of different domain types, even if they
are backed by the same “in-memory” data type (e.g. an integer). I’ll call this the “avoid type confusion”
motivation.
Another very common motivation (which is a subset of the first motivation!) is to ensure that values
of the newtype will uphold some invariant, which is not necessarily true for all values of the type
inside the newtype. A typical example is something like struct Email(String). An Email is backed
by a String, but not all values of strings are valid e-mails! I’ll call this the “ensure invariants”
motivation.
For avoiding type confusion, it is perfectly valid to implement the From trait for the newtype, as it
gives you a standardized way of initializing the newtype from the wrapped value.
For ensuring invariants, implementing From is of course not the right thing to do! From should
always be infallible, and should be able to convert all values of a type into some value of another
type. But in the case of e.g. Email, that of course won’t work. It might make sense to implement
TryFrom in this case, but that is not something that we can easily derive anyway.
It seemed to me that people who expressed concerns about whether we should make implementing
From easier were mostly using newtypes to ensure invariants. In that case I can understand why
they think that implementing From is a bad idea! However, I personally use newtypes more often
for avoiding type confusion (my estimate is that the ratio between the two use-cases in my code is
something like 75:25 in favour of avoiding type confusion). So for most of my newtype use-cases,
implementing From is genuinely useful (which is why I wanted to add this feature to Rust in the
first place :) ).
The second angle to this discussion is more general and unrelated to newtypes.
I personally think that Rust should not dictate what is and what isn’t “good program design”,
of course as long as makes sure that it is sound (in the sense of memory/data race/etc. safety).
Obviously, Rust already makes some strong decisions about program design, for example it does not
allow you to use inheritance, and the borrow checker also makes
expressing certain ownership patterns (like the famous “soup of pointers”) annoying. But in this specific case,
you can already implement From by hand, so forbidding users from deriving this trait, even
though it is technically possible, just to make sure that they don’t do something that might be considered
bad practice, would seem to be a bit silly to me.
Normally, after an RFC is accepted, we have to wait until someone actually implements it (which is
not a given, we shouldn’t forget that!). But luckily, since I’m a compiler contributor, I was able to help myself here :sweat_smile: And after some trial and error, I
managed to implement the feature. It was a pretty
fun exercise, because the From trait is different enough from all the previous traits that
were deriveable, so I also had to make some small changes to the derive infrastructure to make it work.
It was also a chance to work with Nick Nethercote again, which is always awesome.
He even made some improvements to the feature
after my initial PR was merged. Thanks Nick!
While implementing this feature, I understood how the derive macros are actually wired between
the compiler and the standard library, which is actually pretty interesting. The compiler defines
the traits that are deriveable, along with the actual logic to generate the derived code. However,
when I added a new From macro derive implementation to this list, #[derive(From)] still didn’t work.
After digging and tracing of the compiler’s behavior on small Rust programs, and asking the
Foremost Rust Name Resolution Expert™ for help, I finally realized
that the answer is name resolution. When the compiler sees something like #[derive(Trait)], it uses
name resolution to understand what Trait is. One option is that the trait is resolved to a
(derive) proc macro, in which case rustc will call it3 to generate the expanded code.
However, another option is that it will resolve the trait to a macro in the standard library marked
with the #[rustc_builtin_macro] attribute.
And that is the piece of magic that ties the name From to the code that actually generates the right
derive logic. So after I added this:
#[rustc_builtin_macro]
#[unstable(feature = "derive_from", issue = "144889")]
pub macro From($item: item) {
/* compiler built-in */
}
to the standard library, #[derive(From)] finally started working :tada:
However, name resolution (in Rust, but also in general) is a VERY gnarly beast, and my joy from defeating it did not last long. The very next day after my PR was merged, someone created an issue that my PR broke their code in nightly because of name resolution errors. Exactly the thing you want to see when you wake up :laughing:
The issue was (as usually) with glob imports. To make #[derive(From)] work “out of the box”, I added
the From macro to the standard library prelude, so that it can be implicitly resolved, and users
thus wouldn’t have to import anything to use the new derive. This is what makes e.g. #[derive(Hash)]
work even if you don’t have the std::hash::Hash trait imported.
However, adding new things to the prelude is always tricky. Consider this code:
mod foo {
pub use derive_more::From;
}
use foo::*;
#[derive(From)] // ERROR: `From` is ambiguous
struct S(u32);
The code uses a glob import to import a macro named From from the foo module. After my PR landed,
this module also received another macro named From (the one from the standard library), which was
imported with a glob import from the std prelude, which caused an ambiguity. In other words, my change
broke existing (stable) code, even without people using the new feature. That is obviously Very Bad.
To fix this, I had to remove the From macro
from the standard library prelude, and thus for now, it will have to be imported explicitly. I added
the macro to the std::from module; it had to be added to a new module, because no previously
stable Rust code could have had a previous import from a module that did not exist, and thus this
should not break anyone. It is a bit confusing though that the From macro is in std::from, while
the From trait is in std::convert.
To be honest, I don’t know if there is a way to resolve the import ambiguity without an edition bump, so maybe this feature will have to wait until the next edition before it can be used ergonomically. Maybe #139493 might help with issues like this in the future.
It is interesting to note that if the code looked like this instead:
pub use derive_more::From;
#[derive(From)] // Uses `From` from `derive_more`
struct S(u32);
Then there would be no ambiguity, because the explicit import from derive_more would be prioritized
before the From glob import from the std prelude. But having two macros imported both with a
glob import (with one coming from the prelude) causes a problem.
To experiment with this feature, update to a recent nightly release of the compiler, and use #[derive(From)]
on a struct of your choice. Note that it will only work on structs with
exactly a single field, as mentioned before! You will also have to import the macro explicitly,
because of the ambiguity problem described above.
#![feature(derive_from)]
use std::from::From;
#[derive(From)]
struct Foo(u32);
fn main() {
let f: Foo = 1u32.into();
}
You can try the feature and let me know if it works fine for you! Note that I don’t expect that many things could break here, provided that there won’t be any further name resolution errors…
As I already discussed above, there are various improvements and extensions that we could do with
#[derive(From)]. But even more generally, I think that we should make it possible to derive
as many standard library traits as possible. It should be easily possible to generate impls of
AsRef, Deref or even traits like Iterator, as long as you do it on structs with a single field.
If/once #[derive(From)] becomes stabilized, I might try to write RFCs to also support these other
traits, to reduce even more boilerplate.
I hope you enjoyed my adventures with writing an RFC and designing and implementing a Rust feature!
You can let me know what you think about #[derive(From)], and which other traits would you like to
#[derive], on Reddit.
In HyperQueue, we have a struct describing parameters of a queue (for the purpose of this post, it doesn’t really matter what a queue is), which contains a bunch of fields. Something like this:
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct QueueParameters {
pub manager: ManagerType,
pub max_workers_per_alloc: u32,
pub backlog: u32,
pub timelimit: Duration,
pub name: Option<String>,
pub max_worker_count: Option<u32>,
pub min_utilization: Option<f32>,
pub additional_args: Vec<String>,
// ... more fields
}
This struct is just (Plain Old) data, a bag of fields that can be (de)serialized, sent over the network, etc. However, at some point in the program, I want to store these parameters in a way that would prevent other parts of the codebase from modifying them. Of course, I could do that easily by making its fields private, but that would prevent me from using the struct literal syntax for creating values of this struct outside the module (and its children) that defines it.
In short, I want to be able to both:
QueueParameters { manager: ..., backlog: ... })
anywhere in the codebase, so that I can see which fields are initialized with what values (as we
don’t have named parameters in Rust).It would be nice to have something like “init-only” fields, but as far as I know, Rust doesn’t really have anything like that today. So I had to use some workaround. A common approach to sort of emulate named parameters in Rust is to use the builder pattern, but that seemed like overkill to me in this use-case, and its syntax would be more verbose than the normal struct literal syntax anyway. I also didn’t want to complicate the matter with any kind of macro magic.
My original “solution” to avoid the field mutation was to introduce a separate struct that has the
same fields as QueueParameters (let’s call it ReadOnlyQueueParameters),
just with the fields being private, so that they cannot be modified from the outside after initialization.
Because the fields were private, I also had to create a constructor function ReadOnlyQueueParameters::new,
which received values for all the fields. Of course, this meant that I couldn’t use the nice struct
literal syntax anymore, which made the initialization quite awkward, as there are a lot of fields:
let params = ReadOnlyQueueParameters::new(
ManagerType::Slurm,
1,
4,
Duration::from_secs(3600),
None,
Some(10),
None,
...
);
This code was bothering me for a long time, because everytime I had to add a new field to the
struct, I would have to extend this new function with yet another parameter, and the call-sites
would become ever more confusing. This is how such a change would typically look as a diff:
- None,
- None
- )
+ None,
+ None,
+ None,
+ )
Yeah, not great.
After a few years of working with the annoying read-only wrapper, I finally lost my patience with it and decided to refactor it. I quickly realized that the behavior I want can be achieved in a very straightforward and elegant way by not duplicating the fields, but simply storing the mutable struct directly as a field inside a read-only struct. The read-only struct would then expose the fields of the mutable struct in a read-only way:
struct ReadOnlyQueueParameters(QueueParameters);
impl ReadOnlyQueueParameters {
pub fn new(params: QueueParameters) -> Self {
Self(params)
}
pub fn backlog(&self) -> u32 {
self.0.backlog
}
}
With this approach, I can construct the parameters using the struct literal syntax:
let queue = ReadOnlyQueueParameters::new(QueueParameters {
manager: ...,
backlog: ...,
...
});
But at the same time, I cannot modify the fields from the outside after construction anymore.
Truly a “we have init-only fields at home” solution. I could even remove the accessor methods by
implementing the Deref trait for the wrapper, but here I decided to keep the code more explicit.
This is, of course, yet another instance of the newtype pattern, which I use all the time, and even explain it to my students. And the solution is really quite trivial in hindsight. So it was a bit embarrassing to me that it took me so long to realize that it can be used to solve this annoyance. Although frankly, the main issue was to find the motivation to spend five minutes to refactor code that was bothering me for years :)
For these simple newtype wrappers, it would be nice to be able to simplify the implementation of the
Fromtrait, to go from the inner field to a value of the newtype. More about that soon.
Anyway, that’s all. The newtype pattern just keeps on giving! Do you have other useful use-cases for it? Let me know on Reddit.
]]>Recently, in addition to the usual compile times complaints, I also started noticing the following sentiments being expressed by frustrated Rust developers: “Why doesn’t the Rust Project care more about this pressing and known issue? Why don’t they do something about it?”. As a member of the Rust compiler performance working group, I take these questions very seriously, and I of course have some opinions on the topic. In this blog post, I would like to offer some thoughts that might serve as answers to these (and other similar) questions.
Disclaimer: all opinions expressed in this post are mine only and do not necessarily represent the broader views of the Rust Project (the diverse set of contributors and maintainers that contribute to the Rust toolchain).
First, let me assure you - yes, we (as in, the Rust Project) absolutely do care about the performance of our beloved compiler, and we put in a lot of effort to improve it. We triage performance improvements and regressions each week. We run a comprehensive benchmark suite after every merged PR. We enthusiatically welcome any performance improvements (unless they have complicated trade-offs, more on that later), and try to quickly revert (or fix) PRs that introduce performance regressions. Very smart people are continously working on finding bottlenecks and speeding up the compiler. And some significant improvements to make compilation faster by default are currently in the works.
And all this effort shows results; Rust build performance has improved quite a lot over the past few years! When we talk about the long-term trends, we usually show this dashboard, but I find it a bit dull, because it averages results of several benchmarks together and the benchmarks are quite short, so they start running into diminishing returns. Instead, I did a little experiment to show how did the build performance evolve on my favourite test subject, hyperqueue. I took its first commit (from March 20211) and compiled it with a few different Rust compiler versions, with ~1 year increments between them2. I did this on a relatively modest laptop with 8 AMD cores running Linux. Here are the results:
rustc version |
Clean build [s] | Incremental rebuild [s] | Speedup (clean build) |
|---|---|---|---|
1.61.0 (May 2022) |
26.1s | ~0.39 | - |
1.70.0 (June 2023) |
20.2s | ~0.37 | 1.29x |
1.78.0 (May 2024) |
17.0s | ~0.30 | 1.53x |
1.87.0 (May 2025) |
14.7s | ~0.26 | 1.77x |
On this benchmark, the compiler is almost twice as fast than it was three years ago. Of course, this is just a single data point, in different cases the speedup will be larger or smaller, and it will likely be smaller in general on other platforms than x64 Linux, because we spend the most effort optimizing that specific target. But the fact remains that the performance of the compiler is continuously improving.
Now, all that is nice, but it doesn’t change the fact that for many Rust developers, compilation performance is still a big bottleneck, and they could be much more productive if the feedback loop was an order of magnitude (or more) faster. To be fair, this will depend on who you ask, e.g. some C++ developers don’t mind Rust’s compilation times at all, as they are used to the same (or worse) build times, while Python programmers likely aren’t very impressed by Rust’s feedback loop speed. But it’s definitely not a stretch to say that for many users, Rust compilation just isn’t fast enough today, and that is a problem.
Before we examine that problem in more detail, I think that it’s interesting to think about whether the problem is even fundamentally solvable, and what is the end goal that we would eventually like to achieve. Can Rust ever hope to achieve near-instant (re)build speeds, with its complex type system, borrow checking, monomorphization, proc macros and build scripts, large translation units, machine code generation and a “rebuild the world from source” compilation model? When it has a historical track record of favoring runtime (rather than compilation) performance at nearly every opportunity?
I think that it depends on the use-case. If we’re talking about building projects with hundreds of dependencies from scratch or doing super-duper LTO-optimized release builds, I think that we won’t ever get to a point where the build will be truly instant. On the other hand, I personally believe (although others might disagree!) that it is not fundamentally impossible to have a Rust compiler that will be able to almost instantly rebuild a Rust project of nearly any size in incremental (and at least moderately optimized) mode after making a small change to the source code, one where rebuilds are truly O(number of performed changes) and not O(size of the codebase). We might have to make some trade-offs, particularly around the resulting runtime performance, but I think that we can get there. In fact, we can even see examples that can already achieve something akin to this (in somewhat limited scenarios for now)
today!
And it’s not like we don’t have some ideas on how to get (at least closer) to this goal. There are several “northstar” approaches for speeding up the compilation process in various ways, like the parallel frontend, alternative codegen backends, defaulting to a faster linker, deferring code generation (MIR-only rlibs, -Zhint-mostly-unused), avoiding useless workspace rebuilds, having smarter incremental compilation (including incremental linking or even binary hot-patching), and many others.
Some of these approaches are not prepared for wider usage yet, but others are, and you can opt into them today. They usually (but not always) help. I believe that if we had a magic wand and we could land all these changes today, we could get very snappy incremental rebuilds for a large (certainly much larger than today) fraction of Rust projects.
And it’s not like “only” the broader Rust community would benefit from this. Improving build performance makes Rust toolchain contributors themselves much more productive. It reduces the time required to build the compiler itself when making changes to it, it speeds up our CI workflows, reduces the time that we have to wait for tests and compiler performance benchmarks… Everyone would benefit! So what’s blocking us from making progress faster? WHy dON’t We dO SOmEtHIng ABoUt iT??!3
This is a tough question, one that doesn’t have a simple answer. I will try to provide a variety of reasons (in mostly arbitrary order) why I think we have not been making progress faster, based on my personal experiences working within the Rust Project.
First, let’s start with the technical reasons, which are in a way the most straightforward. Simply put, making non-trivial improvements to the performance of the Rust compiler is hard! It is a large codebase4 that has a lot of technical debt risk, as probably all large (compiler) codebases do, and although we provide a guide for contributing to it, it still takes time before you can understand what’s going on. In fact, there is probably not a single person that understands the whole compiler codebase (maybe except for compiler-errors :laughing:).
While it is possible to make performance improvements even without deeply understanding how the compiler works, by profiling and micro-optimizing its various parts, this approach can only get you so far, and most of the low-hanging fruit has already been picked5. A lot of things have been optimized and overfitted to reach a “local performance minimum” on a variety of benchmarks and use-cases, so even if an improvement is made for some use-cases, it often regresses something else, which can lead to that localized improvement not being accepted.
We also have to consider various trade-offs when deciding whether to land certain performance optimizations. For example, we know a few tricks that could make the compiler a bit faster on x64 Linux, the most popular target. We could compile rustc with support for “newer” instruction architectures, e.g. AVX256, which seems to provide some wins. However, that would mean that the compiler would no longer work on older x64 CPUs that we still officially support! Or we could use a different memory allocator (e.g. mimalloc), which also seems to provide some modest performance wins, but at the cost of increased
memory usage, which could
cause rustc to go OOM on devices with smaller amounts of RAM. Maybe you’re thinking why can’t we just ship multiple versions of the compiler (for the same target) with different optimizations and then let people (or rustup) choose which one to use? Well, building the most optimized variant of the compiler for Linux already takes an enormous amount of CI resources right now, not to mention that distributing the toolchain in multiple variants could balloon the amount of data we ship to users over the network, which is already gigantic. There are simply trade-offs everywhere.
I think that to achieve truly significant build performance wins, there are mostly two ways forward. The first one, which I find highly promising, is improving certain compilation workflows. We don’t necessarily have to make the whole compiler faster if we can vastly improve the performance (or reduce the amount of work required) of specific workflows that currently limit the productivity of a large fraction of Rust developers. An example of this is the Relink, don’t rebuild proposal, which has the potential of reducing the amount of compilation that has to be performed when you modify a crate in a large Cargo workspace. Approaches such as this one do not necessarily make the compiler faster per se, rather they make the compilation process smarter, which is an area in which rustc could be improved a lot. Many of these potential workflow improvements are also not strictly tied only to the compiler itself, but they happen at
the intersection of rustc and the used build system (most often Cargo).
I hope that some of these ideas will make the most common Rust compilation workflows much faster in the short-to-medium term, without requiring massive changes to the implementation of the compiler. Nevertheless, even such targeted optimizations are not easy to land. Achieving an impressive performance win with a proof-of-concept is the “easy” part. But then comes the long tail of the really hard work - making sure that your change supports all the edge-cases and quirks of the Rust compilation process, works on all targets, doesn’t regress important use-cases and benchmarks, is not a maintenance nightmare, keeps backwards compatibility, etc.
The second way forward, as you might have guessed, is… performing massive changes and/or refactorings to the implementation of the compiler. However, that is of course challenging, for a number of reasons. First, you need to do a “vibe check”6 with the rest of the compiler team with a Major Change Proposal, to make sure that other compiler maintainers are on board with you performing such a major change. Then you will obviously need to implement these changes, which can take a lot of effort. For example, if you change one thing at the “bottom” layer of the compiler, you will then have to go through hundreds of places and fix them up, and also potentially fix many test cases, which can be very time-consuming7. You will also need to find reviewer(s) with bandwidth to approve your changes, which can span over many PRs and weeks, months or even years of work.
Performing large cross-cutting changes is also tricky because it will necessarily conflict with a lot of other changes being done to the compiler in the meantime. You can try to perform the modifications outside the main compiler tree, but that is almost doomed to fail, given how fast the compiler changes8. Alternatively, you try to land the changes in a single massive PR, which might lead to endless rebases (and might make it harder to find a reviewer). Or, you will need to do the migration incrementally, which might require maintaining two separate implementations of the same thing for a long time, which can be exhausting.
An example of a large internal change (albeit one that is not directly related to performance) is the reimplementation of the compiler’s trait solver. Even though it is being worked on by very smart and incredibly skilled people, this is a process that takes several years to finish. That’s the scale that we’re dealing with if we would like to perform some massive refactoring of the compiler internals, and it should set the expectations for people that ask us when are we finally going to rewrite the whole compiler using Data-oriented Design to make it a bazillion times faster.
Speaking of DoD, an additional thing to consider is the maintainability of the compiler codebase. Imagine that we swung our magic wand again, and rewrote everything over the night using DoD, SIMD vectorization, hand-rolled assembly, etc. It would (possibly) be way faster, yay! However, we do not only care about immediate performance, but also about our ability to make long-term improvements to it. If we overfitted the performance using code that is hard to evolve, understand, debug or maintain, it wouldn’t serve us well in the long term. The compiler receives contributions from hundreds of volunteers, and we want to keep the codebase (at least somewhat) approachable to them. So the long-term maintaintability of the codebase is also something that we need to keep in mind.
While the challenges presented above are very real, they are of course not exactly unique to rustc. The fact that making major changes to a large (compiler) codebase is difficult is probably not very surprising. I’m sure that GCC/LLVM/Clang maintainers also have a lot of ideas on how to improve or speed up things, but it takes a lot of effort and time to get these done. But there are also other reasons why compiler performance is not the only thing we focus on.
Even though many Rust users seem to want compiler performance to be the primary priority of the Rust Project, we should not forget that there are also other priorities that we have to keep in mind.
The Rust compiler is (at least in my opinion) very stable and dependable, and I can count on a new version being released every six weeks. That is no small feat! And we should not take it for granted. It does not happen simply by starting from a working state and then not doing anything. There is so much work that goes into simply making sure that things are not on fire, our infrastructure and CI works, really bad bugs are being fixed in a timely manner, incoming issues are being triaged, backwards compatibility is still being upheld, PRs are being reviewed, security issues are being quickly resolved, we stay on top of changes in external dependencies, such as LLVM, and much, much more. And we have to ensure all of that works on all of our supported targets! At this moment, that’s 8 Tier 1 targets (those have to build and all tests have to pass) and 91 (!) Tier 2 targets (those have to at least build). By a large part, that work is being done by volunteers who care so much about Rust and its toolchain (:heart:) and want to keep it working as well as it was so far. There is simply a ton of work to do on Rust9.
And all that work is of course eating away from time that could be spent e.g. on making the compiler faster. If you do not consider that work to be all that important, ask yourself if you would enjoy a twice faster compiler if it randomly started miscompiling your code :)
Another thing is that both the language and the compiler are getting new features all the time. New compiler flags to support projects like Rust for Linux, syntactical improvements, complex language features, and so many other things. In general, the language seems to be far from “done”, and so many things are in process; there are almost 200 open RFCs pull requests, and many other RFCs were accepted, but haven’t been implemented or stabilized yet. At any given point, we have ~700 opened PRs on the main Rust repository, we have over 10 thousand open issues, there’s just so much going on.
And it seems that Rust users want Rust to evolve as fast as it does today, or even faster, which means more changes! When was the last time you “just wanted this small feature X to be finally stabilized” so that you could make your code nicer? It is perfectly fine to want new features from a language, but we should not forget that (amongst other things, such as design discussions) it requires a lot of experimentation, refactoring, bug fixes, testing and implementation effort in the compiler, which again takes time from optimizing performance. Not to mention that adding new features usually makes compiler performance slightly worse, as it tend to do more work, rather than the opposite.
In the end, the amount of effort that gets spent on optimizing the performance of the compiler ultimately depends on the people who contribute to it and maintain it. They are often volunteers, and they have diverse interests. Some (compiler) contributors are not interested in doing performance optimizations at all, and that is perfectly fine! Remember, Rust is not a company (this is a great blog post by Mara Bos, I recommend you to go read it!). We don’t tell people what they should work on, nor do we assign them tasks or tickets. If there won’t be people interested in making the compiler faster, it mostly likely will not get done, it’s as simple as that.
That being said, there are certain ways of how we can empower people to work on compiler performance. The Rust Project now has the Project Goals program, which specifies a set of goals and projects that we consider to be important, and we explicitly welcome contributions (and promise to provide reviews and support) in these areas. I’m going to try to make compiler performance be one of the flagship goals in the near future, as I think that it really deserves to be one. That could perhaps help motivate more people to work on it.
Even though it may sound a bit banal, making progress faster is also (at least partially) a matter of funding. Sometimes I hear people complaining about certain projects designed to improve compilation performance taking too long to be ready. It sounds like they expect that there is a fully staffed team of well-paid engineers working on these projects. Well, we should not forget that the Rust Project is still mostly built on volunteers, and many people contribute to it in their free time. Many improvements (including some important compiler performance projects) to Rust were literally driven by a single university student who works tirelessly to make Rust better day by day, without maybe even getting paid for it in any way, and the reason why progress pauses can be (for example) that they must finish university exams or do some actual paid work so that they could afford rent and food.
During the recent RustWeek conference and the associated All Hands event, I saw an increased interest from various companies to invest into improving the performance of the Rust compiler, which is great! Improving the performance of rustc requires long-term concentrated effort, and providing stable funding for contributors doing that work is definitely one of the ways of how we could make progress faster. However, it is also important that making improvements is not just about doing the implementation work; someone will also need to review it, and then maintain it in the future. In an ideal world, companies would fund long-term ongoing work on the maintenance of the compiler, rather than paying their employees to implement an optimization that scratches their itch and then disappearing. Sometimes, the best way to help something move forward in Rust is not to implement it, but gather knowledge, become an expert in some compiler area, and help review the code of others, so that you can unblock other people to do the actual work :) But that of course takes a long-term investment (both in terms of time and money).
Based on my experience, the interest amongst Rust compiler contributors to work on compiler performance does not correspond to the amount of discourse this topic gets online (and again, this is fine!). If I used the magic wand again, and could persuade more contributors to care about it, would I do it? Probably yes. Would it be a good thing for Rust, long-term? I don’t know! So far, the development of the language and compiler seems to be going fine, and who am I to determine priorities. I trust the people in the Project to collectively do what’s best for Rust, both today and in the future.
One thing that I find interesting is that maybe more people would actually do performance work if they didn’t feel the need to work on so many other things at the same time. Case in point: when I started contributing to Rust back in 2021, my primary interest was compiler performance. So I started doing some optimization work. Then I noticed that the compiler benchmark suite could use some maintenance, so I started working on that. Then I noticed that we don’t compile the compiler itself with as many optimizations as we could, so I started working on adding support for LTO/PGO/BOLT, which further led to improving our CI infrastructure. Then I noticed that we wait quite a long time for our CI workflows, and started optimizing them. Then I started running the Rust Annual Survey, then our GSoC program, then improving our bots, then… Many of these contributions did (at least I hope) affect the performance of compiler in some indirect way, either by unblocking or supporting other contributors or improving our infrastructure (in various meanings of that word). However, only now, after a few years of getting sidetracked, I’m finally trying to get back to doing direct compiler performance work again.
In other words, the more help we can get with maintaining Rust (regardless in which specific area!), the better the chance that we will collectively have more time to improve the performance of the compiler :)
Funnily enough, after I wrote this post, I realized that I could probably replace the term “compiler performance” with pretty much anything else that people want from Rust (“why doesn’t Rust do X”), and a lot of the same reasoning would still apply. In this regard, compiler performance is not that special, it’s just one of (so) many (very important!) things that we care about and that we try to improve to the best of our ability.
In terms of future compiler performance work, I’m hopeful about several initiatives that were discussed at the All Hands events, and also about making LLD the default linker on Linux, which will hopefully finally land in the upcoming months. I’m also planning to run a compiler performance survey soon, so that we can find out what workflows cause bottlenecks for Rust users, and I also want to build better infrastructure for profiling what the compiler is actually doing during the compilation process.
I hope that this blog post provided a bit of insight into why making progress in build performance is not so easy. If you’d like to join the fray and help with it, we would be happy to see you! Let me know what you think on Reddit.
I used an old commit so that it can be compiled even with old(er) versions of rustc. ↩
Luckily, I still had all the stable compiler versions downloaded from my recent compiler errors experiment :sweat_smile: ↩
Sorry, couldn’t resist. ↩
At least for some definition of large. The compiler consists of ~600k lines of Rust code, and the standard library has a similar size. ↩
I recently saw someone call this vibeck (named after the abbreviated terms typeck and borrowck used in the compiler), which I find super cute. ↩
If you are thinking “why can’t AI do this?”, it probably could help with a lot of things in this area, but especially differences in compiler diagnostics (which is a large part of what our test suite checks) often come down to judgement calls that we still (luckily) leave to humans. ↩
I like to say that if you gave me a hundred full-time engineers, I would find all of them something to work on in the Rust toolchain immediately. ↩
target directory is a commonly cited annoyance with Rust (and Cargo)
– in the last year’s Annual Survey,
it was the third most pressing issue of Rust users, right after slow compilation1 and subpar
debugging experience. Given the “build everything from source” compilation model of Rust, and both debuginfo and incremental compilation being enabled by default in the dev Cargo profile, it is unlikely that the target directory will ever become lean and small. However, there are still ways of how we could reduce the target directory size by a non-trivial amount. I will describe a brand-new method of achieving that in this blog post.
Note that there are also initiatives to reduce the size of the target directory along the temporal axis, i.e. prevent it from ballooning over time (see Cargo Garbage Collection). This post is more about how to reduce the size of the target directory overall.
It is not the focus of this post to dive deep into exploring what exactly takes space in the target
directory, but I still think that it is useful to provide at least some background on this. I took
my favourite work project, hyperqueue, and compiled
it in several modes to compare the resulting target directory sizes2, using rustc 1.89.0-nightly (2eef47813 2025-05-22):
| Optimizations | Incremental | Debuginfo | target size [MiB] |
|---|---|---|---|
No (dev) |
No | No | 462 |
No (dev) |
Yes | No | 677 |
No (dev) |
No | Yes | 870 |
No (dev) |
Yes | Yes | 1316 |
Yes (release) |
No | No | 396 |
From the results, it’s clear that both debuginfo and incremental compilation caches take a lot of
space on disk. It might be that we could do something to reduce either of these, but there is one
additional thing that (kind of unnecessarily) causes target directory bloat, which is not
easily observable in this table, and that is the metadata of the compiled Rust crates.
First, we need to briefly talk about how Rust compilation works (in very, very simplified terms).
When a Rust library crate (rlib) is compiled in a “standard” way (i.e. no LTO or other funny
business) with Cargo, the compiler generates (amongst other things) two main outputs:
The interesting thing about this is that the final object code isn’t actually needed in order to start
compiling Rust code that depends on a said library, metadata is enough (the object code will only be
needed at the final linking step). Cargo makes use of this fact, and when it compiles your crate graph,
it uses a technique called pipelining.
Imagine that you have a binary that depends on crate B, which itself depends on crate A. When
compiling A, Cargo will tell the compiler (using --emit=metadata,...) to emit a .rmeta file
containing the metadata of A as soon as possible. Once that file is ready, we can start compiling
B by passing it the .rmeta file of A, even though the object code of A is not ready yet,
and thus partially overlap the compilation of both crates, which improves compile times. I will borrow
(and slightly extend) a cute ASCII diagram from Alex Crichton’s internals post about pipelining
to succintly show how it works:
--- .rmeta of A generated
v
[-libA----|--------]
[-libB----|--------]
[-binary-----------]
0s 5s 10s 15s 25s
Pipelining has been enabled in Cargo for many years, so it’s nothing new, and if you use Cargo, you are using pipelining every day, perhaps without even knowing about it. However, there is one suboptimal thing in regard to disk space usage with the pipelining approach.
The problem is that once the process finishes, you will be left with two copies of the metadata of
each library on disk. The first copy is in the final .rlib file (along with the object code), and
the second copy is in the .rmeta file. As is usual with Cargo, this problem will balloon if you build
your project with multiple configuration options (different Cargo profiles, different compiler/linker
flags, etc.).
A related issue is that if you are compiling a library as a dylib,
so that you generate a Rust dynamic library (e.g. .so on Linux), that will also contain the
metadata, even though it should not be required to actually consume (call functions from) the
library at runtime. This can be annoying for people who ship Rust dylibs, even though that’s
usually not very common due to Rust currently not having a stable ABI.
So, what can we do about it? Well, last year, I noticed that many years ago, the illustrious @bjorn3 has
proposed an idea to introduce a compiler flag
that would cause the metadata to only be included in the .rmeta files, and no longer be present
in the .rlib/.so library files. I liked this idea, and since there already was a prototype
implementation prepared, it didn’t seem that hard to push it over the finish line. So I
nerd-sniped myself into finishing the implementation of the flag under bjorn3’s mentorship.
I didn’t actually make a lot of progress on it last year (because of… stuff),
but this year I finally filed an MCP, which is the usual process
for introducing new compiler flags, refactored the implementation a bit, added tests, and got it
merged. So long story short, you can now use an
unstable compiler flag in nightly Rust called -Zembed-metadata=no, which avoids the metadata
duplication.
When the flag is used, the .rlib file will only store a “metadata stub”, which contains
the bare minimum of information necessary for the .rlib to be loaded and validated, and the rest
of the metadata will be stored in a .rmeta file.
Normally, when a new unstable compiler flag is added, people can experiment with it using RUSTFLAGS.
However, the way this flag works means that you kind of need to combine it with --emit=metadata,
otherwise there will be no metadata generated at all, which would not be good. Furthermore, you
will now also need to pass the paths to the .rmeta files to the final rustc invocation that links
the top-level binary/library, otherwise the metadata will not be found. This essentially means that
this feature also needs some changes within Cargo, otherwise it wouldn’t be really usable with it.
So I went and implemented support for this rustc flag in Cargo, and exposed it via a new
unstable Cargo flag called -Zno-embed-metadata.
Here are results of building hyperqueue using -Zno-embed-metadata3:
| Optimizations | Incremental | Debuginfo | Before [MiB] | After [MiB] | Reduction |
|---|---|---|---|---|---|
No (dev) |
No | No | 462 | 336 | -27.3% |
No (dev) |
Yes | No | 683 | 558 | -18.3% |
No (dev) |
No | Yes | 874 | 748 | -14.4% |
No (dev) |
Yes | Yes | 1325 | 1199 | -9.5% |
Yes (release) |
No | No | 397 | 253 | -36.3% |
As you can see, the benefits are most notable when building in release mode, or in general without
debuginfo or incremental compilation, as their size will usually dwarf the duplicated metadata contents.
Nevertheless, the disk space savings are quite nice, in my opinion.
Originally, I was also hoping that this could speed up compile times a little bit, as less data has to be written to disk, but from my experiments it seems that the effect is rather miniscule, at least on a Linux system with an SSD disk.
One of the reasons why I wanted this flag to exist was to slightly reduce the size of the distributed
Rust compiler toolchain, particularly the standard library .so file. I didn’t get a chance to do
a full benchmark yet, as it’s a bit annoying to dogfood a Cargo feature, because I have to wait
until the Cargo change reaches the beta channel, so that I could use it for compiling the compiler
itself (the first stage of rustc is compiled using the beta compiler/Cargo), which will take
several weeks.
However, I did some initial local tests, and it seems that using this flag reduces the size of the x86_64-unknown-linux-gnu
standard library .so file from ~13 MiB to ~3 MiB, which seems nice.
I think that unless we find some major issues, we should make the -Zno-embed-metadata behavior
the default in Cargo, to reduce the disk space usage of the target directory for everyone.
Currently, it seems like it might be considered to be a backwards compatibility break though, as the Cargo
team is unsure if some people weren’t relying on the metadata being present in the .rlib
files. In general, it’s quite tricky to determine whether something is a breaking change in Cargo or
not, if it hasn’t been previously documented (Hyrum’s Law is ever-present).
In terms of how to technically perform the migration, I think that we could use the new behavior by default
on the nightly toolchain for some time4 to find potential issues in the wild,
and flip the Cargo flag to allow opting out of (instead of opting in) the new behavior
(so it would essentially become -Zembed-metadata).
If you are interested in the future of this feature, you can observe its status in the rustc and Cargo tracking issues.
I would be interested in how this flag fares in real-world scenarios, whether it causes any issues,
and how much disk space it can save. If you would like to try it out, use a recent rustc and Cargo
and build your project using e.g. cargo +nightly build -Zno-embed-metadata. You can use e.g.
this script as an inspiration for
benchmarking the effect of the flag.
If you try it out, please let me know about your results on Reddit!
Funnily enough, making compilation faster can sometimes increase disk usage; for example, the incremental system of the Rust compiler stores a lot of data on disk, which helps it to make subsequent recompilations faster. ↩
I compiled it with --no-default-features to avoid compiling the C jemalloc dependency, as that takes a lot of space on disk. ↩
The full command was cargo +nightly build --no-default-features -Zno-embed-metadata. ↩