First, some statistics. The GitHub GraphQL API claims that:
1497 PRs in 2025 (+98% from 755 opened PRs in 2024).
1160 (77.49%) out of those were relevant to the rust-lang organization or other upstream Rust work1. I only consider these in this post.50 different upstream-Rust-related repositories in the past year.976 PRs in 2025 (+131% from 421 reviewed PRs in 2024).
753 (77.15%) out of those were relevant to upstream Rust2.I used this script to calculate these statistics.
When I saw these numbers, I was quite surprised. It felt like too much, and I had to go double-check the script I used to compute it. But after examining the list of my opened PRs, and thinking a bit more about my Rust contributions, it started making sense. The high number of PRs that I open(ed) stems from the fact that the kind of work I usually do in Rust is quite maintenance-heavy3. As a member of the Rust Infrastructure team, I do many small and drive-by contributions to various repositories to fix CI, update configs or docs, set up some tooling, etc. I definitely couldn’t open a thousand PRs in a year if they were all implementations of new compiler/languages features or something like that4.
A lot of that work can be quite unglamorous, like doing house chores. There are probably less than 50 pull requests from the past year that I could show to someone and say “here, I did something cool”. The rest is mostly work that “had to be done”. And there’s a lot of that work! Even though 1160 PRs might sound like a lot, it is just a drop in the ocean in the
upstream Rust world. In 2025, just the rust-lang/rust repository
received 10483 (!) pull requests5. I personally opened only 307 out of those PRs. That shows just
how much work goes into maintaining Rust (and note that this is just a single repository, although the most important and extensive one that we have). I try to help this effort by often (but not always) prioritizing unblocking other Rust Project contributors, rather than pushing my own work forward, as that tends to have a larger effect down the line.
It should also be noted that opening pull requests is just a fraction of the work that a typical open-source contributor/maintainer does. There are certainly many other activities that I do, such as code reviews, writing and reading design docs, discussing various ideas with other people on Zulip and GitHub, reading and writing blog posts, triaging issues, governing and making decisions, listening to podcasts to find inspiration, attending and talking at conferences, watching recordings of conference talks that I didn’t see live or thinking about compiler optimizations while walking my dog :) Open-source is about communication as much as it is about writing code (and even that is a form of communication). So quantifying work based on just the number of opened pull requests does not paint the whole picture.
It can be quite hard to demonstrate the value of work like this, especially to companies. That is something that we are trying to improve with the Rust Foundation Maintainer Fund (and other similar efforts). Maintaining Rust so that it keeps working as well today as it did yesterday is a lot of work. And moving it forward by adding new features is even harder6. That is why I think that it is crucial to support contributors that maintain and develop Rust. Without them, Rust could not thrive and evolve.
While I am currently one of the lucky people who are funded for their upstream Rust work7, many Rust contributors do not have access to stable funding. While the Rust Foundation (together with the Rust Project) is currently trying to figure out how we can build a sustainable mechanism for funding full-time Rust maintainers, there are also more targeted ways to help fund individual contributors “in the small”. If you are feeling generous, consider sponsoring individuals who make Rust better on a daily basis.
Here is a short summary of a few things from 2025 that stand out (in no particular order). In 2025, I:
rust-lang-ci/rust repository (finally!), syncing crates.io crate owners in the team repo, moving the Rust web to a static website and adding some cool things to it, and implementing many improvements to our bots (mostly triagebot and bors).cargo build --timings output (not stabilized yet).#[derive(From)] feature), got it accepted :tada: and implemented it. Although it will need to be landed over an edition due to some backwards incompatible name resolution issues, so it might take some time before it gets stabilized.It goes without saying that everything mentioned above was team work that couldn’t be done without the help of the awesome people who are Rust :heart:.
And here is the promised list of the Rust PRs that I opened in 2025. They are categorized by repository (repos are ordered by PR count in descending order) and within each repo they are ordered by creation time in ascending order. Unless otherwise noted, the pull request was merged, although I also kept the opened and closed ones, just to show the open/close/merge ratio.
Below the list you can find a few more thoughts on my 2025 Rust contributions and the conclusion.
x perf directly in bootstrapci.py script in Rustllvm.ccache to build.ccacherun-make-support-Zembed-metadata to allow omitting full metadata from rlibs and dylibsdist::Gcc build step-znostart-stop-gc usage with LLD20RUN_CHECK_WITH_PARALLEL_QUERIESCI_JOB_DOC_URL to DockerDIST_TRY_BUILD for try jobs that were not selected explicitlyrun-make-supportuse-lldcitool to reduce compilation time (closed)opt-dist requires metrics to be enabledrust-cache to speed-up citool compilationopt-dist CLI flag and make it work without LLDdist-x86_64-linux CI job to GitHub temporarilytry-perf branchstdarch into a Josh subtreecreate-docs-artifacts.sh with new bors--json=timings)mingw-check-tidy jobmingw-check-tidy on auto buildscalculate_matrix job on master to cache citool buildsdist-alt jobscitool tests on the auto branchCargo.lock to CI-rustc allowed list for non-CI env”rust.incremental when using download CI rustc-nopt CI jobs--build into account in bootstrapStepMetadata to describe steps--cap-lints (closed)stdarch bootstrap smoke test (open)test_string_interning in bootstrapstd_detect into stdlibtidyToolTarget and port RemoteTestServer and WasmComponentLd to it (closed)doc::Std stepToolTarget to bootstrap--skip-std-check-if-no-download-rustccheck for compiletest and RA using tool macrowasi_sdk_path instead of reading environment variable in bootstrapx test pipelineFn bound when calling a generic parameter (open)rustc_private tools-Zself-profile-events=argsx test on CI-Zprint-mono-items#[derive(From)]stage parameter when setting up stdlib Cargobuild-dir in bootstrappr-check-2 CI jobcompiler_for in bootstrapONLY_HOSTS in bootstrapx check compiler-Zno-embed-metadata in the standard library (open)copy_src_dirs in dry runcopy_src_dirs in bootstrapAssemble stepPartialOrd/Ord from bootstrapFrom derive macro from preludex run mirix test distcheck self-containedriscv64-gc-unknown-linux-musl from Tier 2 with Host tools to Tier 2PartialEq for InvisibleOriginrustc-dev in distcheckToolRustc to ToolRustcPrivategcc.download-ci-gcc to truelibgccjit symlink when we build GCC locallydownload-ci-llvmx <cmd> <path> from a different directoryrustdoc-json-typesx install Cranelift and LLVM bitcode linkerrust.use-lld to rust.bootstrap-override-lldcompiletest-use-stage0-libtest from CIdownload-rustc if LTO is configuredbors environment also to the outcome jobubuntu:24.04 on ghcrx86_64-gnu-miri jobopt-dist (open)eza 0.21.2 benchmarkcranelift-codegen-0.82.1 benchmarkdiesel-1.4.8 benchmarkhyper to 1.6.0clap-3.1.6 benchmarkexa-0.10.1 benchmarkregex-automata-0.4.8 benchmarkhyper-0.14.18image-0.24.1libc-0.2.124regex-1.5.5ripgrep-13.0.0ripgrep-13.0.0-tinydiesel-2.2.10 -new-solver benchmark (closed)nalgebra-0.33.0-new-solver benchmarksyn-2.0.101-new-solverserde-1.0.219-new-solverbitmaps-3.2.1-new-solverhtml5ever-0.31.0-new-solver--exact-match CLI argument to allow exact matching of benchmarksserde-1.0.219-threads4 benchmark for the parallel frontendperf-config.json is not foundlarge-workspace stress test benchmark[workspace] in compile benchmarkstriagebot benchmark (closed)crates.io benchmark (closed)--exact-match instead of --include on the websiteDocJson profile for benchmarking JSON rustdoc outputcollectorcargo stable benchmarkstyle-servo stable benchmarkinclude-blob secondary benchmarkinclude-blobcargo compilable with modern rustc and re-enable stable benchmarks./target/release example commands to cargo run --releaserust directory does not existeprintln profiler invocation failshtml5ever compilable on betamaster-old branch in rustc-dev-guidementorship-programs team--repo argument in dry-run workflowallowed-merge-teamssync-team code for performing dry runsteam to clapsync-team from team directlygsoc-contributors teamgsoc-contributorsmentors teamgsoc-contributors as a subteam of launching-padrust-timer merge bot and try-perf/perf-tmp branch protectionsa-mir-formalityexpect-testexpect-test teamvscode-themes under automation and archive itjosh-sync repositoryjosh-syncinfra-admins to the bors repopromote-releasemainmainmainrust-analyzer/countme repositoryrust-lang/rustrustfmtt-website Zulip stream to t-website website entryT-website zulip group for the website teamannotate-snippets cratemeasureme cratesthorincc-rsannotate-snippetsgenerate-tokens reusable actionfunding repositoryCRATES_IO_USERNAME environment variable in dry-runar_archive_writercrates_io_og_imagegll repositorysqlx-cli in CITreeState in the DBsqlx::Encode implementation for GithubRepoNameDEFAULT clause and add a test for itIF NOT EXISTS to repository creation migrationhelp and info command output nicer@bors tryTestSyncMarker*) in the command parser@bors try commands to replace homu for try builds@bors infoTest timed out commentbors build finished failed jobsenqueue_prs_on_pr_openedmin_ci_timestalled to failedtest job to succeed before starting deploy job@rustbot assign and r? behavior on PRs (closed)IssueData to the DB when it is unchangedassign.review_prefswhoami Zulip commandlookup [github|zulip] <username> command for determining the GitHub/Zulip username mappinglookup zuliplookup githubclapwork showwork showteam-stats Zulip commanddocs-update and ping-goals also through DMsDebug impl@rustbot reroll commandteam@bors rollup=never to rustc-push PR body@bors rollup=never to rustc-push PR body”GITHUB_TOKEN to Zulip CI steprustc-josh-syncrustc-josh-syncjosh-sync@bors try jobscg_gcc invocationAsync Rust is about concurrency, not (just) performance blog postTwo ways of interpreting visibility in Rust blog post (closed)Evolution of Rust compiler errors blog postDisable debuginfo to improve Rust compile times blog postWhy doesn't Rust care more about compiler performance? blog post (closed)Combining struct literal syntax with read-only field access blog postAdding #[derive(From)] to Rust blog postRust compiler performance survey 2025 results blog postReducing binary size of (Rust) programs with debuginfo blog postVariadic Generics Micro Survey official Inside Rust blog postLaunching the 2025 State of Rust Survey Rust blog postGoogle Summer of Code 2025 results blog postMaking it easier to sponsor Rust contributors blog postInvestigating and fixing a nasty clone bug blog post/logosrustc-josh-sync and add a few fixesrust-version and mention this repo in merge commit messagegh PR creation prompt more accuratepull--allow-noop was passed--allow-noop@rustbot rerollproc-macro-src CI jobstd_detect dev dependency in core_archmulti-user@bors prefix for new bors (open)-Zembed-metadata--json=timings behing -Zsection-timings--timings pipeline graphmaster branch instead of stable in CIid-token permissionstdarch test suitetoml to 0.9master to mainrustc-josh-syncno-merges PR titlejosh-sync instead of miri-script for Josh synchronizationrustc-josh-sync merge commit message for pull PR descriptionmaster to mainrust-lang/rust instead of rust-lang-ci/rust for unrolled rollup build commit URLsrustup.sh to use rustc-josh-sync (open)jemalloc-sys finishesstd::thread::available_parallelism for determining the default number of jobs-znostart-stop-gc linker flagfill_tododerive(From) for single-field structsnative-c crateLooking back at the above list, I think I managed to do a lot of work. Even though after finally finishing my PhD I had a lot of energy and time to work on Rust at the beginning of the year, due to some new family responsibilities, teaching at a university, and doing a lot of “non-technical” work (like surveys, mentoring or governance), I felt a bit unproductive in the second half of the year. At least in terms of actual coding, which is still something that I enjoy the most and that “feels” the best, even though other kinds of work are also important.
In fact, my personal Rust TODO list grew (not shrank) in 2025, which is a bit depressing. A large part of that was likely caused by me trying to finish initiatives already started year(s) ago, which I didn’t have the bandwidth to drive forward during my PhD studies, and so new ideas had to wait.
It is also clear that I maintain perhaps definitely too many projects and initiatives. I think that is a good
thing for the Rust Project, because I’m able to help maintain some things that would probably otherwise
be unmaintained or stuck. However, I’m not sure if it’s such a good thing for myself. Even though I think
that I’m relatively good at handling many initiatives at once9 and I’m used to context-switching many times per day hour, that doesn’t mean that I wouldn’t be (or at least feel) more productive and less stressed if I was working on a smaller set of projects at once. It is my short-to-medium-term goal to reduce
the number of Rust projects that I contribute to, so that I can focus on things that I find the most
impactful and most enjoyable for me to work on, such as compiler performance, which I didn’t get to
at all for the past few months. The hard part, as always, is to find people that would continue to
pass the torch so that things I “leave behind” don’t get stuck again.
One area where I’m thinking of scaling down my contributions is the Rust survey team. I mostly only joined it because it seemed like no one else would do it at the time. And even though I created some scripts to partly automate survey analysis, doing surveys doesn’t exactly spark joy to me, so I would like to pass the torch to others. But it’s tricky…
I hope you found this post interesting. I’m thinking of making this a regular yearly “look back” post (for as long as I’m able to work on Rust), so maybe see you here in a year :)
And in case anyone is wondering, none of the PRs listed above were vibe-coded. I don’t use AI much for normal development, although sometimes it is quite useful for creating one-off scripts, such as those I used for gathering my GitHub statistics.
If you found some other interesting new year open-source data visualizations, let me know on Reddit.
It also includes a handful of contributions to the “Rust ecosystem”, i.e. Rust crates on crates.io that are outside the rust-lang organization. But there are not many of those, and I did not include PRs to my own crates, such as delegate or cargo-pgo. ↩
It’s a bit funny how similar is the ratio of upstream Rust work in both my opened and reviewed PRs. ↩
I plan to blog Soon™ about my definition of software maintenance, either on this blog or on the official Rust blog. Stay tuned. ↩
The first one was #134988. On average, it is more than 28 PRs opened per day. ↩
I like to say that if you gave me a hundred full-time engineers, I would find all of them something to work on in the Rust toolchain immediately. ↩
On average, I spend around 70% of my work time on Rust, and I am funded for all that. ↩
Which is sadly kind of stuck on finding more people that understand this part of the compiler (or on becoming such a person myself). ↩
In reality, I probably suck at it the same as everyone else does, but I don’t know how to operate in any other way, so I don’t have a baseline to compare to. I also kind of suck at working on a single thing for a long time, which is probably why I cling to context switching so much, almost as a form of procrastination. ↩
I’m currently trying to finally bring the bors GitHub merge bot to production, so I spent many days in the past weeks fixing various possible race conditions, making its implementation more robust and improving its test suite.
The bors test suite is mostly integration-based; it has very few what you could call “unit tests”. In most tests, we spawn the whole bors web application, connect it to a real running instance of Postgres, apply database migrations, send HTTP requests that emulate GitHub webhooks to it, and let it communicate over the network with a fake HTTP server that emulates GitHub endpoints1. The mocked HTTP GitHub endpoints are implemented using the cool wiremock crate. A typical bors test looks something like this:
#[sqlx::test]
async fn try_build_failed_modify_labels(pool: sqlx::PgPool) {
let gh = GitHub::default().with_default_config(
r#"
[labels]
try_failed = ["+foo", "+bar", "-baz"]
"#,
);
run_test((pool, gh), async |ctx: &mut BorsTester| {
ctx.post_comment("@bors try").await?;
insta::assert_snapshot!(ctx.get_next_comment_text(()).await?, @r"
:hourglass: Trying commit pr-1-sha with merge merge-0-pr-1…
To cancel the try build, run the command `@bors try cancel`.
");
ctx.pr(()).await.expect_added_labels(&[]);
ctx.workflow_full_failure(ctx.try_workflow()).await?;
ctx.expect_comments((), 1).await;
ctx.pr(())
.await
.expect_added_labels(&["foo", "bar"])
.expect_removed_labels(&["baz"]);
Ok(())
})
.await;
}
While doing a particularly big refactoring, a few tests started failing in a very weird way, by panicking in one of the mocked GitHub endpoints. It was a PATCH endpoint that was trying to deserialize a JSON payload from the request body, but the body was empty, so the deserialization of course failed. That was very weird, because bors only sends a PATCH request at one place in its codebase, and there it does fill the request body. Also, the body was empty only sometimes, in some situations it was normally present. That’s usually the sign of a wonderful bughunt starting.
The receiving endpoint mocked with wiremock contained this deserialization code:
#[derive(serde::Deserialize)]
struct SetRefRequest {
sha: String,
}
let data: SetRefRequest = req.body_json().unwrap(); // The test panicked here
bors uses the octocrab crate for communicating with GitHub. The code that is used to send the PATCH request looked like this:
let res = repo
.client()
._patch(
url.as_str(),
Some(&serde_json::json!({
"sha": sha.as_ref(),
"force": matches!(force, ForcePush::Yes)
})),
)
.await?;
Well, clearly I was sending a body on the sender side (a JSON object with two fields, sha and force), and I was trying to deserialize a request type with the correct shape (the sha attribute) on the receiving side. But in some tests, and only after a few calls of the endpoint, the body was suddenly empty. That was super weird.
My usual motto when dealing with weird bugs is “the problem is in your code, not in a dependency”, which is correct approximately 99% of the time. So I started debugging the failing tests, and the place where the request is sent in bors, but everything looked correct. I always seemed to send the request with a body, but sometimes it arrived without it. So I started suspecting that a problem might be in some dependency. I naturally suspected wiremock first, because it is a much less commonly used crate than octocrab, so I started digging into it. The issue tracker didn’t contain anything that would sound related, so I went back to the good ol’ debugger.
Debugging external Rust crates is not actually that easy. Rust is known for being very abstraction heavy (especially in library code), which can be particularly annoying for step-by-step debugging, and even more when it involves async code. Furthermore, if you want to modify the debugged code (e.g. to add more logging), it is not so easy as e.g. in Python, where I could just go to a virtual environment and modify the source code of the dependency manually. If I do that in Rust, Cargo won’t actually pick up that the source code of a crate in $HOME/.cargo/registry/src/... has been modified (and making modifications here is not a good idea anyway). So I git cloned wiremock and depended on it locally with wiremock = { path = "..." }, so that I could add more logging to it to see what’s going on2.
But again, that did not end up with anything interesting. Once I went up the call stack, I saw that wiremock uses the hyper crate to create its HTTP server. And indeed, the hyper request arrived with an empty body (sometimes). I did not find it very probable that hyper would contain a bug that would just randomly remove request bodies, so I decided to take a look elsewhere.
I still wasn’t prepared to believe that this could be a bug in octocrab, so I pulled up wireshark, to find a definitive proof whether the problem is in the sender or the receiver side. Luckily, the communication was on localhost via HTTP (without TLS), so investigating it with TCP/IP dumps was quite easy, and I finally found something useful. The HTTP stream looked something like this:
PATCH /repos/a/b/git/refs/heads/main HTTP/1.1
x-github-api-version: 2022-11-28
content-type: application/json
authorization: Bearer foo
user-agent: octocrab
host: localhost:5555
content-length: 28
{"sha":"foo", "force":true}
HTTP/1.1 500 Internal Server Error
content-length: 0
PATCH /repos/a/b/git/refs/heads/main HTTP/1.1
x-github-api-version: 2022-11-28
content-type: application/json
authorization: Bearer foo
user-agent: octocrab
host: localhost:5555
HTTP/1.1 500 Internal Server Error
content-length: 0
Notice something strange? The body is missing in the second request! So I finally knew for a fact that the sender caused the issue. So at this point, I had three things in mind:
octocrab.I found it quite hard to believe the first two options (especially since bors contains no unsafe code), so I finally set my sights on octocrab. I went to the place where I was creating the PATCH request and stepped through the internals of octocrab, but the body seemed to be there! Although it was not exactly trivial to check, because HTTP bodies are kind of difficult to inspect; in the general case, they can be streamed, so Rust networking APIs are usually built in a way where you can only consume the body once3.
Since I still couldn’t find anything weird even in octocrab, I went back to the drawing board, and tried to approach the problem from a different viewpoint. After some more thinking, I asked myself: why is the request even being sent multiple times in the first place? Originally, I did not find that weird, because the failing tests were returning a 500 HTTP response from the mocked endpoint, to simulate a problem with GitHub, and bors implements request retrying, so it seemed like it was working as intended. However, after inspecting the test logs4 a bit more, I realized that my own retry code was not actually being triggered! That started sounding really suspicious - why was the request retried then?
After taking another look at the octocrab codebase, I realized that it actually implements its own mechanism for request retries, which I had no idea about. And it is enabled by default! And sure enough, when I disabled the retry mechanism:
let client = Octocrab::builder()
...
.add_retry_config(RetryConfig::None)
.build()
The problem with the empty body was gone! :tada: So I finally fixed the problem on the bors side, and since bors was (unintentionally) doing double retries before, disabling the octocrab retry mechanism actually fixed two bugs at once. I think that this was possibly the first time I found an actual bug in a Rust crate dependency that broke my code in ~7 years of using Rust. Pretty good track record for the Rust ecosystem in my experience so far!
But of course, I couldn’t just stop there. I had to know what is the original cause of the bug, so that I could report it and fix it.
Once I finally knew that the bug was caused by the octocrab retry mechanism, creating a minimum reproducible example was relatively easy:
async fn handler(body: String) -> impl IntoResponse {
eprintln!("Received data: {body}");
StatusCode::INTERNAL_SERVER_ERROR
}
#[tokio::main]
async fn main() {
let port = 5555;
std::thread::spawn(move || {
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap();
runtime.block_on(async move {
let app = Router::new().route("/foo", patch(handler));
let listener = tokio::net::TcpListener::bind(
format!("127.0.0.1:{port}")
)
.await
.unwrap();
axum::serve(listener, app).await.unwrap();
});
});
let client = octocrab::OctocrabBuilder::new()
.personal_token("foo")
// .add_retry_config(RetryConfig::None) // <- this fixes the issue
.base_uri(format!("http://localhost:{port}"))
.unwrap()
.build()
.unwrap();
#[derive(serde::Serialize)]
struct Request {
force: bool,
}
let response = client
._patch(
"/foo",
Some(&Request { force: true }),
)
.await
.unwrap();
eprintln!("Response: {}", response.status());
}
This code printed the following:
Received data: {"force":true}
Received data:
Received data:
Received data:
Response: 500 Internal Server Error
Exactly as in bors. When the request was sent for the first time, the body was correctly sent
over the wire. But when octocrab tried to retry the request, suddenly the body disappeared.
Originally, I did not notice this problem when debugging octocrab, because I sort of stopped too soon,
at a place where the HTTP request was passed from octocrab to a stack of tower middleware.
And at that point the request was still fine! But now it became clear that the problem was somewhere inside
that middleware, which is why it was so hard to find originally.
I wasn’t aware of how retrying works in tower, and if this is a bug in octocrab
or in tower (although now I suspected the former). After a bit of digging, I found that
retrying is indeed implemented in octocrab with a retry Policy, which tells tower how to retry a given HTTP request. The most important part is the clone_request method:
fn clone_request(&mut self, req: &Request<OctoBody>)
-> Option<Request<OctoBody>> {
match self {
RetryConfig::None => None,
_ => {
// `Request` can't be cloned
let mut new_req = Request::builder()
.uri(req.uri())
.method(req.method())
.version(req.version());
for (name, value) in req.headers() {
new_req = new_req.header(name, value);
}
let body = req.body().clone();
let new_req = new_req.body(body).expect(
"This should never panic,
as we are cloning a components from existing request",
);
Some(new_req)
}
}
}
Notice how the code says “Request can’t be cloned”, yet it clones the request body like it’s not a big deal :thinking: That looked suspicious, so I looked at how the request body type looks like, at which point the bug became obvious:
#[derive(Debug)]
pub struct OctoBody(Arc<RwLock<BoxBody>>);
Here is the Clone impl, if you haven’t noticed the issue yet:
impl Clone for OctoBody {
fn clone(&self) -> Self {
OctoBody(Arc::clone(&self.0))
}
}
Yes, cloning the body just performs a shallow clone of an Arc, instead of performing a deep copy of
the whole body (!). And since the body is consumed when the request is sent for the first time, the code
simply started sending an empty body when the request was retried.
This is where the relation to the ergonomic cloning initiative comes in. If we had a clearer distinction
between performing a deep copy vs just cloning a (reference-counted) Handle to something, it would likely be much more
obvious that there is a problem with the .clone() call.
It still seemed suspicious why this was a silent bug though. Shouldn’t it panic somewhere if we try to reuse the same body again? :thinking: If we take a look inside the Body trait implementation of the OctoBody, we see that in the poll_frame method, it locks the RwLock and forwards the poll_frame call into its inner boxed body that it abstracts over:
impl http_body::Body for OctoBody {
type Data = Bytes;
type Error = crate::Error;
#[inline]
fn poll_frame(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Option<Result<Frame<Self::Data>, Self::Error>>> {
let inner = Pin::into_inner(self);
let mut boxed_body = inner.0.write()
.expect("RwLock write lock failed");
Pin::new(&mut *boxed_body).poll_frame(cx) // <-- poll forwarding
}
...
}
That sounds like it could cause an issue if the body is being polled even after it has been “consumed” while sending the request for the first time. You might think that if OctoBody stored an RwLock<Option<...>>, which would be replaced by None once the body is consumed, it would make this into a noticeable panic. But OctoBody actually already mostly does that! It usually stores a http_body_util::Full struct as its boxed Body implementation, and that does exactly that. So why haven’t the code panicked, instead of sending an empty body in a retry?
Well, as it turns out, the poll_frame method is actually not ever called again once the request is consumed for the first time. When hyper polls the body, it first uses the Body::is_end_stream method to check if the body is at its end, and if it is, it does not ever poll it again. And the http_body_util::Full implementation of course returns true from this method once its inner body has been consumed. So when the body is consumed for the first time, it will start returning true from is_end_stream, and when we then try to reuse the same body in another request, hyper will just skip reading the body contents and send an empty body instead. Kind of footgunny if bodies are ever reused!
It is interesting to note that this could have caused an issue only because the Arc was combined
with interior mutability (RwLock). If there was no interior mutability, performing a shallow
(instead of a deep) clone would likely result at most in a performance degradation, but not a logical
bug. Interior mutability is useful, but it also relaxes some of the benefits we are normally
used to when using Rust!
It was also surprising to me that this bug has existed in the octocrab crate for more than two years,
but it does not seem like anyone noticed it. I suppose that sending a request to the GitHub API that
both has a body (so it is not a simple GET request) and that gets retried (so GitHub returned e.g. a 500 error
the first time the request was sent) is just not that common. It seems like my heavy usage of HTTP integration
testing in bors is uncovering all sorts of interesting issues :laughing:
The bug might look obvious in retrospect, but it took me several hours to debug it, and I was getting pretty desperate after that time. I’m certainly not used to spending so much time debugging a single issue when working with Rust! In hindsight, I should have tried to send the PATCH request manually, without octocrab. That would give me a hint much sooner that the problem is indeed within octocrab. I guess avoided it because of my usual thought of “the bug has to be in my code”. Well, the next time, I will probably start suspecting dependencies a bit sooner :sweat_smile:.
After finally understanding what was going on, I reported the issue in the octocrab issue tracker and went to sleep. It was a long day :zzz:.
After I rested for a bit and regained enough motivation to take a look at octocrab again a few days later, I took a
shot at fixing the actual issue. The root of the issue was in performing a shallow copy, instead of
copying the whole body contents. That is actually easier said than done, as HTTP request bodies are not exactly
known to be easy to clone, as they could be streamed. However, since octocrab does not necessarily has
to be the most blazingly fast :rocket: zero-copy :fire: library under the sun, I figured that a bit of
copying in the common case won’t hurt that much.
What I did was to add a separate reference-counted copy of the request body, using the venerable bytes crate:
#[derive(Debug)]
pub struct OctoBody {
body: Arc<RwLock<BoxBody>>,
// Copy of the whole body, used for retrying requests
buffered: Option<Bytes>,
}
The buffered field is an Option, because in the most general case, the body might be streamed, in which case I can’t easily turn it into Bytes outside of an async context. However, at that case, we will now at least no longer send an empty body by mistake, but rather do not perform a retry at all.
In cases where the body is available in a buffered way (which I assume will be in 99% of cases when using octocrab), we copy the body contents into Bytes when wrapping it into OctoBody:
impl From<Vec<u8>> for OctoBody {
fn from(buf: Vec<u8>) -> Self {
let buffered: Bytes = Bytes::from(buf.clone());
Self::create(http_body_util::Full::from(buf), Some(buffered))
}
}
This pessimistically clones the body contents even if we never actually do a retry, but at least it makes the code work as it should. It might be better to only perform this copy if the retry feature of octocrab is enabled. Could be an interesting follow-up pull request, if you’re interested!
The final piece of the puzzle was adding a method for deep copying the body, which I called try_clone:
/// Try to perform a deep clone of this body
pub fn try_clone(&self) -> Option<Self> {
self.buffered.as_ref().map(|buffered| {
Self::create(
http_body_util::Full::from(buffered.clone()),
Some(buffered.clone()),
)
})
}
and then using this method in the clone_request method:
fn clone_request(&mut self, req: &Request<OctoBody>)
-> Option<Request<OctoBody>> {
match self {
RetryConfig::None => None,
_ => {
let body = req.body().try_clone()?;
...
}
Notice how try_clone returns an Option. If the request body cannot be cloned, the clone_request
method will also return None, and the request simply will not be retried. Which is much better than
sending an invalid request!
My fix was sent in this pull request, which was
merged soon after, and included in the 0.49.1 octocrab release. If you use octocrab and do not disable the automatic retry mechanism, you might want to update to it.
While writing this blog post, I wondered if LLMs could actually find this bug sooner, and save me the hours of debugging pain. I’m not a very skilled vibe-coder, but I do use Claude sometimes, so out of curiosity, I asked it to locate the bug.
First, I showed it the minimal reproducible example from my octocrab issue. It found out that the problem is with retries pretty much immediately, although with this example that was already quite obvious. After asking it to dig deeper, it also found out the exact cause of the underlying clone bug directly in octocrab. That was relatively impressive, although it also pointed me to the patched 0.49.1 release, and cited text that was suspiciously similar to my original issue and my PR. So it clearly saw my fix :laughing: And thus it’s unclear whether it would be able to find the issue even before I actually fixed it. Well, too late to test that now.
In a more realistic scenario, I reverted bors to a commit right before I disabled the retries, and asked Claude to try to find the issue there. At first, it produced some nonsense suggestions, such as claiming that I send an empty string for the sha field (which is quite different from sending an empty request body…). Then it kind of forgot that I’m using Rust and not C++ and claimed that the issue is a “subtle temporary lifetime problem with async/await”:
I think I found the issue! It's a subtle temporary lifetime problem
with async/await in Rust.
Look at this code in src/github/api/operations.rs:172-181:
let res = repo
.client()
._patch(
url.as_str(),
Some(&serde_json::json!({
"sha": sha.as_ref(),
"force": matches!(force, ForcePush::Yes)
})),
)
.await?;
The problem is that serde_json::json!(...) creates a temporary value,
and you're passing a reference to it (&serde_json::json!(...)).
Even though octocrab's build_request is synchronous and serializes
the body before the .await, there can be subtle issues with temporary lifetime
extension in async contexts, especially across different compiler versions
or optimization levels.
Here's what might be happening:
1. The json! macro creates a temporary serde_json::Value
2. A reference to this temporary is passed to _patch
3. Due to the async context, there's a possibility the temporary
gets dropped before serialization completes
4. This results in either garbage data or an empty body being sent
The Fix
Store the JSON value in a variable first to ensure it lives long enough:
let body = serde_json::json!({
"sha": sha.as_ref(),
"force": matches!(force, ForcePush::Yes)
});
let res = repo
.client()
._patch(url.as_str(), Some(&body))
.await?;
After I patiently explained that this is not the kind of issue that can just happen (without unsafe) in Rust, and hinted that the problem might also be in a dependency, it finally started looking at octocrab, and actually did find the shallow clone problem (although it is again unclear whether that would happen without it knowing about my fix):
Ah! I found something suspicious. Look at the OctoBody implementation -
it uses Arc<RwLock<BoxBody>> and implements Clone by cloning the Arc:
impl Clone for OctoBody {
fn clone(&self) -> Self {
OctoBody(Arc::clone(&self.0))
}
}
This means when the body is cloned, both clones share the same underlying
body stream! If one clone consumes the body (by polling it), the other clone
will have an empty body!
However, even though it found the actual bug, it then started to lose confidence that this bug is the culprit, in a very funny way. It examined my Cargo.toml:
Wait, I don't see the "retry" feature enabled for octocrab in
bors's Cargo.toml. Let me check if bors has its own retry logic
that might be cloning requests:
It failed to realize that retry is a default feature of octocrab, so even if I do not enable it myself, it was actually enabled by default. This was enough for it to stop suspecting octocrab, and go on a hunt to find (hopefully fictitious) retry bugs in the logic of bors. At that point I lost my patience with it and stopped it.
I think that this shows both the advantages and disadvantages of debugging using LLMs. They can sometimes find even pretty tricky issues, but they can also get sidetracked, at which point they need someone else (either a human that actually understands what is going on, or another LLM :) ) to guide them. I can imagine someone less knowledgeable about Rust wasting quite some time debugging imaginary issues in bors found by Claude, in an attempt to locate what might be causing the empty request bodies.
Anyway, next time I start a similar bughunt, I will probably just start Claude in a terminal and let it run concurrently. Then we can see who can get to the core of the bug sooner :)
I’m really glad that I managed to find the underlying issue, as it was a satisfying conclusion to an otherwise pretty maddening debugging experience. And in the end, it helped fix other pre-existing bugs in bors and its test suite and also fix a nasty bug in a popular Rust crate at the same time. So, a win-win?
By the way, this post is not taking a shot at octocrab in any way. It’s a very useful library that
saved me a lot of time over the years! Kudos to XAMPPRocky for creating
it and for merging my bugfix PR and releasing a new version so promptly.
If you have similar Rust bug stories, perhaps related to wrong usage of clone, I’d be glad to hear them on Reddit.
I recently had a talk about my experiences with Rust testing, inspired primarily by the bors test suite. Since the recording will probably not be available online, I plan to turn that talk into a blog post Soon™. Stay tuned. ↩
If it was a transitive dependency, I could use the [patch] section in Cargo.toml. ↩
This is actually kind of foreshadowing the cause of the bug :) ↩
Using the traced-test crate. ↩
Perhaps contrary to popular belief, Rust binaries can actually be quite small if you spend some effort on it. However, once you start including debug information (debuginfo) in them, the binary sizes typically start increasing at an alarming rate.
If you wonder why we should care about binary size when debuginfo is enabled, as that’s typically used for debug builds, it is also commonly used for distributing release/production binaries that include at least debug line tables, which are needed to produce quality backtraces.
In this post, I’ll shortly describe why debuginfo takes so much space in Rust programs and how we can optimize it. Note that I will only talk about the DWARF debuginfo format and Linux ELF binaries. Furthermore, I will only be considering debuginfo embedded directly in the binary. If you distribute “split” debuginfo in a separate file, you might not care about its size so much, but that approach has other trade-offs, and I won’t consider it in this post.
Apart from (DWARF) debuginfo generally taking quite a lot of space on its own, there is a less obvious reason why debuginfo in binaries can be so large, and that is because linkers typically do not “tree-shake” unused debuginfo (unlike unused code, which gets removed). I only found out about this recently, which was the motivation for writing this post.
Let’s try to demonstrate that with a simple example. Consider the following crate (let’s call it dep), which has a single function foo:
pub fn foo() { println!("foo"); }
and another crate called mycrate that depends on dep and calls its foo function:
fn main() {
dep::foo();
}
When compiled with Rust 1.90.0 on x64 Linux in --release mode, the resulting mycrate binary has exactly 449328 bytes.
If I add another function bar to the dep crate, without changing anything else:
pub fn foo() { println!("foo"); }
pub fn bar() { println!("bar"); }
and recompile, the resulting binary still has the exact same size as before. And that is what I would expect, because the bar function is not called anywhere, so it should not take space in my binary.
However, the situation changes if I add debuginfo. If I compile the main crate with two versions of dep (just foo vs foo + bar) in release mode with debug = "line-tables-only", the size of the binary jumps from 3 874 080 bytes to 3 874 232 bytes! So even if the bar function is not used anywhere, its debuginfo is still taking space in my binary :angry: This is not so bad for a single function, but if you consider the amount of code that Rust programs typically depend on, and how much of that code is actually unused, it starts to quickly add up.
This also explains why the binary size suddenly jumped to almost 4 MiB with debuginfo enabled; that is the debuginfo of the standard library being included in it. Even though a lot of it is actually unused/dead code (because my program doesn’t use these parts of the stdlib), its whole debuginfo is still included. Even though the stdlib is a bit special in this regard, because it is precompiled, the same would actually happen with a “normal” crate dependency that you build from scratch.
I’m not really an expert on linkers, but from what I understand, when linkers see debuginfo entries that are unused, they do not remove them, but merely mark them with special “tombstone” values. So they effectively become just dead weight that takes space in the binary, but don’t do anything. At least this is what MaskRay (who is probably one of the most knowledgeable persons about linkers) says. Apparently, removing these debuginfo sections in linkers could be very costly in terms of performance, due to DWARF’s tree representation. I found this forum post where someone claims that they tried to implement it in the LLD linker, and that made it seven times slower. I’m somewhat dubious that this is the best that we could achieve, but for now it seems that linkers simply do not remove unused debuginfo entries and we have to live with that. Maybe a sufficiently smart linker will be able to do it one day :)
Of course, this problem is not unique to Rust, as it also happens for other “natively compiled” languages, such as C or C++. However, for Rust I think that the issue is typically worse than for C/C++, because Rust programs tend to both have a lot of dependencies, and crucially if you compile them with debuginfo, by default you will be building debuginfo for every. single. dependency. that you have, even though most of that debuginfo will likely never be needed. I think that this is not so prevalent for C or C++, because there people often use prebuilt libraries for which they either might not have debuginfo available, or it is fetched through some external mechanism (e.g. -dbg packages from the distro package repository) and is thus not embedded directly into the final binary.
As usually, I’ll use hyperqueue as a test subject for demonstrating how we can reduce the binary size when debuginfo is included. I tried it in two configurations, the first is simply compiling with --release, while the second is compiling with --profile dist, which in HQ’s case enables more optimizations (1 CGU, thin LTO, etc.). This is the actual configuration that we use for distributing hyperqueue to end-users. Both configurations include line-tables-only debuginfo.
One approach that can be used to reduce debuginfo binary size is to perform garbage collection. That finds the unused placeholder (“tombstone”) values and gets rid of them. There are probably more ways of doing that, but I found that the llvm-dwarfutil1 tool can do it, so I tried it:
$ llvm-dwarfutil hq hq-gc
Here are the results:
| Version | release profile |
dist profile |
|---|---|---|
| Original | 70 924 912 (100%) |
50 832 304 (100%) |
| GC | 59 046 440 (83%) |
53 162 184 (104%) |
For the release profile, it resulted in a nice 17% binary improvement, not bad! However, for the more optimized (and smaller) dist profile, the resulting binary is actually larger. So it seems that sometimes it doesn’t really help.
However, when it does help, this approach doesn’t really have any disadvantages, as far as I am aware (but if you know of any, I’d be glad to learn!), so it still seems quite useful. It did print a warning about not supporting the .debug_gdb_scripts section, and some other warnings, but the resulting binary seems to work and produce correct backtraces. The garbage collection took under two seconds.
When I tried to apply the garbage collection to a hello world Rust application compiled in --release with debuginfo, it reduces the size from the original ~4 MiB down to ~2 MiB. Not bad at all! Seems like it is able to remove a lot of the unused debuginfo from the Rust standard library.
The second approach, which we actually use on our CI, and which is probably more widely known, is using debuginfo compression. This can be done using the venerable objcopy tool:
$ objcopy --compress-debug-sections=zlib hq hq-zlib
I tried both zlib compression (which took ~1s to compress) and zstd (which took ~0.1s to compress). Here are the results:
| Version | release profile |
dist profile |
|---|---|---|
| Original | 70 924 912 (100%) |
50 832 304 (100%) |
zlib |
24 775 848 (35%) |
18 751 280 (37%) |
zstd |
23 413 992 (33%) |
17 472 960 (34%) |
The results are pretty great! Going from ~50 MiB down to ~18 MiB is really impressive.
I suppose that the disadvantage of this approach is the risk that some tools that work with debuginfo (such as debuggers, profilers or symbolizers) won’t be able to understand the compressed debuginfo sections or won’t be able to decompress them. This is especially problematic for zstd, which is less supported “in the wild”. Even here, when I ran the zstd compressed binary and it panicked, the stacktrace no longer contained proper line numbers (I guess that gimli, which is included in the stdlib and AFAIK deals with decoding the backtraces, can’t deal with zstd?). That is why we use zlib2 for our distributed hyperqueue artifacts, even though it produces slightly larger binaries than zstd.
EDIT: thecakeisalie16 showed on Reddit that you can also perform debuginfo compression directly using the linker with RUSTFLAGS="-Clink-arg=-Wl,--compress-debug-sections=zlib". Didn’t know about that, very cool!
Interestingly, these approaches can be combined to produce even smaller binaries, by first running gargage collection and then compression (the other direction doesn’t really make sense). And funnily enough, compressing the garbage-collected dist binary, which is actually larger than the original binary, produces a smaller file than compressing the original dist binary!
Here are the final results with all the approaches I described above:
| Version | release profile |
dist profile |
|---|---|---|
| Original | 70 924 912 (100%) |
50 832 304 (100%) |
| GC | 59 046 440 (83%) |
53 162 184 (105%) |
zlib |
24 775 848 (35%) |
18 751 280 (37%) |
GC + zlib |
20 457 568 (29%) |
17 318 760 (34%) |
zstd |
23 413 992 (33%) |
17 472 960 (34%) |
GC + zstd |
19 777 752 (28%) |
16 492 768 (32%) |
You can find the script that I used to generate this table here, in case you wanted to perform a similar experiment on your own binaries.
On hyperqueue’s CI, we currently only compress the debuginfo using objcopy, but based on these results, we might want to also use the debuginfo GC in the future (although getting ahold of llvm-dwarfutil on CI might be a bit annoying).
A natural question to ask is whether the Rust compiler should do any of the above by default. Last year, I made --release Rust binaries smaller by default by stripping the standard library debuginfo from them when it wasn’t required. I think that was a pretty successful initiative, and it is relatively similar to what I described in this post (the compiler invokes a strip binary on the binary to remove the debuginfo, while here we invoke llvm-dwarfutil/objcopy), so it might not be too far-fetched?
That being said, I think that compressing debuginfo goes too far in what I would expect that a compiler invocation would do for me, and has the trade-offs mentioned above. Maybe GC would be a better fit (in fact I expected that the linker already does it for me), but the linkers that we currently use don’t seem to have this ability, and doing it post-compilation using e.g. llvm-dwarfutil might be unnecessarily slow? Testing this on some ludicrously large binary would probably be needed to get a better intuition about trade-offs involved in this.
Of course, if we ever wanted to do this, we would have to decide whether to do it by default (which would likely increase compile times) or whether we want to make it configurable. But in general, it probably seems too specialized to me to be in rustc. Maybe we could at least document it e.g. in the Cargo book or in min-sized-rust.
Anyway, I wanted to share this post to increase the awareness of compressing and garbage collecting debuginfo in (Rust) binaries, so that we can all ship less data to our users and avoid needless waste. Hopefully you found it useful.
I’m definitely not an expert in this area though, so if you have any suggestions/corrections/tips, I’d be glad to hear from you on Reddit.
]]>#[derive(From)] can be used in nightly now. See here on how to use it.
You can follow this tracking issue for more updates.
I use the newtype pattern in Rust a lot,
and I would like to make certain use-cases around it (even) simpler to express. A typical thing to
do with newtypes is that you implement a bunch of standard traits for them, which will delegate
to the inner field. The easiest way of achieveing that is of course with a #[derive(...)]:
#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Default)]
struct WorkerId(u32);
However, this approach cannot be currently used with all “built-in” traits from the standard library.
One of those traits is the From trait,
which is used to convert between values of two types. Even though sometimes it might not be a good idea
to implement From when using a newtype (more on that later), in other cases it’s quite useful,
as it makes it easier to go from a value of the inner field to a value of the newtype (especially
in generic contexts). A manual implementation of the trait will usually look something like this:
impl From<u32> for WorkerId {
fn from(value: u32) -> Self {
Self(value)
}
}
This code is trivial to implement, of course, and there’s not much space for bugs here, but it’s also pure boilerplate, and especially if you have a lot of newtypes in your crate, it just seems wasteful, as it’s clear that the compiler should be able to generate this implementation easily.
In fact, writing this impl over and over again usually annoys me enough that I end up writing a
macro to generate newtypes, so that I don’t have to copy this (and other similar) code all over the
place. But writing and using such macros has trade-offs (same as using third-party crates for
this, such as derive_more), and
I thought that rustc should just give me the option to #[derive(From)].
After running into this missing feature yet again, I realized that… I could just add it to Rust :laughing: Making language changes can be daunting, but this specific change seemed quite simple, mostly because the behavior of the feature seems obvious. And so I decided to try it and write an RFC. To test the waters, I first drafted a “Pre-RFC” on the IRLO forum. It received a relatively positive response, even from a member of the Language team, which made me confident enough to turn it into an actual RFC, which I did a few weeks later.
I have to say that I was quite pleasantly surprised by the positive response to my RFC. I’m used to reading big controversial RFCs that generate hundreds of comments and many arguments, but here the response was mostly “yep, this sounds like a good idea” (with a few exceptions that I will discuss later). I even received a few suggestions on how this feature could be improved in a future RFC.
The PR itself also received a lot of positive emojis!
At the time of writing of this blog post, it has over one hundred emoji reactions, which actually
makes it into the top ~60 most “liked” RFCs in the repository, even though it’s very recent. Emojis
are of course not a good way to decide which RFCs should be accepted, but it is a signal that many people seem to share the same annoyance of lacking an easy way to derive From impls.
I think that one of the strengths of this RFC is that it is arguably not even a new feature, per se.
It is one of those things that “just” fill a gap in Rust; it allows combining two features that were working
separately before (the From trait and deriving standard library traits), but couldn’t be used together.
One thing that was a bit unclear with the RFC though was determining which Rust team(s) should actually be responsible for deciding its fate. The first decision process (FCP) was started for the Language team. Then it was suggested that it should actually be handled solely by the Libs API team. And in the end we did a dual Language + Libs API FCP. This is actually a recurrent problem in the Rust Project (and I’m sure in many/most other organizational structures), where we are relatively often unsure who should actually be responsible for deciding (or doing) something, which can sometimes block progress.
Luckily, that didn’t happen here, and I can happily announce that after a few minor road bumps, the FCP has been finished, and my first ever Rust RFC was accepted :tada:.
Now that I have spoiled the outcome of the RFC, I should probably talk about the actual design of the feature. Even though it sounds simple, there are a few things to consider.
First, let’s make it clear what is the goal of #[derive(From)]. When applied on an ADT, i.e. a struct
or an enum1, it should generate an implementation of the From trait for it. Let’s start
with structs – when you apply #[derive(From)] to a struct, you should get a From impl for the
struct that creates it from a value of its field, like this:
#[derive(From)]
struct Foo(u32);
// generates:
impl From<u32> for Foo {
fn from(value: u32) -> Self {
Self(value)
}
}
If you look at the example above, an obvious problem presents itself: what to do if there is not exactly one field in the struct?
struct Foo(u32, bool);
There are three solutions that come to mind:
From impl from all the values at once, i.e. From<(u32, bool)>From impl#[derive(From)] in this caseI think that option 1) would be quite unintuitive, and it would run into an issue for the most common
use-case where you have a single field - should we generate From<FieldType>, and thus break consistency
with other field counts, or From<(FieldType,)> (i.e. generate the struct from a tuple
of size one), which is quite unergonomic and uncommon? This approach also wouldn’t really work for
structs with named fields.
I left option 2) as a future possibility. One of the suggestions that were proposed was that we could
annotate a single field with an attribute (e.g. #[from]), and then create the rest of the fields
using their Default impl:
#[derive(From)]
struct Foo(#[from] u32, bool);
impl From<u32> for Foo {
fn from(value: u32) -> Self {
Self(value, Default::default())
}
}
I like this option a lot, and I think that we should explore it in the future. A similar
approach is also being used for selecting the default enum variant using the #[default] attribute
when deriving Default, so there is a precedent for doing something like this. However, since I wanted
to keep this RFC minimal in order to improve its chances of being accepted without falling into
deep “design holes”, I didn’t include it in the RFC and left it as a “future improvement”.
And thus I chose to go with option 3), which means that you can currently only use #[derive(From)] on
structs that have exactly one field. I don’t think that it’s a big limitation though, because I expect
that in most cases this derive will be used for newtypes (which typically store exactly one field).
In general, the From trait is most commonly used for converting a single value of one type into a
single value of another type, so it makes sense to have this limitation in the initial design.
In terms of enums, the situation is more complicated. In theory, we could allow the macro to be used
on enums that have exactly one variant with exactly one field, but that doesn’t sound like a very
useful scenario. For simplicity, I thus also decided to forbid using #[derive(From)] on enums,
at least for now.
Apart from this main design question, some concerns were expressed by several people on the RFC.
One noted concern was whether #[derive(From)] struct Foo(u32); should generate From<u32> for Foo,
From<Foo> for u32, or both. To me, the answer is obvious (it should be From<u32> for Foo),
because that is how all the other derive macros work; they always generate an impl for the type on
which the derive macro is applied. I think that this behavior is the most intuitive one, and we should
keep consistency with the previous deriveable macros. This direction also makes the newtype use-case
work, and I consider that to be the most important driving use-case for this feature.
However, it is actually interesting to also consider how we could support the second direction, because
it can also be useful to go from a value of the struct back into the inner field, of course. I omitted
this functionality from my RFC, but one of the proposed solutions was to add something like #[derive(Into)],
which would generate From<StructType> for FieldType. Of course, this naming could be quite confusing,
because we would not actually generate an impl of the Into trait, because that is generated
automatically by a blanket impl in the
standard library. But that is a discussion for another RFC.
This is the usual worry with any (Rust) RFC that adds a new feature – are we sure that we got the right design for it, that it will be extensible in the future, and won’t close doors to a better solution for the same problem?
Since I kept my RFC minimal, this is a valid concern. Maybe we should first figure out how to support
the From impl in the other direction? Or flesh out the #[from] extensions? Some reviewers wanted
to see the “grand picture” of this feature before it will be stabilized (or even before the RFC would
get accepted).
I sympathize with those views, as I am myself also often worried about accepting a feature too soon, when we don’t know all the details yet. However, in this specific case, I am quite confident that the feature that I proposed will be perfectly compatible with any relevant extension, and thus we should land it as soon as possible, without waiting on the more complicated discussions about the possible extensions, to achieve incremental progress.
My confidence comes from the fact that (at least to me), the behavior of #[derive(From)]
is simply obvious, and there is no other way how it could reasonably behave.
If rustc generated anything else than the following for #[derive(From)] struct Foo(u32):
impl From<u32> for Foo {
fn from(value: u32) -> Self {
Self(value)
}
}
Then I think that the feature would be wrong. It might have some other extensions or configuration points, but this core functionality is in my opinion set in stone, and there is no other way how it could work. And thus I wanted to get my minimalistic RFC out the door, so that we can start testing the feature in the wild, and mainly so that it can start providing value to Rust users.
It should be possible to lift the limitations (such as allowing it on structs with multiple fields
and implementing the #[from] attribute), or add related functionality (such as #[derive(Into)])
to this feature in the future in a backwards-compatible way2.
From on newtypes even a good idea?One additional, a more philosophical, concern discussed on my RFC was whether implementing
From for newtypes is even a good idea in the first place, and thus whether we should teach rustc
to do it.
There are two angles to this discussion. The first one is whether newtypes should implement the
From trait or not. To answer that question, I’ll try to examine the various motivations
for using newtypes in a bit more detail.
I stress motivations here so much because newtype is a pattern, and the motivation for using a pattern is one of the things that defines it. You can have two identical pieces of code, where one upholds a certain pattern, and the other doesn’t, which can happen if those pieces of code were created because of a different motivation. That is why there are some patterns in Gang of Four that can have pretty much the same source code representation, but they still form different patterns because they have a different motivation.
The primary motivation for using a newtype, one that is upheld by every usage of a newtype, is to
create a new type in the type system (d’uh). This allows us distinguish between e.g. WorkerId and
TaskId, so that it is harder to accidentally combine values of different domain types, even if they
are backed by the same “in-memory” data type (e.g. an integer). I’ll call this the “avoid type confusion”
motivation.
Another very common motivation (which is a subset of the first motivation!) is to ensure that values
of the newtype will uphold some invariant, which is not necessarily true for all values of the type
inside the newtype. A typical example is something like struct Email(String). An Email is backed
by a String, but not all values of strings are valid e-mails! I’ll call this the “ensure invariants”
motivation.
For avoiding type confusion, it is perfectly valid to implement the From trait for the newtype, as it
gives you a standardized way of initializing the newtype from the wrapped value.
For ensuring invariants, implementing From is of course not the right thing to do! From should
always be infallible, and should be able to convert all values of a type into some value of another
type. But in the case of e.g. Email, that of course won’t work. It might make sense to implement
TryFrom in this case, but that is not something that we can easily derive anyway.
It seemed to me that people who expressed concerns about whether we should make implementing
From easier were mostly using newtypes to ensure invariants. In that case I can understand why
they think that implementing From is a bad idea! However, I personally use newtypes more often
for avoiding type confusion (my estimate is that the ratio between the two use-cases in my code is
something like 75:25 in favour of avoiding type confusion). So for most of my newtype use-cases,
implementing From is genuinely useful (which is why I wanted to add this feature to Rust in the
first place :) ).
The second angle to this discussion is more general and unrelated to newtypes.
I personally think that Rust should not dictate what is and what isn’t “good program design”,
of course as long as makes sure that it is sound (in the sense of memory/data race/etc. safety).
Obviously, Rust already makes some strong decisions about program design, for example it does not
allow you to use inheritance, and the borrow checker also makes
expressing certain ownership patterns (like the famous “soup of pointers”) annoying. But in this specific case,
you can already implement From by hand, so forbidding users from deriving this trait, even
though it is technically possible, just to make sure that they don’t do something that might be considered
bad practice, would seem to be a bit silly to me.
Normally, after an RFC is accepted, we have to wait until someone actually implements it (which is
not a given, we shouldn’t forget that!). But luckily, since I’m a compiler contributor, I was able to help myself here :sweat_smile: And after some trial and error, I
managed to implement the feature. It was a pretty
fun exercise, because the From trait is different enough from all the previous traits that
were deriveable, so I also had to make some small changes to the derive infrastructure to make it work.
It was also a chance to work with Nick Nethercote again, which is always awesome.
He even made some improvements to the feature
after my initial PR was merged. Thanks Nick!
While implementing this feature, I understood how the derive macros are actually wired between
the compiler and the standard library, which is actually pretty interesting. The compiler defines
the traits that are deriveable, along with the actual logic to generate the derived code. However,
when I added a new From macro derive implementation to this list, #[derive(From)] still didn’t work.
After digging and tracing of the compiler’s behavior on small Rust programs, and asking the
Foremost Rust Name Resolution Expert™ for help, I finally realized
that the answer is name resolution. When the compiler sees something like #[derive(Trait)], it uses
name resolution to understand what Trait is. One option is that the trait is resolved to a
(derive) proc macro, in which case rustc will call it3 to generate the expanded code.
However, another option is that it will resolve the trait to a macro in the standard library marked
with the #[rustc_builtin_macro] attribute.
And that is the piece of magic that ties the name From to the code that actually generates the right
derive logic. So after I added this:
#[rustc_builtin_macro]
#[unstable(feature = "derive_from", issue = "144889")]
pub macro From($item: item) {
/* compiler built-in */
}
to the standard library, #[derive(From)] finally started working :tada:
However, name resolution (in Rust, but also in general) is a VERY gnarly beast, and my joy from defeating it did not last long. The very next day after my PR was merged, someone created an issue that my PR broke their code in nightly because of name resolution errors. Exactly the thing you want to see when you wake up :laughing:
The issue was (as usually) with glob imports. To make #[derive(From)] work “out of the box”, I added
the From macro to the standard library prelude, so that it can be implicitly resolved, and users
thus wouldn’t have to import anything to use the new derive. This is what makes e.g. #[derive(Hash)]
work even if you don’t have the std::hash::Hash trait imported.
However, adding new things to the prelude is always tricky. Consider this code:
mod foo {
pub use derive_more::From;
}
use foo::*;
#[derive(From)] // ERROR: `From` is ambiguous
struct S(u32);
The code uses a glob import to import a macro named From from the foo module. After my PR landed,
this module also received another macro named From (the one from the standard library), which was
imported with a glob import from the std prelude, which caused an ambiguity. In other words, my change
broke existing (stable) code, even without people using the new feature. That is obviously Very Bad.
To fix this, I had to remove the From macro
from the standard library prelude, and thus for now, it will have to be imported explicitly. I added
the macro to the std::from module; it had to be added to a new module, because no previously
stable Rust code could have had a previous import from a module that did not exist, and thus this
should not break anyone. It is a bit confusing though that the From macro is in std::from, while
the From trait is in std::convert.
To be honest, I don’t know if there is a way to resolve the import ambiguity without an edition bump, so maybe this feature will have to wait until the next edition before it can be used ergonomically. Maybe #139493 might help with issues like this in the future.
It is interesting to note that if the code looked like this instead:
pub use derive_more::From;
#[derive(From)] // Uses `From` from `derive_more`
struct S(u32);
Then there would be no ambiguity, because the explicit import from derive_more would be prioritized
before the From glob import from the std prelude. But having two macros imported both with a
glob import (with one coming from the prelude) causes a problem.
To experiment with this feature, update to a recent nightly release of the compiler, and use #[derive(From)]
on a struct of your choice. Note that it will only work on structs with
exactly a single field, as mentioned before! You will also have to import the macro explicitly,
because of the ambiguity problem described above.
#![feature(derive_from)]
use std::from::From;
#[derive(From)]
struct Foo(u32);
fn main() {
let f: Foo = 1u32.into();
}
You can try the feature and let me know if it works fine for you! Note that I don’t expect that many things could break here, provided that there won’t be any further name resolution errors…
As I already discussed above, there are various improvements and extensions that we could do with
#[derive(From)]. But even more generally, I think that we should make it possible to derive
as many standard library traits as possible. It should be easily possible to generate impls of
AsRef, Deref or even traits like Iterator, as long as you do it on structs with a single field.
If/once #[derive(From)] becomes stabilized, I might try to write RFCs to also support these other
traits, to reduce even more boilerplate.
I hope you enjoyed my adventures with writing an RFC and designing and implementing a Rust feature!
You can let me know what you think about #[derive(From)], and which other traits would you like to
#[derive], on Reddit.
In HyperQueue, we have a struct describing parameters of a queue (for the purpose of this post, it doesn’t really matter what a queue is), which contains a bunch of fields. Something like this:
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct QueueParameters {
pub manager: ManagerType,
pub max_workers_per_alloc: u32,
pub backlog: u32,
pub timelimit: Duration,
pub name: Option<String>,
pub max_worker_count: Option<u32>,
pub min_utilization: Option<f32>,
pub additional_args: Vec<String>,
// ... more fields
}
This struct is just (Plain Old) data, a bag of fields that can be (de)serialized, sent over the network, etc. However, at some point in the program, I want to store these parameters in a way that would prevent other parts of the codebase from modifying them. Of course, I could do that easily by making its fields private, but that would prevent me from using the struct literal syntax for creating values of this struct outside the module (and its children) that defines it.
In short, I want to be able to both:
QueueParameters { manager: ..., backlog: ... })
anywhere in the codebase, so that I can see which fields are initialized with what values (as we
don’t have named parameters in Rust).It would be nice to have something like “init-only” fields, but as far as I know, Rust doesn’t really have anything like that today. So I had to use some workaround. A common approach to sort of emulate named parameters in Rust is to use the builder pattern, but that seemed like overkill to me in this use-case, and its syntax would be more verbose than the normal struct literal syntax anyway. I also didn’t want to complicate the matter with any kind of macro magic.
My original “solution” to avoid the field mutation was to introduce a separate struct that has the
same fields as QueueParameters (let’s call it ReadOnlyQueueParameters),
just with the fields being private, so that they cannot be modified from the outside after initialization.
Because the fields were private, I also had to create a constructor function ReadOnlyQueueParameters::new,
which received values for all the fields. Of course, this meant that I couldn’t use the nice struct
literal syntax anymore, which made the initialization quite awkward, as there are a lot of fields:
let params = ReadOnlyQueueParameters::new(
ManagerType::Slurm,
1,
4,
Duration::from_secs(3600),
None,
Some(10),
None,
...
);
This code was bothering me for a long time, because everytime I had to add a new field to the
struct, I would have to extend this new function with yet another parameter, and the call-sites
would become ever more confusing. This is how such a change would typically look as a diff:
- None,
- None
- )
+ None,
+ None,
+ None,
+ )
Yeah, not great.
After a few years of working with the annoying read-only wrapper, I finally lost my patience with it and decided to refactor it. I quickly realized that the behavior I want can be achieved in a very straightforward and elegant way by not duplicating the fields, but simply storing the mutable struct directly as a field inside a read-only struct. The read-only struct would then expose the fields of the mutable struct in a read-only way:
struct ReadOnlyQueueParameters(QueueParameters);
impl ReadOnlyQueueParameters {
pub fn new(params: QueueParameters) -> Self {
Self(params)
}
pub fn backlog(&self) -> u32 {
self.0.backlog
}
}
With this approach, I can construct the parameters using the struct literal syntax:
let queue = ReadOnlyQueueParameters::new(QueueParameters {
manager: ...,
backlog: ...,
...
});
But at the same time, I cannot modify the fields from the outside after construction anymore.
Truly a “we have init-only fields at home” solution. I could even remove the accessor methods by
implementing the Deref trait for the wrapper, but here I decided to keep the code more explicit.
This is, of course, yet another instance of the newtype pattern, which I use all the time, and even explain it to my students. And the solution is really quite trivial in hindsight. So it was a bit embarrassing to me that it took me so long to realize that it can be used to solve this annoyance. Although frankly, the main issue was to find the motivation to spend five minutes to refactor code that was bothering me for years :)
For these simple newtype wrappers, it would be nice to be able to simplify the implementation of the
Fromtrait, to go from the inner field to a value of the newtype. More about that soon.
Anyway, that’s all. The newtype pattern just keeps on giving! Do you have other useful use-cases for it? Let me know on Reddit.
]]>Recently, in addition to the usual compile times complaints, I also started noticing the following sentiments being expressed by frustrated Rust developers: “Why doesn’t the Rust Project care more about this pressing and known issue? Why don’t they do something about it?”. As a member of the Rust compiler performance working group, I take these questions very seriously, and I of course have some opinions on the topic. In this blog post, I would like to offer some thoughts that might serve as answers to these (and other similar) questions.
Disclaimer: all opinions expressed in this post are mine only and do not necessarily represent the broader views of the Rust Project (the diverse set of contributors and maintainers that contribute to the Rust toolchain).
First, let me assure you - yes, we (as in, the Rust Project) absolutely do care about the performance of our beloved compiler, and we put in a lot of effort to improve it. We triage performance improvements and regressions each week. We run a comprehensive benchmark suite after every merged PR. We enthusiatically welcome any performance improvements (unless they have complicated trade-offs, more on that later), and try to quickly revert (or fix) PRs that introduce performance regressions. Very smart people are continously working on finding bottlenecks and speeding up the compiler. And some significant improvements to make compilation faster by default are currently in the works.
And all this effort shows results; Rust build performance has improved quite a lot over the past few years! When we talk about the long-term trends, we usually show this dashboard, but I find it a bit dull, because it averages results of several benchmarks together and the benchmarks are quite short, so they start running into diminishing returns. Instead, I did a little experiment to show how did the build performance evolve on my favourite test subject, hyperqueue. I took its first commit (from March 20211) and compiled it with a few different Rust compiler versions, with ~1 year increments between them2. I did this on a relatively modest laptop with 8 AMD cores running Linux. Here are the results:
rustc version |
Clean build [s] | Incremental rebuild [s] | Speedup (clean build) |
|---|---|---|---|
1.61.0 (May 2022) |
26.1s | ~0.39 | - |
1.70.0 (June 2023) |
20.2s | ~0.37 | 1.29x |
1.78.0 (May 2024) |
17.0s | ~0.30 | 1.53x |
1.87.0 (May 2025) |
14.7s | ~0.26 | 1.77x |
On this benchmark, the compiler is almost twice as fast than it was three years ago. Of course, this is just a single data point, in different cases the speedup will be larger or smaller, and it will likely be smaller in general on other platforms than x64 Linux, because we spend the most effort optimizing that specific target. But the fact remains that the performance of the compiler is continuously improving.
Now, all that is nice, but it doesn’t change the fact that for many Rust developers, compilation performance is still a big bottleneck, and they could be much more productive if the feedback loop was an order of magnitude (or more) faster. To be fair, this will depend on who you ask, e.g. some C++ developers don’t mind Rust’s compilation times at all, as they are used to the same (or worse) build times, while Python programmers likely aren’t very impressed by Rust’s feedback loop speed. But it’s definitely not a stretch to say that for many users, Rust compilation just isn’t fast enough today, and that is a problem.
Before we examine that problem in more detail, I think that it’s interesting to think about whether the problem is even fundamentally solvable, and what is the end goal that we would eventually like to achieve. Can Rust ever hope to achieve near-instant (re)build speeds, with its complex type system, borrow checking, monomorphization, proc macros and build scripts, large translation units, machine code generation and a “rebuild the world from source” compilation model? When it has a historical track record of favoring runtime (rather than compilation) performance at nearly every opportunity?
I think that it depends on the use-case. If we’re talking about building projects with hundreds of dependencies from scratch or doing super-duper LTO-optimized release builds, I think that we won’t ever get to a point where the build will be truly instant. On the other hand, I personally believe (although others might disagree!) that it is not fundamentally impossible to have a Rust compiler that will be able to almost instantly rebuild a Rust project of nearly any size in incremental (and at least moderately optimized) mode after making a small change to the source code, one where rebuilds are truly O(number of performed changes) and not O(size of the codebase). We might have to make some trade-offs, particularly around the resulting runtime performance, but I think that we can get there. In fact, we can even see examples that can already achieve something akin to this (in somewhat limited scenarios for now)
today!
And it’s not like we don’t have some ideas on how to get (at least closer) to this goal. There are several “northstar” approaches for speeding up the compilation process in various ways, like the parallel frontend, alternative codegen backends, defaulting to a faster linker, deferring code generation (MIR-only rlibs, -Zhint-mostly-unused), avoiding useless workspace rebuilds, having smarter incremental compilation (including incremental linking or even binary hot-patching), and many others.
Some of these approaches are not prepared for wider usage yet, but others are, and you can opt into them today. They usually (but not always) help. I believe that if we had a magic wand and we could land all these changes today, we could get very snappy incremental rebuilds for a large (certainly much larger than today) fraction of Rust projects.
And it’s not like “only” the broader Rust community would benefit from this. Improving build performance makes Rust toolchain contributors themselves much more productive. It reduces the time required to build the compiler itself when making changes to it, it speeds up our CI workflows, reduces the time that we have to wait for tests and compiler performance benchmarks… Everyone would benefit! So what’s blocking us from making progress faster? WHy dON’t We dO SOmEtHIng ABoUt iT??!3
This is a tough question, one that doesn’t have a simple answer. I will try to provide a variety of reasons (in mostly arbitrary order) why I think we have not been making progress faster, based on my personal experiences working within the Rust Project.
First, let’s start with the technical reasons, which are in a way the most straightforward. Simply put, making non-trivial improvements to the performance of the Rust compiler is hard! It is a large codebase4 that has a lot of technical debt risk, as probably all large (compiler) codebases do, and although we provide a guide for contributing to it, it still takes time before you can understand what’s going on. In fact, there is probably not a single person that understands the whole compiler codebase (maybe except for compiler-errors :laughing:).
While it is possible to make performance improvements even without deeply understanding how the compiler works, by profiling and micro-optimizing its various parts, this approach can only get you so far, and most of the low-hanging fruit has already been picked5. A lot of things have been optimized and overfitted to reach a “local performance minimum” on a variety of benchmarks and use-cases, so even if an improvement is made for some use-cases, it often regresses something else, which can lead to that localized improvement not being accepted.
We also have to consider various trade-offs when deciding whether to land certain performance optimizations. For example, we know a few tricks that could make the compiler a bit faster on x64 Linux, the most popular target. We could compile rustc with support for “newer” instruction architectures, e.g. AVX256, which seems to provide some wins. However, that would mean that the compiler would no longer work on older x64 CPUs that we still officially support! Or we could use a different memory allocator (e.g. mimalloc), which also seems to provide some modest performance wins, but at the cost of increased
memory usage, which could
cause rustc to go OOM on devices with smaller amounts of RAM. Maybe you’re thinking why can’t we just ship multiple versions of the compiler (for the same target) with different optimizations and then let people (or rustup) choose which one to use? Well, building the most optimized variant of the compiler for Linux already takes an enormous amount of CI resources right now, not to mention that distributing the toolchain in multiple variants could balloon the amount of data we ship to users over the network, which is already gigantic. There are simply trade-offs everywhere.
I think that to achieve truly significant build performance wins, there are mostly two ways forward. The first one, which I find highly promising, is improving certain compilation workflows. We don’t necessarily have to make the whole compiler faster if we can vastly improve the performance (or reduce the amount of work required) of specific workflows that currently limit the productivity of a large fraction of Rust developers. An example of this is the Relink, don’t rebuild proposal, which has the potential of reducing the amount of compilation that has to be performed when you modify a crate in a large Cargo workspace. Approaches such as this one do not necessarily make the compiler faster per se, rather they make the compilation process smarter, which is an area in which rustc could be improved a lot. Many of these potential workflow improvements are also not strictly tied only to the compiler itself, but they happen at
the intersection of rustc and the used build system (most often Cargo).
I hope that some of these ideas will make the most common Rust compilation workflows much faster in the short-to-medium term, without requiring massive changes to the implementation of the compiler. Nevertheless, even such targeted optimizations are not easy to land. Achieving an impressive performance win with a proof-of-concept is the “easy” part. But then comes the long tail of the really hard work - making sure that your change supports all the edge-cases and quirks of the Rust compilation process, works on all targets, doesn’t regress important use-cases and benchmarks, is not a maintenance nightmare, keeps backwards compatibility, etc.
The second way forward, as you might have guessed, is… performing massive changes and/or refactorings to the implementation of the compiler. However, that is of course challenging, for a number of reasons. First, you need to do a “vibe check”6 with the rest of the compiler team with a Major Change Proposal, to make sure that other compiler maintainers are on board with you performing such a major change. Then you will obviously need to implement these changes, which can take a lot of effort. For example, if you change one thing at the “bottom” layer of the compiler, you will then have to go through hundreds of places and fix them up, and also potentially fix many test cases, which can be very time-consuming7. You will also need to find reviewer(s) with bandwidth to approve your changes, which can span over many PRs and weeks, months or even years of work.
Performing large cross-cutting changes is also tricky because it will necessarily conflict with a lot of other changes being done to the compiler in the meantime. You can try to perform the modifications outside the main compiler tree, but that is almost doomed to fail, given how fast the compiler changes8. Alternatively, you try to land the changes in a single massive PR, which might lead to endless rebases (and might make it harder to find a reviewer). Or, you will need to do the migration incrementally, which might require maintaining two separate implementations of the same thing for a long time, which can be exhausting.
An example of a large internal change (albeit one that is not directly related to performance) is the reimplementation of the compiler’s trait solver. Even though it is being worked on by very smart and incredibly skilled people, this is a process that takes several years to finish. That’s the scale that we’re dealing with if we would like to perform some massive refactoring of the compiler internals, and it should set the expectations for people that ask us when are we finally going to rewrite the whole compiler using Data-oriented Design to make it a bazillion times faster.
Speaking of DoD, an additional thing to consider is the maintainability of the compiler codebase. Imagine that we swung our magic wand again, and rewrote everything over the night using DoD, SIMD vectorization, hand-rolled assembly, etc. It would (possibly) be way faster, yay! However, we do not only care about immediate performance, but also about our ability to make long-term improvements to it. If we overfitted the performance using code that is hard to evolve, understand, debug or maintain, it wouldn’t serve us well in the long term. The compiler receives contributions from hundreds of volunteers, and we want to keep the codebase (at least somewhat) approachable to them. So the long-term maintaintability of the codebase is also something that we need to keep in mind.
While the challenges presented above are very real, they are of course not exactly unique to rustc. The fact that making major changes to a large (compiler) codebase is difficult is probably not very surprising. I’m sure that GCC/LLVM/Clang maintainers also have a lot of ideas on how to improve or speed up things, but it takes a lot of effort and time to get these done. But there are also other reasons why compiler performance is not the only thing we focus on.
Even though many Rust users seem to want compiler performance to be the primary priority of the Rust Project, we should not forget that there are also other priorities that we have to keep in mind.
The Rust compiler is (at least in my opinion) very stable and dependable, and I can count on a new version being released every six weeks. That is no small feat! And we should not take it for granted. It does not happen simply by starting from a working state and then not doing anything. There is so much work that goes into simply making sure that things are not on fire, our infrastructure and CI works, really bad bugs are being fixed in a timely manner, incoming issues are being triaged, backwards compatibility is still being upheld, PRs are being reviewed, security issues are being quickly resolved, we stay on top of changes in external dependencies, such as LLVM, and much, much more. And we have to ensure all of that works on all of our supported targets! At this moment, that’s 8 Tier 1 targets (those have to build and all tests have to pass) and 91 (!) Tier 2 targets (those have to at least build). By a large part, that work is being done by volunteers who care so much about Rust and its toolchain (:heart:) and want to keep it working as well as it was so far. There is simply a ton of work to do on Rust9.
And all that work is of course eating away from time that could be spent e.g. on making the compiler faster. If you do not consider that work to be all that important, ask yourself if you would enjoy a twice faster compiler if it randomly started miscompiling your code :)
Another thing is that both the language and the compiler are getting new features all the time. New compiler flags to support projects like Rust for Linux, syntactical improvements, complex language features, and so many other things. In general, the language seems to be far from “done”, and so many things are in process; there are almost 200 open RFCs pull requests, and many other RFCs were accepted, but haven’t been implemented or stabilized yet. At any given point, we have ~700 opened PRs on the main Rust repository, we have over 10 thousand open issues, there’s just so much going on.
And it seems that Rust users want Rust to evolve as fast as it does today, or even faster, which means more changes! When was the last time you “just wanted this small feature X to be finally stabilized” so that you could make your code nicer? It is perfectly fine to want new features from a language, but we should not forget that (amongst other things, such as design discussions) it requires a lot of experimentation, refactoring, bug fixes, testing and implementation effort in the compiler, which again takes time from optimizing performance. Not to mention that adding new features usually makes compiler performance slightly worse, as it tend to do more work, rather than the opposite.
In the end, the amount of effort that gets spent on optimizing the performance of the compiler ultimately depends on the people who contribute to it and maintain it. They are often volunteers, and they have diverse interests. Some (compiler) contributors are not interested in doing performance optimizations at all, and that is perfectly fine! Remember, Rust is not a company (this is a great blog post by Mara Bos, I recommend you to go read it!). We don’t tell people what they should work on, nor do we assign them tasks or tickets. If there won’t be people interested in making the compiler faster, it mostly likely will not get done, it’s as simple as that.
That being said, there are certain ways of how we can empower people to work on compiler performance. The Rust Project now has the Project Goals program, which specifies a set of goals and projects that we consider to be important, and we explicitly welcome contributions (and promise to provide reviews and support) in these areas. I’m going to try to make compiler performance be one of the flagship goals in the near future, as I think that it really deserves to be one. That could perhaps help motivate more people to work on it.
Even though it may sound a bit banal, making progress faster is also (at least partially) a matter of funding. Sometimes I hear people complaining about certain projects designed to improve compilation performance taking too long to be ready. It sounds like they expect that there is a fully staffed team of well-paid engineers working on these projects. Well, we should not forget that the Rust Project is still mostly built on volunteers, and many people contribute to it in their free time. Many improvements (including some important compiler performance projects) to Rust were literally driven by a single university student who works tirelessly to make Rust better day by day, without maybe even getting paid for it in any way, and the reason why progress pauses can be (for example) that they must finish university exams or do some actual paid work so that they could afford rent and food.
During the recent RustWeek conference and the associated All Hands event, I saw an increased interest from various companies to invest into improving the performance of the Rust compiler, which is great! Improving the performance of rustc requires long-term concentrated effort, and providing stable funding for contributors doing that work is definitely one of the ways of how we could make progress faster. However, it is also important that making improvements is not just about doing the implementation work; someone will also need to review it, and then maintain it in the future. In an ideal world, companies would fund long-term ongoing work on the maintenance of the compiler, rather than paying their employees to implement an optimization that scratches their itch and then disappearing. Sometimes, the best way to help something move forward in Rust is not to implement it, but gather knowledge, become an expert in some compiler area, and help review the code of others, so that you can unblock other people to do the actual work :) But that of course takes a long-term investment (both in terms of time and money).
Based on my experience, the interest amongst Rust compiler contributors to work on compiler performance does not correspond to the amount of discourse this topic gets online (and again, this is fine!). If I used the magic wand again, and could persuade more contributors to care about it, would I do it? Probably yes. Would it be a good thing for Rust, long-term? I don’t know! So far, the development of the language and compiler seems to be going fine, and who am I to determine priorities. I trust the people in the Project to collectively do what’s best for Rust, both today and in the future.
One thing that I find interesting is that maybe more people would actually do performance work if they didn’t feel the need to work on so many other things at the same time. Case in point: when I started contributing to Rust back in 2021, my primary interest was compiler performance. So I started doing some optimization work. Then I noticed that the compiler benchmark suite could use some maintenance, so I started working on that. Then I noticed that we don’t compile the compiler itself with as many optimizations as we could, so I started working on adding support for LTO/PGO/BOLT, which further led to improving our CI infrastructure. Then I noticed that we wait quite a long time for our CI workflows, and started optimizing them. Then I started running the Rust Annual Survey, then our GSoC program, then improving our bots, then… Many of these contributions did (at least I hope) affect the performance of compiler in some indirect way, either by unblocking or supporting other contributors or improving our infrastructure (in various meanings of that word). However, only now, after a few years of getting sidetracked, I’m finally trying to get back to doing direct compiler performance work again.
In other words, the more help we can get with maintaining Rust (regardless in which specific area!), the better the chance that we will collectively have more time to improve the performance of the compiler :)
Funnily enough, after I wrote this post, I realized that I could probably replace the term “compiler performance” with pretty much anything else that people want from Rust (“why doesn’t Rust do X”), and a lot of the same reasoning would still apply. In this regard, compiler performance is not that special, it’s just one of (so) many (very important!) things that we care about and that we try to improve to the best of our ability.
In terms of future compiler performance work, I’m hopeful about several initiatives that were discussed at the All Hands events, and also about making LLD the default linker on Linux, which will hopefully finally land in the upcoming months. I’m also planning to run a compiler performance survey soon, so that we can find out what workflows cause bottlenecks for Rust users, and I also want to build better infrastructure for profiling what the compiler is actually doing during the compilation process.
I hope that this blog post provided a bit of insight into why making progress in build performance is not so easy. If you’d like to join the fray and help with it, we would be happy to see you! Let me know what you think on Reddit.
I used an old commit so that it can be compiled even with old(er) versions of rustc. ↩
Luckily, I still had all the stable compiler versions downloaded from my recent compiler errors experiment :sweat_smile: ↩
Sorry, couldn’t resist. ↩
At least for some definition of large. The compiler consists of ~600k lines of Rust code, and the standard library has a similar size. ↩
I recently saw someone call this vibeck (named after the abbreviated terms typeck and borrowck used in the compiler), which I find super cute. ↩
If you are thinking “why can’t AI do this?”, it probably could help with a lot of things in this area, but especially differences in compiler diagnostics (which is a large part of what our test suite checks) often come down to judgement calls that we still (luckily) leave to humans. ↩
I like to say that if you gave me a hundred full-time engineers, I would find all of them something to work on in the Rust toolchain immediately. ↩
target directory is a commonly cited annoyance with Rust (and Cargo)
– in the last year’s Annual Survey,
it was the third most pressing issue of Rust users, right after slow compilation1 and subpar
debugging experience. Given the “build everything from source” compilation model of Rust, and both debuginfo and incremental compilation being enabled by default in the dev Cargo profile, it is unlikely that the target directory will ever become lean and small. However, there are still ways of how we could reduce the target directory size by a non-trivial amount. I will describe a brand-new method of achieving that in this blog post.
Note that there are also initiatives to reduce the size of the target directory along the temporal axis, i.e. prevent it from ballooning over time (see Cargo Garbage Collection). This post is more about how to reduce the size of the target directory overall.
It is not the focus of this post to dive deep into exploring what exactly takes space in the target
directory, but I still think that it is useful to provide at least some background on this. I took
my favourite work project, hyperqueue, and compiled
it in several modes to compare the resulting target directory sizes2, using rustc 1.89.0-nightly (2eef47813 2025-05-22):
| Optimizations | Incremental | Debuginfo | target size [MiB] |
|---|---|---|---|
No (dev) |
No | No | 462 |
No (dev) |
Yes | No | 677 |
No (dev) |
No | Yes | 870 |
No (dev) |
Yes | Yes | 1316 |
Yes (release) |
No | No | 396 |
From the results, it’s clear that both debuginfo and incremental compilation caches take a lot of
space on disk. It might be that we could do something to reduce either of these, but there is one
additional thing that (kind of unnecessarily) causes target directory bloat, which is not
easily observable in this table, and that is the metadata of the compiled Rust crates.
First, we need to briefly talk about how Rust compilation works (in very, very simplified terms).
When a Rust library crate (rlib) is compiled in a “standard” way (i.e. no LTO or other funny
business) with Cargo, the compiler generates (amongst other things) two main outputs:
The interesting thing about this is that the final object code isn’t actually needed in order to start
compiling Rust code that depends on a said library, metadata is enough (the object code will only be
needed at the final linking step). Cargo makes use of this fact, and when it compiles your crate graph,
it uses a technique called pipelining.
Imagine that you have a binary that depends on crate B, which itself depends on crate A. When
compiling A, Cargo will tell the compiler (using --emit=metadata,...) to emit a .rmeta file
containing the metadata of A as soon as possible. Once that file is ready, we can start compiling
B by passing it the .rmeta file of A, even though the object code of A is not ready yet,
and thus partially overlap the compilation of both crates, which improves compile times. I will borrow
(and slightly extend) a cute ASCII diagram from Alex Crichton’s internals post about pipelining
to succintly show how it works:
--- .rmeta of A generated
v
[-libA----|--------]
[-libB----|--------]
[-binary-----------]
0s 5s 10s 15s 25s
Pipelining has been enabled in Cargo for many years, so it’s nothing new, and if you use Cargo, you are using pipelining every day, perhaps without even knowing about it. However, there is one suboptimal thing in regard to disk space usage with the pipelining approach.
The problem is that once the process finishes, you will be left with two copies of the metadata of
each library on disk. The first copy is in the final .rlib file (along with the object code), and
the second copy is in the .rmeta file. As is usual with Cargo, this problem will balloon if you build
your project with multiple configuration options (different Cargo profiles, different compiler/linker
flags, etc.).
A related issue is that if you are compiling a library as a dylib,
so that you generate a Rust dynamic library (e.g. .so on Linux), that will also contain the
metadata, even though it should not be required to actually consume (call functions from) the
library at runtime. This can be annoying for people who ship Rust dylibs, even though that’s
usually not very common due to Rust currently not having a stable ABI.
So, what can we do about it? Well, last year, I noticed that many years ago, the illustrious @bjorn3 has
proposed an idea to introduce a compiler flag
that would cause the metadata to only be included in the .rmeta files, and no longer be present
in the .rlib/.so library files. I liked this idea, and since there already was a prototype
implementation prepared, it didn’t seem that hard to push it over the finish line. So I
nerd-sniped myself into finishing the implementation of the flag under bjorn3’s mentorship.
I didn’t actually make a lot of progress on it last year (because of… stuff),
but this year I finally filed an MCP, which is the usual process
for introducing new compiler flags, refactored the implementation a bit, added tests, and got it
merged. So long story short, you can now use an
unstable compiler flag in nightly Rust called -Zembed-metadata=no, which avoids the metadata
duplication.
When the flag is used, the .rlib file will only store a “metadata stub”, which contains
the bare minimum of information necessary for the .rlib to be loaded and validated, and the rest
of the metadata will be stored in a .rmeta file.
Normally, when a new unstable compiler flag is added, people can experiment with it using RUSTFLAGS.
However, the way this flag works means that you kind of need to combine it with --emit=metadata,
otherwise there will be no metadata generated at all, which would not be good. Furthermore, you
will now also need to pass the paths to the .rmeta files to the final rustc invocation that links
the top-level binary/library, otherwise the metadata will not be found. This essentially means that
this feature also needs some changes within Cargo, otherwise it wouldn’t be really usable with it.
So I went and implemented support for this rustc flag in Cargo, and exposed it via a new
unstable Cargo flag called -Zno-embed-metadata.
Here are results of building hyperqueue using -Zno-embed-metadata3:
| Optimizations | Incremental | Debuginfo | Before [MiB] | After [MiB] | Reduction |
|---|---|---|---|---|---|
No (dev) |
No | No | 462 | 336 | -27.3% |
No (dev) |
Yes | No | 683 | 558 | -18.3% |
No (dev) |
No | Yes | 874 | 748 | -14.4% |
No (dev) |
Yes | Yes | 1325 | 1199 | -9.5% |
Yes (release) |
No | No | 397 | 253 | -36.3% |
As you can see, the benefits are most notable when building in release mode, or in general without
debuginfo or incremental compilation, as their size will usually dwarf the duplicated metadata contents.
Nevertheless, the disk space savings are quite nice, in my opinion.
Originally, I was also hoping that this could speed up compile times a little bit, as less data has to be written to disk, but from my experiments it seems that the effect is rather miniscule, at least on a Linux system with an SSD disk.
One of the reasons why I wanted this flag to exist was to slightly reduce the size of the distributed
Rust compiler toolchain, particularly the standard library .so file. I didn’t get a chance to do
a full benchmark yet, as it’s a bit annoying to dogfood a Cargo feature, because I have to wait
until the Cargo change reaches the beta channel, so that I could use it for compiling the compiler
itself (the first stage of rustc is compiled using the beta compiler/Cargo), which will take
several weeks.
However, I did some initial local tests, and it seems that using this flag reduces the size of the x86_64-unknown-linux-gnu
standard library .so file from ~13 MiB to ~3 MiB, which seems nice.
I think that unless we find some major issues, we should make the -Zno-embed-metadata behavior
the default in Cargo, to reduce the disk space usage of the target directory for everyone.
Currently, it seems like it might be considered to be a backwards compatibility break though, as the Cargo
team is unsure if some people weren’t relying on the metadata being present in the .rlib
files. In general, it’s quite tricky to determine whether something is a breaking change in Cargo or
not, if it hasn’t been previously documented (Hyrum’s Law is ever-present).
In terms of how to technically perform the migration, I think that we could use the new behavior by default
on the nightly toolchain for some time4 to find potential issues in the wild,
and flip the Cargo flag to allow opting out of (instead of opting in) the new behavior
(so it would essentially become -Zembed-metadata).
If you are interested in the future of this feature, you can observe its status in the rustc and Cargo tracking issues.
I would be interested in how this flag fares in real-world scenarios, whether it causes any issues,
and how much disk space it can save. If you would like to try it out, use a recent rustc and Cargo
and build your project using e.g. cargo +nightly build -Zno-embed-metadata. You can use e.g.
this script as an inspiration for
benchmarking the effect of the flag.
If you try it out, please let me know about your results on Reddit!
Funnily enough, making compilation faster can sometimes increase disk usage; for example, the incremental system of the Rust compiler stores a lot of data on disk, which helps it to make subsequent recompilations faster. ↩
I compiled it with --no-default-features to avoid compiling the C jemalloc dependency, as that takes a lot of space on disk. ↩
The full command was cargo +nightly build --no-default-features -Zno-embed-metadata. ↩
This blog post is essentially a PSA on how you can easily improve dev incremental rebuild performance.
At RustWeek, we led many discussions on compiler performance, and what we can do to improve it. One of the discussion points in this area was the generation of debug information (debuginfo), which happens by default in the Cargo dev (unoptimized) profile. Now, I suppose that it’s not surprising that generating debuginfo is not free and costs some compilation time. But I’m not sure if people actually realize just how expensive it currently is, especially for incremental (re)builds.
Here are results of a run of the compiler benchmark suite where I forcefully disabled debuginfo generation. In some situations, especially for incremental rebuilds, disabling debuginfo can make your compilation 30-40% faster! And this benchmark was already performed with the fast lld linker, which is not used by default on the x64 Linux target (yet); I expect that with the default Linux linker (BFD), the improvements would be actually way higher, because linking debuginfo can be quite slow. You won’t get such a large improvement in all situations, of course, but it can definitely be worth it to consider disabling debuginfo.
There are essentially two ways how we can improve this situation. We could make debuginfo generation faster, as I suspect that it is currently quite suboptimal, particularly in incremental builds, or we can avoid generating it by default. But until either of that happens, you can speed up your (unoptimized) incremental rebuilds today by disabling debuginfo in your Cargo.toml file:
[profile.dev]
debug = false
Note that this will also remove source code lines from stack traces, making them much less useful. If you want to keep that information, you can use debug = "line-tables-only" instead, which is a compromise between full and no debuginfo. It still provides a pretty nice speedup vs the default (full) debuginfo.
I personally think that generating debuginfo by default is a bit wasteful. Based on various polls and survey results that I saw, many Rust programmers simply don’t use a debugger at all. But even if you do use it, I would estimate that you don’t end up actually running a debugger on a large fraction of your builds, even though you do pay the debuginfo generation cost for every build (by default).
I’m planning to work on removing debuginfo (or changing it to line-table-only) from the default dev profile in Cargo, but it will take some design work and time (and it’s unclear if it will actually happen). We should probably also make debuginfo generation faster (and smarter for incremental builds) in the compiler.
Removing the debuginfo generation is actually the only optimization that I do for the stable compiler in the cargo-wizard tool; other approaches designed to speed up compilation (such as the parallel frontend, the Cranelift backend or using a different linker) are either unstable or cannot be easily configured directly through
Cargo.toml.
If you have any thoughts on this, let me know on Reddit!
]]>I recently attended RustWeek (which was totally awesome) and the talks by Alex Crichton (on the history of Rust) and Pietro Albini (on the importance of error messages) inspired me to do a little archaeology into the way Rust compiler messages have evolved over time.
I wrote a script that downloaded all stable Rust releases all the way back to 1.01, executed each stable version of the compiler on a set of small programs containing an error and gathered the compiler standard (error) output.
The widget below visualizes how the error messages evolved over time. You can use the select box to examine different Rust programs to see their error:
There are a couple of interesting things to note:
1.0.0 contained pretty solid error reporting, and it got much better over time.1.2.0 introduced numerical error codes. Edit: they were actually added in 1.0.0, just that this specific error did not have a code until 1.2.0, as was pointed out to me by Steve Klabnik.1.26.0 introduced colorful error messages. It sounds like a small change, but you can see what an improvement it makes! It also added the rustc --explain <error-code> hint.error: aborting due to 2 previous errors has switched to ...previous error(s) in 1.19.0 and then back to ...2 previous errors in 1.20.0, which seems like an unintended change in 1.19.0. Sometimes the difference is only in a single space, which is not even visible in the visualization above.Wrong field program change in 1.87.0.But I think that ultimately, the most interesting thing about this is the evolution process of these messages itself, which demonstrates that a lot of effort has to be put into the messages to make them really good. To someone, it might seem like these messages are somehow automatically derived from the compilation process, and we get them “for free”, but that couldn’t be further from the truth. It is the result of a continuous design, implementation, review and testing effort that has been performed by hundreds of individual contributors over the span of more than ten years. Thank you to everyone who has worked on the Rust compiler and contributed to these awesome error messages!
If you’d like to test this out on more programs than the ones I have shown here, you can check out my script here2.
If you have your own favourite example of a Rust compiler error message, share it with others on Reddit!
This is actually possible to do just with rustup (at least on x64 Linux), even though some of the oldest releases actually predate rustup. How cool is that? ↩
For a moment I thought about making the widget interactive, so that you could write any Rust code and see the evolution of the compiler errors for it directly in the browser, but that would essentially mean that I would have to reimplement the Rust Playground for almost a hundred different compiler versions, which seemed… too much for a blog post :) ↩
GOTO 1., with the fix/refactor becoming the “new feature” that you’re now implementing.This happens to me all the time. After a few iterations of this cycle, where I repeatedly keep starting new and new refactorings, I usually end up with a git workspace that contains a ton of unrelated changes. And since I usually code in Rust these days, it’s also very likely that my project does not even compile, because I started (but did not finish) a lot of changes. At this point I sometimes used to just give up, git checkout . the whole thing and start from scratch, to avoid having to dig myself out of the mess and cleanly separate the unrelated changes into individual commits.
If you also sometimes fall into these endless refactoring cycles, I found a pretty simple workflow that makes it easier to untangle them (at least for me). Everytime you notice something suboptimal in the codebase that is not directly a part of what you’re currently implementing and that you want to “just slightly refactor”, use git stash to stash all your current changes away, and start working on the refactoring that you just thought of. If you encounter another thing that should be refactored or fixed during that, apply the workflow recursively - git stash your changes away and start working on the latest thing that you have in mind. After you finally get to a change that you can finish from start to end, commit it, and then restore the previous state with git stash pop and continue onwards. With this approach, the changes are effectively applied “inside-out”.
It’s nothing ground-breaking, of course, but I feel that this workflow really helps me to focus on a single thing at a time. I don’t have to consider the uncommitted changes I made to something unrelated previously. I know that I can always purge all workspace changes without worrying that I will remove work on the previous feature that spawned this refactoring. And most importantly, I do not have to think about the previous in-progress work. This is similar to how using asserts in code help me to avoid thinking about certain possibilities and code paths, as I know that they cannot happen1.
Of course, if the individual refactorings are too unrelated, you might want to merge them in separate PRs, so sometimes it makes sense to throw in a bunch of cherry-picks or interactive rebases to move the work to a different branch and merge it separately. But the main thing is to avoid working on multiple things at once, to avoid getting distracted, which is what git stash is really useful for.
By the way, here are two small tips related to git stash:
git stash -- <path> or git stash push <path>-m, to make it easier to recall what you stashed away a week ago :)If you have any comments, let me know on Reddit.
Without triggering an assert anyway. ↩