![\"A](\"media/kebab-king-duplicates.png\")
In case you missed the announcement,\nthe reqwest crate has a new and very important release out!\nreqwest is an opinionated, high-level HTTP client for Rust,\nand the main feature of this release is that rustls\nis now the default TLS backend.\nRead the excellent blog posts from Sean and others on why rustls\nsafer and often faster than native TLS.\nIt's also a lot more convenient most of the time!
This post is about one of the more mundane parts of the release.\nPreviously there were a lot of somewhat confusing features related to certificate verification.\nThese have been condensed down to a smaller number of feature flags.\nThe summary of these changes took a bit to "click" for me so here's a rephrasing in my own words.
\ntls_certs_merge.tls_certs_only\nto limit verification to only the certificates you specify.The documentation and release notes also mention that tls_certs_merge is not always supported.\nI frankly have no idea what conditions cause this to be supported or not.\nBut tls_certs_only apparently can't fail. ¯\\_(ツ)_/¯
The reason I'm interested in this in mostly because at $DAYJOB, just about everything is deployed in containers.\nFor reasons that I don't fully understand (something about image size maybe??),\nthe popular container images like debian:trixie-slim do not include any root CAs.\nYou have to apt-get install them yourself.\nThis is to say that most TLS applications will straight up break in the out-of-the-box config.
Previously I had seen this solved in two ways.\nThe first is to install the certs from your distribution's package manager like so:
\nRUN apt-get update \\\n && apt-get install -y --no-install-recommends ca-certificates \\\n && rm -rf /var/lib/apt/lists/*\nThe second is to add the WebPKI roots to your cargo dependencies.\nThis actually requires some manual work; adding the crate isn't enough.\nYou then have to add all of the roots (e.g. via tls_certs_merge or tls_certs_only).
The net result is approximately the same, but not entirely.\nThe system-level approach is more flexible.\nPresumably you would get updates in some cases without having to rebuild your application\n(though you do not get these automatically; the certs are only loaded once on app startup\nby rustls_platform_verifier!).\nPresumably you would also get any, say, enterprise-level trust, distrust, CRLs, etc.\nthat are dictated by your corporate IT department.
The WebPKI approach on the other hand is baked at build time.\nThe crate\nhas a pretty strong, if slightly obtuse warning about this:
\n\n\nThis library is suitable for use in applications that can always be recompiled and instantly deployed. For applications that are deployed to end-users and cannot be recompiled, or which need certification before deployment, consider a library that uses the platform native certificate verifier such as
\nrustls-platform-verifier. This has the additional benefit of supporting OS provided CA constraints and revocation data.
Attempting to read between the lines, past that "instantly deployed" jargon,\nI think they are really just saying "if you use this, certs are baked at compile time and you never get automatic updates. Be careful with that."
\nSo it's clear to me you shouldn't ship, say, a static binary to users with certs baked like this.\nBut I'm building server-side software.\nAnd as of February 2026, people look at you funny if you don't deploy using containers.\nI can deploy sufficiently instantly,\nthough to be honest I would have no idea when I should.\nMost apps get deployed frequently enough that I would assume this just doesn't matter,\nand so I'm not sure the warning as-written does much to help a lot of the Rust devs I know.
\nMy conclusion is that if you're deploying containerized apps, there is approximately no functional difference.\nYour container is a static image anyways.\nThey don't typically run background tasks of any sort.\nAnd even if they did, the library won't reload the trusted store during application.\nSo it's functionally the same (delta any minor differences between WebPKI and Debian, which should be minimal).\nSimilarly, unless you work for a large enterprises / government,\nyou probably don't have mandated, hand-picked set of CAs and CRLs.\nSo again here there really is no difference as far as I can tell.
\nIn spite of that, I decided to switch away from using WebPKI in one of our containers that I upgraded.\nThe reason is that structuring this way\n(provided that the sources are copied from a previous layer!)\nensures that every image build always has the latest certs from Debian.\ncargo build is a lot more deterministic,\nand will use whatever you have in the lockfile unless you explicitly run cargo update.
And even though I'm fortunate to not have an IT apparatus dictating cert policy today,\nyou never know... this approach seems to be both more flexible and creates a "pit of success"\nrather than a landmine where the trust store may not see an update for a year\ndespite regular rebuilds.
\nIn other words, I think Sean made the right choice, and you should probably delegate to the system,\nunless you have a particular reason to do otherwise.
\nHope this helps; I wrote this because I didn't understand the tradeoffs initially,\nand had some trouble parsing the existing writing on the subject.
\n", "summary": "", "date_published": "2026-02-13T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "rust", "cryptography" ], "language": "en" }, { "id": "https://ianwwagner.com//even-safer-rust-with-miri.html", "url": "https://ianwwagner.com//even-safer-rust-with-miri.html", "title": "Even Safer Rust with Miri", "content_html": "Recently some of the Miri contributors published a paper that was accepted to POPL.\nI've been using Rust professionally for about 7 years now,\nand while I'd heard of Miri several times over the years,\nI think there's a wide lack of knowledge about what it does, and why anyone should care.\nI only recently started using it myself, so I'm writing this post to share\nwhat Miri is, why you should care, and how you can get started easily.
\nMiri is an interpreter for Rust's mid-level intermediate representation (MIR; hence the acronym).\nThat's how I first remember seeing it described years ago,\nand that's what the GitHub project description still says.
\nThe latest README is a bit more helpful though: it's a tool for detecting undefined behavior (UB) in Rust code.\nIn other words, it helps you identify code that's unsafe or unsound.\nWhile it would be a bug to hit such behaviors in safe Rust,\nif you're using unsafe (or any of your dependency chain does!),\nthen this is a real concern!\nMiri has in fact even found soundness bugs in the Rust standard library,\nso even a transitive sort of #![forbid(unsafe_code)] won't help you.
I think to understand why Miri matters,\nwe first need to understand why UB is bad.\nThis is not something that most professional programmers have a great understanding of (myself included).
\nIn abstract, UB can mean "anything that isn't specified", or something like that...\nBut that's not very helpful!\nAnd it doesn't really explain the stakes if we don't avoid it.\nThe Rust Reference has a list\nof behaviors that are considered to be undefined in Rust,\nbut they note that this list is not exhaustive.
\nWhen searching for a better understanding,\nI've seen people online make statements like\n"UB means your program can do literally anything at this point, like launch nuclear missiles."\nWhile this is technically true, this isn't particularly helpful to most readers.\nI want something more concrete...
\nThe authors of the paper put UB's consequences in terms which really "clicked" for me\nusing a logical equivalence, which I'll quote here:
\n\n\nFurthermore, Undefined Behavior is a massive security problem. Around 70% of critical security vulnerabilities are caused by memory safety violations [38, 18, 32], and all of these memory safety violations are instances of Undefined Behavior. After all, if the attacker overflows a buffer to eventually execute their own code, this is not something that the program does because the C or C++ specification says so—the specification just says that doing out-of-bounds writes (or overwriting the vtable, or calling a function pointer that does not actually point to a function, or doing any of the other typical first steps of an exploit chain) is Undefined Behavior, and executing the attacker’s code is just how Undefined Behavior happens to play out in this particular case.
\n
I never made this connection on my own.\nI equate UB most often with things like data races between threads,\nwhere you can have unexpected update visibility without atomics or locks.\nOr maybe torn reads of shared memory that's not properly synchronized.\nBut this is a new way of looking at it that makes the stakes more clear,\nespecially if you're doing anything with pointers.
\nAnother connection I never made previously is that UB is relative to a very specific context.\nHere's another quote from the paper:
\n\n\nThe standard random number crate used across the Rust ecosystem performed an unaligned memory access. Interestingly, the programmers seemed to have been aware that alignment is a problem in this case: there were dedicated code paths for x86 and for other architectures. Other architectures used read_unaligned, but the x86 code path had a comment saying that x86 allows unaligned reads, so we do not need to use this (potentially slower) operation. Unfortunately, this is a misconception: even though x86 allows unaligned accesses, Rust does not, no matter the target architecture—and this can be relevant for optimizations.
\n
This is REALLY interesting to me!\nIt makes sense in retrospect, but it's not exactly obvious.\nLanguages are free to define their own semantics in addition to or independently of hardware.\nI suspect Rust's specification here is somehow related to its concept of allocations\n(which the paper goes into more detail about).
\nIt is obviously not "undefined" what the hardware will do when given a sequence of instructions.\nBut it is undefined in Rust, which controls how those instructions are generated.\nAnd here the Rust Reference is explicit in calling this UB.\n(NOTE: I don't actually know what the "failure modes" are here, but you can imagine they could be very bad\nsince it could enable the compiler to make a bad assumption that leads to a program correctness or memory safety vulnerability.)
\nI actually encountered the same confusion re: what the CPU guarantees vs what Rust guarantees for unaligned reads in one of my own projects,\nas a previous version of this function didn't account for alignment.\nI addressed the issue by using the native zerocopy U32 type,\nwhich is something I'd have needed to do anyways to ensure correctness regardless of CPU endianness.\n(If you need to do something like this at a lower level for some reason, there's a read_unaligned function in std::ptr).
TL;DR - UB is both a correctness and a security issue, so it's really bad!
\nOne of the reasons I write pretty much everything that I can in Rust is because\nit naturally results in more correct and maintainable software.\nThis is a result of the language guarantees of safe Rust,\nthe powerful type system,\nand the whole ecosystem of excellent tooling.\nIt's a real pit of success situation.
\nWhile you can run a program under Miri as a one-shot test,\nthis isn't a practical approach to ensuring correctness long-term.\nMiri is a complementary tool to existing things that you should be doing already.\nAutomated testing is the most obvious one,\nbut fuzzing and other strategies may also be relevant for you.
\nIf you're already running automated tests in CI, adding Miri is easy.\nHere's an example of how I use it in GitHub actions:
\nsteps:\n - uses: actions/checkout@v4\n - uses: taiki-e/install-action@nextest\n\n - name: Build workspace\n run: cargo build --verbose\n\n - name: Run tests\n run: cargo nextest run --no-fail-fast\n\n - name: Run doc tests (not currently supported by nextest https://github.com/nextest-rs/nextest/issues/16)\n run: cargo test --doc\n\n - name: Install big-endian toolchain (s390x)\n run: rustup target add s390x-unknown-linux-gnu\n\n - name: Install s390x cross toolchain and QEMU (Ubuntu only)\n run: sudo apt-get update && sudo apt-get install -y gcc-s390x-linux-gnu g++-s390x-linux-gnu libc6-dev-s390x-cross qemu-user-static\n\n - name: Run tests (big-endian s390x)\n run: cargo nextest run --no-fail-fast --target s390x-unknown-linux-gnu\n\n - name: Install Miri\n run: rustup +nightly component add miri\n\n - name: Run tests in Miri\n run: cargo +nightly miri nextest run --no-fail-fast\n env:\n RUST_BACKTRACE: 1\n MIRIFLAGS: -Zmiri-disable-isolation\n\n - name: Run doc tests in Miri\n run: cargo +nightly miri test --doc\n env:\n RUST_BACKTRACE: 1\n MIRIFLAGS: -Zmiri-disable-isolation\n\n - name: Install nightly big-endian toolchain (s390x)\n run: rustup +nightly target add s390x-unknown-linux-gnu\n\n - name: Run tests in Miri (big-endian s390x)\n run: cargo +nightly miri nextest run --no-fail-fast --target s390x-unknown-linux-gnu\n env:\n RUST_BACKTRACE: 1\n MIRIFLAGS: -Zmiri-disable-isolation\nI know that's a bit longer than what you'll find in the README,\nbut I wanted to highlight my usage in a more complex codebase\nsince these examples are less common.\n(NOTE: I assume an Ubuntu runner here, since Linux has the best support for Miri right now.)\nSome things to highlight:
\nMIRIFLAGS to disable host isolation for my tests, since they require direct filesystem access. You may not need this for your project, but I do for mine.s390x-unknown-linux-gnu is the "big-endian target of choice" from the Miri authors. This requires a few dependencies and flags.Hopefully you learned something from this post.\nI'm pretty sure I wrote my first line of unsafe Rust less than a year ago\n(after using it professionally for over 6 years prior),\nso even if you don't need this today, file it away for later.\nAs I said at the start, I'm still not an expert,\nso if you spot any errors, please reach out to me on Mastodon!
\n", "summary": "", "date_published": "2026-01-07T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "rust", "software-reliability" ], "language": "en" }, { "id": "https://ianwwagner.com//const-assertions.html", "url": "https://ianwwagner.com//const-assertions.html", "title": "Const Assertions", "content_html": "I'm currently working on a project which involves a lot of lower level\ndata structures.\nBy lower level I mean things like layout and bit positions, and exact sizes being important.\nAs such, I have a number of pedantic lints enabled.
\nOne of the lints I use is cast_precision_loss.\nFor example, casting from usize to f32 using the as keyword is not guaranteed to be exact,\nsince f32 can only precisely represent integrals up to 23 bits of precision (due to how floating point is represented).\nAbove this you can have precision loss.
This lint is pedantic because it can generate false positives where you know the input can't ever exceed some threshold.\nBut wouldn't it be nice if we could go from "knowing" we can safely disable a lint to actually proving it?
\nThe first thing that came to mind was runtime assertions, but this is kind of ugly.\nIt requires that we actually exercise the code at runtime, for one.\nWe should be able to cover this in unit tests, but even if we do that,\nan assertion isn't as good as a compile time guarantee.
\nconstOne thing I didn't mention, and the reason I "know" that the lint would be fine is that I'm using a const declaration.\nHere's a look at what that's like:
pub const BUCKET_SIZE_MINUTES: u32 = 5;\npub const BUCKETS_PER_WEEK: usize = (7 * 24 * 60) as usize / BUCKET_SIZE_MINUTES as usize;\nThis isn't the same as a static or a let binding.\nconst expressions are actually evaluated at compile time.\n(Well, most of the time... there's a funny edge case where const blocks which can never executed at runtime\nis not guaranteed to evaluate.)
You can't do everything in const contexts, but you can do quite a lot, including many kinds of math.\nNot all math; some things like square root and trigonometry are not yet usable in const contexts\nsince they are not reproducible across architectures (and sometimes even on the same machine, it seems).
assert! in a const blockAnd now for the cool trick!\nI want to do a division here, and to do so, I need to ensure the types match.\nThis involves casting a usize to f32,\nwhich can cause truncation as noted above.
But since BUCKETS_PER_WEEK is a constant value,\nwe can actually do an assertion against it in our const context.\nThis lets us safely enable the lint, while ensuring we'll get a compile-time error if this ever changes!\nThis has no runtime overhead.
#[allow(clippy::cast_precision_loss, reason = "BUCKETS_PER_WEEK is always <= 23 bits")]\nconst PI_BUCKET_CONST: f32 = {\n // Asserts the invariant; panics at compile time if violated\n assert!(BUCKETS_PER_WEEK < 2usize.pow(24));\n\t// Computes the value\n std::f32::consts::PI / BUCKETS_PER_WEEK as f32\n};\nThis is all possible in stable Rust at the time of this writing (tested on 1.89).\nI saw some older crates out there which appeared to do this,\nbut as far as I can tell, they are no longer necessary.
\nHere's a Rust Playground\npreloaded with the sample code\nwhere you can verify that changing BUCKETS_PER_WEEK to a disallowed value causes a compile-time error.
Recently I've been doing some work using Apache DataFusion for some high-throughput data pipelines.\nOne of the interesting things I noticed on the user guide was the suggestion to set\nRUSTFLAGS='-C target-cpu=native'.\nThis is actually a pretty common optimization (which I periodically forget about and rediscover),\nso I thought I'd do a quick writeup on this.
A compiler translates your "idiomatic" code into low-level instructions.\nModern optimizing compilers are pretty good at figuring out ways to cleverly rewrite your code\nto make it faster, while still being functionally equivalent at execution time.\nThe instructions may be reordered from what your simple mental model expects,\nand they may even have no resemblance.\nThis includes rewriting some loop-like (or iterator) patterns into "vectorized" code using SIMD instructions\nthat perform some operation on multiple values at once.
\nSpecial instruction families like this often vary within a single architecture,\nwhich may be surprising at first.\nThe compiler can be configured to enable (or disable!) specific "features",\noptimizing for compatibility or speed.
\nIn rustc, each target triple has a default set of CPU features enabled.\nIn the case of my work laptop, that's aarch64-apple-darwin.\nSince this architecture doesn't have a lot of variation among chips,\nthe compiler can make some pretty good assumptions about what's available.\n(In fact, for my specific CPU, the M1 Max, it's perfect!)\nBut we'll soon see this is not the case for the most common target: x86_64 Linux.
To figure out what features we could theoretically enable,\nwe need some CPU info from the machine we intend to deploy on.\nThe canonical way of checking CPU features on Linux is probably to cat /proc/cpuinfo.\nThis gives a lot more output that you probably need though.\nHelpfully, rustc includes a simple command that shows you the config\nfor the native CPU capabilities: rustc --print=cfg -C target-cpu=native.\nHere's what it looks like on one Linux machine:
debug_assertions\npanic="unwind"\ntarget_abi=""\ntarget_arch="x86_64"\ntarget_endian="little"\ntarget_env="gnu"\ntarget_family="unix"\ntarget_feature="adx"\ntarget_feature="aes"\ntarget_feature="avx"\ntarget_feature="avx2"\ntarget_feature="bmi1"\ntarget_feature="bmi2"\ntarget_feature="cmpxchg16b"\ntarget_feature="f16c"\ntarget_feature="fma"\ntarget_feature="fxsr"\ntarget_feature="lzcnt"\ntarget_feature="movbe"\ntarget_feature="pclmulqdq"\ntarget_feature="popcnt"\ntarget_feature="rdrand"\ntarget_feature="rdseed"\ntarget_feature="sse"\ntarget_feature="sse2"\ntarget_feature="sse3"\ntarget_feature="sse4.1"\ntarget_feature="sse4.2"\ntarget_feature="ssse3"\ntarget_feature="xsave"\ntarget_feature="xsavec"\ntarget_feature="xsaveopt"\ntarget_feature="xsaves"\ntarget_has_atomic="16"\ntarget_has_atomic="32"\ntarget_has_atomic="64"\ntarget_has_atomic="8"\ntarget_has_atomic="ptr"\ntarget_os="linux"\ntarget_pointer_width="64"\ntarget_vendor="unknown"\nunix\nAside: I'm not quite sure why, but this isn't a 1:1 match with /proc/cpuinfo on this box!\nIt definitely does support some AVX512 instructions,\nbut those don't show up in the native CPU options.\nIf anyone knows why, let me know!
UPDATE: Shortly after publishing this post, Rust 1.89 was released.\nThis PR linked in the release notes caught my eye.\nApparently the target features for AVX512 were not actually stable at the time of writing,\nbut they are now.\nRe-running the above command with version 1.89 of rustc now includes the AVX512 instructions.
\nPerhaps the more interesting question which motivates this investigation is\nwhat the defaults are.\nYou can get this with rustc --print cfg.\nThis shows what you get when you run cargo build without any special configuration.\nHere's the output for the same machine:
debug_assertions\npanic="unwind"\ntarget_abi=""\ntarget_arch="x86_64"\ntarget_endian="little"\ntarget_env="gnu"\ntarget_family="unix"\ntarget_feature="fxsr"\ntarget_feature="sse"\ntarget_feature="sse2"\ntarget_has_atomic="16"\ntarget_has_atomic="32"\ntarget_has_atomic="64"\ntarget_has_atomic="8"\ntarget_has_atomic="ptr"\ntarget_os="linux"\ntarget_pointer_width="64"\ntarget_vendor="unknown"\nunix\nWell, that's disappointing, isn't it?\nBy default, you'd only get up to SSE2, which is over 20 years old by now!\nThis is a consequence of the diversity of the x86_64 architecture.\nIf you want your binary to run everywhere, this is the price you'd have to pay.
While -C target-cpu=native will usually make your code faster on the build machine,\na lot of modern software is built by a CI pipeline on cheap runners, but deployed elsewhere.\nTo reliably target a specific set of features, use the target-feature flag.\nThis lets you specifically enable features you know will be available on the machine running the code.\nHere's an example of RUSTFLAGS that incorporates all of the above features.\nThis should enable builds to proceed from any other x86_64 Linux machine and while producing a binary\nthat supports the exact features of the deployment machine.
RUSTFLAGS="-C target-feature=+adx,+aes,+avx,+avx2,+bmi1,+bmi2,+cmpxchg16b,+f16c,+fma,+fxsr,+lzcnt,+movbe,+pclmulqdq,+popcnt,+rdrand,+rdseed,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+xsave,+xsavec,+xsaveopt,+xsaves"\nA few days after writing this, I accidentally stumbled upon something else when working out target flags\nfor a program I knew would have wider support across several datacenters.\nIt sure would be nice if there were some "groups" of commonly supported features, right?
\nTurns out this exists, and it was staring right at me in the CPU list: microarchitecture levels!\nIf you list out all the available target CPUs via rustc --print target-cpus on a typical x86_64 Linux box,\nyou'll see that your default target CPU is x86-64.\nThis means it will run on all x86_64 CPUs, and as we discussed above, this doesn't give much of a baseline.\nBut there are 4 versions in total, going up to x86-64-v4.\nIt turns out that AMD, Intel, RedHat, and SUSE got together in 2020 to define these,\nand came up with some levels which are specifically designed for our use case of optimizing compilers!\nYou can find the full list of supported features by level on Wikipedia\n(search for "microarchitecture levels").
rustc --print target-cpus will also tell you which specific CPU you're on.\nYou can use this info to find which "level" you support.\nBut a more direct way to map to level support is to run /lib64/ld-linux-x86-64.so.2 --help.\nThanks, internet!\nYou'll get some output like this on a modern CPU:
Subdirectories of glibc-hwcaps directories, in priority order:\n x86-64-v4 (supported, searched)\n x86-64-v3 (supported, searched)\n x86-64-v2 (supported, searched)\nAnd if you run on slightly older hardware, you might get something like this:
\nSubdirectories of glibc-hwcaps directories, in priority order:\n x86-64-v4\n x86-64-v3 (supported, searched)\n x86-64-v2 (supported, searched)\nThis should help if you're trying to aim for broader distribution rather than enabling specific features for some known host.\nThe line to target an x86_64 microarch level is a lot shorter.\nFor example:
\nRUSTFLAGS="-C target-cpu=x86-64-v3"\nNOTE: As mentioned above, Rust 1.89 was released shortly after this post.\nThis incidentally brings support for AVX512 CPU features in the x86-64-v4 target CPU,\nwhich were previously marked unstable.
Enabling CPU features doesn't always make things faster.\nIn fact, in some cases, it can even do the opposite!\nThis thread\nhas some interesting anecdotes.
\nIn conclusion, here's a quick reference of the useful commands we covered:
\nrustc --print cfg - Shows the compiler configuration that your toolchain will use by default.rustc --print=cfg -C target-cpu=native - List the configuration if you were to specifically target for your CPU. Use this to see the delta between the defaults and the featurse supported for a specific CPU.rustc --print target-cpus - List all known target CPUs. This also tells you what your current CPU and what the default CPU is for your current toolchain./lib64/ld-linux-x86-64.so.2 --help - Specifically for x86_64 Linux users, will show you what microarchitecture levels your CPU supports.rustc --print target-features - List all available target features with a short description. You can scope to a specific CPU with -C target-cpu=. Useful mostly to see what you're missing, I guess.Rust's ownership system is well-known for the ways it enforces memory safety guaranteees.\nFor example, you can't use some value after it's been freed.\nFurther, it also ensures that mutability is explicit,\nand it enforces some extra rules that make most data races impossible.\nBut the ownership system has benefits beyond this which don't get as much press.
\nLet's look at a fairly common design pattern: the builder.\nA builder typically takes zero or few arguments to create.\nIn Rust, it's often implemented as a struct that implements the Default trait.\nThen, you progressively "chain" method invocations to ergonomically\nto specify how to build the thing you want.\nFor example:
let client = reqwest::Client::builder()\n .user_agent(APP_USER_AGENT)\n .timeout(Duration::from_secs(3))\n .build()?;\nThis pattern is useful because it avoids bloated constructors with dozens of arguments.\nIt also lets you encode failability into the process:\nsome combinations of arguments may be invalid.
\nIf you look at the signature for the timeout function,\nyou'll find it takes self as its first parameter and returns a value of the same type.\nThe key thing to note is that a non-reference self parameter\nwill "consume" the receiver!\nSince it takes ownership, you can't hold on to a reference to the original value!
This prevents a whole class of subtle bugs.\nPython, for example, doesn't prevent you from modifying inputs,\nand it's not always clear if a function/method is supposed to return a new value,\nwhether that value has the same contents as the original reference (which is still valid!)\nor if it's completely fresh,\nand so on.
\nA few other languages, mostly in the purely functional tradition (Haskell comes to mind)\nalso have a similar property.\nThey don't use a concept of "ownership" but rather remove mutability from the language.\nRust makes what I consider to be a nice compromise\nwhich retains most of the benefits while being easier to use.
\nIn summary, the borrow checker is a powerful ally,\nand you can leverage it to make truly better APIs,\nsaving hours of debugging in the future.
\n", "summary": "", "date_published": "2025-05-31T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "rust", "functional programming" ], "language": "en" }, { "id": "https://ianwwagner.com//unicode-normalization.html", "url": "https://ianwwagner.com//unicode-normalization.html", "title": "Unicode Normalization", "content_html": "Today I ran into an amusingly named place,\nthanks to some sharp eyes on the OpenStreetMap US Slack.\nThe name of this restaurant is listed as "𝐊𝐄𝐁𝐀𝐁 𝐊𝐈𝐍𝐆 𝐘𝐀𝐍𝐆𝐎𝐍".\nThat isn't some font trickery; it's a bunch of Unicode math symbols\ncleverly used to emphasize the name.\n(Amusingly, this does not actually show up properly on most maps, but that's another story for another post).
\nI was immediately curious how well the geocoder I spent the last few months building handles this.
\n
Well, at least it found the place, despite the very SEO-unfriendly name!\nBut what's up with the second search result?
\nWell, that's a consequence of us pulling in data from multiple sources.\nIn this case, the second result comes from the Foursquare OS Places dataset.\nIt seems that either the Placemaker validators decided to clean this up,\nor the Foursquare user who added the place didn't have that key on their phone keyboard.
\nOne of the things our geocoder needs to do when combining results is deduplicating results.\n(Beyond that, it needs to decide which results to keep, but that's a much longer post!)\nWe use a bunch of factors to make that decision, but one of them is roughly\n"does this place have the same name," where same is a bit fuzzy.
\nOne of the ways we can do this is normalizing away things like punctuation and diacritics.\nThese are quite frequently inconsistent across datasets, so two nearby results with similar enough names\nare probably the same place.\nFortunately, Unicode provides a few standardized transformations into canonical forms\nthat make this easier.
\nWhat we think of as a "character" does not necessarily have a single representation in Unicode.\nFor example, there are multiple ways of encoding "서울" which will look the same when rendered,\nbut have a different binary representation.\nThe Korean writing system is perhaps a less familiar case for many,\nbut characters with diacritical marks such as accents are usually the same.\nThey can be either "composed" or "decomposed" into the component parts\nat the binary level.
\nThis composition and decomposition transform is useful for (at least) two reasons:
\nI use the unicode_normalization crate\nto do this "decompose and filter" operation.\nSpecifically, the UnicodeNormalization trait,\nwhich has helpers which will work on most string-like types.
You might notice there are four confusingly named methods in the trait:\nnfd, nfkd, nfc, and nfkc.\nThe nf stands for "normalization form".\nThese functions normalize your strings.\nc and d stand for composition and decomposition.\nThe composed form is, roughly, the more compact form,\nwhereas the decomposed form is the version where you separate the base from the modifiers,\nthe jamo from the syllables, etc.
We were already decomposing strings so that we could remove the diacritics, using form NFD.\nThis works great for diacritics and even Hangul,\nbut 𝐊𝐄𝐁𝐀𝐁 𝐊𝐈𝐍𝐆 𝐘𝐀𝐍𝐆𝐎𝐍 shows that we were missing something.
\nThat something is the k, which stands for "compatibility."\nYou can refer to Unicode Standard Annex #15\nfor a full definition,\nbut the intuition is that compatibility equivalence of two characters\nis a bit more permissive than the stricter canonical equivalence.\nBy reducing two characters (or strings) to their canonical form,\nyou will be able to tell if they represent the same "thing" with the same visual appearance,\nbehavior, semantic meaning, etc.\nCompatibility equivalence is a weaker form.
Compatibility equivalence is extremely useful in our quest for determining whether two nearby place names\nare a fuzzy match.\nIt reduces things like ligatures, superscripts, and width variations into a standard form.\nIn the case of "𝐊𝐄𝐁𝐀𝐁 𝐊𝐈𝐍𝐆 𝐘𝐀𝐍𝐆𝐎𝐍," compatibility decomposition transforms it into the ASCII\n"KEBAB KING YANGON."\nAnd now we can correctly coalesce the available information into a single search result.
\nHopefully this shines a light on one small corner of the complexities of unicode!
\n", "summary": "", "date_published": "2025-05-09T00:00:00-00:00", "image": "media/kebab-king-duplicates.png", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "unicode", "rust" ], "language": "en" }, { "id": "https://ianwwagner.com//databases-as-an-alternative-to-application-logging.html", "url": "https://ianwwagner.com//databases-as-an-alternative-to-application-logging.html", "title": "Databases as an Alternative to Application Logging", "content_html": "In my work, I've been doing a lot of ETL pipeline design recently for our geocoding system.\nThe system processes on the order of a billion records per job,\nand failures are part of the process.\nWe want to log these.
\nMost applications start by dumping logs to stderr.\nUntil they overflow their terminal scrollback buffer.\nThe next step is usually text files.\nBut getting insights from 10k+ of lines of text with grep is a chore.\nIt may even be impossible unless you've taken extra care with how your logs are formatted.
In this post we'll explore some approcahes to do application logging better.
\nMy first introduction to logs with a structural element was probably Logcat for Android.\nLogcat lets you filter the fire hose of Android logs down to a specific application,\nand can even refine the scope further if you learn how to use it.\nLogcat is a useful tool, but fundamentally all it can do is filter logs from a stream\nand it has most of the same drawbacks as grepping plain text files.
\nLarger systems often benefit from something like the tracing crate,\nwhich integrates with services like journald and Grafana Loki.\nThis is a great fit for a long-running service,\nbut is total overkill for an application that does some important stuff ™\nand exits.\nLike our ETL pipeline example.
(Aside: I have a love/hate relationship with journalctl.\nI mostly interact with it through Ctrl+R in my shell history,\nwhich is problemating when connecting to a new server.\nBut it does have the benefit of being a nearly ubiquitous local structured logging system!)
Using a database as an application log can be a brilliant level up for many applications\nbecause you can actually query your logs with ease.\nI'll give a few examples, and then show some crazy cool stuff you can do with that.
\nOne type of failure we frequently encounter is metadata that looks like a URL where it shouldn't be.\nFor example, the name of a shop being http://spam.example.com/,\nor having a URL in an address or phone number field.\nIn this case, we usually drop the record, but we also want to log it so we can clean up the source data.\nSome other common failures are missing required fields, data in the wrong format, and the like.
Rather than logging these to stderr or some plain text files, we write to a DuckDB database.\nThis has a few benefits beyond the obvious.\nFirst, using a database forces you to come up with a schema.\nAnd just like using a language with types, this forces you to clarify your thinking a bit upfront.\nIn our case, we log things like the original data source, an ID, a log level (warn, error, info, etc.),\na failure code, and additional details.
From here, we can do meaningufl analytical queries like\n"how many records were dropped due to invalid geographic coordinates"\nor "how many records were rejected due to metadata mismatches"\n(ex: claiming to be a US address but appearing in North Korea).
\nIf this query uncovers a lot of rejected records from one data source,\nwouldn't it be nice if we could look at a sample?\nWe have the IDs right there in the log, and the data source identifier, after all.\nBut since we're in DuckDB rather than a plain text file,\nwe can pretty much effortlessly join on the data files!\n(This assumes that your data is in some halfway sane format like JSON, CSV, Parquet, or even another database).
\nWe can even take this one step further and compare logs across imports!\nWhat's up with that spike in errors compared to last month's release from that data source?
\nThese are the sort of insights which are almost trivial to uncover when your log is a database.
\nNow that I've described all the awesome things you can do,\nlet's get down to the practical questions like how you'd do this in your app.\nMy goals for the code were to make it easy to use and impossible to get wrong at the use site.\nFortunately that's pretty easy in Rust!
\n#[derive(Clone)]\npub struct ImportLogger {\n pool: Pool<DuckdbConnectionManager>,\n // Implementation detail for our case: we have multiple ETL importers that share code AND logs.\n // If you have any such attributes that will remain fixed over the life of a logger instance,\n // consider storing them as struct fields so each event is easier to log.\n importer_name: String,\n}\nPretty standard struct setup using DuckDB and r2d2 for connection pooling.\nWe put this in a shared logging crate in a workspace containing multiple importers.\nThe importer_name is a field that will get emitted with every log,\nand doesn't change for a logger instance.\nIf your logging has any such attributes (ex: a component name),\nstoring them as struct fields makes each log invocation easier!
Note
\nAt the time of this writing, I couldn't find any async connection pool integrations for DuckDB.\nIf anyone knows of one (or wants to add it to bb8), let me know!
pub fn new(config: ImportLogConfig, importer_name: String) -> anyhow::Result<ImportLogger> {\n let manager = DuckdbConnectionManager::file(config.import_log_path)?;\n let pool = Pool::new(manager)?;\n\n pool.get()?.execute_batch(include_str!("schema.sql"))?;\n\n Ok(Self {\n pool,\n importer_name,\n })\n}\nThe constructor isn't anything special; it sets up a DuckDB connection to a file-backed database\nbased on our configuration.\nIt also initializes the schema from a file.\nThe schema file lives in the source tree, but the lovely include_str!\nmacro bakes it into a static string at compile time (so we can still distribute a single binary).
pub fn log(&self, level: Level, source: &str, id: Option<&str>, code: &str, reason: &str) {\n log::log!(level, "{code}\\t{source}\\t{id:?}\\t{reason}");\n let conn = match self.pool.get() {\n Ok(conn) => conn,\n Err(e) => {\n log::error!("failed to get connection: {}", e);\n return;\n }\n };\n match conn.execute(\n "INSERT INTO logs VALUES (current_timestamp, ?, ?, ?, ?, ?, ?)",\n params![level.as_str(), self.importer_name, source, id, code, reason],\n ) {\n Ok(_) => (),\n Err(e) => log::error!("Failed to insert log entry: {}", e),\n }\n}\nAnd now the meat of the logging!\nThe log method does what you'd expect.\nThe signature is a reflection of the schema:\nwhat you need to log, what you may optionally log, and what type of data you're logging.
For our use case, we decided to additionally log via the log crate.\nThis way, we can see critical errors on the console to as the job is running.
And that's pretty much it!\nIt took significantly more time to write this post than to actually write the code.\nSomeone could probably write a macro-based crate to generate these sorts of loggers if they had some spare time ;)
\nfilter_logWe have a pretty common pattern in our codebase,\nwhere most operations / pipeline stages yield results,\nand we want to chain these together.\nWhen it succeeds, we pass the result on to the next stage.\nOtherwise, we want to log what went wrong.
\nWe called this filter_log because it usually shows up in filter_map over streams\nand as such yields an Option<T>.
This was extremely easy to add to our logging struct,\nand saves loads of boilerplate!
\n/// Converts a result to an option, logging the failure if the result is an `Err` variant.\npub fn filter_log<T, E: Debug>(\n &self,\n level: Level,\n source: &str,\n id: Option<&str>,\n code: &str,\n result: Result<T, E>,\n) -> Option<T> {\n match result {\n Ok(result) => Some(result),\n Err(err) => {\n self.log(level, source, id, code, &format!("{:?}", err));\n None\n }\n }\n}\nThe concept of logging to a database is not at all original with me\nMany enterprise services log extensively to special database tables.\nBut I think the technique is rarely applied to applications.
\nHopefully this post convinced you to give it a try in the next situation where it makes sense.
\n", "summary": "", "date_published": "2025-01-13T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "software-engineering", "duckdb", "databases", "rust" ], "language": "en" }, { "id": "https://ianwwagner.com//the-rust-toolchain-toml-file.html", "url": "https://ianwwagner.com//the-rust-toolchain-toml-file.html", "title": "The rust-toolchain.toml file", "content_html": "This isn't so much a TIL as a quick PSA.\nIf you're a Rust developer and need to ensure specific things about your toolchain,\nthe rust-toolchain.toml file is a real gem!
I don't quite remember how, but I accidentally discovered this file a year or two ago.\nSince then, I've spread the good news to at least half a dozen other devs,\nand most of them simply had no idea it existed.\nSo, without further ado...
\nrust-toolchain.toml is a file that lets you specify certain things about your Rust toolchain.\nFor example, if you need to use nightly rust for a project,\nyou can specify that in your toolchain file.\nIt also lets you specify other cargo components to install\nand specify cross-compilation targets you want to have available.
The headline use case in The rustup book\nis to pin to a specific release.\nThis is pretty rare in practice I think, unless you need nightly.\nYou can specify channels like nightly, stable, and beta in addition to specific releases.
The killer use case in my opinion is for easier cross-compilation.\nI do a lot of cross compiling, and codifying all required targets in a single file makes life much easier!
\nThe best part is that, as long as you're using rustup, everything is automatic!\nFor projects with a large number of collaborators (like an open-source library),\nthis makes it a lot easier to onboard new devs.
Not everyone uses rustup.\nFor example, some devs I know use nix.\nWhen I asked one of them about how to do this without duplicating work,\nthey suggested Fenix,\nwhich is able to consume the rust-toolchain.toml.
If you have suggestions or experiences with other invironments,\nlet me know and I'll update this post.\nContact links in the footer.
\nHere's what the file looks like for a cross-platform mobile library that I maintain:
\n[toolchain]\nchannel = "stable"\ntargets = [\n # iOS\n "aarch64-apple-ios",\n "x86_64-apple-ios",\n "aarch64-apple-ios-sim",\n\n # Android\n "armv7-linux-androideabi",\n "i686-linux-android",\n "aarch64-linux-android",\n "x86_64-linux-android",\n "x86_64-unknown-linux-gnu",\n "x86_64-apple-darwin",\n "aarch64-apple-darwin",\n "x86_64-pc-windows-gnu",\n "x86_64-pc-windows-msvc",\n\n # WebAssembly\n "wasm32-unknown-unknown"\n]\ncomponents = ["clippy", "rustfmt"]\nIn the weeks since my previous post on Working with Arrow and DuckDB in Rust,\nI've found a few gripes that I'd like to address.
\nquery_arrow and stream_arrowIn the previous post, I used the query_arrow API.\nIt's pretty straightforward and gives you iterator-compatible access to the query results.\nHowever, there's one small problem: its memory consumption scales roughly linearly with your result set.
This isn't a problem for many uses of DuckDB, but if your datasets are in the tens or hundreds of gigabytes\nand you're wanting to process a large number of rows, the RAM requirements can be excessive.\nThe memory profile of query_arrow seems to be "create all of the RecordBatches upfront\nand keep them around for as long as you hold the Arrow handle.
Disclaimer
\nI have not done extensive allocation-level memory profiling as of this writing.\nIt's quite possible that I've missed something, but this seems to be what's happening\nfrom watching Activity Monitor.\nPlease let me know if I've misrepresented anything!
\nFortunately, DuckDB also has another API: stream_arrow.\nThis appears to allocate RecordBatches on demand rather than all at once.\nThere is also some overhead, which I'll revisit later that varies with result size.\nBut overall, profiling indicates that stream_arrow requires significantly less RAM over the life of a large Arrow iterator.
Unfortunately, none of the above information about memory consumption appears to be documented,\nand there are no (serious) code samples demonstrating the use of stream_arrow!
\n\n[!question] Down the rabbit hole...\nDigging into the code in duckdb-rs raises even more questions,\nsince several underlying C functions, like
\nduckdb_execute_prepared_streaming\nare marked as deprecated.\nPresumably, alternatives are being developed or the methods are just not stable yet.
SchemaRefThe signature of stream_arrow is a bit different from that of query_arrow.\nHere's what it looks like as of crate version 1.1.1:
pub fn stream_arrow<P: Params>(\n &mut self,\n params: P,\n schema: SchemaRef,\n) -> Result<ArrowStream<'_>>\nThis looks pretty familiar at first if you've used query_arrow,\nbut there's a new third parameter: schema.\nSchemaRef is just a type alias for Arc<Schema>.\nArrow objects have a schema associated with them,\nso this is a reasonable detail for a low-level API.\nBut DuckDB is fine at inferring this when needed!\nSurely there is a way of getting it from a query, right?\n(After all, query_arrow has to do something similar, but doesn't burden the caller.)
My first attempt at getting a Schema object was to call the schema() method on Statement.\nThe Statement type in duckdb-rs is actually a high-level wrapper around RawStatement,\nand at the time of this writing, the schema getter hides an unwrap.\nThe docs do tell you this (using a somewhat nonstandard heading?),\nbut basically you can't get a schema without executing a query.\nI wish they used the Typestate pattern\nor at least made the result an Option, but alas...
This leaves developers with three options.
\nStatement that is the same SQL, but with a LIMIT 0 clause at the end.Manually constructing the schema is a non-starter for me.\nA program which has a hand-written code dependency on a SQL string is a terrible idea\non several levels.\nBesides, DuckDB clearly can infer the schema in query_arrow, so why not here?
The second idea is, amusingly, what ChatGPT o1 suggested (after half a dozen prompts;\nit seems like it will just confidently refuse to fetch documentation now,\nand hallucinates new APIs based off its outdated training data).\nThe basic idea is to add LIMIT 0 to the end of the original query\nso it's able to get the schema, but doesn't actually return any results.
fn fetch_schema_for_query(db: &Connection, sql: &str) -> duckdb::Result<SchemaRef> {\n // Append "LIMIT 0" to the original query, so we don't actually fetch anything\n // NB: This does NOT handle cases such as the original query ending in a semicolon!\n let schema_sql = format!("{} LIMIT 0", sql);\n\n let mut statement = db.prepare(&schema_sql)?;\n let arrow_result = statement.query_arrow([])?;\n\n Ok(arrow_result.get_schema())\n}\nThere is nothing fundamentally unsound about this approach.\nBut it requires string manipulation, which is less than ideal.\nThere is also at least one obvious edge case.
\nThe third option is not as straightforward as I expected it to be.\nAt first, I tried the row_count method,\nbut internally this just calls a single FFI function.\nThis doesn't actually update the internal schema field.\nYou really do need to run through a more "normal" execution path.
A solution that seems reasonably clean is to do what the docs say and call stmt.execute().\nIt's a bit strange to do this on a SELECT query to be honest,\nbut the API does indeed internally mutate the Schema property,\nand returns a row count.\nSo it seems semantically equivalent to a SELECT COUNT(*) FROM (...)\n(and in my case, getting the row count was helpful too).
In my testing, it appears that this may actually allocate a non-trivial amount of memory,\nwhich may be mildly surprising.\nHowever, the max amount of memory we require during execution is definitely less overall.\nAny ideas why this is?
\nstream_arrowLet's bring what we've learned into a "real" example.
\n// let sql = "SELECT * FROM table;";\nlet mut stmt = conn.prepare(sql)?;\n// Execute the query (so we have a usable schema)\nlet size = stmt.execute([])?;\n// Now we run the "real" query using `stream_arrow`.\n// This returned in a few hundred milliseconds for my dataset.\nlet mut arrow = stmt.stream_arrow([], stmt.schema())?;\n// Iterate over arrow...\nWhen you structure your code like this rather than using the easier query_arrow,\nyou can significantly reduce your memory footprint for large datasets.\nIn my testing, there was no appreciable impact on performance.
The above leaves me with a few open questions.\nFirst, with my use case (a dataset of around 12GB of Parquet files), execute took several seconds.\nThe "real" stream_arrow query took a few hundred milliseconds.\nWhat's going on here?\nPerhaps it's doing a scan and/or caching some data initially the way to make subsequent queries faster?
Additionally, the memory profile does have a "spike" which makes me wonder what exactly each step loads into RAM,\nand thus, the memory requirements for working with extremely large datasets.\nIn my testing, adding a WHERE clause that significantly reduces the result set\nDOES reduce the memory footprint.\nThat's somewhat worrying to me, since it implies there is still measurable overhead\nproportional to the dataset size.\nWhat practical limits does this impose on dataset size?
Note
\nAn astute reader may be asking whether the memory profile of the LIMIT 0 and execute approaches are equivalent.\nThe answer appears to be yes.
I've opened issue #418\nasking for clarification.\nIf any readers have any insights, post them in the issue thread!
\n", "summary": "", "date_published": "2024-12-31T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "rust", "apache arrow", "parquet", "duckdb", "big data", "data engineering" ], "language": "en" }, { "id": "https://ianwwagner.com//how-and-why-to-work-with-arrow-and-duckdb-in-rust.html", "url": "https://ianwwagner.com//how-and-why-to-work-with-arrow-and-duckdb-in-rust.html", "title": "How (and why) to work with Arrow and DuckDB in Rust", "content_html": "My day job involves wrangling a lot of data very fast.\nI've heard a lot of people raving about several technologies like DuckDB,\n(Geo)Parquet, and Apache Arrow recently.\nBut despite being an "early adopter,"\nit took me quite a while to figure out how and why to leverage these practiclaly.
\nLast week, a few things "clicked" for me, so I'd like to share what I learned in case it helps you.
\n(Geo)Parquet is quite possibly the best understood tech in the mix.\nIt is not exactly new.\nParquet has been around for quite a while in the big data ecosystem.\nIf you need a refresher, the Cloud-optimized Geospatial Formats Guide\ngives a great high-level overview.
\nHere are the stand-out features:
\nPractically speaking, parquet lets you can distribute large datasets in one or more files\nwhich will be significantly smaller and faster to query than other familiar formats.
\nThe value proposition is clear for big data processing.\nIf you're trying to get a record of all traffic accidents in California,\nor find the hottest restaurants in Paris based on a multi-terabyte dataset,\nparquet provides clear advantages.\nYou can skip row groups within the parquet file or even whole files\nto narrow your search!\nAnd since datasets can be split across files,\nyou can keep adding to the dataset over time, parallelize queries,\nand other nice things.
\nBut what if you're not doing these high-level analytical things?\nWhy not just use a more straightforward format like CSV\nthat avoids the need to "rotate" back into rows\nfor non-aggregation use cases?\nHere are a few reasons to like Parquet:
\nThe upshot of all this is that it generally gets both easier and faster\nto work with your data...\nprovided that you have the right tools to leverage it.
\nDuckDB describes itself as an in-process, portable, feature-rich, and fast database\nfor analytical workloads.\nDuckDB was that tool that triggered my "lightbulb moment" last week.\nFoursquare, an app which I've used for a decade or more,\nrecently released an open data set,\nwhich was pretty cool!\nIt was also in Parquet format (just like Overture's data sets).
\nYou can't just open up a Parquet file in a text editor or spreadsheet software like you can a CSV.\nMy friend Oliver released a web-based demo\na few weeks ago which lets you inspect the data on a map at the point level.\nBut to do more than spot checking, you'll probably want a database that can work with Parquet.\nAnd that's where DuckDB comes in.
\nI understood the in-process part of DuckDB's value proposition right away.\nIt's similar to SQLite, where you don't have to go through a server\nor over an HTTP connection.\nThis is both simpler to reason about and is usually quite a bit faster\nthan having to call out to a separate service!
\nDuckDB is pretty quick to compile from source.\nYou probably don't need to muck around with this if you're just using the CLI,\nbut I wanted to eventually use it embedded in some Rust code.\nCompiling from source turned out to be the easiest way to get their crate working.\nIt looks for a shared library by default, but I couldn't get this working after a brew install.\nThis was mildly annoying, but on the other hand,\nvendoring the library does make consistent Docker builds easier 🤷🏻♂️
DuckDB includes a mind boggling number of features.\nNot in a confusing way; more in a Python stdlib way where just about everything you'd want is already there.\nYou can query a whole directory (or bucket) of CSV files,\na Postgres database, SQLite, or even an OpenStreetMap PBF file 🤯\nYou can even write a SQL query against a glob expression of Parquet files in S3\nas your "table."\nThat's really cool!\n(If you've been around the space, you may recognize this concept from\nAWS Athena and others.)
\nWriting a query against a local directory of files is actually really fast!\nIt does a bit of munging upfront, and yes,\nit's not quite as fast as if you'd prepped the data into a clean table,\nbut you actually can run quite efficient queries this way locally!
\nWhen running a query against local data,\nDuckDB will make liberal use of your system memory\n(the default is 80% of system RAM)\nand as many CPUs as you can throw at it.\nBut it will reward you with excellent response times,\ncourtesy of the "vectorized" query engine.\nWhat I've heard of the design reminds me of how array-oriented programming languages like APL\n(or less esoteric libraries like numpy) are often implemented.
\nI was able to do some spatial aggregation operations\n(bucketing a filtered list of locations by H3 index)\nin about 10 seconds on a dataset of more than 40 million rows!\n(The full dataset is over 100 million rows, so I also got to see the selective reading in action.)\nThat piqued my interest, to say the least.\n(Here's the result of that query, visualized).
\n![\"A](\"media/foursquare-os-places-density-2024.png\")
And now for the final buzzword in DuckDB's marketing: analytical.\nDuckDB frequently describes itself as optimized for OLAP (OnLine Analytical Processing) workloads.\nThis is contrasted with OLTP (OnLine Transaction Processing).\nWikipedia will tell you some differences\nin a lot of sweepingly broad terms, like being used for "business reporting" and read operations\nrather than "transactions."
\nWhen reaching for a definition, many sources focus on things like aggregation queries\nas a differentiator.\nThis didn't help, since most of my use cases involve slurping most or all of the data set.\nThe DuckDB marketing and docs didn't help clarify things either.
\nLet me know on Mastodon if you have a better explanatation of what an "analytical" database is 🤣
\nI think a better explanation is probably 1) you do mostly read queries,\nand 2) it can execute highly parallel queries.\nSo far, DuckDB has been excellent for both the "aggregate" and the "iterative" use case.\nI assume it's just not the best choice per se if your workload is a lot of single-record writes?
\nEmbedding DuckDB in a Rust project allowed me to deliver something with a better end-user experience,\nis easier to maintain,\nand saved writing hundreds of lines of code in the process.
\nMost general-purpose languages like Python and Rust\ndon't have primitives for expressing things like joins across datasets.\nDuckDB, like most database systems, does!\nYes, I could write some code using the parquet crate\nthat would filter across a nested directory tree of 5,000 files.\nBut DuckDB does that out of the box!
It feels like this is a "regex moment" for data processing.\nJust like you don't (usually) need to hand-roll string processing,\nthere's now little reason to hand-roll data aggregation.
\nFor the above visualization, I used the Rust DuckDB crate for the data processing,\nconverted the results to JSON,\nand served it up from an Axum web server.\nAll in a single binary!\nThat's lot nicer than a bash script that executes SQL,\ndumps to a file, and then starts up a Python or Node web server!\nAnd breaks when you don't have Python or Node installed,\nyour OS changes its default shell,\nyou forget that some awk flag doesn't work on the GNU version,\nand so on.
\nThe final thing I want to touch on is Apache Arrow.\nThis is yet another incredibly useful technology which I've been following for a while,\nbut never quite figured out how to properly use until last week.
\nArrow is a language-independent memory format\nthat's optimized for efficient analytic operations on modern CPUs and GPUs.\nThe core idea is that, rather than having to convert data from one format to another (this implies copying!),\nArrow defines as shared memory format which many systems understand.\nIn practice, this ends up being a bunch of standards which define common representations for different types,\nand libraries for working with them.\nFor example, the GeoArrow spec\nbuilds on the Arrow ecosystem to enable operations on spatial data in a common memory format.\nPretty cool!
\nIt turns out that copying and format shifting data can really eat into your processing times.\nArrow helps you sidestep that by reducing the amount of both you'll need to do,\nand by working on data in groups.
\nArrow is mostly hidden from view beneath other libraries.\nSo most of the time, especially if you're writing in a very high level language like Python,\nyou won't even see it.
\nBut if you're writing something at a slightly lower level,\nit's something you may have to touch for critical sections.\nThe DuckDB crate\nincludes an Arrow API\nwhich will give you an iterator over RecordBatches.\nThis is pretty convenient, since you can use DuckDB to gather all your data\nand just consume the stream of batches!
So, how do we work with RecordBatches?\nThe Arrow ecosystem, like Parquet, takes a lot of work to understand,\nand using the low-level libraries directly is difficult.\nEven as a seasoned Rustacean, I found the docs rather obtuse.
After some searching, I finally found serde_arrow.\nIt builds on the serde ecosystem with easy-to-use methods that operate on RecordBatches.\nFinally; something I can use!
I was initilaly worried about how performant the shift from columns to rows + any (minimal) serde overhead would be,\nbut this turned out to not be an issue.
Here's how the code looks:
\nserde_arrow::from_record_batch::<Vec<FoursquarePlaceRecord>>(&batch)\nA few combinators later and you've got a proper data pipeline!
\nWhat this ultimately enabled for me was being able to get a lot closer to "scripting"\na pipeline in Rust.\nMost people turn to Python or JavaScript for tasks like this,\nbut Rust has something to add: strong typing and all the related guarantees which can only come with some level of formalism.\nBut that doesn't necessarily have to get in the way of productivity!
\nHopefully this sparks some ideas for making your next data pipeline both fast and correct.
\n", "summary": "", "date_published": "2024-12-08T00:00:00-00:00", "image": "media/foursquare-os-places-density-2024.png", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "rust", "apache arrow", "parquet", "duckdb", "big data", "data engineering", "gis" ], "language": "en" }, { "id": "https://ianwwagner.com//quadrupling-the-performance-of-a-data-pipeline.html", "url": "https://ianwwagner.com//quadrupling-the-performance-of-a-data-pipeline.html", "title": "Quadrupling the Performance of a Data Pipeline", "content_html": "Over the past two weeks, I've been focused on optimizing some data pipelines.\nI inherited some old ones which seemed especially slow,\nand I finally hit a limit where an overhaul made sense.\nThe pipelines process and generate data on the order of hundreds of gigabytes,\nrequiring correlation and conflated across several datasets.
\nThe pipelines in question happened to be written in Node.js,\nwhich I will do my absolute best not to pick on too much throughout.\nNode is actually a perfectly fine solution for certain problems,\nbut was being used especially badly in this case.\nThe rewritten pipeline, using Rust, clocked in at 4x faster than the original.\nBut as we'll soon see, the choice of language wasn't even the main factor in the sluggishness.
\nSo, let's get into it...
\nNode.js made a splash in the early 2010s,\nand I can remember a few years where it was the hot new thing to write everything in.\nOne of the selling points was its ability to handle thousands (or tens of thousands)\nof connections with ease; all from JavaScript!\nThe key to this performance is async I/O.\nModern operating systems are insanely good at this, and Node made it really easy to tap into it.\nThis was novel to a lot of developers at the time, but it's pretty standard now\nfor building I/O heavy apps.
\nNode performs well as long as you were dealing with I/O-bound workloads,\nbut the magic fades if your workload requires a lot of CPU work.\nBy default, Node is single-threaded.\nYou need to bring in libuv, worker threads (Node 10 or so), or something similar\nto access parallel processing from JavaScript.\nI've only seen a handful of Node programs use these,\nand the pipelines in question were not among them.
If you ingest data files (CSV and the like) record-by-record in a naïve way,\nyou'll just read one record at a time, process, insert to the database, and so on in a loop.\nThe original pipeline code was fortunately not quite this bad (it did have batching at least),\nbut had some room for improvemnet.
\nThe ingestion phase, where you're just reading data from CSV, parquet, etc.\nmaps naturally to Rust's streams\n(the cousin of futures).\nThe original node code was actually fine at this stage,\nif a bit less elegant.\nBut the Rust structure we settled on is worth a closer look.
\nfn csv_record_stream<'a, S: AsyncRead + Unpin + Send + 'a, T: TryFrom<StringRecord>>(\n stream: S,\n delimiter: u8,\n) -> impl Stream<Item = T> + 'a\nwhere\n <T as TryFrom<StringRecord>>::Error: Debug,\n{\n let reader = AsyncReaderBuilder::new()\n .delimiter(delimiter)\n // Other config elided...\n .create_reader(stream);\n reader.into_records().filter_map(|res| async move {\n let Ok(record) = res else {\n log::error!("Error reading from the record stream: {:?}", res);\n return None;\n };\n\n match T::try_from(record) {\n Ok(parsed) => Some(parsed),\n Err(e) => {\n log::error!("Error parsing record: {:?}.", e);\n None\n }\n }\n })\n}\nIt starts off dense, but the concept is simple.\nWe'll take an async reader,\nconfigure a CSV reader to pull records for it,\nand map them to another data type using TryFrom.\nIf there are any errors, we just drop them from the stream and log an error.\nThis usually isn't a reason to stop processing for our use case.
You should not be putting expensive code in your TryFrom implementation.\nBut really quick things like verifying that you have the right number of fields,\nor that a field contains an integer or is non-blank are usually fair game.
Rust's trait system really shines here.\nOur code can turn any CSV(-like) file\ninto an arbitrary record type.\nAnd the same techniques can apply to just about any other data format too.
\nNow that we've done the light format shifting and discarded some obviously invalid records,\nlet's turn to the heavier processing.
\nlet available_parallelism = std::thread::available_parallelism()?.get();\n// let record_pipeline = csv_record_stream(...);\nrecord_pipeline\n .chunks(500) // Batch the work (your optimal size may vary)\n .for_each_concurrent(available_parallelism, |chunk| {\n // Clone your database connection pool or whatnot before `move`\n // Every app is different, but this is a pretty common pattern\n // for sqlx, Elastic Search, hyper, and more which use Arcs and cheap clones for pools.\n let db_pool = db_pool.clone();\n async move {\n // Process your records using a blocking threadpool\n let documents = tokio::task::spawn_blocking(move || {\n // Do the heavy work here!\n chunk\n .into_iter()\n .map(do_heavy_work)\n .collect()\n })\n .await\n .expect("Problem spawning a blocking task");\n\n // Insert processesd data to your database\n db_pool.bulk_insert(documents).await.expect("You probably need an error handling strategy here...");\n }\n })\n .await;\nWe used the chunks\nadaptor to pull hundreds of items at a time for more efficient processing in batches.\nThen, we used for_each_concurrent\nin conjunction with spawn_blocking\nto introduce parallel processing.
Note that neither chunks nor even for_each_concurrent imply any amount of parallelism\non their own.\nspawn_blocking is the only thing that can actually create a new thread of execution!\nChunking simply splits the work into batches (most workloads like this tend to benefit from batching).\nAnd for_each_concurrent allows for concurrent operations over multiple batches.\nBut spawn_blocking is what enables computation in a background thread.\nIf you don't use spawn_blocking,\nyou'll end up blocking Tokio's async workers,\nand your performance will tank.\nJust like the old Node.js code.
The astute reader may point out that using spawn_blocking like this\nis not universally accepted as a solution.\nTokio is (relatively) optimized for non-blocking workloads, so some claim that you should avoid this pattern.\nBut my experience having done this for 5+ years in production code serving over 2 billion requests/month,\nis that Tokio can be a great scheduler for heavier tasks too!
One thing that's often overlooked in these discussions\nis that not all "long-running operations" are the same.\nOne category consists of graphics event loops,\nlong-running continuous computations,\nor other things that may not have an obvious "end."\nBut some tasks can be expected to complete within some period of time,\nthat's longer than a blink.
\nIn the case of the former ("long-lived" tasks), then spawning a dedicated thread often makes sense.\nIn the latter scenario though, Tokio tasks with spawn_blocking can be a great choice.
For our workload, we were doing a lot of the latter sort of operation.\nOne helpful rule of thumb I've seen is that if your task takes longer than tens of microseconds,\nyou should move it off the Tokio worker threads.\nUsing chunks and spawn_blocking avoids this death by a thousand cuts.\nIn our case, the parallelism resulted in a VERY clear speedup.
The original data pipeline was very careful to not overload the data store.\nPerhaps a bit too careful!\nThis may have been necessary at some point in the distant past,\nbut most data storage, from vanilla databases to multi-node clustered storage,\nhave some level of natural backpressure built-in.\nThe Node implementation was essentially limiting the amount of work in-flight that hadn't been flushed.
\nThis premature optimization and the numerous micro-pauses it introduced\nwere another death by a thousand cuts problem.\nDropping the artificial limits approximately doubled throughput.\nIt turned out that our database was able to process 2-4x more records than under the previous implementation.
\nTL;DR — set a reasonable concurrency, let the server tell you when it's chugging (usually via slower response times),\nand let your async runtime handle the rest!
\nSerde, or serialization + deserialization, can be a silent killer.\nAnd unless you're tracking things carefully, you often won't notice!
\nI recently listened to Recoding America at the recommendation of a friend.\nOne of the anecdotes made me want to laugh and cry at the same time.\nEngineers had designed a major improvemnet to GPS, but the rollout is delayed\ndue to a performance problem that renders it unusable.
\nThe project is overseen by Raytheyon, a US government contractor.\nAnd they can't deliver because some arcane federal guidance (not even a regulation proper)\n"recommends" an "Enterprise Service Bus" in the architecture.\nThe startuppper in me dies when I hear such things.\nThe "recommendation" boils down to a data exchange medium where one "service" writes data and another consumes it.\nThink message queues like you may have used before.
\nThis is fine (even necessary) for some applications,\nbut positively crippling for others.\nIn the case of the new positioning system,\nwhich was heavily dependent on timing,\nthis was a wildly inefficient architecture.\nEven worse, the guidelines stated that it should be encrypted.
\nThis wasn't even "bad" guidance, but in the context of the problem,\nwhich depended on rapid exchange of time-sensitive messages,\nit was a horrendously bad fit.
\nIn our data pipeline, I discovered a situation with humorous resemblance in retrospect.\nThe pipeline was set up using a microservice architecture,\nwhich I'm sure souded like a good idea at the time,\nbut it introduced some truly obscene overhead.\nAll services involved were capable of working with data in the same format,\nbut the Node.js implementation was split into multiple services with HTTP and JSON round trips in the middle!\nDouble whammy!
\nThe new data pipeline simply imports the "service" as a crate,\nand gets rid of all the overhead by keeping everything in-process.\nIf you do really need to have a microservice architecture (ex: to scale another service up independently),\nthen other communication + data exchange formats may improve your performance.\nBut if it's possible to keep everything in-process, your overhead is roughly zero.\nThat's hard to beat!
\nIn the end, the new pipeline was 4x the speed of the old.\nI happened to rewrite it in Rust, but Rust itself wasn't the source of all the speedups:\nunderstanding the architecture was.\nYou could achieve similar results in Node.js or Python,\nbut Rust makes it significantly easy to reason about the architecture and correctness of your code.\nThis is especially important when it comes to parallelizing sections of a pipeline,\nwhere Rust's type system will save you from the most common mistakes.
\nThese and other non-performance-related reasons to use Rust will be the subject of a future blog post (or two).
\n", "summary": "", "date_published": "2024-11-29T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "algorithms", "rust", "elasticsearch", "nodejs", "data engineering", "gis" ], "language": "en" } ] }