Over the past two weeks, I've been focused on optimizing some data pipelines.\nI inherited some old ones which seemed especially slow,\nand I finally hit a limit where an overhaul made sense.\nThe pipelines process and generate data on the order of hundreds of gigabytes,\nrequiring correlation and conflated across several datasets.
\nThe pipelines in question happened to be written in Node.js,\nwhich I will do my absolute best not to pick on too much throughout.\nNode is actually a perfectly fine solution for certain problems,\nbut was being used especially badly in this case.\nThe rewritten pipeline, using Rust, clocked in at 4x faster than the original.\nBut as we'll soon see, the choice of language wasn't even the main factor in the sluggishness.
\nSo, let's get into it...
\nNode.js made a splash in the early 2010s,\nand I can remember a few years where it was the hot new thing to write everything in.\nOne of the selling points was its ability to handle thousands (or tens of thousands)\nof connections with ease; all from JavaScript!\nThe key to this performance is async I/O.\nModern operating systems are insanely good at this, and Node made it really easy to tap into it.\nThis was novel to a lot of developers at the time, but it's pretty standard now\nfor building I/O heavy apps.
\nNode performs well as long as you were dealing with I/O-bound workloads,\nbut the magic fades if your workload requires a lot of CPU work.\nBy default, Node is single-threaded.\nYou need to bring in libuv, worker threads (Node 10 or so), or something similar\nto access parallel processing from JavaScript.\nI've only seen a handful of Node programs use these,\nand the pipelines in question were not among them.
If you ingest data files (CSV and the like) record-by-record in a naïve way,\nyou'll just read one record at a time, process, insert to the database, and so on in a loop.\nThe original pipeline code was fortunately not quite this bad (it did have batching at least),\nbut had some room for improvemnet.
\nThe ingestion phase, where you're just reading data from CSV, parquet, etc.\nmaps naturally to Rust's streams\n(the cousin of futures).\nThe original node code was actually fine at this stage,\nif a bit less elegant.\nBut the Rust structure we settled on is worth a closer look.
\nfn csv_record_stream<'a, S: AsyncRead + Unpin + Send + 'a, T: TryFrom<StringRecord>>(\n stream: S,\n delimiter: u8,\n) -> impl Stream<Item = T> + 'a\nwhere\n <T as TryFrom<StringRecord>>::Error: Debug,\n{\n let reader = AsyncReaderBuilder::new()\n .delimiter(delimiter)\n // Other config elided...\n .create_reader(stream);\n reader.into_records().filter_map(|res| async move {\n let Ok(record) = res else {\n log::error!("Error reading from the record stream: {:?}", res);\n return None;\n };\n\n match T::try_from(record) {\n Ok(parsed) => Some(parsed),\n Err(e) => {\n log::error!("Error parsing record: {:?}.", e);\n None\n }\n }\n })\n}\nIt starts off dense, but the concept is simple.\nWe'll take an async reader,\nconfigure a CSV reader to pull records for it,\nand map them to another data type using TryFrom.\nIf there are any errors, we just drop them from the stream and log an error.\nThis usually isn't a reason to stop processing for our use case.
You should not be putting expensive code in your TryFrom implementation.\nBut really quick things like verifying that you have the right number of fields,\nor that a field contains an integer or is non-blank are usually fair game.
Rust's trait system really shines here.\nOur code can turn any CSV(-like) file\ninto an arbitrary record type.\nAnd the same techniques can apply to just about any other data format too.
\nNow that we've done the light format shifting and discarded some obviously invalid records,\nlet's turn to the heavier processing.
\nlet available_parallelism = std::thread::available_parallelism()?.get();\n// let record_pipeline = csv_record_stream(...);\nrecord_pipeline\n .chunks(500) // Batch the work (your optimal size may vary)\n .for_each_concurrent(available_parallelism, |chunk| {\n // Clone your database connection pool or whatnot before `move`\n // Every app is different, but this is a pretty common pattern\n // for sqlx, Elastic Search, hyper, and more which use Arcs and cheap clones for pools.\n let db_pool = db_pool.clone();\n async move {\n // Process your records using a blocking threadpool\n let documents = tokio::task::spawn_blocking(move || {\n // Do the heavy work here!\n chunk\n .into_iter()\n .map(do_heavy_work)\n .collect()\n })\n .await\n .expect("Problem spawning a blocking task");\n\n // Insert processesd data to your database\n db_pool.bulk_insert(documents).await.expect("You probably need an error handling strategy here...");\n }\n })\n .await;\nWe used the chunks\nadaptor to pull hundreds of items at a time for more efficient processing in batches.\nThen, we used for_each_concurrent\nin conjunction with spawn_blocking\nto introduce parallel processing.
Note that neither chunks nor even for_each_concurrent imply any amount of parallelism\non their own.\nspawn_blocking is the only thing that can actually create a new thread of execution!\nChunking simply splits the work into batches (most workloads like this tend to benefit from batching).\nAnd for_each_concurrent allows for concurrent operations over multiple batches.\nBut spawn_blocking is what enables computation in a background thread.\nIf you don't use spawn_blocking,\nyou'll end up blocking Tokio's async workers,\nand your performance will tank.\nJust like the old Node.js code.
The astute reader may point out that using spawn_blocking like this\nis not universally accepted as a solution.\nTokio is (relatively) optimized for non-blocking workloads, so some claim that you should avoid this pattern.\nBut my experience having done this for 5+ years in production code serving over 2 billion requests/month,\nis that Tokio can be a great scheduler for heavier tasks too!
One thing that's often overlooked in these discussions\nis that not all "long-running operations" are the same.\nOne category consists of graphics event loops,\nlong-running continuous computations,\nor other things that may not have an obvious "end."\nBut some tasks can be expected to complete within some period of time,\nthat's longer than a blink.
\nIn the case of the former ("long-lived" tasks), then spawning a dedicated thread often makes sense.\nIn the latter scenario though, Tokio tasks with spawn_blocking can be a great choice.
For our workload, we were doing a lot of the latter sort of operation.\nOne helpful rule of thumb I've seen is that if your task takes longer than tens of microseconds,\nyou should move it off the Tokio worker threads.\nUsing chunks and spawn_blocking avoids this death by a thousand cuts.\nIn our case, the parallelism resulted in a VERY clear speedup.
The original data pipeline was very careful to not overload the data store.\nPerhaps a bit too careful!\nThis may have been necessary at some point in the distant past,\nbut most data storage, from vanilla databases to multi-node clustered storage,\nhave some level of natural backpressure built-in.\nThe Node implementation was essentially limiting the amount of work in-flight that hadn't been flushed.
\nThis premature optimization and the numerous micro-pauses it introduced\nwere another death by a thousand cuts problem.\nDropping the artificial limits approximately doubled throughput.\nIt turned out that our database was able to process 2-4x more records than under the previous implementation.
\nTL;DR — set a reasonable concurrency, let the server tell you when it's chugging (usually via slower response times),\nand let your async runtime handle the rest!
\nSerde, or serialization + deserialization, can be a silent killer.\nAnd unless you're tracking things carefully, you often won't notice!
\nI recently listened to Recoding America at the recommendation of a friend.\nOne of the anecdotes made me want to laugh and cry at the same time.\nEngineers had designed a major improvemnet to GPS, but the rollout is delayed\ndue to a performance problem that renders it unusable.
\nThe project is overseen by Raytheyon, a US government contractor.\nAnd they can't deliver because some arcane federal guidance (not even a regulation proper)\n"recommends" an "Enterprise Service Bus" in the architecture.\nThe startuppper in me dies when I hear such things.\nThe "recommendation" boils down to a data exchange medium where one "service" writes data and another consumes it.\nThink message queues like you may have used before.
\nThis is fine (even necessary) for some applications,\nbut positively crippling for others.\nIn the case of the new positioning system,\nwhich was heavily dependent on timing,\nthis was a wildly inefficient architecture.\nEven worse, the guidelines stated that it should be encrypted.
\nThis wasn't even "bad" guidance, but in the context of the problem,\nwhich depended on rapid exchange of time-sensitive messages,\nit was a horrendously bad fit.
\nIn our data pipeline, I discovered a situation with humorous resemblance in retrospect.\nThe pipeline was set up using a microservice architecture,\nwhich I'm sure souded like a good idea at the time,\nbut it introduced some truly obscene overhead.\nAll services involved were capable of working with data in the same format,\nbut the Node.js implementation was split into multiple services with HTTP and JSON round trips in the middle!\nDouble whammy!
\nThe new data pipeline simply imports the "service" as a crate,\nand gets rid of all the overhead by keeping everything in-process.\nIf you do really need to have a microservice architecture (ex: to scale another service up independently),\nthen other communication + data exchange formats may improve your performance.\nBut if it's possible to keep everything in-process, your overhead is roughly zero.\nThat's hard to beat!
\nIn the end, the new pipeline was 4x the speed of the old.\nI happened to rewrite it in Rust, but Rust itself wasn't the source of all the speedups:\nunderstanding the architecture was.\nYou could achieve similar results in Node.js or Python,\nbut Rust makes it significantly easy to reason about the architecture and correctness of your code.\nThis is especially important when it comes to parallelizing sections of a pipeline,\nwhere Rust's type system will save you from the most common mistakes.
\nThese and other non-performance-related reasons to use Rust will be the subject of a future blog post (or two).
\n", "summary": "", "date_published": "2024-11-29T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "algorithms", "rust", "elasticsearch", "nodejs", "data engineering", "gis" ], "language": "en" } ] }