In my work, I've been doing a lot of ETL pipeline design recently for our geocoding system.\nThe system processes on the order of a billion records per job,\nand failures are part of the process.\nWe want to log these.
\nMost applications start by dumping logs to stderr.\nUntil they overflow their terminal scrollback buffer.\nThe next step is usually text files.\nBut getting insights from 10k+ of lines of text with grep is a chore.\nIt may even be impossible unless you've taken extra care with how your logs are formatted.
In this post we'll explore some approcahes to do application logging better.
\nMy first introduction to logs with a structural element was probably Logcat for Android.\nLogcat lets you filter the fire hose of Android logs down to a specific application,\nand can even refine the scope further if you learn how to use it.\nLogcat is a useful tool, but fundamentally all it can do is filter logs from a stream\nand it has most of the same drawbacks as grepping plain text files.
\nLarger systems often benefit from something like the tracing crate,\nwhich integrates with services like journald and Grafana Loki.\nThis is a great fit for a long-running service,\nbut is total overkill for an application that does some important stuff ™\nand exits.\nLike our ETL pipeline example.
(Aside: I have a love/hate relationship with journalctl.\nI mostly interact with it through Ctrl+R in my shell history,\nwhich is problemating when connecting to a new server.\nBut it does have the benefit of being a nearly ubiquitous local structured logging system!)
Using a database as an application log can be a brilliant level up for many applications\nbecause you can actually query your logs with ease.\nI'll give a few examples, and then show some crazy cool stuff you can do with that.
\nOne type of failure we frequently encounter is metadata that looks like a URL where it shouldn't be.\nFor example, the name of a shop being http://spam.example.com/,\nor having a URL in an address or phone number field.\nIn this case, we usually drop the record, but we also want to log it so we can clean up the source data.\nSome other common failures are missing required fields, data in the wrong format, and the like.
Rather than logging these to stderr or some plain text files, we write to a DuckDB database.\nThis has a few benefits beyond the obvious.\nFirst, using a database forces you to come up with a schema.\nAnd just like using a language with types, this forces you to clarify your thinking a bit upfront.\nIn our case, we log things like the original data source, an ID, a log level (warn, error, info, etc.),\na failure code, and additional details.
From here, we can do meaningufl analytical queries like\n"how many records were dropped due to invalid geographic coordinates"\nor "how many records were rejected due to metadata mismatches"\n(ex: claiming to be a US address but appearing in North Korea).
\nIf this query uncovers a lot of rejected records from one data source,\nwouldn't it be nice if we could look at a sample?\nWe have the IDs right there in the log, and the data source identifier, after all.\nBut since we're in DuckDB rather than a plain text file,\nwe can pretty much effortlessly join on the data files!\n(This assumes that your data is in some halfway sane format like JSON, CSV, Parquet, or even another database).
\nWe can even take this one step further and compare logs across imports!\nWhat's up with that spike in errors compared to last month's release from that data source?
\nThese are the sort of insights which are almost trivial to uncover when your log is a database.
\nNow that I've described all the awesome things you can do,\nlet's get down to the practical questions like how you'd do this in your app.\nMy goals for the code were to make it easy to use and impossible to get wrong at the use site.\nFortunately that's pretty easy in Rust!
\n#[derive(Clone)]\npub struct ImportLogger {\n pool: Pool<DuckdbConnectionManager>,\n // Implementation detail for our case: we have multiple ETL importers that share code AND logs.\n // If you have any such attributes that will remain fixed over the life of a logger instance,\n // consider storing them as struct fields so each event is easier to log.\n importer_name: String,\n}\nPretty standard struct setup using DuckDB and r2d2 for connection pooling.\nWe put this in a shared logging crate in a workspace containing multiple importers.\nThe importer_name is a field that will get emitted with every log,\nand doesn't change for a logger instance.\nIf your logging has any such attributes (ex: a component name),\nstoring them as struct fields makes each log invocation easier!
Note
\nAt the time of this writing, I couldn't find any async connection pool integrations for DuckDB.\nIf anyone knows of one (or wants to add it to bb8), let me know!
pub fn new(config: ImportLogConfig, importer_name: String) -> anyhow::Result<ImportLogger> {\n let manager = DuckdbConnectionManager::file(config.import_log_path)?;\n let pool = Pool::new(manager)?;\n\n pool.get()?.execute_batch(include_str!("schema.sql"))?;\n\n Ok(Self {\n pool,\n importer_name,\n })\n}\nThe constructor isn't anything special; it sets up a DuckDB connection to a file-backed database\nbased on our configuration.\nIt also initializes the schema from a file.\nThe schema file lives in the source tree, but the lovely include_str!\nmacro bakes it into a static string at compile time (so we can still distribute a single binary).
pub fn log(&self, level: Level, source: &str, id: Option<&str>, code: &str, reason: &str) {\n log::log!(level, "{code}\\t{source}\\t{id:?}\\t{reason}");\n let conn = match self.pool.get() {\n Ok(conn) => conn,\n Err(e) => {\n log::error!("failed to get connection: {}", e);\n return;\n }\n };\n match conn.execute(\n "INSERT INTO logs VALUES (current_timestamp, ?, ?, ?, ?, ?, ?)",\n params![level.as_str(), self.importer_name, source, id, code, reason],\n ) {\n Ok(_) => (),\n Err(e) => log::error!("Failed to insert log entry: {}", e),\n }\n}\nAnd now the meat of the logging!\nThe log method does what you'd expect.\nThe signature is a reflection of the schema:\nwhat you need to log, what you may optionally log, and what type of data you're logging.
For our use case, we decided to additionally log via the log crate.\nThis way, we can see critical errors on the console to as the job is running.
And that's pretty much it!\nIt took significantly more time to write this post than to actually write the code.\nSomeone could probably write a macro-based crate to generate these sorts of loggers if they had some spare time ;)
\nfilter_logWe have a pretty common pattern in our codebase,\nwhere most operations / pipeline stages yield results,\nand we want to chain these together.\nWhen it succeeds, we pass the result on to the next stage.\nOtherwise, we want to log what went wrong.
\nWe called this filter_log because it usually shows up in filter_map over streams\nand as such yields an Option<T>.
This was extremely easy to add to our logging struct,\nand saves loads of boilerplate!
\n/// Converts a result to an option, logging the failure if the result is an `Err` variant.\npub fn filter_log<T, E: Debug>(\n &self,\n level: Level,\n source: &str,\n id: Option<&str>,\n code: &str,\n result: Result<T, E>,\n) -> Option<T> {\n match result {\n Ok(result) => Some(result),\n Err(err) => {\n self.log(level, source, id, code, &format!("{:?}", err));\n None\n }\n }\n}\nThe concept of logging to a database is not at all original with me\nMany enterprise services log extensively to special database tables.\nBut I think the technique is rarely applied to applications.
\nHopefully this post convinced you to give it a try in the next situation where it makes sense.
\n", "summary": "", "date_published": "2025-01-13T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "software-engineering", "duckdb", "databases", "rust" ], "language": "en" } ] }