I have been a fan of JetBrains products for over a decade by now,\nand an unapologetic lover of IDEs generally.\nI've used PyCharm since shortly after it launched,\nand over the years I've used IntelliJ IDEA,\nWebStorm, DataGrip, RustRover, and more.\nI literally have the all products pack (and have for many years).
\nI truly believe that a good IDE can be a productivity multiplier.\nYou get refactoring, jump-to-definition, symbol-aware search,\nsaved build/run configurations, a nice and consistent interface\nto otherwise terrible tooling (looking at you CMake and the half dozen Python package managers\nof the last decade and change).
\nBut something has changed over the past few years.\nThe quality of the product has generally deteriorated in several ways.\nWith the advent of LSP, the massive lead JetBrains had in "code intelligence"\nhas eroded, and in many cases no longer exists.\nThe resource requirements of the IDE have also ballooned massively,\neven occasionally causing memory pressure on my amply equipped MacBook Pro with 32GB of RAM.
\n(Side note: I regularly have 3 JetBrains IDEs open at once because I need to work in many languages,\nand for some reason they refuse to ship a single product that does that.\nI would have paid for such a product.)
\nAnd as if that weren't enough, it seems like I have to restart to install some urgent nagging update\nseveral times/week, usually related to one of their confusing mess of AI plugins\n(is AI Chat what we're supposed to use? Or Junie? Or... what?).\nTo top it all off, stability has gone out the window.\nAt least once/week, I will open my laptop from sleep,\nonly to find out that one or more of my JetBrains IDEs has crashed.\nUsually RustRover.\nWhich also eats up like 30GB of extra disk space for things like macro expansions\nand other code analysis.\nThe taxes are high and increasing on every front.
\nSo, I decided the time was right to give Emacs another shot.
\nIf you know me personally, you may recall that I made some strong statements in the past\nto the effect that spending weeks writing thousands of lines of Lua to get the ultimate Neovim config was silly.\nAnd my strongly worded statements of the past were partially based on my own experiences with such editors,\nincluding Emacs.\nBasically, I appreciate that you can "build your own lightsaber",\nbut I did not consider that to be a good use of my time.\nOne of the reasons I like(d) JetBrains is that I didn't ever need to think about tweaking configs!
\nBut things have gotten so bad that I figured I'd give it a shot with a few stipulations.
\nWith these constraints, I set off to see if I needed to revise my philosophy of editors.
\nAside: why not (Helix|Neovim|Zed|something else)?\nA few reasons, in no particular order:
\nSo, how's it going so far?\nHere are a few of the highlights.
\nIt used to be the case that JetBrains had a dominant position in code analysis.\nThis isn't the case anymore, and most of the languages I use that would benefit from an LSP\nhave a great one available.\nThings have improved a lot, particularly in terms of Emacs integrations,\nover the past decade!\neglot is now bundled with Emacs,\nso you don't even need to go out of your way to get some funky packages hooked up\n(like I had to with some flycheck plugin for Haskell back in the day).
The LSP-guided tools for refactoring have also improved a lot.\nIt used to be that only a "real IDE" had much better than grep and replace.\nI was happy to find that eglot-rename "just worked".
I'm used to hovering my mouse over any bit of code, waiting a few seconds,\nand being greeted by a docs popover.\nThis is now possible in Emacs too with eldoc + your LSP.\nI added the eldoc-box plugin and configured it to my liking.
So far, every single quick-fix action that I'm used to in RustRover\nseems to be there in the eglot integration with rust-analyzer.\nIt took me a few minutes to realize that this was called eglot-code-actions),\nbut once I figured that out, I was rolling.
I frequently use the jump-to-definition feature in IDEs.\nUsually by command+clicking.\nYou can do the same in Emacs with M-., which is a bit weird, but okay.\nI picked up the muscle memory after less than an hour.\nThe weird thing though is what happens next.\nI'm used to JetBrains and most other well-designed software (glares in the general direction of Apple)\n"just working" with the forward+back buttons that many input devices have.\nEmacs did not out of the box.
One thing JetBrains did fairly well was bookmarking where you were in a file, and even letting you jump back after\nnavigating to the definition or to another file.\nThis had some annoying side effects with multiple tabs, which I won't get into but it worked overall.\nIn Emacs, you can return from a definition jump with M-,, but there is no general navigate forward/backward concept.\nThis is where the build-your-own-lightsaber philosophy comes in I guess.\nI knew I'd hit it eventually.
I tried out a package called better-jumper but it didn't immediately do what I wanted,\nso I abandoned it.\nI opted instead to simple backward and forward navigation.\nIt works alright.
(global-set-key (kbd "<mouse-3>") #'previous-buffer)\n(global-set-key (kbd "<mouse-4>") #'next-buffer)\nAside: I had to use C-h k (describe-key) to figure out what the mouse buttons were.\nAdvice I saw online apparently isn't universally applicable,\nand Xorg, macOS, etc. may number the buttons differently!
The emacs shell mode is terrible.\nIt's particularly unusable if you're running any sort of TUI application.\nA friend recommended eat as an alternative.\nThis worked pretty well out of the box with most things,\nbut when I ran cargo nextest for the first time,\nI was shocked at how slow it was.\nMy test suite which normally runs in under a second took over 30!\nYikes.\nI believe the slowness is because it's implemented in elisp,\nwhich is still pretty slow even when native compilation is enabled.
Another Emacs user recommended I try out vterm, so I did.\nHallelujah!\nIt's no iTerm 2, and it does have a few quirks,\nbut it's quite usable and MUCH faster.\nIt also works better with full-screen TUI apps like Claude Code.
I'm not going to get into the pros and cons of LLMs in this post.\nBut if you use these tools in your work,\nI think you'll be surprised by how good the experience is with vterm and the claude CLI.\nI have been evaluating JetBrains' disjoint attempts at integrations with Junie,\nand more recently Claude Code and Codex.
Junie is alright for some things.\nThe only really good thing I have to say about the product is that at least it let me select a GPT model.\nAnthropic models have been severely hampered in their ability to do anything useful in most codebases I work in,\ndue to tiny context windows.\nThat recently changed when Anthropic rolled out a 1 million token context window to certain users.
\nJetBrains confusingly refers to Claude Code as "Claude Agent" and team subscriptions automatically include some monthly credits.\nEvery single JetBrains IDE will install its own separate copy of Claude Code (yay).\nBut it is really just shelling out to Claude Code it seems\n(it asks for your permission to download the binary.\nCodex is the same.)
\nGiven this, I assumed the experience and overall quality would be similar.\nWell, I was VERY wrong there.\nClaude Code in the terminal is far superior for a number of reasons.\nNot just access to the new model though that helps.\nYou can also configure "effort" (lol), and the "plan" mode seems to be far more sophisticated than what you get in the JetBrains IDEs.
\nSo yeah, if you're going to use these tools, just use the official app.\nIt makes sense; they have an incentive to push people to buy direct.\nAnd it so happens that Claude Code fits comfortably in my Emacs environment.
\nMore directly relevant to this post,\nLLMs (any of them really) are excellent at recommending Emacs packages and config tweaks.\nSo it's never been easier to give it a try.\nI've spent something like 2-3x longer writing this post than I did configuring Emacs.\n(And yes, before you ask, this post is 100% hand-written.)\nMy basic flow was to work, get annoyed (thats pretty easy for me),\nand describe my problem to ChatGPT or Claude.\nI am nowhere near the hours I budgeted for config fiddling.\nThat surprised me!
\nWhile I'm no stranger to hacking around with nothing more than a console,\nI really don't like the git CLI.\nI've heard jj is better, but honestly I think GUIs are pretty great most of the time.\nI will probably try magit at some point,\nbut for now I'm very happy with Sublime Merge.
\nBut one thing I MUST have in my editor is a "gutter" view of lines that are new/changed,\nand a way to get a quick inline diff.\nJetBrains had a great UX for this which I used daily.\nAnd for Emacs, I found something just as great: diff-hl.
My config for this is very simple:
\n(unless (package-installed-p 'diff-hl)\n (package-install 'diff-hl))\n(use-package diff-hl\n :config\n (global-diff-hl-mode))\nTo get a quick diff of a section that's changed,\nI use diff-hl-show-chunk.\nI might even like the hunk review experience here better than in JetBrains!
I think JetBrains has the best search around with their double-shift, cmd+shift+o, and cmd-shift-f views.\nI have not yet gotten my Emacs configured to be as good.\nBut C-x p g (project-find-regexp) is pretty close.\nI'll look into other plugins later for fuzzy filename/symbol search.\nI do miss that.
The final pleasant surprise is that I don't miss JetBrains run configurations as much as I expected.\nI instead switch to putting a justfile in my repo and populating that with my run configurations\n(much of the software I work on has half a dozen switches which vary by environment).\nThis also has the side effect of cleaning up some of my CI configuration (just run the same thing!)\nand also serves as useful documentation to LLMs.
I have typos configured for most of my projects in CI,\nbut it drives me nuts when an editor doesn't flag typos for me.\nJetBrains did this well.\nEmacs has nothing out of the box (Zed also annoyingly doesn't ship with anything, which is really confusing to me).\nBut it's easy to add.
I went with Jinx.\nThere are other options, but this one seemed pretty modern and worked without any fuss, so I stuck with it.
\nThis is all a lot more positive than I was expecting to be honest!\nI am not going to cancel my JetBrains subscription tomorrow;\nthey still do make the best database tool I know of.\nBut I've moved all my daily editing to Emacs.
\nThat said, there are still some papercuts I need to address:
\neglot-x which I'll look into later.cargo fmt)!mut variables. I would love to get that back in Emacs.Here's a fun one!\nYou may already know that tarball is a pure archive format,\nand that any compression is applied to the whole archive as a unit.\nThat is to say that compression is not actually applied at the file level,\nbut to the entire archive.
\nThis is a trade-off the designers made to limit complexity,\nand as a side-effect, is the reason why you can't randomly access parts of a compressed tarball.
\nWhat you may not know is that the tar utility has built-in support for a few formats!\nGZIP is probably the most commonly used for historical reasons,\nbut zstd and lz4 are built-in options on my Mac.\nThis is probably system-dependent, so check your local manpages.
Here's an example of compressing and decompressing with zstd:
tar --zstd -cf directory.tar.zst directory/\ntar --zstd -xf directory.tar.zst\nYou can also use this with any (de)compression program that operates on stdin and stdout!
\ntar --use-compress-program zstd -cf directory.tar.zst directory/\nPretty cool, huh?\nIt's no different that using pipes at the end of the day,\nbut it does simplify the invocation a bit in my opinion.
\nAfter I initially published this article,\n@cartocalypse@norden.social noted that some versions of tar include the\n-a/--auto-compress option which will automatically determine format and compression based on the suffix!\nCheck your manpages for details; it appears to work on FreeBSD, macOS (which inherits the FreeBSD implementation), and GNU tar.
I've been using nushell as my daily driver for about six months now,\nand wanted to show a few simple examples of why I'm enjoying it so much.\nI think it's a breath of fresh air compared to most shells.
\nIn case you've never heard of it before, nushell is a, well, new shell ;)\nbash has been the dominant shell for as long as I can remember,\nthough zsh have their fair share of devotees.\nfish is the only recent example I can think of as a "challenger" shell.\nfish gained enough traction that it's supported by tooling such as Python virtualenv\n(which only has integrations out of the box for a handful of shells).\nI think fish is popular because it had some slightly saner defaults out of the box,\nwas easier to "customize" with flashy prompts (which can make your shell SUPER slow to init),\nand had a saner scripting language than bash.\nBut it still retained a lot of the historical baggage from POSIX shells.
Nushell challenges two common assumptions about shells\nand asks "what if things were different?"
\nls and du) are replaced by builtins.By dropping the goal of POSIX compliance,\nnushell frees itself from decades of baggage.\nThis means you get a scripting language that feels a lot more like Rust.\nYou'll actually get errors by default when you try to do something stupid,\nunlike most shells which will happily proceed,\nusually doing something even more stupid.\nMaybe treating undefined variables as empty string make sense in the 1970s,\nbut that's almost never helpful.
\nnushell also takes a relatively unique approach to utilities.\nWhen you type something like ls or ps in nushell,\nthis is handled by a shell builtin!\nIt's just Rust code baked into the shell rather than calling out to GNU coreutils\nor whatever your base system includes.\nThis means that whether you type ps on FreeBSD, Debian, or macOS,\nyou'll get the same behavior!
I can already hear some readers thinking "doesn't this just massively bloat the shell?"\nNo, not really.\nThe code for these is for less than that of the typical GNU utility,\nbecause nushell actually (IMO) embraces UNIX philosophy even better than the original utilities.\nThey are all extremely minimal and work with other builtins.\nFor example, there are no sorting flags for ls,\nand no format/unit flags for du.
The reason that nushell can take this approach is because they challenge the notion that\n"text is the universal API."\nYou can't meaningfully manipulate text without lots of lossy heuristics.\nBut you can do this for structured data!\nI admit I'm a bit of a chaos monkey, so I love to see a project taking a rare new approach\nin space where nothing has fundamentally changed since the 1970s.
\nOkay, enough about philosophy... here are a few examples of some shell pipelines I found delightful.
\nFirst up: I do a lot of work with Elasticsearch during $DAYJOB.\nOne workflow I have to do fairly often is spin up a cluster and restore from a snapshot.\nThe Elasticsearch API is great... for programs.\nBut I have a hard time grokking hundreds of lines of JSON.\nHere's an example of a pipeline I built which culls the JSON response down to just the section I care about.
http get "http://localhost:9200/myindex/_recovery"\n | get restored-indexname\n | get shards\n | get index\n | get size\nIn case it's not obvious, the http command makes HTTP requests.\nThis is another nushell builtin that is an excellent alternative to curl.\nIt's not as feature-rich (curl has a few decades of head start),\nbut it brings something new to the table: it understands from the response that the content is JSON,\nand converts it into structured data!\nEverything in this pipeline is nushell builtins.\nAnd it'll work on any OS that nushell supports.\nEven Windows!\nThat's wild!
Pro tip: you can press option+enter to add a new line when typing in the shell.
\nHere's an example I hinted at earlier.\nIf you type help du (to get built-in docs),\nyou won't find any flags for changing the units.\nBut you can do it using formatters like so:
du path/to/bigfile.bin | format filesize B apparent\nThe du command always shows human-readable units by default. Which I very much appreciate!\nAnd did you notice apparent at the end there?\nWell, the version of du you'd find with a typical Linux distro doesn't exactly lie to you,\nbut it withholds some very important information.\nThe physical size occupied on disk is not necessarily the same as how large the file\n(in an abstract platonic sense) actually is.
There are a bunch of reasons for this, but the most impactful one is compressed filesystems.\nIf I ask Linux du how large a file is in an OpenZFS dataset,\nit will report the physical size by default, which may be a few hundred megabytes\nwhen the file is really multiple gigabytes.\nNot necessarily helpful.
Anyways, the nushell builtin always gives you columns for both physical and apparent.\nSo you can't ignore the fact that these sizes are often different.\nI like that!
\nIf you want to give nushell a try,\nthey have some great documentation.\nRead the basics, but also check out their specific pages on, e.g.\ncoming from bash.
\nFinally, here are two more that tripped me up at first.
\n$nu.pid instead of $$.$env. On the plus side, you can now explicitly differentiate from environment variables.If you're a developer or sysadmin, there's a pretty good chance you've had to transfer files back and forth.\nBack in the old days, you may have used the ftp utility or something similar\n(I think my first one was probably CuteFTP).\nThen you probably thought better of doing things in plaintext,\nand switched to SFTP/SCP, which operate over an SSH connection.
My quick post today is not exactly a bombshell of new information,\nbut these tools are not the fastest way to transfer.\nThere's often not a huge difference if you're transferring between machines in the same datacenter,\nbut you can do many times better then transferring from home or the office.
\nOf course I'm talking about rsync, which has a lot of features like compression, partial transfer resume,\nchecksumming so just the deltas are sent in a large file, and more.\nI don't know why, but it's rarely in my consciousness, and always strikes me as a bit quirky.\nYou need quite a few flags for what I would consider to be reasonable defaults.\nBut if you remember those (or use shel history, like me),\nit can save you a ton of time.
In fact, this morning, using rsync saved me more time than it took to write this blog post.\nI was transferring a ~40GB file from a server halfway around the world,\nbut I only had to transfer bytes equivalent to 20% of the total.
\nHere's a look at the rsync command I often use for pulling files from a remote server\n(I usually do this to download a large build artifact without going through a cloud storage intermediary,\nwhich is both slower and more eexpensive):
rsync -Pzhv example.com:/remote/path/to/big/file ~/file\nIt's not really that bad compared to an scp invocation, but those flags make all the difference.\nHere's what they do:
-P - Keeps partially transferred files (in case of interruption) and shows progress during the transfer.-z - Compresses data on the source server before sending to the destination. This probably isn't a great idea for an intra-datacenter transfer (it just wastes CPU), but it's perfect for long distance transfers over "slower" links (where I'd say slower is something less than like 100Mbps between you and the server... the last part is important because you may have a gigabit link, but the peering arrangements or other issues conspire to limit effective transfer rates to something much lower).-h - Makes the output more "human-readable." This shows previfexs like K, M, or G. You can add it twice if you'd like 1024-based values instead of 1000-based. To be honest, I don't know why this isn't the default.-v - Verbose mode. By default, rsync is silent. This is another behavior that I find strange in the present, but probably made more sense in the era of teletype terminals and very slow links. It's not really that verbose; it just tells you what files are being transferred and a brief summary at the end. You actually have to give two v's (-vv) for rsync to tell you which files it's skipping!Hope this helps speed up your next remote file transfer.\nIf there's any options you like which I may have missed, hit me up on Mastodon with a suggestion!
\nBonus pro tip: I had (until I recently switched to nushell) over a decade of accumulated shell history,\nand sometimes it's hard to keep the options straight.\nNaturally my history has a few different variations on a command like rsync.\nRather than searching through manpages with the equivalent of grep,\nI usually go to explainshell.com.\nIt seems to be a frontend to the manapages that understands the various sections,\nproviding a much quicker explanation of what your switches do!
Today I had a rather peculiar need to search through features from TIGER\nmatching specific attributes.\nThese files are not CSV or JSON, but rather ESRI Shapefiles.\nShapefiles are a binary format which have long outlived their welcome\naccording to many in the industry, but they still persist today.
\nYeah, so this post probably isn't interesting to very many people,\nbut here's a bit of context in case you don't know what's going on and you're still reading.\nTIGER is a geospatial dataset published by the US government.\nThere's far more to this dataset than fits in this TIL post,\nbut my interest in it lies in finding addresses.\nSpecifically, guessing at where an address might be.
\nWhen you type an address into your maps app,\nthey might not actually have the exact address in their database.\nThis happens more than you might imagine,\nbut you can usually get a pretty good guess of where the address is\nvia a process called interpolation.\nThe basic idea is that you take address data from multiple sources and use that to make a better guess.
\nSome of the input to this is existing address points.\nBut there's one really interesting form of data that brings us to today's TIL:\naddress ranges.\nOne of the TIGER datasets is a set of lines (for the roads.\nEach segment is annotated with info letting us know the range of house numbers on each side of the road.
\nI happen to use this data for my day job at Stadia Maps,\nwhere I was investigating a data issue today related to our geocoder and TIGER data.
\nIn case you find yourself in a similar situation,\nyou may notice that the data from the government is sitting in an FTP directory,\nwhich contains a bunch of confusingly named ZIP files.\nThe data that I'm interested in (address features)\nhas names like tl_2024_48485_addrfeat.zip.
The year might be familiar, but what's that other number?\nThat's a FIPS code for the county whose data is contained in the archive.\nYou can find a list here.\nThis is somewhat interesting in itself, since the first 2 characters are a state code.\nTexas, in this case.\nThe full number makes up a county: Wichita County, in this case.\nYou can suck down the entire dataset, just one file, or anything in-between\nfrom the Census website.
\nSo, now you have a directory full of ZIP files.\nEach of which has a bunch of files necessary to interpret the shapefile.\nIsn't GIS lovely?
\nThe following script will let you write a simple "WHERE" clause,\nfiltering the data exactly as it comes from the Census Bureau!
\n#!/bin/bash\nset -e;\n\nfind "$1" -type f -iname "*.zip" -print0 |\\\n while IFS= read -r -d $'\\0' filename; do\n\n filtered_json=$(ogr2ogr -f GeoJSON -t_srs crs:84 -where "$2" /vsistdout/ /vsizip/$filename);\n # Check if the filtered GeoJSON has any features\n feature_count=$(echo "$filtered_json" | jq '.features | length')\n\n if [ "$feature_count" -gt 0 ]; then\n # echo filename to stderr\n >&2 echo $(date -u) "Match(es) found in $filename";\n echo "$filtered_json";\n fi\n\n done;\nYou can run it like so:
\n./find-tiger-features.sh $HOME/Downloads/tiger-2021/ "TFIDL = 213297979 OR TFIDR = 213297979"\nThis ends up being a LOT easier and faster than QGIS in my experience\nif you want to search for specific known attributes.\nEspecially if you don't know the specific area that you're looking for.\nI was surprised that so such tool for things like ID lookps existed already!
\nNote that this isn't exactly "fast" by typical data processing workload standards.\nIt takes around 10 minutes to run on my laptop.\nBut it's a lot faster than the alternatives in many circumstances,\nespecilaly if you don't know exactly which file the data is in!
\nFor details on the fields available,\nrefer to the technical documentation on the Census Bureau website.
\n", "summary": "", "date_published": "2024-11-09T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "gis", "shell", "ogr2ogr" ], "language": "en" }, { "id": "https://ianwwagner.com//copying-and-unarchiving-from-a-server-without-a-temp-file.html", "url": "https://ianwwagner.com//copying-and-unarchiving-from-a-server-without-a-temp-file.html", "title": "Copying and Unarchiving From a Server Without a Temp File", "content_html": "Sometimes I want to copy files from a remote machine--usually a server I control.\nEasy; just use scp, right?
Well, today I had a subtly different twist to the usual problem.\nI needed to transfer a ~100GB tarball to my local machine,\nand I really wanted to unarchive it so that I could get at the internal data directly.\nAnd I wanted to do it in one step, since I didn't have 200GB of free space.
\nI happened to remember that this should be possible with pipes or something.\nThe tarball format is helpfully designed to allow streaming.\nBut it took me a bit to come up with the right set of commands to do this.
\nscp is really designed for dumping to files first.\nI found a few suggestions on StackOverflow that looked like they might work,\nbut didn't for me (might have been my shell? I use fish rather than bash).\nBut I noticed that almost all of the answers recommended using ssh instead,\nsince it's a bit more suited to the purpose.
The basic idea is to dump the file to standard out on the remote host,\nthen pipe the ssh output into tar locally.\nThe tar flags are probably familiar or easily understandable: -xvf - in my case.\nThis puts tar into extract mode,\nenables verbose logging (so you see its progress),\nand tells it to read from stdin (-).\nMy tarball was not compressed.\nIf yours is, add the appropriate decompression flags.
The SSH flags were a bit trickier.\nI discovered the -C flag, which enables gzip compression.\nI happen to know this dataset compresses well with gzip,\nand further that the network link between me and the remote is not the best,\nso I enabled it.\nDon't use this if your data does not compress well,\nor if it is already compressed.
Another flag, -e none,\nI found via this unix.com forum post.\nThis seemed like a good thing to enable after some research,\nsince sequences like ~. will not be interpreted as "kill the session."\nIt also prevents more subtle bugs which would look like data corruption.
-T was suggested after I pressed ChatGPT o1-preview for other flags that might be helpful.\nIt just doesn't allocate a pseudo-terminal.\nWhich we didn't need anyways.\n(Aside: ChatGPT 4o will give you some hot garbage suggestions; o1-preview was only helpful in suggesting refinements.)
Finally, the command executes cat on the remote host to dump the tarball to stdout.\nI saw suggestions as well to use dd since you can set the block size explicitly.\nThat might improve perf in some situations if you know your hardware well.\nOr it might just be a useless attempt at premature optimization ;)
Here's the final command:
\nssh -C -e none -T host.example.com 'cat /path/to/archive.tar' | tar -xvf -\n