![\"A](\"media/kebab-king-duplicates.png\")
Today's episode of "things that make you go 'wat'" is sponsored by unzip.\nYes, the venerable utility ubiquitous on UNIX-like systems.\nI mean, what could possibly go wrong?
This morning I was minding my own business sipping a coffee,\nwhen suddenly the "simple" CI pipeline I was working on failed.\nI literally copied the command that failed out of a shell script.\nWhich ran on the exact same machine previously.\nAnd the command that failed was a simple unzip.\nHuh?
I initially assumed the file may have been corrupted, or maybe the archive had failed in a weird way\n(I don't control the process, and was downloading it from the internet).\nWhile it was running again, I started hitting Page Down looking for oddities in the log.\nI was greeted by this near the end of the output:
se/municipality_of_savsj#U00f6-addresses-city.geojson: mismatching "local" filename (se/municipality_of_savsjΓΆ-addresses-city.geojson),\n continuing with "central" filename version\nWeird, huh?\nIt looks like it's trying to unzip a file with a UTF-8 name, which should be supported.
\nI ssh'd into the remote machine to give it a try in the terminal.\nI've run this command dozens of times, and wanted to see if I got the same output in a standard bash remote terminal.\nGiven that log output is sometimes inscrutable, I thought "maybe it wasn't the unzip itself that failed directly?"\nOr something...
Well, the unzip worked in the regular bash login shell on the same host.\nSo there must be something different about the CI environment.
As a first resort, I started digging through the help and man pages.\nThe first amusing factoid I learned was that apparently the last official release of this utility happened in 2009.\nI guess zip doesn't evolve much, eh?
\nThe man page was actually pretty detailed,\nbut it didn't have a lot to say about Unicode, or anything else obvious related to the error.\nAnd the project doesn't exactly have an active central issue tracker (there is one on SourceForge, but replies are few and far between).\nI couldn't find anyone else talking about this specific error in the usual places on the internet either.
\nI did however find this interesting bug report via a Debian mailing list\nfrom 2019.\nThe thing that caught my eye was the code sample and mention of locales.
\nOh no...\nWhat if this actually changes behavior based on your system locale?\nThen it started coming back to me... something about a dozen or so environment variables\nthat determine the behavior of C programs in bizarre ways...
\nSo I ran locale on the ssh session and got something reasonable back:
$ locale\nLANG=C.UTF-8\nLANGUAGE=\nLC_CTYPE="C.UTF-8"\nLC_NUMERIC="C.UTF-8"\nLC_TIME="C.UTF-8"\nLC_COLLATE="C.UTF-8"\nLC_MONETARY="C.UTF-8"\nLC_MESSAGES="C.UTF-8"\nLC_PAPER="C.UTF-8"\nLC_NAME="C.UTF-8"\nLC_ADDRESS="C.UTF-8"\nLC_TELEPHONE="C.UTF-8"\nLC_MEASUREMENT="C.UTF-8"\nLC_IDENTIFICATION="C.UTF-8"\nLC_ALL=\nThen, to avoid wasting even more hours of CI time (it's a long job whose source is a ~100GB ZIP file),\nI set about replicating the environment of the runner.\nFortunately it left a Docker volume behind for me.\nSo, I launched an interactive Docker container using the same image that the CI job used (python:3.13).\nSomewhat surprisingly, the output of locale was a bunch of variables set to.... nothing!\nNow we're getting somewhere!
To confirm that I could indeed reproduce the failure, I ran unzip again in the container,\nand sure enough, I got the same log message, and the exit code was 1.\nNow we've confirmed how to reproduce the issue at least, so we can test a fix.
unzipSince unzip is single threaded (read: SLOW),\nI spent my time during the tests looking through the source code to try and confirm my theory about the locale variables being the issue.\nThe official version seems to be hosted on SourceForge, which is apparently still a thing.\n(I'm sure there are a lot of Debian patches, but I just wanted a quick way to peruse the code).\nAmusingly, there were 47,074 downloads/week of unzip60.tar.gz, and only 36 downloads/week of the ZIP version.
Eventually, I found what I was looking for in unzip.c:
int unzip(__G__ argc, argv)\n __GDEF\n int argc;\n char *argv[];\n{\n#ifndef NO_ZIPINFO\n char *p;\n#endif\n#if (defined(DOS_FLX_H68_NLM_OS2_W32) || !defined(SFX))\n int i;\n#endif\n int retcode, error=FALSE;\n#ifndef NO_EXCEPT_SIGNALS\n#ifdef REENTRANT\n savsigs_info *oldsighandlers = NULL;\n# define SET_SIGHANDLER(sigtype, newsighandler) \\\n if ((retcode = setsignalhandler(__G__ &oldsighandlers, (sigtype), \\\n (newsighandler))) > PK_WARN) \\\n goto cleanup_and_exit\n#else\n# define SET_SIGHANDLER(sigtype, newsighandler) \\\n signal((sigtype), (newsighandler))\n#endif\n#endif /* NO_EXCEPT_SIGNALS */\n\n /* initialize international char support to the current environment */\n SETLOCALE(LC_CTYPE, "");\n\n#ifdef UNICODE_SUPPORT\n /* see if can use UTF-8 Unicode locale */\n# ifdef UTF8_MAYBE_NATIVE\n {\n char *codeset;\n# if !(defined(NO_NL_LANGINFO) || defined(NO_LANGINFO_H))\n /* get the codeset (character set encoding) currently used */\n# include <langinfo.h>\n\n codeset = nl_langinfo(CODESET);\n# else /* NO_NL_LANGINFO || NO_LANGINFO_H */\n /* query the current locale setting for character classification */\n codeset = setlocale(LC_CTYPE, NULL);\n if (codeset != NULL) {\n /* extract the codeset portion of the locale name */\n codeset = strchr(codeset, '.');\n if (codeset != NULL) ++codeset;\n }\n# endif /* ?(NO_NL_LANGINFO || NO_LANGINFO_H) */\n /* is the current codeset UTF-8 ? */\n if ((codeset != NULL) && (strcmp(codeset, "UTF-8") == 0)) {\n /* successfully found UTF-8 char coding */\n G.native_is_utf8 = TRUE;\n } else {\n /* Current codeset is not UTF-8 or cannot be determined. */\n G.native_is_utf8 = FALSE;\n }\n /* Note: At least for UnZip, trying to change the process codeset to\n * UTF-8 does not work. For the example Linux setup of the\n * UnZip maintainer, a successful switch to "en-US.UTF-8"\n * resulted in garbage display of all non-basic ASCII characters.\n */\n }\n# endif /* UTF8_MAYBE_NATIVE */\nAnd there we have it, right at the start of the program...
\nThis is a bit rough to grok if you don't regularly read C,\nbut the gist of it is that it tries to look at the system locale\nusing setlocale.\nThe LC_CTYPE specifies the types of character used in the locale.\nThe second argument is... very C.\nThe function will "install the specified system locale... as the new C locale."
It defaults to "C" at startup, per the docs.\nIn unzip.c though, the authors use the magic value "",\nwhose behavior is to set the locale to the "user-preferred locale".\nThis comes from those environment variables.\nA few lines down, past the #ifdefs gating unicode support,\nthey parse the codeset portion by passing NULL, which is ANOTHER magic value\nthat simply returns the current value.\n(So this two-step dance loads the preferred locale from the environment variables,\nwhich are empty in this case, and then inspects the codeset.).
If your environment variables do not explicitly specify UTF-8,\nyou will get some.... strange and undesirable behavior, apparently,\nif your archive contains unicode file names!
\nI'm pretty sure there's a good historical reason for this.\nGiven that many C library functions are locale-sensitive,\nand UTF-8 support is much younger than UNIX and (obviously) C.\nBut I found this behavior from unzip to be a bit surprising nonetheless.\nAnd rather than setting a sensible default like C.UTF-8,\nit turns out most Docker containers start with none!
Fortunately this is pretty easy to fix.\nJust export LC_ALL="C.UTF-8" before using unzip.\n(There is probably a more granular approach, but the LC_ALL sledgehammer does the job.)\nKinda crazy that you have to do this,\nbut hopefully this saves someone else an afternoon of debugging!
Today I ran into an amusingly named place,\nthanks to some sharp eyes on the OpenStreetMap US Slack.\nThe name of this restaurant is listed as "πππππ ππππ ππππππ".\nThat isn't some font trickery; it's a bunch of Unicode math symbols\ncleverly used to emphasize the name.\n(Amusingly, this does not actually show up properly on most maps, but that's another story for another post).
\nI was immediately curious how well the geocoder I spent the last few months building handles this.
\n
Well, at least it found the place, despite the very SEO-unfriendly name!\nBut what's up with the second search result?
\nWell, that's a consequence of us pulling in data from multiple sources.\nIn this case, the second result comes from the Foursquare OS Places dataset.\nIt seems that either the Placemaker validators decided to clean this up,\nor the Foursquare user who added the place didn't have that key on their phone keyboard.
\nOne of the things our geocoder needs to do when combining results is deduplicating results.\n(Beyond that, it needs to decide which results to keep, but that's a much longer post!)\nWe use a bunch of factors to make that decision, but one of them is roughly\n"does this place have the same name," where same is a bit fuzzy.
\nOne of the ways we can do this is normalizing away things like punctuation and diacritics.\nThese are quite frequently inconsistent across datasets, so two nearby results with similar enough names\nare probably the same place.\nFortunately, Unicode provides a few standardized transformations into canonical forms\nthat make this easier.
\nWhat we think of as a "character" does not necessarily have a single representation in Unicode.\nFor example, there are multiple ways of encoding "μμΈ" which will look the same when rendered,\nbut have a different binary representation.\nThe Korean writing system is perhaps a less familiar case for many,\nbut characters with diacritical marks such as accents are usually the same.\nThey can be either "composed" or "decomposed" into the component parts\nat the binary level.
\nThis composition and decomposition transform is useful for (at least) two reasons:
\nI use the unicode_normalization crate\nto do this "decompose and filter" operation.\nSpecifically, the UnicodeNormalization trait,\nwhich has helpers which will work on most string-like types.
You might notice there are four confusingly named methods in the trait:\nnfd, nfkd, nfc, and nfkc.\nThe nf stands for "normalization form".\nThese functions normalize your strings.\nc and d stand for composition and decomposition.\nThe composed form is, roughly, the more compact form,\nwhereas the decomposed form is the version where you separate the base from the modifiers,\nthe jamo from the syllables, etc.
We were already decomposing strings so that we could remove the diacritics, using form NFD.\nThis works great for diacritics and even Hangul,\nbut πππππ ππππ ππππππ shows that we were missing something.
\nThat something is the k, which stands for "compatibility."\nYou can refer to Unicode Standard Annex #15\nfor a full definition,\nbut the intuition is that compatibility equivalence of two characters\nis a bit more permissive than the stricter canonical equivalence.\nBy reducing two characters (or strings) to their canonical form,\nyou will be able to tell if they represent the same "thing" with the same visual appearance,\nbehavior, semantic meaning, etc.\nCompatibility equivalence is a weaker form.
Compatibility equivalence is extremely useful in our quest for determining whether two nearby place names\nare a fuzzy match.\nIt reduces things like ligatures, superscripts, and width variations into a standard form.\nIn the case of "πππππ ππππ ππππππ," compatibility decomposition transforms it into the ASCII\n"KEBAB KING YANGON."\nAnd now we can correctly coalesce the available information into a single search result.
\nHopefully this shines a light on one small corner of the complexities of unicode!
\n", "summary": "", "date_published": "2025-05-09T00:00:00-00:00", "image": "media/kebab-king-duplicates.png", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "unicode", "rust" ], "language": "en" } ] }