Case-insensitive ext4
Case-insensitive ext4
Posted Mar 27, 2019 18:12 UTC (Wed) by clugstj (subscriber, #4020)Parent article: Case-insensitive ext4
Posted Mar 27, 2019 19:08 UTC (Wed)
by marcH (subscriber, #57642)
[Link] (1 responses)
Without overly complicated code security researchers wouldn't have any work to do!
> Case-insensitivity would be set on a per-directory basis
Insanity has no limit. I was using the (otherwise pretty cool) Windows Subsystem for Linux. This is what happened:
Because I was using the same project sometimes from WSL and sometimes from Windows, some directories *in the same project* were created case-sensitive and others not. Hilarity ensued.
Posted Mar 27, 2019 19:47 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link]
rule copy
saying that no rule makes FOO even though technically it will exist if you build foo. Basically, build tools that exist today need cases to match everywhere. And yes, ninja could figure this out right now, but if `dir/foo` and `dir/FOO` is used and `dir` is made by some rule during the build, its case sensitive flag can't be known at the start.
Case insensitivity in filesystems is broken. Conditional case sensitivity at a per-filesystem level means even ninja needs to add ioctl queries to figure that out, but `--one-file-system` is something that is at least enforceable. Per-directory flags which require magical "what will the flag on this directory be in the future" is even more broken.
I'd be surprised if "doesn't work in case insensitive ext4 directories" (nevermind an environment with a mix of case sensitive and insensitive directories) issues don't get closed as WONTFIX in many tools.
Posted Mar 27, 2019 19:10 UTC (Wed)
by Karellen (subscriber, #67644)
[Link] (20 responses)
Posted Mar 27, 2019 19:25 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (19 responses)
Posted Mar 27, 2019 20:02 UTC (Wed)
by rweikusat2 (subscriber, #117920)
[Link] (15 responses)
Posted Mar 27, 2019 20:05 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (11 responses)
Posted Mar 27, 2019 21:17 UTC (Wed)
by rweikusat2 (subscriber, #117920)
[Link] (8 responses)
Posted Mar 27, 2019 21:26 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link]
I've seen this firsthand - I'm using a Linux server for TimeMachine backups for Mac OS X. TimeMachine is braindead - it creates hundreds of thousands files in the same directory. With the default settings Samba slowed down to a crawl.
Fortunately, TimeMachine doesn't care about file name cases. So by following steps from here: https://wiki.samba.org/index.php/Performance_Tuning I was able to speed up backups by something like 10x. This is not insignificant and it would be nice for Linux to handle similar use-cases natively.
Posted Mar 28, 2019 0:50 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link] (6 responses)
Have you talked to Samba developers and asked them if they are happy with the current performance or would like to see better support from the kernel? If you haven't I would encourage you to do that or talk to enterprises supporting Samba or even large customers. I think you will find that perspectives useful to add to your opinions.
Posted Mar 28, 2019 16:10 UTC (Thu)
by rweikusat2 (subscriber, #117920)
[Link] (5 responses)
Posted Mar 28, 2019 18:36 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
Posted Mar 29, 2019 3:03 UTC (Fri)
by pabs (subscriber, #43278)
[Link] (2 responses)
Posted Mar 29, 2019 4:21 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
fanotify is better, but it also can drop events from time to time under high load.
Posted Oct 4, 2023 18:51 UTC (Wed)
by calumapplepie (guest, #143655)
[Link]
Posted Mar 29, 2019 22:55 UTC (Fri)
by jra (subscriber, #55261)
[Link]
Unfortunately it isn't enough. Cache misses are the problem. If the SMB client sends a filename "foo" and it isn't in the directory, we don't know if it doesn't exist, or exists under another case (e.g. as "Foo"). In that case we need to scan the directory. This gets really expensive, really quickly.
We don't negatively cache as we're often used to export filesystems that local processes are also modifying.
I've been wanting a case-insensitive filesystem lookup option in Linux for a long time (I think ZFS and XFS already have it, however flawed).
Posted Mar 28, 2019 7:28 UTC (Thu)
by patrakov (subscriber, #97174)
[Link] (1 responses)
Posted Mar 28, 2019 7:35 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link]
But this will break a ton of other software that wants to directly modify the disk files. It will also mean that Linux's VFS is inadequate for a fairly common use-case.
Posted Mar 27, 2019 20:17 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link] (2 responses)
Sorry for the snark, it's not in response to your comment in particular, but my mind coming up with all the Pandora's boxes this is threatening to open.
Posted Mar 27, 2019 21:11 UTC (Wed)
by rweikusat2 (subscriber, #117920)
[Link] (1 responses)
It's possible to implement case-insensitive open in user space without doing a second linear search through a directory for every open.
Posted Mar 27, 2019 21:19 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link]
There's also the problem of making sure that no duplicate files exist.
Posted Mar 29, 2019 9:16 UTC (Fri)
by Karellen (subscriber, #67644)
[Link] (2 responses)
How is a call to open() getting the filename to open? Either it's going to from an existing directory scan, in which case the capitalisation/normal form should already be correct, or it's going to be because a user has selected a file - in which case the shell/picker/whatever should be able to do that work already?
Where would calls to open() be getting these correctly named but incorrectly capitalised/normalised filenames from?
Posted Mar 29, 2019 9:20 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
You can try a happy case and just attempt an open() with the provided name. If it fails, you need to scan the directory to find a matching file with a different case.
And you can't really cache the negative result, patterns like "if !exists(fname) {creat(fname);}" are exceedingly common.
Posted Apr 4, 2019 17:09 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
The user, maybe?
What about the use case where I type in a name in a picker, and it displays a bunch of matches?
Or what about the case where I typed in the name on the command line? Some of us still use a command line, you know ...
Cheers,
Posted Mar 28, 2019 2:05 UTC (Thu)
by dw (subscriber, #12017)
[Link] (1 responses)
Was disgusted just last night to discover a Gtk chooser dialog's autocomplete was case sensitive. In a GUI. Total disconnect between Linux and what the real world has been doing successfully for decades now..
Posted Mar 28, 2019 10:44 UTC (Thu)
by mpr22 (subscriber, #60784)
[Link]
I agree that the Gtk file chooser having case-sensitive autocomplete is daft, but... I don't actually care, because I hate the Gtk file chooser anyway for other, more fundamental design decisions.
Posted Mar 28, 2019 2:10 UTC (Thu)
by dw (subscriber, #12017)
[Link] (5 responses)
I was disgusted just last night to discover a Gtk chooser dialog's autocomplete was case sensitive. In a GUI. In 2019. Total disconnect between Linux and what the real world has been doing successfully for decades, and what actual users expect. No doubt someone will pop up to say 'but I prefer it that way', well, you're free patch whatever brainwrong you like into your desktop, but most people cannot and do not want that -- it's why contemporary developers are walking around with MacBooks rather than Linux boxes
Posted Mar 28, 2019 3:56 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link] (4 responses)
foo:
Because if so, this means that tools now need to make a syscall just to do path manipulation to be accurate (something like canonpath() that would give a path which is the same for all equivalent input paths maybe by doing tolower() and normalization). And it has to work for paths that don't exist yet. And I don't think that can even be correct because that path might end up having a bind mount in there at some point which changes behavior (yeah, low chance, but kernels don't always have that luxury).
Yeah, case insensitivity might be useful at the UI level, but even there you still have to deal with paths using binary data or invalid utf8 because a file that the GUI can't delete is a wonderful thing to diagnose and resolve. Personally, I don't find it that useful (but I encourage you to file an issue against GTK for the completion thing).
Posted Mar 28, 2019 19:11 UTC (Thu)
by jccleaver (subscriber, #127418)
[Link] (3 responses)
Classic Mac OS was designed with case-insensitivity in mind, had no manual tools that needed to be imported with minimal effort rather than a complete rewrite, and had no shell mechanics to emulate.
Case Insensitivity #JustWorks when people expect it and are going through translation layers (and aren't in the business of writing drivers), and doesn't when people assume low level access.
Posted Mar 28, 2019 20:14 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link] (2 responses)
Would you have expected the shown Makefile snippet to work on Classic Mac OS or would an error that "no rule to make FOO" be acceptable?
[1]Making a path appear in Explorer via a network share with the name "CON1" renders as some mangled name. Creating a file with that mangled name then shows two files with the same name appear. Deleting either one via the UI deletes the one with the real mangled name first (I assume given a HANDLE, they can be differentiated).
Posted Mar 28, 2019 20:35 UTC (Thu)
by k8to (guest, #15413)
[Link]
I think that was the approach taken by other people too, probably one of Apple Single or Apple Double representations which probably had some solution for NFS which was still in vogue in the 90s.
It wasn't that nice an experience for the Mac users or the non-mac users. I never programmed against it to experience the extra sharp edges, though.
Posted Mar 28, 2019 21:01 UTC (Thu)
by jccleaver (subscriber, #127418)
[Link]
I think by System 7.5 (or 7.1 Pro) you did, because if I recall correctly that's how File Exchange/PC Exchange did its work.
Remember, in classic Mac OS the colon ':' was the directory separator in paths, and you could use '/'s to your heart's content. Actually, you could use pretty much anything to your heart's content, including spaces, punctuation (since no one in the Mac side cared about extensions) and even weird graphs like the f-hook or florin https://en.wikipedia.org/wiki/%C6%91#Appearance_in_comput... , which I still find myself occasionally doing on OS X 20 years later.
Anyway, with /. \. and : being used in different locations, there was definitely path-mangling going on below the interface. But general users didn't have to care, and most Mac programs didn't deal with constructed path names, and *never* had to worry about shell-quoting for spaces and whatnot.
Between this freeform text attitude, the resource and data fork dichotomy, and the use of Type and Creator codes, I definitely feel like we've lost some good capabilities on the Mac side in the quest for broader interoperability.
Posted Mar 28, 2019 8:11 UTC (Thu)
by daniels (subscriber, #16193)
[Link]
No, everyone involved is just doing this for absolutely no reason at all. Weird.
Posted Mar 28, 2019 8:39 UTC (Thu)
by nim-nim (subscriber, #34454)
[Link] (2 responses)
So any shared filesystem will need to export to userspace the encoding used for each part of its tree (either a single encoding for everything, or separate encodings per subtree).
Casing is something else but once you get past the encoding point casing becomes a less harder to tackle.
Posted Mar 28, 2019 15:58 UTC (Thu)
by nybble41 (subscriber, #55106)
[Link] (1 responses)
Not much less. Casing rules depend not just on encoding but also locale, and while it may be practical to enforce a single universal encoding and normalization scheme you're definitely not going to get away with enforcing a single universal locale.
The logical way to handle normalization is to simply disallow non-normalized filenames. The kernel doesn't change the encoding or compare different normal forms, it just verifies that the names of new files are in a particular normal form and returns an error if they aren't. Since all names are already in the same normal form comparisons reduce to exact binary matches. The equivalent for case would be to disallow either lowercase or uppercase characters in filenames (assuming you could even clearly define what is "uppercase" or "lowercase"—it depends on the locale). People put up with that in the DOS era but I don't think it would be considered acceptable today.
The odds that encoding or normalization would be permitted to vary per-filesystem or per-subtree are negligible. Applications aren't prepared to deal with that, nor should they be expected to do so. Any conversions needed for shared filesystems should be handled at the lowest layers of the filesystem, between the storage or network and the kernel.
Posted Mar 29, 2019 10:52 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link]
That's the part people object to, because they are used to the simplicity of pushing encoding problems somewhere else, with "filenames are streams of bytes". Which was not true even for original UNIX. Actual original Unix filename bytes were 7bit ASCII bytes and nothing else.
But 7bit ASCII is useless in a modern i18n world. So you need to record other pivot encoding(s) in filesystems¹.
¹ Record, not reproduce the mistake of original UNIX, that assumed there was a single encoding that would never evolve so there was no need to make it explicit; easy mistake to made in the simpler computer age they lived in; inexcusable mistake to make today.
Posted Mar 28, 2019 14:28 UTC (Thu)
by smurf (subscriber, #17840)
[Link]
Case-insensitive ext4
https://github.com/vector-of-bool/vscode-cmake-tools/issu...
Case-insensitive ext4
command = cp $in $out
build foo: copy in
build bar: copy FOO
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Nope. There is no way to maintain this cache with any sort of consistency guarantees. Linux filesystem change notifications are not up to it.
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
You have an SMB request to open a file, with a file name. There's nothing else.
Case-insensitive ext4
Wol
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
touch foo
bar: Foo
cp Foo bar
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4
Case-insensitive ext4