Tickets #2325 & #4480: cd into directories with long names and/or embedded \n

[egmontkob]

Proposed changes

More robust reading of pwd, namely:

• support trailing newlines
• don't send long lines that might clog the kernel's cooked mode buffer
• make sure to read the entire pwd if the writer is slow

• Resolves: "Cannot change to" directory\n warning when entering directory ending with newline #2325
• Resolves: mc hangs for 1-2s with subshell bash when entering paths > 128 characters on Solaris #4480
• Resolves: mc-wrapper.sh: Can't change to directory ending in \n #4884

Checklist

👉 Our coding style can be found here: https://midnight-commander.org/coding-style/ 👈

• I have referenced the issue(s) resolved by this PR (if any)
• I have signed-off my contribution with git commit --amend -s
• Lint and unit tests pass locally with my changes (make indent && make check)
• I have added tests that prove my fix is effective or that my feature works
• I have added the necessary documentation (if appropriate)

[egmontkob]

There's a regression with tcsh:

Previously it could enter directories with a non-alphanumeric UTF-8 symbol (e.g. heart ❤) in their name, now it cannot. (It can still enter alphanumeric UTF-8 characters.)

Let me play around a little bit with tcsh to see if I can fix this.

Let's hold off this PR for now.

[ossilator]

3nd commit msg: typo in 'platforms'

[egmontkob]

Back to tcsh:

I think I can do either of these two things:

• Use the $'...' string constant notation. As strings aren't byte-safe, they're forced to the locale (practically UTF-8), it means that in an UTF-8 environment directories with invalid UTF-8 in their name won't work. But the method handles newline characters just fine.

• Use `printf ...` command substitution. This method breaks newlines – in case of tcsh not just the trailing ones but any internal newlines. On the other hand it's binary-safe, we can enter any directory, even if invalid UTF-8. Note though that it doesn't work backwards: if you hide the panels and do a cd to a directory with an invalid UTF-8 name, the subshell reporting it back to mc using echo $cwd:q will also mangle it and mc won't be able to cd there (already buggy in current mc). To which a workaround could be to invoke the external pwd -L utility.

So it's either-or: invalid UTF-8 but no newlines, or newlines but not invalid UTF-8.

And no, I'm not going to implement both and pick runtime so that we only fail if a path contains both :)

Which one do you guys vote for?

[egmontkob]

tcsh: I was wrong, command substitution isn't binary safe either.

So, without terrible hacks, I can get newline working but not invalid UTF-8.

---

To get invalid UTF-8 working, I think this would do it:

• save the value of LC_ALL
• setenv LC_ALL C
• cd target_directory
• restore LC_ALL (or most likely: the lack thereof)
• set cwd=target_directory # to fix the accents shown in the prompt

and when a command completes and tcsh sends it working directory to mc, invoke the external pwd -L.

I'm not gonna do this, it's just not worth it.

[zyv]

Which one do you guys vote for?

My thinking is that if I had to choose, I'd take newlines.

[egmontkob]

Yup I'm going with newlines.

New commit pushed to fix regression with tcsh.

How confident are we that placing unquoted 128..255 bytes in the command line is safe in every shell, they don't have any special meaning in the shell?

[egmontkob]

New ideas for fully fixing tcsh filed in #4851.

But let's not do everything in this PR, let's leave that for another day.

Pushed an unchanged version, just rebase.

Please review.

[zyv]

In as far as I'm concerned, I'm satisfied with the current state. I have to admit that neither have I looked at the code too closely, nor have I tested it on my machines, but I really like the new structure and the documentation around it.

Just as a general comment, if I leave line comments with just one statement (and not more than one sentence), it looks better to me to omit the final full stop. Which is of course not pursued consistently throughout the whole codebase due to age.

[egmontkob]

/rebase

Just as a general comment, if I leave line comments with just one statement (and not more than one sentence), it looks better to me to omit the final full stop. Which is of course not pursued consistently throughout the whole codebase due to age.

I myself am inconsistent with that, too. Also whether to begin with uppercase or lowercase.

Fixed one such occurrence.

In the last commit, there's a one-liner and a two-liner that I'd like to keep consistent, so I kept the trailing dot.

[zyv]

/rebase

[zyv]

As I've got the bot to rebase this one and had another look, I've noticed 2 fixmes in the code. One relates to the pipes, and the other one to our read buffer.

As to the buffer fixme, my understanding is that our buffer is set to something like MAXPATHLEN, which appears to be about 4K. This is not too big, but then the issue is unlikely, because the system wouldn't probably push more than that anyways. We could also make it MAXPATHLEN x 2 or something?

In any case, I would remove the word "fixme" here and adjust the comment to that effect (saying that it's unlikely, because our buffer size is about this big). The comment itself is valuable if some problem arises and it will have to be debugged, but "fixme" somehow means to me that there is an immediate problem to take care of.

In as far as the pipes are concerned, I tried to understand how bad that is. Apparently, it isn't bad, as the default seems to be around 64K. Still, I think it would be nice to adjust the comment to say something like that. Otherwise, one has no feeling at all how bad that is. And given the findings above, maybe say that the other limit will be hit earlier than the pipe buffer, and remove "fixme" on these grounds?

https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer

[egmontkob]

(the previous push was a rebase only)

[egmontkob]

I would remove the word "fixme" [...] "fixme" somehow means to me that there is an immediate problem to take care of.

Softened both to "Note" and slightly expanded one of the comments. (By the way, for me "fixme" comments are exactly these long-term super-low-priority things. Anything more important or more urgent deserves a separate issue. But it's a matter of taste and project conventions.)

Added another commit to fix #4884 for sh and fish (not tcsh tough).

Sorry guys that it took me too long to get back to this. @mc-worker @zyv could you please do another round of review?

[ossilator]

i wouldn't take bets that this works as a fallback for really old shells, as they were funky with quoting inside backticks.

how about mc_newdir=`cat <&$cd_pipe`? this would also significantly simplify and unify the code between shells. concept-wise, it would be quite complementary to the pwd query.
note that we can't cat the tty itself, as that would have the same cooked buffer size limitation, only without a way to express line continuations.

ah, well, whatever ...

[egmontkob]

I don't think we support the old Bourne shell, nor anything old-fashioned like that. We only support a fixed list of shells, something like ash(busybox) bash dash fish ksh mksh tcsh zsh, and I'm fairly certain I tested my change with all of them. (Also note that in precmd_fallback we already use $(...) rather than backticks, which as far as I know also didn't work in really old shells, and no one complained so far.)

I'm afraid that cat <&$cd_pipe wouldn't work with unnamed pipes, cat would wait for EOF that would never arrive. It could work with names pipes or plain files, but I really don't want to refactor to using one of them in this PR.

Also, I wouldn't want to complicate the architecture, it's already complicated enough. Whenever mc needs to get some data from the subshell, it's understandable that we need side channels like a separate fd. Here it's the opposite direction: mc wants to send a command to the subshell, wants to tell it to change to a directory. It sounds ridiculous that it'd need a side channel for that, not being able to directly say so. If that's the case, I'd rather just limit support to decent-looking paths where the simplest cd '/over/there'-like commands work, or even claim that we don't support such shells.

[ossilator]

i wonder whether the compiler is smart enough to rewrite that as constant assignments inside the branches? probably not.
[...]
nope, despite being smart enough to use table lookups. both gcc and clang.

(i was just curious. ignore me.)

[ossilator]

s/final/trailing/

[egmontkob]

Can you please elaborate on the subtle details?

I can already see two final '\n's in that file, and also final byte, final whitespace elsewhere in mc. Several "how to remove final characters from string"-like web pages. Possibly not by native English speakers, I dont know. How bad is it to say "final character"?

This PR also adds final '\n' at three different places. At one of them changing it to trailing would introduce an ugly linebreak :)

[ossilator]

Can you please elaborate on the subtle details?

to me, "final" sounds distinctly temporal, while we want a positional meaning, preferably using terminology that is conventional for strings.

This PR also adds final '\n' at three different places. At one of them changing it to trailing would introduce an ugly linebreak :)

i'm assuming that you are bright enough to extrapolate the conclusion. ;-)

[egmontkob]

i'm assuming that you are bright enough to extrapolate the conclusion. ;-)

What I forgot to add in my last commit is that possibly I chose the shorter word to fit the comment in one line :) I can't remember.

Anyway, changed to "trailing". Instead of the linebreak I decided it's pointless to link to the bug.