In defense of extending language tags

[pchampin]

BCP47 is indeed a complex specification, but extending it looks relatively lightweight: the U extension is 7 pages long, counting a lot of boilerplate text...

The regular expression mentioned in the document is very complex because it aims at identifying every part of the language tag. But unless I'm misreading the ABNF grammar, if you are only interested in the direction, the regular expression -d-(ltr|rtl|auto) would do the trick.

[gkellogg]

It seems to me that text direction is equivalent to a style attribute. To my mind, two language-tagged strings that differ only in direction at least have the same value, but can’t be considered to be the same term.

I could see that -auto would be a syntax-level consideration similar to xsd:string and rdf:landString. Any language tagged string without an explicit direction implicitly is “auto” and serializers should not emit this.

[pchampin]

@gkellogg I included auto because it was part of the two other proposals, but I tend to agree with you: the extension d to BCP47 should only define ltr and rtl. The auto mode is conveyed by the absence of the d extension.

[r12a]

I'm quite reluctant to see this as a good solution, because it bends the language tag to something it isn't designed for.

(Note that the original hack suggestion in the i18n document was to deduce the direction from the language, rather than to specify it explicitly. (And we're not even very keen on that idea.))

(I'm also quite worried about the running battles i'm sure we'll have to face where people working with other formats (such as HTML) might (in my mind, erroneously) think this is a good idea (i already know some who have suggested it), and might want to use language tags everywhere instead of direction tags. Not to mention that we'd then have 4 ways of applying direction in HTML.)

[dlongley]

@r12a,

I think that if the direction of a language can be gleaned from the language tag (e.g., you can already do this with English and most? other languages), then it seems to me wholly appropriate that you can specify a direction explicitly using the same mechanism. It also seems like the simplest and least disruptive solution across various implementations and platforms.

As you mentioned, HTML already has a number of different ways of applying direction, what's the harm in adding another one if it finally gets this issue solved across the board? It seems like any system that already supports language tags would get direction support with this change and it would remove the need to argue any further about this for the variety of syntaxes in the world. What would be so bad if people did start using language tags everywhere instead of direction tags? What are the other drawbacks besides a feeling that it isn't "the best fit"?

As the old adage goes, let's not let the best be the enemy of the good.

[chaals]

@dlongley you cannot necessarily get the direction information just from the language.

You asked

HTML already has a number of different ways of applying direction, what's the harm in adding another one

The first harm in adding another way to check direction in HTML is that it requires all HTML implementations to implement it, which is generally quite difficult to arrange. Browsers are nervous about changes to the layout engines because those are already very complex and can have major performance impact. Other software (e.g. editing software) often takes a long time to work out what to do and then implement it.

The second harm is that we have to specify what to do in each possible combination of methods, including working out what should happen if there is conflicting information. Apart from being a complex task with potential for us to make mistakes that don't get picked up until the first round of implementations make them happen in practice, the more complex the system the more likely that implementations will confuse something and introduce a bug even in the historically unlikely situation that the specification is perfect first time.

[gkellogg]

We should consider not doing a blanket change to language tags for direction, but allow them for certain syntaxes, such as Turtle/SPARQL. Other serializations, such as HTML and JSON-LD (with @dir) can express it differently.

[chaals]

Forking language tagging according to what spec is being applied seems like a pretty bad idea to me.

I don't think it will take long for confusion to arise and implementors who were promised they wouldn't get language tags with directions in them will find that they are appearing, and in practical terms they need to work out how to deal with them.

That seems something worth avoiding if we can.

[dlongley]

@chaals,

you cannot necessarily get the direction information just from the language.

Sorry, I meant that this is already true for many languages, but not all. For those languages where it is possible, that information comes from the language tag (e.g., "en" means the direction is "ltr"). That indicates to me that a sensible solution to this problem is to add the direction information to the language tags that do not imply it already (via an -d- language tag extension). This would unify how it is dealt with today across all languages and it would work across all syntaxes.

The first harm in adding another way to check direction in HTML is that it requires all HTML implementations to implement it, which is generally quite difficult to arrange.

This is true, but it seems an even more onerous task to update all implementations of all syntax processors except HTML. My understanding is that various efforts have been made to shoehorn HTML into other syntaxes because it's the only syntax with the necessary direction support. It seems to me that we should solve the problem then not by having a change to every other syntax and processor, but by extending the language tags that they already process.

The second harm is that we have to specify what to do in each possible combination of methods, including working out what should happen if there is conflicting information.

This is a valid concern, however, a language tag extension seems like the easier path than having a separate debate for how to update every syntax followed by updating all processors for all syntaxes (sans HTML). Instead, we could focus on a clear precedence order for dealing with this in HTML. That would be the only thing we need to concern ourselves with as opposed to having to sort out what to do with everything else in the world. That seems to be the more intractable problem to me -- and the lack of a solution in the other direction despite many passing years seems to be evidence of this as well.

So, for example, for HTML, we could state that if direction is specified in one of the three current ways that HTML handles it today -- those ways take precedence over any conflicting information (according to whatever precedence order for those three there is today). Only if those mechanisms are not used, would the language tag -d- extension be used. This seems, at least naively, to be a good start for upgrading browsers to support it. I imagine browsers already guess at the direction when no information is given today -- and this behavior would not change at all when encountering a language tag with a -d- extension until they are upgraded to understand it.

Even if browsers never support the -d- extension, we'd be in a very similar situation to today: HTML does one thing, other syntaxes do something else. The only difference would be that the "something else" would become "express and preserve language direction information" instead of "do not express or preserve language direction information".

All that to say that I'm still in the camp that a language tag extension is the simplest fix here. It is not without drawbacks, but the stated challenges seem far more surmountable than the alternatives so far.

[iherman]

We are facing an RDF problem. I believe we have to solve it within the realm of RDF, and even that is not obvious in terms of deployment.

By touching BCP47 we would change a standard that is used, referred to, and deployed by a way larger palette of technologies including, but not limiting to, HTML. We would have to convince a huge community, or more exactly a huge number of different communities, that this is a way to solve a problem which they may not even have (the last remark also includes HTML, which has solved the problem of directions for a while now). I do not think we should go there.

There was a remark in an earlier mail (not on this issue) to extend BCP47 for RDF usage only. I think forking a standard this way is also a bad idea, and would create bad precedence in the way standards are treated....

[pchampin]

We are facing an RDF problem. I believe we have to solve it within the realm of RDF

This problem arose in the RDF community, but is it intrinsically an RDF problem? Could it not arise in other contexts?

That being said, I admit that I didn't consider the complex interactions that the -d- extension could have on HTML (or other technologies having their own way of encoding direction). But may be this could be mitigated by carefully writing the RFC, making it clear that the -d- extension is merely a hint (no MUST involved). In other words, any technology specifying its own way to convey direction (e.g. HTML) MAY supersede this hint.

[pchampin]

I realize that the proposal in my comment above is a "feature, not a bug" kind of solutions. If we make the -d- extension available, people will immediately want to write something like that

<p lang="he-d-rtl">פעילות הבינאום, W3C</p>

and expect the hint to have an efffect...

sigh

[iherman]

This problem arose in the RDF community, but is it intrinsically an RDF problem? Could it not arise in other contexts?

The problem, ie, to indicate the base direction, obviously does arise elsewhere. But, e.g., in a pure JSON context it can simply be solved by adding an extra tag. See https://www.w3.org/TR/string-meta that touches upon that. The problems come when we are bound by the JSON-LD rules that are bound by the RDF rules...