Encode {PROXY-,}BASIC-AUTHORIZATION using the default external format

[phoe]

Fixes #152

Also fixed an inconsistency between the docs XML/HTML files.

[hanshuebner]

I don't think that using *drakma-default-external-format* encoding for credentials is a good choice. I'd rather see an keyword parameter credentials-external-format be added, as the authentication encoding and the payload encoding may well be different. I would suggest that using non-ASCII credentials should fail unless credentials-external-format is specified. In most cases, utf-8 would be the right choice for the external format, but some servers may not be compliant and use some other encoding. With the keyword parameter, it will be relatively straightforward to select the right encoding without having to mess with special variables or run into conflict with a possibly different body encoding.

From reading https://datatracker.ietf.org/doc/html/rfc7617, I gather that the character set for the credentials is defined by the server's 401 response, specifically by the charset parameter, and that the only allowed charset is UTF-8. As DRAKMA does not negotiate basic authentication, it has no way to see whether the server is standards compliant. In fact, RFC7617 seems a little pointless.

[phoe]

IIUC, RFC7617 requires something else in the loop that makes the choice about the encoding to choose - in particular, if there's no previous HTTP 401, there's no way to figure out the encoding to use except "guess".

Sure, I can implement and document :credentials-external-format instead. But:

I would suggest that using non-ASCII credentials should fail unless credentials-external-format is specified.

How do you want to check this? Verify if the credentials are encodable in Latin-1 before actually sending them?

[phoe]

Also, "non-ASCII credentials" is poorly specified - e.g. £££££££££££££ is valid in both Latin-1 and UTF-8, but results in different encodings. We can catch characters that are non-encodable in general, but we can't catch everything.

[hanshuebner]

How do you want to check this? Verify if the credentials are encodable in Latin-1 before actually sending them?

I believe that HTTP headers are encoded as US-ASCII in general, so a check would have to verify that. I'm not entirely sure about the check, however, because it could break existing code that just blindly worked for some reason. On the other hand, it has been a wee while since I encountered Latin-1 as character set in the wild (which is what *drakma-default-external-format* defaults to). In the sake of being implicitly compatible with the modern world, defaulting to utf-8 encoding might be more useful than failing. What do you think?

[phoe]

The headers are going to be US-ASCII, it's not a problem. But basic authorization uses base64, so the question is what actually hides underneath that layer - Latin-1, UTF-8, or something actually else.

Defaulting to UTF-8 would break Drakma for people whose code currently assumes that authorization always uses Latin-1. I'd go for :credentials-external-format, default it to Latin-1 for backwards compatibility, and let people provide UTF-8 explicitly if they want to live in the 21st century.