Skip to content

Document protocol implementations #220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wojtekmach merged 1 commit into from
Feb 11, 2025
Merged

Document protocol implementations #220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
elixir: 1.8.2
otp: 20.3.x
- pair:
elixir: 1.17.x
elixir: 1.18.x
otp: 27.x
lint: lint
steps:
Expand Down
57 changes: 51 additions & 6 deletions lib/decimal.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,45 @@ defmodule Decimal do
according to the specification, return a number that "underflows" 0 is returned
instead of Etiny. This may happen when dividing a number with infinity.
Additionally, overflow, underflow and clamped may never be signalled.

## Protocol Implementations

`Decimal` implements the following protocols:

### `Inspect`

iex> inspect(Decimal.new("1.00"))
"Decimal.new(\\"1.00\\")"

### `String.Chars`

iex> to_string(Decimal.new("1.00"))
"1.00"

### `JSON.Encoder`

_(If running Elixir 1.18+.)_

By default, decimals are encoded as strings to preserve precision:

iex> JSON.encode!(Decimal.new("1.00"))
"\\"1.00\\""

To change that, pass a custom encoder to `JSON.encode!/2`. The following encodes
decimals as floats:

iex> encoder = fn
...> %Decimal{} = decimal, _encoder ->
...> decimal |> Decimal.to_float() |> :json.encode_float()
Copy link

@azizk azizk Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we can't directly use Decimal.to_string/1 but the precision loss via float shouldn't be assumed acceptable. If we had a Decimal.finite?/1 or Decimal.number?/1 function this example could be more accurate:

if Decimal.number?(decimal), do: Decimal.to_string(decimal)
# Or:
true = Decimal.number?(decimal)
Decimal.to_string(decimal)

Copy link
Collaborator Author

@wojtekmach wojtekmach Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but the precision loss via float shouldn't be assumed acceptable

Not sure I follow. I believe this is Eric's and mine point all along.

In any case, sure, we could add something like the following to the example encoder.

if Decimal.inf?(decimal) or Decimal.nan?(decimal) do
  raise ArgumentError, "#{inspect(decimal)} cannot be encoded to JSON"
end

However, Decimal.to_string("1.00") #=> "1.00" and that is not the result we want.

Copy link

@azizk azizk Feb 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I see what you mean by the 1.00 example. But let's say you want to encode Decimal.div(1, 3). You will lose quite a few decimal places when you convert to a 64-bit float first...

Copy link
Collaborator Author

@wojtekmach wojtekmach Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am misunderstanding something or we are talking past each other. Yes, we are losing precision, that's why we want to use strings not floats!

iex> Decimal.div(1, 3) |> Decimal.to_string() |> JSON.decode!() |> dbg
Decimal.div(1, 3) #=> Decimal.new("0.3333333333333333333333333333")
|> Decimal.to_string() #=> "0.3333333333333333333333333333"
|> JSON.decode!() #=> 0.3333333333333333

Same with Deno:

$ deno
> JSON.parse("0.3333333333333333333333333333")
0.3333333333333333
>

I don't see the point of encoding to a higher precision something that is expected to be a float. My understanding is decoders convert to float and lose that precision anyway.

Copy link

@azizk azizk Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're basically talking about the same thing, but your intention is now clearer to me.

Since the default (protocol impl) behaviour is to encode a Decimal as a string, isn't the point of overriding the protocol encoder in the example to show how to encode it as a JSON number instead? If the client you intend to send the JSON to, cannot handle JSON floats properly, it would of course not make any sense to do that. So we normally would do that when the API requires it or we want to store floats in a database JSON column.

If you return a string like "0.3333333333333333333333333333" in the encoder function it will become a JSON float. No problem there unless you have to parse it with JS. But even in Elixir you will get an inaccurate float, unless you provide the :float option and construct a Decimal using Decimal.new/1.

I hope I could clear the confusion. :)

Copy link
Owner

@ericmj ericmj Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have examples on JSON implementations that can decode literal JSON floats lossless? My understanding is that most implementations, including Elixir which we are using in the example here, would decode it as lossy IEEE 754 float.

Copy link
Collaborator Author

@wojtekmach wojtekmach Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think implementations would decode as floats by default but at least in ruby you can opt-in:

$ irb -r json -r bigdecimal
irb> JSON.parse('0.3333333333333333333333333333')
=> 0.3333333333333333
irb> JSON.parse('0.3333333333333333333333333333', decimal_class: BigDecimal)
=> 0.3333333333333333333333333333e0

but yeah. maybe cd6aa59 is a mistake because I have a feeling it will cause confusion to most people. The purpose of this piece of documentation is not to educate people on subtleties of having or not having IEEE 754 but to show that we can customize encoding. @ericmj feel free to revert cd6aa59, I'm done.

Copy link

@LostKobrakai LostKobrakai Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Json itself is actually quite unaware of floats. It just has arbitrary presision numbers being a text based format. It completely depends on the parser or encoder to deal with how to map between runtime level number values and their json text representation.

Copy link
Owner

@ericmj ericmj Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what we have now is enough. By default we have a safe arbitrary precision encoding using strings, we also have examples of custom encoding to IEEE 754 floats in the docs, and this example can of course be tweaked to whatever precision encoding you want.

Thank you @wojtekmach for the protocol implementation and documentation improvements.

Copy link

@azizk azizk Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wojtekmach for adjusting the example.

I think the example doesn't need to give a complete education but it should make sense. And the only good reason to override the protocol encoder is the need to do something different than the default. I don't see how the adjusted example could be confusing. I'm sure it's helpful for developers to know that you can output lossless JSON floats if required.

To cover a more general case for overriding protocol encoders, an example could be included in the Elixir documentation for JSON.encode!/2. There isn't one currently.

Do you have examples on JSON implementations that can decode literal JSON floats lossless?

Wojteck already gave an example with Ruby. In the issue #219 referenced by this PR, I mentioned a few other languages that can do lossless decoding and encoding. Afaics, JS is probably the only major language that isn't capable of doing this.

Thanks for the improvements. 👍

...>
...> other, encoder ->
...> JSON.protocol_encode(other, encoder)
...> end
...>
iex> JSON.encode!(%{x: Decimal.new("1.00")}, encoder)
"{\\"x\\":1.0}"

Note: `Decimal.to_float/1` crashes on infinite and NaN decimals.
"""

import Bitwise
Expand Down Expand Up @@ -1588,7 +1627,7 @@ defmodule Decimal do
@doc """
Returns the decimal represented as an integer.

Fails when loss of precision will occur.
Raises when loss of precision will occur.

## Examples

Expand Down Expand Up @@ -1623,28 +1662,30 @@ defmodule Decimal do
@doc """
Returns the decimal converted to a float.

The returned float may have lower precision than the decimal. Fails if
the decimal cannot be converted to a float.
The returned float may have lower precision than the decimal.

Raises if the decimal cannot be converted to a float.

## Examples

iex> Decimal.to_float(Decimal.new("1.5"))
1.5

iex> Decimal.to_float(Decimal.new("-1.79769313486231581e308"))
** (Decimal.Error) : negative number smaller than DBL_MAX: Decimal.new("-1.79769313486231581E+308")


iex> Decimal.to_float(Decimal.new("-1.79769313486231581e308"))
** (Decimal.Error) : negative number smaller than DBL_MAX: Decimal.new("-1.79769313486231581E+308")


iex> Decimal.to_float(Decimal.new("2.22507385850720139e-308"))
** (Decimal.Error) : number smaller than DBL_MIN: Decimal.new("2.22507385850720139E-308")


iex> Decimal.to_float(Decimal.new("-2.22507385850720139e-308"))
** (Decimal.Error): negative number bigger than DBL_MIN: Decimal.new(\"-2.22507385850720139E-308\")

iex> Decimal.to_float(Decimal.new("inf"))
** (ArgumentError) Decimal.new("Infinity") cannot be converted to float

"""
@spec to_float(t) :: float
def to_float(%Decimal{coef: coef} = decimal) when is_integer(coef) do
Expand All @@ -1669,6 +1710,10 @@ defmodule Decimal do
end
end

def to_float(%Decimal{} = decimal) do
raise ArgumentError, "#{inspect(decimal)} cannot be converted to float"
end

@doc """
Returns the scale of the decimal.

Expand Down
22 changes: 19 additions & 3 deletions test/decimal_test.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,13 @@ defmodule DecimalTest do

require Decimal

doctest Decimal
elixir_json_available? = Version.match?(System.version(), ">= 1.18.0-rc")

if elixir_json_available? do
doctest Decimal
else
doctest Decimal, except: [:moduledoc]
end

test "parse/1" do
assert Decimal.parse("123") == {d(1, 123, 0), ""}
Expand Down Expand Up @@ -684,7 +690,7 @@ defmodule DecimalTest do
assert Decimal.to_float(~d"2251799813685248") === 2_251_799_813_685_248.0
assert Decimal.to_float(~d"9007199254740992") === 9_007_199_254_740_992.0

assert_raise FunctionClauseError, fn ->
assert_raise ArgumentError, fn ->
Decimal.to_float(d(1, :NaN, 0))
end
end)
Expand Down Expand Up @@ -983,9 +989,19 @@ defmodule DecimalTest do
end
end

if Version.match?(System.version(), ">= 1.18.0-rc") do
if elixir_json_available? do
test "JSON.Encoder implementation" do
assert JSON.encode!(%{x: Decimal.new("1.0")}) == "{\"x\":\"1.0\"}"

encoder = fn
%Decimal{} = decimal, _encode ->
decimal |> Decimal.to_float() |> :json.encode_float()

other, encode ->
JSON.protocol_encode(other, encode)
end

assert JSON.encode!(%{x: Decimal.new("1.0")}, encoder) == "{\"x\":1.0}"
end
end
end
Loading