diff --git a/CHANGELOG.md b/CHANGELOG.md index 46da0f0..6fc1947 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [unreleased] +### Breaking Changes +- JSON path will now be used in full for virtual fields, instead of being cut off at the first section (for all virtual fields except a list of Maps). + +### Added +- The `any` predicate has been added for single and virtual fields. + +### Changed +- Empty `path` now throws an error in most cases. Exception: sub-predicates inside `any`. + ## [0.5.0] - 2025-10-31 ### Breaking Changes diff --git a/Operators.md b/Operators.md index 7fe52e5..393d59b 100644 --- a/Operators.md +++ b/Operators.md @@ -203,6 +203,24 @@ relationship destination itself, and `"id"` targets the `id` column of the relat } ``` +### Stored arrays + +One-dimensional, native SQL arrays can be referenced by including an empty `path` in the sub-predicate. + +The following example will search for an `"blue"` value inside the `"colors"` array: + +```json +{ + "op": "any", + "path": "colors", + "arg": { + "op": "eq", + "path": "", + "arg": "blue" + } +} +``` + ## Plain Value Predicate These special predicates always evaluate to true or false. diff --git a/README.md b/README.md index 31cbf58..17f749e 100644 --- a/README.md +++ b/README.md @@ -169,7 +169,7 @@ The first path segment is converted to atom and looked-up on the model: values within the JSON structure. 3. If the field is stored, its value is used for the comparison. The remaining path is discarded. 4. If the field is a virtual field, PredicateConverter invokes `get_virtual_field` as described in [Virtual - Fields](#virtual-fields). The remaining path is discarded. + Fields](#virtual-fields). 5. If the segment points to an association, an `any` predicate is created with the semantics of "is there a related entity for which the original predicate evaluates to true?". This behaves the same for both one-to-one and one-to-many relationships. The remaining path is applied to the related entity. @@ -180,8 +180,7 @@ Virtual fields in Ecto are fields defined in your schema that do not exist in th useful for computed or derived values, such as combining multiple columns, formatting data, or performing temporary calculations. -To allow PredicateConverter to use virtual fields, the schema module must implement a `get_virtual_field/2` (or `/1`) -returning a query fragment that evaluates to a value. +To allow PredicateConverter to use virtual fields, the schema module must implement a `get_virtual_field/2` (or `/1`) returning a query fragment that evaluates to a value. Special case for the `any` predicate is explained in the [Virtual fields in the `any` predicate](#virtual-fields-in-the-any-predicate) When using sub-queries, refer to to the parent query using the named binding from [Model Awareness](#model-awareness). @@ -213,6 +212,20 @@ defmodule Author do end ``` +### Virtual fields in the `any` predicate +When virtual fields are used within the `any` predicate, the value must be wrapped and bound to `__value__`. +The `get_virtual_field/2` (or `/1`) function in this case must return a subquery (not a dynamic query). Not following these (current) restrictions will result in an error. + +```elixir +def get_virtual_field(:oldest_post_date, original_query), do: + subquery( + # ... + select: %{ + __value__: value + } + ) +``` + ## Multi-Tenancy Multi-tenancy is not enforced by default. When using "soft multi tenancy" (where data is stored in shared tables and diff --git a/lib/predicate_converter.ex b/lib/predicate_converter.ex index 68e8911..6905357 100644 --- a/lib/predicate_converter.ex +++ b/lib/predicate_converter.ex @@ -97,27 +97,43 @@ defmodule Predicates.PredicateConverter do end # Quantor Predicate - def convert_query(queryable, %{"op" => "any", "path" => path, "arg" => sub_predicate}, meta) do + def convert_query( + queryable, + %{"op" => "any", "path" => path, "arg" => sub_predicate}, + meta + ) do case process_path(queryable, path, meta) do {:assoc, field, _} -> convert_any({:assoc, field}, sub_predicate, queryable, meta) + {:single, field} -> + convert_any({:single, field}, sub_predicate, queryable, meta) + + {:virtual, field, {:array, :map} = type, path} -> + convert_any({:virtual, field, type, path}, sub_predicate, queryable, meta) + _ -> raise PredicateError, - message: "Operator 'any' is currently only supported on associations" + message: "Operator 'any' is not supported for this field" end end # Comparator Predicates def convert_query(queryable, %{"op" => op, "path" => path, "arg" => value} = predicate, meta) do - convert_comparator(op, process_path(queryable, path, meta), value, queryable, meta) - rescue - FunctionClauseError -> - # credo:disable-for-next-line Credo.Check.Warning.RaiseInsideRescue - raise PredicateError, message: "Operator #{inspect(op)} not supported", predicate: predicate + path = process_path(queryable, path, meta) - e -> - reraise e, __STACKTRACE__ + try do + convert_comparator(op, path, value, queryable, meta) + rescue + FunctionClauseError -> + # credo:disable-for-next-line Credo.Check.Warning.RaiseInsideRescue + raise PredicateError, + message: "Operator #{inspect(op)} not supported", + predicate: predicate + + e -> + reraise e, __STACKTRACE__ + end end # Value Predicate @@ -184,8 +200,9 @@ defmodule Predicates.PredicateConverter do defp convert_eq({:single, field}, nil), do: dynamic([q], is_nil(field(q, ^field))) - defp convert_eq({:virtual, field, _}, nil), - do: dynamic(is_nil(^field)) + defp convert_eq({:virtual, field, _type, json_path}, nil) do + dynamic(is_nil(^maybe_use_path(field, json_path))) + end defp convert_eq({:json, field, path}, nil), do: @@ -205,8 +222,8 @@ defmodule Predicates.PredicateConverter do dynamic([q], field(q, ^field) == ^value) end - defp convert_eq({:virtual, field, type}, value) do - dynamic(^field == ^maybe_cast(value, type)) + defp convert_eq({:virtual, field, type, json_path}, value) do + dynamic(^maybe_use_path(field, json_path) == ^maybe_cast(value, type)) end defp convert_eq({:json, field, path}, value), @@ -219,8 +236,8 @@ defmodule Predicates.PredicateConverter do defp convert_not_eq({:single, field}, nil), do: dynamic([q], not is_nil(field(q, ^field))) - defp convert_not_eq({:virtual, field, _}, nil), - do: dynamic(not is_nil(^field)) + defp convert_not_eq({:virtual, field, _, json_path}, nil), + do: dynamic(not is_nil(^maybe_use_path(field, json_path))) defp convert_not_eq({:json, field, path}, nil), do: dynamic([q], not is_nil(fragment("?#>>?", field(q, ^field), ^path))) @@ -244,13 +261,17 @@ defmodule Predicates.PredicateConverter do defp convert_not_eq({:single, field}, value), do: dynamic([q], field(q, ^field) != ^value or is_nil(field(q, ^field))) - defp convert_not_eq({:virtual, field, type}, value), - do: dynamic(^field != ^maybe_cast(value, type) or is_nil(^field)) + defp convert_not_eq({:virtual, field, type, json_path}, value), + do: + dynamic( + ^maybe_use_path(field, json_path) != ^maybe_cast(value, type) or + is_nil(^maybe_use_path(field, json_path)) + ) defp convert_gt({:single, field}, value), do: dynamic([q], field(q, ^field) > ^value) - defp convert_gt({:virtual, field, type}, value), - do: dynamic(^field > ^maybe_cast(value, type)) + defp convert_gt({:virtual, field, type, json_path}, value), + do: dynamic(^maybe_use_path(field, json_path) > ^maybe_cast(value, type)) defp convert_gt({:json, field, path}, value), do: dynamic([q], fragment("?#>? > ?", field(q, ^field), ^path, ^value)) @@ -258,16 +279,16 @@ defmodule Predicates.PredicateConverter do defp convert_ge({:single, field}, value), do: dynamic([q], field(q, ^field) >= ^value) - defp convert_ge({:virtual, field, type}, value), - do: dynamic(^field >= ^maybe_cast(value, type)) + defp convert_ge({:virtual, field, type, json_path}, value), + do: dynamic(^maybe_use_path(field, json_path) >= ^maybe_cast(value, type)) defp convert_ge({:json, field, path}, value), do: dynamic([q], fragment("?#>? >= ?", field(q, ^field), ^path, ^value)) defp convert_lt({:single, field}, value), do: dynamic([q], field(q, ^field) < ^value) - defp convert_lt({:virtual, field, type}, value), - do: dynamic(^field < ^maybe_cast(value, type)) + defp convert_lt({:virtual, field, type, json_path}, value), + do: dynamic(^maybe_use_path(field, json_path) < ^maybe_cast(value, type)) defp convert_lt({:json, field, path}, value), do: dynamic([q], fragment("?#>? < ?", field(q, ^field), ^path, ^value)) @@ -275,8 +296,8 @@ defmodule Predicates.PredicateConverter do defp convert_le({:single, field}, value), do: dynamic([q], field(q, ^field) <= ^value) - defp convert_le({:virtual, field, type}, value), - do: dynamic(^field <= ^maybe_cast(value, type)) + defp convert_le({:virtual, field, type, json_path}, value), + do: dynamic(^maybe_use_path(field, json_path) <= ^maybe_cast(value, type)) defp convert_le({:json, field, path}, value), do: dynamic([q], fragment("?#>? <= ?", field(q, ^field), ^path, ^value)) @@ -284,7 +305,7 @@ defmodule Predicates.PredicateConverter do defp convert_like({:single, field}, value), do: dynamic([q], like(field(q, ^field), ^"%#{search_to_like_pattern(value)}%")) - defp convert_like({:virtual, field, _type}, value), + defp convert_like({:virtual, field, _type, _}, value), do: dynamic(like(type(^field, :string), ^"%#{search_to_like_pattern(value)}%")) defp convert_like({:json, field, path}, value), @@ -297,7 +318,7 @@ defmodule Predicates.PredicateConverter do defp convert_ilike({:single, field}, value), do: dynamic([q], ilike(field(q, ^field), ^"%#{search_to_like_pattern(value)}%")) - defp convert_ilike({:virtual, field, _type}, value), + defp convert_ilike({:virtual, field, _type, _}, value), do: dynamic(ilike(type(^field, :string), ^"%#{search_to_like_pattern(value)}%")) defp convert_ilike({:json, field, path}, value), @@ -316,8 +337,14 @@ defmodule Predicates.PredicateConverter do defp convert_starts_with({:single, field}, value), do: dynamic([q], like(field(q, ^field), ^"#{search_to_like_pattern(value)}%")) - defp convert_starts_with({:virtual, field, _type}, value), - do: dynamic(like(type(^field, :string), ^"#{search_to_like_pattern(value)}%")) + defp convert_starts_with({:virtual, field, _type, json_path}, value), + do: + dynamic( + like( + type(^maybe_use_path(field, json_path), :string), + ^"#{search_to_like_pattern(value)}%" + ) + ) defp convert_starts_with({:json, field, path}, value), do: @@ -329,8 +356,14 @@ defmodule Predicates.PredicateConverter do defp convert_ends_with({:single, field}, value), do: dynamic([q], like(field(q, ^field), ^"%#{search_to_like_pattern(value)}")) - defp convert_ends_with({:virtual, field, _type}, value), - do: dynamic(like(type(^field, :string), ^"%#{search_to_like_pattern(value)}")) + defp convert_ends_with({:virtual, field, _type, json_path}, value), + do: + dynamic( + like( + type(^maybe_use_path(field, json_path), :string), + ^"%#{search_to_like_pattern(value)}%" + ) + ) defp convert_ends_with({:json, field, path}, value), do: @@ -353,11 +386,11 @@ defmodule Predicates.PredicateConverter do else: dynamic([q], ^query or is_nil(field(q, ^field))) end - defp convert_in({:virtual, field, type}, value) do + defp convert_in({:virtual, field, type, json_path}, value) do # `nil` values will never match with `IN` operator, so we need to handle them separately. {values, nil_values} = Enum.split_with(value, &(!is_nil(&1))) - query = dynamic(^field in ^maybe_cast_array(values, type)) + query = dynamic(^maybe_use_path(field, json_path) in ^maybe_cast_array(values, type)) if nil_values == [], do: query, @@ -392,15 +425,15 @@ defmodule Predicates.PredicateConverter do else: dynamic(^query and not is_nil(^db_field)) end - defp convert_not_in({:virtual, field, type}, value) do + defp convert_not_in({:virtual, field, type, json_path}, value) do # `nil` values will never match with `NOT IN` operator, so we need to handle them separately. {values, nil_values} = Enum.split_with(value, &(!is_nil(&1))) - query = dynamic(^field not in ^maybe_cast_array(values, type)) + query = dynamic(^maybe_use_path(field, json_path) not in ^maybe_cast_array(values, type)) if nil_values == [], - do: dynamic(^query or is_nil(^field)), - else: dynamic(^query and not is_nil(^field)) + do: dynamic(^query or is_nil(^maybe_use_path(field, json_path))), + else: dynamic(^query and not is_nil(^maybe_use_path(field, json_path))) end defp convert_not_in({:json, field, path}, value) do @@ -442,6 +475,41 @@ defmodule Predicates.PredicateConverter do end end + defp convert_any({:single, field}, sub_predicate, queryable, meta) do + parent_table = get_table_name(queryable) + + subquery = + from( + s in fragment("select unnest(?) as __element__", field(parent_as(^parent_table), ^field)), + select: s.__element__ + ) + + subquery = + subquery + |> where(^convert_query(subquery, Map.put(sub_predicate, "path", :__element__), meta)) + + dynamic(exists(subquery)) + end + + defp convert_any({:virtual, field, {:array, :map}, _json_path}, sub_predicate, _queryable, meta) do + subquery = + subquery( + from(t in field, + select: %{ + __element__: fragment("jsonb_array_elements(?)", t.__value__) + } + ) + ) + + subquery = + subquery + |> where( + ^convert_query(subquery, sub_predicate, Map.put(meta, :__nested_virtual_json__, true)) + ) + + dynamic(exists(subquery)) + end + defp maybe_cast(value, :utc_datetime), do: dynamic(type(^value, :utc_datetime)) defp maybe_cast(value, :utc_datetime_usec), do: dynamic(type(^value, :utc_datetime_usec)) defp maybe_cast(value, _), do: value @@ -477,13 +545,23 @@ defmodule Predicates.PredicateConverter do defp build_sub_query(queryable, sub_predicate, query), do: build_query(queryable, sub_predicate, query) - # check for existing fields and throw an error if the field does not exist + # Used by convert_any({:single, ...) to anchor the sub-predicate on the special __element__ field + defp process_path(_queryable, :__element__, _meta), do: {:single, :__element__} + + defp process_path(_queryable, "", _meta), + do: raise(PredicateError, message: "Empty path is not allowed") + defp process_path(queryable, path, meta) when is_binary(path), do: process_path(queryable, String.split(path, ".", trim: true), meta) defp process_path(queryable, path, meta) when is_atom(path), do: process_path(queryable, [path], meta) + # Used by convert_any({:virtual, ...) for an array of maps to anchor the sub-predicate on the special __element__ + # field + defp process_path(%Ecto.SubQuery{}, json_path, _meta), do: {:json, :__element__, json_path} + + # check for existing fields and throw an error if the field does not exist # credo:disable-for-next-line Credo.Check.Refactor.CyclomaticComplexity defp process_path(queryable, [field | json_path], meta) do schema = get_schema(queryable) @@ -517,7 +595,15 @@ defmodule Predicates.PredicateConverter do end type = get_virtual_field_type(schema, atom_field) - {:virtual, virtual_field, type} + + if json_path != [] and type == {:array, :map}, + do: + raise(PredicateError, + message: + "Can't use JSON path on virtual field '#{field}' of type {:array, :map}, use explicit 'any' instead (remaining path: #{inspect(json_path)})" + ) + + {:virtual, virtual_field, type, json_path} Enum.member?(associations, atom_field) -> # it is an association field and we probably have a sub path into the associations @@ -541,4 +627,9 @@ defmodule Predicates.PredicateConverter do ArgumentError -> raise PredicateError, message: "Field '#{field}' does not exist" end + + defp maybe_use_path(field, []), do: field + + defp maybe_use_path(field, json_path), + do: dynamic(fragment("?#>>?", ^field, ^json_path)) end diff --git a/test/predicates_test.exs b/test/predicates_test.exs index b1cc8b1..4433148 100644 --- a/test/predicates_test.exs +++ b/test/predicates_test.exs @@ -15,6 +15,7 @@ defmodule PredicatesTest do field :likes, :integer field :publisher_id, :string field :meta, :map, default: %{} + field :tags, {:array, :string} field :slug, :string, virtual: true @@ -31,6 +32,7 @@ defmodule PredicatesTest do likes int DEFAULT 0, publisher_id text, meta jsonb DEFAULT '{}', + tags text[], author_id int REFERENCES pred_authors(id) ON DELETE CASCADE ON UPDATE CASCADE, inserted_at timestamp with time zone NOT NULL default now(), updated_at timestamp with time zone NOT NULL default now() @@ -56,6 +58,8 @@ defmodule PredicatesTest do field :birthday, :utc_datetime field :post_count, :integer, virtual: true field :oldest_post, :utc_datetime, virtual: true + field :oldest_post_object, :map, virtual: true + field :total_tags, {:array, :map}, virtual: true has_many :posts, Post end @@ -96,6 +100,55 @@ defmodule PredicatesTest do ) ) ) + + def get_virtual_field(:oldest_post_object), + do: + dynamic( + subquery( + from(p in Post, + where: p.author_id == parent_as(:pred_authors).id, + order_by: [asc: p.inserted_at], + limit: 1, + select: + fragment( + """ + jsonb_build_object('id', ?, 'name', ?) + """, + p.id, + p.name + ) + ) + ) + ) + + def get_virtual_field(:total_tags), + do: + subquery( + from( + a in subquery( + from( + t in subquery( + from(p in Post, + where: p.author_id == parent_as(:pred_authors).id, + select: %{ + tag: fragment("unnest(?)", p.tags) + } + ) + ), + select: %{ + tag: t.tag, + count: count(t.tag) + }, + group_by: t.tag + ) + ), + select: %{ + __value__: + fragment("jsonb_agg(jsonb_build_object('tag', ?, 'count', ?))", a.tag, a.count) + |> coalesce("[]") + } + ) + ) end alias __MODULE__.Author @@ -517,7 +570,7 @@ defmodule PredicatesTest do |> Predicates.Repo.all() end - test "using contains to check against array lists" do + test "using contains to check against JSON array lists" do Predicates.Repo.insert_all(Post, [ %{name: "Post 1", likes: 13, meta: %{"tags" => ["foo", "bar"]}}, %{name: "Post 2", likes: 42, meta: %{"tags" => ["fizz", "buzz"]}} @@ -560,6 +613,13 @@ defmodule PredicatesTest do |> Predicates.Repo.one() end + # FIXME: make sure we support these cases + # test "contains works for stored arrays" do + # end + + # test "contains work for virtual arrays" do + # end + test "starts_with and ends_with operators" do Predicates.Repo.insert_all(Author, [ %{name: "Goethe"}, @@ -626,7 +686,67 @@ defmodule PredicatesTest do |> Predicates.Repo.one() end - test "resolves path into json field" do + test "resolves nested path for a virtual JSON field" do + {2, [goethe, schiller]} = + Predicates.Repo.insert_all(Author, [%{name: "Goethe"}, %{name: "Schiller"}], + returning: true + ) + + Predicates.Repo.insert_all( + Post, + [ + %{ + author_id: goethe.id, + name: "Die Laune des Verliebten", + inserted_at: ~U[1767-01-01T12:00:00Z] + }, + %{ + author_id: goethe.id, + name: "Die Mitschuldigen", + inserted_at: ~U[1769-01-01T12:00:00Z] + }, + %{ + author_id: schiller.id, + name: "Kabale und Liebe", + inserted_at: ~U[1784-01-01T12:00:00Z] + } + ] + ) + + assert goethe == + Converter.build_query(Author, %{ + "op" => "eq", + "path" => "oldest_post_object.name", + "arg" => "Die Laune des Verliebten" + }) + |> Predicates.Repo.one() + + assert schiller == + Converter.build_query(Author, %{ + "op" => "not_eq", + "path" => "oldest_post_object.name", + "arg" => "Die Laune des Verliebten" + }) + |> Predicates.Repo.one() + + assert nil == + Converter.build_query(Author, %{ + "op" => "not_in", + "path" => "oldest_post_object.name", + "arg" => ["Die Laune des Verliebten", "Kabale und Liebe"] + }) + |> Predicates.Repo.one() + + assert goethe == + Converter.build_query(Author, %{ + "op" => "like", + "path" => "oldest_post_object.name", + "arg" => "Die" + }) + |> Predicates.Repo.one() + end + + test "resolves path into stored JSON field" do Predicates.Repo.insert_all(Author, [%{name: "Goethe", meta: %{born: 1749}}]) assert %{name: "Goethe"} = @@ -662,6 +782,30 @@ defmodule PredicatesTest do }) |> Predicates.Repo.one() end + + test "errors for predicates with empty path (unless explicitly allowed)" do + assert_raise PredicateError, "Empty path is not allowed", fn -> + Converter.build_query(Author, %{ + "op" => "eq", + "path" => "", + "arg" => "foo" + }) + |> Predicates.Repo.one() + end + end + + test "errors when looking into nested field of virtual list of maps" do + assert_raise PredicateError, + ~r"Can't use JSON path on virtual field 'total_tags' of type {:array, :map}, use explicit 'any' instead", + fn -> + Converter.build_query(Author, %{ + "op" => "eq", + "path" => "total_tags.count", + "arg" => "drang" + }) + |> Predicates.Repo.one() + end + end end describe "any" do @@ -688,6 +832,60 @@ defmodule PredicatesTest do }) |> Predicates.Repo.one() end + + test "works on stored list of values" do + {2, [_post1, post2]} = + Predicates.Repo.insert_all( + Post, + [ + %{name: "Post 1", tags: ["sturm"]}, + %{name: "Post 2", tags: ["sturm", "drang"]} + ], + returning: true + ) + + assert [post2] == + Converter.build_query( + Post, + %{ + "op" => "any", + "path" => "tags", + "arg" => %{ + "op" => "eq", + "path" => "", + "arg" => "drang" + } + } + ) + |> Predicates.Repo.all() + end + + test "works on virtual list of values" do + {2, [goethe, _]} = + Predicates.Repo.insert_all(Author, [%{name: "Goethe"}, %{name: "Schiller"}], + returning: true + ) + + Predicates.Repo.insert_all(Post, [ + %{name: "Post 1", author_id: goethe.id, tags: ["sturm", "drang"]}, + %{name: "Post 2", author_id: goethe.id, tags: ["sturm"]} + ]) + + assert [goethe] == + Converter.build_query( + Author, + %{ + "op" => "any", + "path" => "total_tags", + "arg" => %{ + "op" => "eq", + "path" => "count", + "arg" => 2 + } + } + ) + |> Predicates.Repo.all() + end end describe "conjunctions" do