From 35307a468078a28e7ce4a3b224164cae272937e1 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Mon, 15 Dec 2025 15:01:11 +0100 Subject: [PATCH 01/12] feat: dynamic actor memory --- .../dynamic_actor_memory/index.md | 170 ++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md new file mode 100644 index 000000000..1b5b840bd --- /dev/null +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -0,0 +1,170 @@ +--- +title: Dynamic Actor memory +description: Learn how to automatically adjust your Actor's memory based on input size and run options, so you can optimize performance and reduce costs without manual configuration.. +slug: /actors/development/actor-definition/dynamic-actor-memory +--- + +**Learn how to automatically adjust your Actor's memory based on input size and run options, so you can optimize performance and reduce costs without manual configuration.** + +--- + +Dynamic Actor memory allows Actors to _automatically adjust its memory allocation based on the input and run options_. Instead of always using a fixed memory value, Actor can use just the right amount of memory for each run. + +This helps: +- _Optimize performance_ for large inputs (more memory for bigger tasks). +- _Reduce costs_ for small runs (less memory when it’s not needed). +- _Provide better user experience_, so users get optimal performance without having to manually configure memory. + +:::info +_Important_: The memory calculated by dynamic expression is used as the default memory for the run. Users can still override it manually for each run. +::: + +--- + +## Why dynamic memory matters +Currently, Actors often use a _static default memory_, but the optimal memory usually depends on the input: +- A small input (e.g., 10 URLs) might run fine on 512 MB. +- A large input (e.g., 1,000 URLs) could require 4 GB+ to run efficiently. + +Setting a single default either _wastes resources or slows down execution_. Dynamic memory solves this by calculating memory just before the run starts. + +--- + +## How to define dynamic memory expression + +You can define a dynamic memory expression in your `actor.json`: + +```json +{ + "defaultMemoryMbytes": "get(input, 'startUrls.length' * 1024)" +} +``` +- The expression is evaluated _before the run starts_. +- The expression can reference variables from input and run options to calculate memory (e.g., `input.startUrls`, `runOptions.maxItems`). +- The result becomes the default memory for the run, but users can still override it. + +--- + +### Writing Expressions + +Expressions are based on [MathJS](https://mathjs.org/), extended with custom helper function `get`. + +#### Variable Access + +You can access variables in two ways: + +1. Direct property access +```js +input.foo + 512 +runOptions.maxItems + 256 +``` +2. Double-brace syntax +```js +{{input.foo}} +{{runOptions.maxItems}} +``` + +_You can mix both styles._ + +#### Supported Operations + +- Arithmetic: `+`, `-`, `*`, `/` +- Math functions: `min()`, `max()`, `ceil()`, `floor()`, `round()`, `log()`, `exp()`, `log10()` +- Conditional logic: +``` +condition ? valueIfTrue : valueIfFalse +``` +- Variable assignment: +```js +memoryPerUrl = 64; +get(input, 'startUrls') * memoryPerUrl +``` + +#### Custom `get()` function +Use `get()` to safely read nested properties or provide fallback values: + +```js +get(obj, 'path.to.property', defaultValue) +``` +Examples: +```js +// Safely get array length +get(input, 'startUrls.length', 1) // returns length or 1 if undefined + +// Safely get nested property +get(input, 'foo.bar.baz.count') // safely access nested objects + +// Fallback +get(input, 'foo', 1024) // returns 1024 if 'foo' doesn't exist + +// Safely get an array element +get(input, 'numbers.1') // element at index 1 of the numbers array +``` + +### Memory limits and clamping +After evaluation: +1. The result is rounded up to the nearest power of two +- 900 → 1024 MB +- 3,600 → 4096 MB +2. It is clamped to actor-defined min/max (`minMemoryMbytes` / `maxMemoryMbytes`). +3. It is clamped to platform limits (128 MB to 32 GB). + +:::info +If the calculation results in an error, the Actor will start with a fixed default memory, which can be configured in the Actor's UI settings. +::: + +### Example expressions +1. Simple memory based on URL count +```js +get(input, 'startUrls.length', 1) * 512 +``` +2. Conditional logic +```js +get(input, 'scrapeDetailed', false) ? 4096 : 1024 +``` +3. More complex calculation +```js +urls = get(input, 'startUrls.length', 0); +reviewsMultiplier = max(get(input, 'maxReviews', 1) / 10, 1); +urls * reviewsMultiplier * 128 +``` +4. Using double-brace variables +```js +{{input.itemsToProcess}} * 64 +``` + +### Testing expressions + +#### Use NPM package +You can use our [NPM package](https://www.npmjs.com/package/@apify/actor-memory-expression) not only to calculate memory for your expression, but also to write unit tests and verify the behavior of your expressions locally. + +```shell +npm install @apify/actor-memory-expression +``` + +```js +import { calculateRunDynamicMemory } from '@apify/actor-memory-expression'; + +await calculateRunDynamicMemory( + "get(input, 'urls.length', 1) * 256", + { + input: { urls: ["a", "b", "c"] }, + runOptions: { maxTotalChargeUsd: 10 } + } +); +``` + +#### Use CLI +You can use [Apify CLI](https://docs.apify.com/cli) to quickly evaluate expressions without writing code. It supports reading input from a JSON file and passing run options as flags. + +```shell +apify actor calculate-memory --input ./input.json --maxTotalChargeUsd=25 +``` + +--- + +### Key Takeaways +- Dynamic memory _automatically adjusts memory_ based on input and run options. +- The result is used as default memory, but users can override it. +- Use `get()`, arithmetic, math functions, and conditional logic to define expressions. +- Test expressions locally with the NPM package or CLI. \ No newline at end of file From 0305a252599d16b2cc9a0369a2d5b1b5ea233f39 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Mon, 15 Dec 2025 15:17:11 +0100 Subject: [PATCH 02/12] chore: add backlinks --- .../openapi/components/schemas/actors/ActorDefinition.yaml | 5 +++++ .../actors/development/actor_definition/actor_json.md | 2 ++ sources/platform/actors/running/input_and_output.md | 3 +++ sources/platform/actors/running/usage_and_resources.md | 2 +- 4 files changed, 11 insertions(+), 1 deletion(-) diff --git a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml index 54cf46634..dd225896e 100644 --- a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml +++ b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml @@ -42,6 +42,11 @@ properties: dataset: type: object description: Defines the schema of items in your dataset, the full specification can be found in [Apify docs](https://docs.apify.com/platform/actors/development/actor-definition/dataset-schema) + defaultMemoryMbytes: + oneOf: + - type: integer + - type: string + description: Specifies the default amount of memory in megabytes to be used when the Actor is started. Can be an integer or a dynamic memory expression. minMemoryMbytes: type: integer minimum: 256 diff --git a/sources/platform/actors/development/actor_definition/actor_json.md b/sources/platform/actors/development/actor_definition/actor_json.md index 8023a1fce..95a363721 100644 --- a/sources/platform/actors/development/actor_definition/actor_json.md +++ b/sources/platform/actors/development/actor_definition/actor_json.md @@ -25,6 +25,7 @@ import TabItem from '@theme/TabItem'; "name": "name-of-my-scraper", "version": "0.0", "buildTag": "latest", + "defaultMemoryMbytes": "get(input, 'startUrls.length', 1) * 1024", "minMemoryMbytes": 256, "maxMemoryMbytes": 4096, "environmentVariables": { @@ -77,6 +78,7 @@ Actor `name`, `version`, `buildTag`, and `environmentVariables` are currently on | `input` | Optional | You can embed your [input schema](./input_schema/index.md) object directly in `actor.json` under the `input` field. You can also provide a path to a custom input schema. If not provided, the input schema at `.actor/INPUT_SCHEMA.json` or `INPUT_SCHEMA.json` is used, in this order of preference. | | `changelog` | Optional | The path to the CHANGELOG file displayed in the Information tab of the Actor in Apify Console next to Readme. If not provided, the CHANGELOG at `.actor/CHANGELOG.md` or `CHANGELOG.md` is used, in this order of preference. Your Actor doesn't need to have a CHANGELOG but it is a good practice to keep it updated for published Actors. | | `storages.dataset` | Optional | You can define the schema of the items in your dataset under the `storages.dataset` field. This can be either an embedded object or a path to a JSON schema file. [Read more](/platform/actors/development/actor-definition/dataset-schema) about Actor dataset schemas. | +| `defaultMemoryMbytes` | Optional | Specifies the default amount of memory in megabytes to be used when the Actor is started. Can be an integer or a [dynamic memory expression string](./dynamic_actor_memory/index.md). | | `minMemoryMbytes` | Optional | Specifies the minimum amount of memory in megabytes required by the Actor to run. Requires an _integer_ value. If both `minMemoryMbytes` and `maxMemoryMbytes` are set, then `minMemoryMbytes` must be equal or lower than `maxMemoryMbytes`. Refer to the [Usage and resources](https://docs.apify.com/platform/actors/running/usage-and-resources#memory) for more details about memory allocation. | | `maxMemoryMbytes` | Optional | Specifies the maximum amount of memory in megabytes required by the Actor to run. It can be used to control the costs of run, especially when developing pay per result Actors. Requires an _integer_ value. Refer to the [Usage and resources](https://docs.apify.com/platform/actors/running/usage-and-resources#memory) for more details about memory allocation. | | `usesStandbyMode` | Optional | Boolean specifying whether the Actor will have [Standby mode](../programming_interface/actor_standby.md) enabled. | diff --git a/sources/platform/actors/running/input_and_output.md b/sources/platform/actors/running/input_and_output.md index 1a9277630..89f823633 100644 --- a/sources/platform/actors/running/input_and_output.md +++ b/sources/platform/actors/running/input_and_output.md @@ -39,6 +39,9 @@ As part of the input, you can also specify run options such as [Build](../develo | Timeout | Timeout for the Actor run in seconds. Zero value means there is no timeout. | | Memory | Amount of memory allocated for the Actor run, in megabytes. | +:::info Dynamic memory +If the Actor is configured by developer to use [dynamic memory](../development/actor_definition/dynamic_actor_memory/index.md), the system will calculate the optimal memory allocation based on your input. In this case, the **Memory** option acts as an override - if you set it, the calculated value will be ignored. +::: ## Output diff --git a/sources/platform/actors/running/usage_and_resources.md b/sources/platform/actors/running/usage_and_resources.md index 49bb245f5..310819f7c 100644 --- a/sources/platform/actors/running/usage_and_resources.md +++ b/sources/platform/actors/running/usage_and_resources.md @@ -21,7 +21,7 @@ Check out the [Limits](../../limits.md) page for detailed information on Actor m ### Memory -When invoking an Actor, the caller must specify the memory allocation for the Actor run. The memory allocation must follow these requirements: +When invoking an Actor, the caller can specify the memory allocation for the Actor run. If not specified, the Actor's default memory is used (which can be [dynamic](../development/actor_definition/dynamic_actor_memory/index.md)). The memory allocation must follow these requirements: - It must be a power of 2. - The minimum allowed value is `128MB` From 6f1618def16647ebcf905e51c615430ba3c6e84a Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Mon, 15 Dec 2025 15:24:36 +0100 Subject: [PATCH 03/12] chore: fix lint --- .../dynamic_actor_memory/index.md | 37 +++++++++++++++---- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index 1b5b840bd..dfb093c77 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -11,6 +11,7 @@ slug: /actors/development/actor-definition/dynamic-actor-memory Dynamic Actor memory allows Actors to _automatically adjust its memory allocation based on the input and run options_. Instead of always using a fixed memory value, Actor can use just the right amount of memory for each run. This helps: + - _Optimize performance_ for large inputs (more memory for bigger tasks). - _Reduce costs_ for small runs (less memory when it’s not needed). - _Provide better user experience_, so users get optimal performance without having to manually configure memory. @@ -22,7 +23,9 @@ _Important_: The memory calculated by dynamic expression is used as the default --- ## Why dynamic memory matters + Currently, Actors often use a _static default memory_, but the optimal memory usually depends on the input: + - A small input (e.g., 10 URLs) might run fine on 512 MB. - A large input (e.g., 1,000 URLs) could require 4 GB+ to run efficiently. @@ -39,6 +42,7 @@ You can define a dynamic memory expression in your `actor.json`: "defaultMemoryMbytes": "get(input, 'startUrls.length' * 1024)" } ``` + - The expression is evaluated _before the run starts_. - The expression can reference variables from input and run options to calculate memory (e.g., `input.startUrls`, `runOptions.maxItems`). - The result becomes the default memory for the run, but users can still override it. @@ -54,11 +58,13 @@ Expressions are based on [MathJS](https://mathjs.org/), extended with custom hel You can access variables in two ways: 1. Direct property access + ```js input.foo + 512 runOptions.maxItems + 256 ``` -2. Double-brace syntax +1. Double-brace syntax + ```js {{input.foo}} {{runOptions.maxItems}} @@ -71,22 +77,28 @@ _You can mix both styles._ - Arithmetic: `+`, `-`, `*`, `/` - Math functions: `min()`, `max()`, `ceil()`, `floor()`, `round()`, `log()`, `exp()`, `log10()` - Conditional logic: + ``` condition ? valueIfTrue : valueIfFalse ``` + - Variable assignment: + ```js memoryPerUrl = 64; get(input, 'startUrls') * memoryPerUrl ``` #### Custom `get()` function + Use `get()` to safely read nested properties or provide fallback values: ```js get(obj, 'path.to.property', defaultValue) ``` + Examples: + ```js // Safely get array length get(input, 'startUrls.length', 1) // returns length or 1 if undefined @@ -102,33 +114,41 @@ get(input, 'numbers.1') // element at index 1 of the numbers array ``` ### Memory limits and clamping + After evaluation: + 1. The result is rounded up to the nearest power of two + - 900 → 1024 MB - 3,600 → 4096 MB -2. It is clamped to actor-defined min/max (`minMemoryMbytes` / `maxMemoryMbytes`). -3. It is clamped to platform limits (128 MB to 32 GB). +1. It is clamped to actor-defined min/max (`minMemoryMbytes` / `maxMemoryMbytes`). +2. It is clamped to platform limits (128 MB to 32 GB). :::info If the calculation results in an error, the Actor will start with a fixed default memory, which can be configured in the Actor's UI settings. ::: ### Example expressions + 1. Simple memory based on URL count + ```js get(input, 'startUrls.length', 1) * 512 ``` -2. Conditional logic +1. Conditional logic + ```js get(input, 'scrapeDetailed', false) ? 4096 : 1024 ``` -3. More complex calculation +1. More complex calculation + ```js urls = get(input, 'startUrls.length', 0); reviewsMultiplier = max(get(input, 'maxReviews', 1) / 10, 1); urls * reviewsMultiplier * 128 ``` -4. Using double-brace variables +1. Using double-brace variables + ```js {{input.itemsToProcess}} * 64 ``` @@ -136,6 +156,7 @@ urls * reviewsMultiplier * 128 ### Testing expressions #### Use NPM package + You can use our [NPM package](https://www.npmjs.com/package/@apify/actor-memory-expression) not only to calculate memory for your expression, but also to write unit tests and verify the behavior of your expressions locally. ```shell @@ -155,6 +176,7 @@ await calculateRunDynamicMemory( ``` #### Use CLI + You can use [Apify CLI](https://docs.apify.com/cli) to quickly evaluate expressions without writing code. It supports reading input from a JSON file and passing run options as flags. ```shell @@ -164,7 +186,8 @@ apify actor calculate-memory --input ./input.json --maxTotalChargeUsd=25 --- ### Key Takeaways + - Dynamic memory _automatically adjusts memory_ based on input and run options. - The result is used as default memory, but users can override it. - Use `get()`, arithmetic, math functions, and conditional logic to define expressions. -- Test expressions locally with the NPM package or CLI. \ No newline at end of file +- Test expressions locally with the NPM package or CLI. From 90f48e965b2312df04b0f6576a867a8df1e60705 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Tue, 16 Dec 2025 10:21:32 +0100 Subject: [PATCH 04/12] refactor: clean up --- .../components/schemas/actors/ActorDefinition.yaml | 1 + .../actor_definition/dynamic_actor_memory/index.md | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml index dd225896e..c8afb3353 100644 --- a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml +++ b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml @@ -47,6 +47,7 @@ properties: - type: integer - type: string description: Specifies the default amount of memory in megabytes to be used when the Actor is started. Can be an integer or a dynamic memory expression. + example: get(input, 'startUrls.length', 1) * 1024 minMemoryMbytes: type: integer minimum: 256 diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index dfb093c77..54f1b7a61 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -63,6 +63,7 @@ You can access variables in two ways: input.foo + 512 runOptions.maxItems + 256 ``` + 1. Double-brace syntax ```js @@ -121,6 +122,7 @@ After evaluation: - 900 → 1024 MB - 3,600 → 4096 MB + 1. It is clamped to actor-defined min/max (`minMemoryMbytes` / `maxMemoryMbytes`). 2. It is clamped to platform limits (128 MB to 32 GB). @@ -130,24 +132,27 @@ If the calculation results in an error, the Actor will start with a fixed defaul ### Example expressions -1. Simple memory based on URL count +#### Simple memory based on URL count ```js get(input, 'startUrls.length', 1) * 512 ``` -1. Conditional logic + +#### Conditional logic ```js get(input, 'scrapeDetailed', false) ? 4096 : 1024 ``` -1. More complex calculation + +#### More complex calculation ```js urls = get(input, 'startUrls.length', 0); reviewsMultiplier = max(get(input, 'maxReviews', 1) / 10, 1); urls * reviewsMultiplier * 128 ``` -1. Using double-brace variables + +#### Using double-brace variables ```js {{input.itemsToProcess}} * 64 From e203dcecaf51ec8ab4bb2f250a864a1b10364a6f Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Tue, 16 Dec 2025 10:26:52 +0100 Subject: [PATCH 05/12] refactor: clean up --- .../actor_definition/dynamic_actor_memory/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index 54f1b7a61..cf2a75fca 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -79,7 +79,7 @@ _You can mix both styles._ - Math functions: `min()`, `max()`, `ceil()`, `floor()`, `round()`, `log()`, `exp()`, `log10()` - Conditional logic: -``` +```js condition ? valueIfTrue : valueIfFalse ``` @@ -160,9 +160,9 @@ urls * reviewsMultiplier * 128 ### Testing expressions -#### Use NPM package +#### Use npm package -You can use our [NPM package](https://www.npmjs.com/package/@apify/actor-memory-expression) not only to calculate memory for your expression, but also to write unit tests and verify the behavior of your expressions locally. +You can use our [npm package](https://www.npmjs.com/package/@apify/actor-memory-expression) not only to calculate memory for your expression, but also to write unit tests and verify the behavior of your expressions locally. ```shell npm install @apify/actor-memory-expression @@ -195,4 +195,4 @@ apify actor calculate-memory --input ./input.json --maxTotalChargeUsd=25 - Dynamic memory _automatically adjusts memory_ based on input and run options. - The result is used as default memory, but users can override it. - Use `get()`, arithmetic, math functions, and conditional logic to define expressions. -- Test expressions locally with the NPM package or CLI. +- Test expressions locally with the npm package or CLI. From b3e4628707d5b2d76e78f0a59ee68a7f40cabbc3 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Tue, 16 Dec 2025 11:23:14 +0100 Subject: [PATCH 06/12] refactor: clean up --- .../openapi/components/schemas/actors/ActorDefinition.yaml | 5 +++-- .../actors/development/actor_definition/actor_json.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml index c8afb3353..679e9a07c 100644 --- a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml +++ b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml @@ -44,10 +44,11 @@ properties: description: Defines the schema of items in your dataset, the full specification can be found in [Apify docs](https://docs.apify.com/platform/actors/development/actor-definition/dataset-schema) defaultMemoryMbytes: oneOf: - - type: integer - type: string + example: get(input, 'startUrls.length', 1) * 1024 + - type: integer + example: 1024 description: Specifies the default amount of memory in megabytes to be used when the Actor is started. Can be an integer or a dynamic memory expression. - example: get(input, 'startUrls.length', 1) * 1024 minMemoryMbytes: type: integer minimum: 256 diff --git a/sources/platform/actors/development/actor_definition/actor_json.md b/sources/platform/actors/development/actor_definition/actor_json.md index 95a363721..9fc80c184 100644 --- a/sources/platform/actors/development/actor_definition/actor_json.md +++ b/sources/platform/actors/development/actor_definition/actor_json.md @@ -67,7 +67,7 @@ Actor `name`, `version`, `buildTag`, and `environmentVariables` are currently on | Property | Type | Description | | --- | --- | --- | -| `actorSpecification` | Required | The version of the Actor specification. This property must be set to `1`, which is the only version available. | +| `actorSpecification` | Required | The version of the Actor specification. This property must be set to `1`, which is the only version available. | | `name` | Required | The name of the Actor. | | `version` | Required | The version of the Actor, specified in the format `[Number].[Number]`, e.g., `0.1`, `0.3`, `1.0`, `1.3`, etc. | | `buildTag` | Optional | The tag name to be applied to a successful build of the Actor. If not specified, defaults to `latest`. Refer to the [builds](../builds_and_runs/builds.md) for more information. | From 133af16cae52dc6a99b17eb74588a06a26da39ad Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Thu, 18 Dec 2025 10:22:05 +0100 Subject: [PATCH 07/12] refactor: clean up --- .../schemas/actors/ActorDefinition.yaml | 2 +- .../dynamic_actor_memory/index.md | 58 ++++++++++--------- 2 files changed, 31 insertions(+), 29 deletions(-) diff --git a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml index 679e9a07c..1978ecbc1 100644 --- a/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml +++ b/apify-api/openapi/components/schemas/actors/ActorDefinition.yaml @@ -48,7 +48,7 @@ properties: example: get(input, 'startUrls.length', 1) * 1024 - type: integer example: 1024 - description: Specifies the default amount of memory in megabytes to be used when the Actor is started. Can be an integer or a dynamic memory expression. + description: Specifies the default amount of memory in megabytes to be used when the Actor is started. Can be an integer or a [dynamic memory expression](/platform/actors/development/actor-definition/dynamic-actor-memory). minMemoryMbytes: type: integer minimum: 256 diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index cf2a75fca..c9410c5bb 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -1,6 +1,6 @@ --- title: Dynamic Actor memory -description: Learn how to automatically adjust your Actor's memory based on input size and run options, so you can optimize performance and reduce costs without manual configuration.. +description: Learn how to automatically adjust your Actor's memory based on input size and run options, so you can optimize performance and reduce costs without manual configuration. slug: /actors/development/actor-definition/dynamic-actor-memory --- @@ -16,8 +16,10 @@ This helps: - _Reduce costs_ for small runs (less memory when it’s not needed). - _Provide better user experience_, so users get optimal performance without having to manually configure memory. -:::info -_Important_: The memory calculated by dynamic expression is used as the default memory for the run. Users can still override it manually for each run. +:::info Manual override + +The memory calculated by dynamic expression is used as the default memory for the run. Users can still override it manually for each run. + ::: --- @@ -49,46 +51,46 @@ You can define a dynamic memory expression in your `actor.json`: --- -### Writing Expressions +### How to write expressions Expressions are based on [MathJS](https://mathjs.org/), extended with custom helper function `get`. -#### Variable Access +#### Access run input and options You can access variables in two ways: 1. Direct property access -```js -input.foo + 512 -runOptions.maxItems + 256 -``` + ```js + input.foo + 512 + runOptions.maxItems + 256 + ``` -1. Double-brace syntax +2. Double-brace syntax -```js -{{input.foo}} -{{runOptions.maxItems}} -``` + ```js + {{input.foo}} + {{runOptions.maxItems}} + ``` _You can mix both styles._ -#### Supported Operations +#### Supported operations - Arithmetic: `+`, `-`, `*`, `/` - Math functions: `min()`, `max()`, `ceil()`, `floor()`, `round()`, `log()`, `exp()`, `log10()` - Conditional logic: -```js -condition ? valueIfTrue : valueIfFalse -``` + ```js + condition ? valueIfTrue : valueIfFalse + ``` - Variable assignment: -```js -memoryPerUrl = 64; -get(input, 'startUrls') * memoryPerUrl -``` + ```js + memoryPerUrl = 64; + get(input, 'startUrls') * memoryPerUrl + ``` #### Custom `get()` function @@ -120,13 +122,13 @@ After evaluation: 1. The result is rounded up to the nearest power of two -- 900 → 1024 MB -- 3,600 → 4096 MB + - 900 → 1024 MB + - 3,600 → 4096 MB -1. It is clamped to actor-defined min/max (`minMemoryMbytes` / `maxMemoryMbytes`). -2. It is clamped to platform limits (128 MB to 32 GB). +2. It is clamped to actor-defined min/max (`minMemoryMbytes` / `maxMemoryMbytes`). +3. It is clamped to platform limits (128 MB to 32 GB). -:::info +:::info Fallback value If the calculation results in an error, the Actor will start with a fixed default memory, which can be configured in the Actor's UI settings. ::: @@ -162,7 +164,7 @@ urls * reviewsMultiplier * 128 #### Use npm package -You can use our [npm package](https://www.npmjs.com/package/@apify/actor-memory-expression) not only to calculate memory for your expression, but also to write unit tests and verify the behavior of your expressions locally. +You can use the [actor-memory-expressions](https://www.npmjs.com/package/@apify/actor-memory-expression) npm package not only to calculate memory for your expression, but also to write unit tests and verify the behavior of your expressions locally. ```shell npm install @apify/actor-memory-expression From 52099e77ee3f1b77f8ed788de1905431a35518f0 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Thu, 18 Dec 2025 10:32:06 +0100 Subject: [PATCH 08/12] fix: lint --- sources/platform/actors/running/input_and_output.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sources/platform/actors/running/input_and_output.md b/sources/platform/actors/running/input_and_output.md index 89f823633..07186674d 100644 --- a/sources/platform/actors/running/input_and_output.md +++ b/sources/platform/actors/running/input_and_output.md @@ -34,13 +34,15 @@ As part of the input, you can also specify run options such as [Build](../develo ![Run options](./images/input_and_output/actor-options.png) | Option | Description | -|:---|:---| +| :--- | :--- | | Build | Tag or number of the build to run (e.g. **latest** or **1.2.34**). | | Timeout | Timeout for the Actor run in seconds. Zero value means there is no timeout. | | Memory | Amount of memory allocated for the Actor run, in megabytes. | :::info Dynamic memory + If the Actor is configured by developer to use [dynamic memory](../development/actor_definition/dynamic_actor_memory/index.md), the system will calculate the optimal memory allocation based on your input. In this case, the **Memory** option acts as an override - if you set it, the calculated value will be ignored. + ::: ## Output From 9e853c809f4f0796e2d45421e427af89cae6f4c4 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Fri, 19 Dec 2025 14:56:32 +0100 Subject: [PATCH 09/12] refactor: clean up --- .../dynamic_actor_memory/index.md | 147 +++++++++++------- .../actors/running/input_and_output.md | 2 +- 2 files changed, 91 insertions(+), 58 deletions(-) diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index c9410c5bb..43d72a37b 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -4,36 +4,54 @@ description: Learn how to automatically adjust your Actor's memory based on inpu slug: /actors/development/actor-definition/dynamic-actor-memory --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + **Learn how to automatically adjust your Actor's memory based on input size and run options, so you can optimize performance and reduce costs without manual configuration.** --- -Dynamic Actor memory allows Actors to _automatically adjust its memory allocation based on the input and run options_. Instead of always using a fixed memory value, Actor can use just the right amount of memory for each run. +Dynamic Actor memory allows Actor to automatically adjust its memory allocation based on the input and run options. Instead of always using a fixed memory value, Actor can use just the right amount of memory for each run. -This helps: +Currently, Actors often use a static default memory, but the optimal memory usually depends on the input size: +- A small input (for example, 10 URLs) might run fine on 512 MB. +- A large input (for example, 1,000 URLs) could require 4 GB or more to run efficiently. + +_Setting a single default value either wastes resources on small runs or slows down execution for large ones._ Dynamic memory solves this by calculating the required memory just before the run starts, based on the actual input and run options. +This helps: - _Optimize performance_ for large inputs (more memory for bigger tasks). - _Reduce costs_ for small runs (less memory when it’s not needed). - _Provide better user experience_, so users get optimal performance without having to manually configure memory. -:::info Manual override +:::info Dynamic memory is not runtime auto-scaling. + +_This feature does not change memory while the Actor is running._ -The memory calculated by dynamic expression is used as the default memory for the run. Users can still override it manually for each run. +Memory is calculated once, right before the run begins. Each new run (for example, when the user provides different input) starts with memory calculated by the expression. + +Users can still override it manually for each run. ::: ---- -## Why dynamic memory matters +## How is memory for a run determined? -Currently, Actors often use a _static default memory_, but the optimal memory usually depends on the input: +The final memory assigned to an Actor run is determined in the following order: -- A small input (e.g., 10 URLs) might run fine on 512 MB. -- A large input (e.g., 1,000 URLs) could require 4 GB+ to run efficiently. +1. _Run-level override (highest priority)._ + If the user explicitly sets memory when starting a run (via UI or API), this value is always used. -Setting a single default either _wastes resources or slows down execution_. Dynamic memory solves this by calculating memory just before the run starts. +2. _Dynamic memory expression._ + If no run-level override is provided, the platform evaluates the dynamic memory expression defined in actor.json. The expression can use values from input and run options to calculate memory. ---- +3. _Actor default memory._ + If no dynamic expression is defined, or if the expression fails to evaluate, the Actor falls back to its fixed default memory configured in the Actor’s UI settings. + +4. _Platform limits._ + The final value is always rounded and clamped to platform-supported memory limits. + +_In all cases, the memory value is finalized before the run starts and remains constant during execution._ ## How to define dynamic memory expression @@ -45,17 +63,10 @@ You can define a dynamic memory expression in your `actor.json`: } ``` -- The expression is evaluated _before the run starts_. -- The expression can reference variables from input and run options to calculate memory (e.g., `input.startUrls`, `runOptions.maxItems`). -- The result becomes the default memory for the run, but users can still override it. - ---- - -### How to write expressions - Expressions are based on [MathJS](https://mathjs.org/), extended with custom helper function `get`. -#### Access run input and options + +### Access run input and options You can access variables in two ways: @@ -75,7 +86,7 @@ You can access variables in two ways: _You can mix both styles._ -#### Supported operations +### Supported operations - Arithmetic: `+`, `-`, `*`, `/` - Math functions: `min()`, `max()`, `ceil()`, `floor()`, `round()`, `log()`, `exp()`, `log10()` @@ -92,7 +103,7 @@ _You can mix both styles._ get(input, 'startUrls') * memoryPerUrl ``` -#### Custom `get()` function +### Custom `get()` function Use `get()` to safely read nested properties or provide fallback values: @@ -107,7 +118,7 @@ Examples: get(input, 'startUrls.length', 1) // returns length or 1 if undefined // Safely get nested property -get(input, 'foo.bar.baz.count') // safely access nested objects +get(input, 'foo.bar.baz') // safely access nested objects // Fallback get(input, 'foo', 1024) // returns 1024 if 'foo' doesn't exist @@ -116,17 +127,18 @@ get(input, 'foo', 1024) // returns 1024 if 'foo' doesn't exist get(input, 'numbers.1') // element at index 1 of the numbers array ``` -### Memory limits and clamping +### Memory limits -After evaluation: +After the expression is evaluated, the memory value goes through these steps: -1. The result is rounded up to the nearest power of two +1. The result is rounded to the nearest power of two + - 300 -> 256 MB - 900 → 1024 MB - 3,600 → 4096 MB -2. It is clamped to actor-defined min/max (`minMemoryMbytes` / `maxMemoryMbytes`). -3. It is clamped to platform limits (128 MB to 32 GB). +1. If the Actor has minimum or maximum memory limits defined (`minMemoryMbytes` / `maxMemoryMbytes`), the value is adjusted to stay within those limits. +1. The value is adjusted to stay within platform limits (128 MB to 32 GB). :::info Fallback value If the calculation results in an error, the Actor will start with a fixed default memory, which can be configured in the Actor's UI settings. @@ -134,31 +146,61 @@ If the calculation results in an error, the Actor will start with a fixed defaul ### Example expressions -#### Simple memory based on URL count + + + This expression calculates memory based on the number of URLs you want to process. + It multiplies the number of URLs by 512 MB, so more URLs automatically get more memory. -```js -get(input, 'startUrls.length', 1) * 512 -``` + ```js + get(input, 'startUrls.length', 1) * 512 + ``` -#### Conditional logic + Explanation: + - `get(input, 'startUrls.length', 1)` → Safely reads length of `startUrls` array; defaults to 1 if not provided. + - Allocates 512 MB per URL. -```js -get(input, 'scrapeDetailed', false) ? 4096 : 1024 -``` + + -#### More complex calculation + You can adjust memory based on a condition, for example user wants detailed scraping. -```js -urls = get(input, 'startUrls.length', 0); -reviewsMultiplier = max(get(input, 'maxReviews', 1) / 10, 1); -urls * reviewsMultiplier * 128 -``` + ```js + get(input, 'scrapeDetailed', false) ? 4096 : 1024 + ``` -#### Using double-brace variables + Explanation: + - `get(input, 'scrapeDetailed', false)` → Reads a boolean flag from `input`; defaults to `false`. + - `? 4096 : 1024` → If `scrapeDetailed` is `true`, allocate 4096 MB; otherwise, allocate 1024 MB. -```js -{{input.itemsToProcess}} * 64 -``` + + + + For more complex cases, you can assign intermediate variables to simplify calculations. + + ```js + urlsCount = get(input, 'startUrls.length', 0); + reviewsMultiplier = max(get(input, 'maxReviews', 1) / 10, 1); + urlsCount * reviewsMultiplier * 128 + ``` + Explanation: + - `urlsCount` → Number of URLs to process. + - `reviewsMultiplier` → Adjusts memory based on the number of reviews; ensures at least 1. + - `urlsCount * reviewsMultiplier * 128` → Final memory allocation, scaling with both URLs and review count. + + + + + You can also use double-brace syntax to refer to input variables. + + ```js + {{input.itemsToProcess}} * 64 + ``` + + Explanation: + - `{{input.itemsToProcess}}` → Reads the number of items to process. + - Allocates 64 MB per item. + + ### Testing expressions @@ -166,7 +208,7 @@ urls * reviewsMultiplier * 128 You can use the [actor-memory-expressions](https://www.npmjs.com/package/@apify/actor-memory-expression) npm package not only to calculate memory for your expression, but also to write unit tests and verify the behavior of your expressions locally. -```shell +```bash npm install @apify/actor-memory-expression ``` @@ -186,15 +228,6 @@ await calculateRunDynamicMemory( You can use [Apify CLI](https://docs.apify.com/cli) to quickly evaluate expressions without writing code. It supports reading input from a JSON file and passing run options as flags. -```shell +```bash apify actor calculate-memory --input ./input.json --maxTotalChargeUsd=25 ``` - ---- - -### Key Takeaways - -- Dynamic memory _automatically adjusts memory_ based on input and run options. -- The result is used as default memory, but users can override it. -- Use `get()`, arithmetic, math functions, and conditional logic to define expressions. -- Test expressions locally with the npm package or CLI. diff --git a/sources/platform/actors/running/input_and_output.md b/sources/platform/actors/running/input_and_output.md index 07186674d..05f5a441a 100644 --- a/sources/platform/actors/running/input_and_output.md +++ b/sources/platform/actors/running/input_and_output.md @@ -41,7 +41,7 @@ As part of the input, you can also specify run options such as [Build](../develo :::info Dynamic memory -If the Actor is configured by developer to use [dynamic memory](../development/actor_definition/dynamic_actor_memory/index.md), the system will calculate the optimal memory allocation based on your input. In this case, the **Memory** option acts as an override - if you set it, the calculated value will be ignored. +If the Actor is configured by developer to use [dynamic memory](../development/actor_definition/dynamic_actor_memory/index.md), the system will calculate the optimal memory allocation based on your input. In this case, the **Memory** option acts as an override — if you set it, the calculated value will be ignored. ::: From 7ab0ccc961353746b993a6da8703b5ee3ac8fea9 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Fri, 19 Dec 2025 14:56:46 +0100 Subject: [PATCH 10/12] fix: lint --- .../dynamic_actor_memory/index.md | 28 ++++++++++++------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index 43d72a37b..1289b9f51 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -14,12 +14,14 @@ import TabItem from '@theme/TabItem'; Dynamic Actor memory allows Actor to automatically adjust its memory allocation based on the input and run options. Instead of always using a fixed memory value, Actor can use just the right amount of memory for each run. Currently, Actors often use a static default memory, but the optimal memory usually depends on the input size: + - A small input (for example, 10 URLs) might run fine on 512 MB. - A large input (for example, 1,000 URLs) could require 4 GB or more to run efficiently. _Setting a single default value either wastes resources on small runs or slows down execution for large ones._ Dynamic memory solves this by calculating the required memory just before the run starts, based on the actual input and run options. This helps: + - _Optimize performance_ for large inputs (more memory for bigger tasks). - _Reduce costs_ for small runs (less memory when it’s not needed). - _Provide better user experience_, so users get optimal performance without having to manually configure memory. @@ -28,7 +30,7 @@ This helps: _This feature does not change memory while the Actor is running._ -Memory is calculated once, right before the run begins. Each new run (for example, when the user provides different input) starts with memory calculated by the expression. +Memory is calculated once, right before the run begins. Each new run (for example, when the user provides different input) starts with memory calculated by the expression. Users can still override it manually for each run. @@ -156,8 +158,9 @@ If the calculation results in an error, the Actor will start with a fixed defaul ``` Explanation: - - `get(input, 'startUrls.length', 1)` → Safely reads length of `startUrls` array; defaults to 1 if not provided. - - Allocates 512 MB per URL. + +- `get(input, 'startUrls.length', 1)` → Safely reads length of `startUrls` array; defaults to 1 if not provided. +- Allocates 512 MB per URL. @@ -169,8 +172,9 @@ If the calculation results in an error, the Actor will start with a fixed defaul ``` Explanation: - - `get(input, 'scrapeDetailed', false)` → Reads a boolean flag from `input`; defaults to `false`. - - `? 4096 : 1024` → If `scrapeDetailed` is `true`, allocate 4096 MB; otherwise, allocate 1024 MB. + +- `get(input, 'scrapeDetailed', false)` → Reads a boolean flag from `input`; defaults to `false`. +- `? 4096 : 1024` → If `scrapeDetailed` is `true`, allocate 4096 MB; otherwise, allocate 1024 MB. @@ -182,10 +186,12 @@ If the calculation results in an error, the Actor will start with a fixed defaul reviewsMultiplier = max(get(input, 'maxReviews', 1) / 10, 1); urlsCount * reviewsMultiplier * 128 ``` + Explanation: - - `urlsCount` → Number of URLs to process. - - `reviewsMultiplier` → Adjusts memory based on the number of reviews; ensures at least 1. - - `urlsCount * reviewsMultiplier * 128` → Final memory allocation, scaling with both URLs and review count. + +- `urlsCount` → Number of URLs to process. +- `reviewsMultiplier` → Adjusts memory based on the number of reviews; ensures at least 1. +- `urlsCount * reviewsMultiplier * 128` → Final memory allocation, scaling with both URLs and review count. @@ -197,9 +203,11 @@ If the calculation results in an error, the Actor will start with a fixed defaul ``` Explanation: - - `{{input.itemsToProcess}}` → Reads the number of items to process. - - Allocates 64 MB per item. + +- `{{input.itemsToProcess}}` → Reads the number of items to process. +- Allocates 64 MB per item. + ### Testing expressions From 2f7561c634e645ca2c7f36d5d5973436d0b78005 Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Fri, 19 Dec 2025 15:03:51 +0100 Subject: [PATCH 11/12] refactor: clean up --- .../actor_definition/dynamic_actor_memory/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index 1289b9f51..fadcabd98 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -13,7 +13,7 @@ import TabItem from '@theme/TabItem'; Dynamic Actor memory allows Actor to automatically adjust its memory allocation based on the input and run options. Instead of always using a fixed memory value, Actor can use just the right amount of memory for each run. -Currently, Actors often use a static default memory, but the optimal memory usually depends on the input size: +Optimal memory usually depends on the input size: - A small input (for example, 10 URLs) might run fine on 512 MB. - A large input (for example, 1,000 URLs) could require 4 GB or more to run efficiently. @@ -79,7 +79,7 @@ You can access variables in two ways: runOptions.maxItems + 256 ``` -2. Double-brace syntax +1. Double-brace syntax ```js {{input.foo}} From 50b78baac5ab16287305ba162898921f799aeedc Mon Sep 17 00:00:00 2001 From: Daniil Poletaev <44584010+danpoletaev@users.noreply.github.com> Date: Fri, 19 Dec 2025 15:13:32 +0100 Subject: [PATCH 12/12] refactor: clean up --- .../actor_definition/dynamic_actor_memory/index.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md index fadcabd98..843bbba0c 100644 --- a/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md +++ b/sources/platform/actors/development/actor_definition/dynamic_actor_memory/index.md @@ -26,6 +26,14 @@ This helps: - _Reduce costs_ for small runs (less memory when it’s not needed). - _Provide better user experience_, so users get optimal performance without having to manually configure memory. +For example, the developer of an Actor could define an expression like: + +```js +min(get(input, 'startUrls.length', 1) * 64, 4096) +``` + +This expression calculates memory based on the number of URLs provided by the user, making sure that for large inputs the Actor doesn’t exceed 4 GB. + :::info Dynamic memory is not runtime auto-scaling. _This feature does not change memory while the Actor is running._