Merge branch 'master' into jdstrand/fix-rpm-install2

Author: jstirnaman

content/telegraf/v1/configuration.md

@@ -512,6 +512,26 @@ The following config parameters are available for all inputs:
 - **name_prefix**: Specifies a prefix to attach to the measurement name.
 - **name_suffix**: Specifies a suffix to attach to the measurement name.
 
+### Data formats
+
+Some output plugins support the `data_format` option, which specifies a serializer
+to convert metrics before writing.
+Common serializers include `json`, `influx`, `prometheus`, and `csv`.
+
+Output plugins that support serializers may also offer `use_batch_format`, which
+controls whether the serializer receives metrics individually or as a batch.
+Batch mode enables more efficient encoding for formats like JSON arrays.
+
+```toml
+[[outputs.file]]
+  files = ["stdout"]
+  data_format = "json"
+  use_batch_format = true
+```
+
+For available serializers and configuration options, see
+[output data formats](/telegraf/v1/data_formats/output/).
+
 ## Aggregator configuration
 
 The following config parameters are available for all aggregators:

content/telegraf/v1/configure_plugins/output_plugins/_index.md

@@ -1,18 +1,41 @@
 ---
 title: Write data with output plugins
 description: |
-  Output plugins define where Telegraf will deliver the collected metrics.
+  Output plugins define where Telegraf delivers the collected metrics.
 menu:
   telegraf_v1:
-
      name: Output plugins
      weight: 20
      parent: Configure plugins
 related:
   - /telegraf/v1/output-plugins/
+  - /telegraf/v1/data_formats/output/
 ---
-Output plugins define where Telegraf will deliver the collected metrics. Send metrics to InfluxDB or to a variety of other datastores, services, and message queues, including Graphite, OpenTSDB, Datadog, Librato, Kafka, MQTT, and NSQ.
 
-For a complete list of output plugins and links to their detailed configuration options, see [output plugins](/telegraf/v1/plugins/#output-plugins).
+Output plugins define where Telegraf delivers collected metrics.
+Send metrics to InfluxDB or to a variety of other datastores, services, and
+message queues, including Graphite, OpenTSDB, Datadog, Kafka, MQTT, and NSQ.
+
+For a complete list of output plugins and links to their detailed configuration
+options, see [output plugins](/telegraf/v1/output-plugins/).
+
+## Output plugins and data formats
+
+Output plugins control *where* metrics go.
+Many output plugins also support *data formats* (serializers) that control how
+metrics are formatted before writing.
+
+Configure a serializer using the `data_format` option in your output plugin:
+
+```toml
+[[outputs.http]]
+  url = "http://example.com/metrics"
+  data_format = "json"
+```
+
+Some output plugins (like `influxdb_v2` or `prometheus_client`) use a fixed
+format and don't support `data_format`.
+Others (like `file`, `http`, `kafka`) support multiple serializers.
 
-In addition to plugin-specific data formats, Telegraf supports a set of [common data formats](/telegraf/v1/data_formats/output/) available when configuring many of the Telegraf output plugins.
+For available serializers and their options, see
+[output data formats](/telegraf/v1/data_formats/output/).

content/telegraf/v1/data_formats/output/_index.md

@@ -7,25 +7,73 @@ menu:
     name: Output data formats
     weight: 1
     parent: Data formats
+related:
+  - /telegraf/v1/configure_plugins/output_plugins/
+  - /telegraf/v1/configuration/
 ---
 
-In addition to output-specific data formats, Telegraf supports the following set
-of common data formats that may be selected when configuring many of the Telegraf
-output plugins.
+Telegraf uses **serializers** to convert metrics into output data formats.
+Many [output plugins](/telegraf/v1/configure_plugins/output_plugins/) support the `data_format` option, which lets you choose
+how metrics are formatted before writing.
 
-{{< children >}}
+- [How output plugins use serializers](#how-output-plugins-use-serializers)
+- [Choosing an output approach](#choosing-an-output-approach)
+- [Available serializers](#available-serializers)
+
+## How output plugins use serializers
+
+When you configure `data_format` in an output plugin, Telegraf uses a **serializer**
+to convert metrics into that format before writing.
+The output plugin controls *where* data goes; the serializer controls *how* it's formatted.
 
-You will be able to identify the plugins with support by the presence of a
-`data_format` configuration option, for example, in the File (`file`) output plugin:
+Some output plugins support `use_batch_format`, which changes how the serializer
+processes metrics.
+When enabled, the serializer receives all metrics in a batch together rather than
+one at a time, enabling more efficient encoding and formats that represent multiple
+metrics as a unit (like JSON arrays).
 
 ```toml
 [[outputs.file]]
-  ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout"]
 
-  ## Data format to output.
-  ## Each data format has its own unique set of configuration options, read
-  ## more about them here:
-  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
-  data_format = "influx"
+  ## Output plugin option: process metrics as a batch
+  use_batch_format = true
+
+  ## Serializer selection: format metrics as JSON
+  data_format = "json"
 ```
+
+Output plugins that support `use_batch_format`:
+`file`, `http`, `amqp`, `kafka`, `nats`, `mqtt`, `exec`, `execd`, `remotefile`
+
+## Choosing an output approach
+
+### By destination
+
+| Destination | Recommended Approach |
+|-------------|---------------------|
+| **Prometheus scraping** | [`prometheus_client`](/telegraf/v1/output-plugins/prometheus_client/) output plugin (exposes `/metrics` endpoint) |
+| **InfluxDB** | [`influxdb`](/telegraf/v1/output-plugins/influxdb/) or [`influxdb_v2`](/telegraf/v1/output-plugins/influxdb_v2/) output plugin (native protocol) |
+| **Remote HTTP endpoints** | [`http`](/telegraf/v1/output-plugins/http/) output + serializer |
+| **Files** | [`file`](/telegraf/v1/output-plugins/file/) output + serializer |
+| **Message queues** | [`kafka`](/telegraf/v1/output-plugins/kafka/), [`nats`](/telegraf/v1/output-plugins/nats/), [`amqp`](/telegraf/v1/output-plugins/amqp/) + serializer |
+
+### By metric type
+
+Some metric types require state across collection intervals:
+
+- **Histograms** accumulate observations into buckets
+- **Summaries** track quantiles over a sliding window
+
+Serializers process each batch independently and cannot maintain this state.
+When a histogram or summary spans multiple batches, the serializer may produce
+incomplete or incorrect output.
+
+For these metric types, use a dedicated output plugin that maintains state--for example:
+
+- **Prometheus metrics**: Use [`prometheus_client`](/telegraf/v1/output-plugins/prometheus_client/)
+  instead of the prometheus serializer
+
+## Available serializers
+
+{{< children >}}

content/telegraf/v1/data_formats/output/binary.md

@@ -0,0 +1,80 @@
+---
+title: Binary output data format
+list_title: Binary
+description: Use the `binary` output data format (serializer) to serialize Telegraf metrics into binary protocols using user-specified configurations.
+menu:
+  telegraf_v1_ref:
+    name: Binary
+    weight: 10
+    parent: Output data formats
+    identifier: output-data-format-binary
+---
+
+Use the `binary` output data format (serializer) to serialize Telegraf metrics into binary protocols using user-specified configurations.
+
+## Configuration
+
+```toml
+[[outputs.socket_writer]]
+  address = "tcp://127.0.0.1:54000"
+  metric_batch_size = 1
+
+  ## Data format to output.
+  data_format = "binary"
+
+  ## Specify the endianness of the data.
+  ## Available values are "little" (little-endian), "big" (big-endian) and "host",
+  ## where "host" means the same endianness as the machine running Telegraf.
+  # endianness = "host"
+
+  ## Definition of the message format and the serialized data.
+  ## Please note that you need to define all elements of the data in the
+  ## correct order as the binary format is position-dependent.
+  ##
+  ## Entry properties:
+  ##  read_from         --  Source of the data: "field", "tag", "time" or "name".
+  ##                        Defaults to "field" if omitted.
+  ##  name              --  Name of the field or tag. Can be omitted for "time" and "name".
+  ##  data_format       --  Target data-type: "int8/16/32/64", "uint8/16/32/64",
+  ##                        "float32/64", "string".
+  ##                        For time: "unix" (default), "unix_ms", "unix_us", "unix_ns".
+  ##  string_length     --  Length of the string in bytes (for "string" type only).
+  ##  string_terminator --  Terminator for strings: "null", "0x00", etc.
+  ##                        Defaults to "null" for strings.
+  entries = [
+    { read_from = "name", data_format = "string", string_length = 32 },
+    { read_from = "tag", name = "host", data_format = "string", string_length = 64 },
+    { read_from = "field", name = "value", data_format = "float64" },
+    { read_from = "time", data_format = "unix_ns" },
+  ]
+```
+
+### Configuration options
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `endianness` | string | Byte order: `"little"`, `"big"`, or `"host"` (default) |
+| `entries` | array | Ordered list of data elements to serialize |
+
+### Entry properties
+
+Each entry in the `entries` array defines how to serialize a piece of metric data:
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `read_from` | string | Data source: `"field"`, `"tag"`, `"time"`, or `"name"` |
+| `name` | string | Field or tag name (required for `"field"` and `"tag"`) |
+| `data_format` | string | Target type: `"int8/16/32/64"`, `"uint8/16/32/64"`, `"float32/64"`, `"string"` |
+| `string_length` | integer | Fixed string length in bytes (for `"string"` type) |
+| `string_terminator` | string | String terminator: `"null"`, `"0x00"`, etc. |
+
+## Type conversion
+
+If the original field type differs from the target type, the serializer converts the value.
+A warning is logged if the conversion may cause loss of precision.
+
+## String handling
+
+For string fields:
+- If the string is longer than `string_length`, it is truncated so that the string and its terminator together fit within `string_length` bytes
+- If the string is shorter than `string_length`, it is padded with the terminator character so that the string and its terminator together occupy `string_length` bytes

content/telegraf/v1/data_formats/output/cloudevents.md

@@ -0,0 +1,68 @@
+---
+title: CloudEvents output data format
+list_title: CloudEvents
+description: Use the `cloudevents` output data format (serializer) to format Telegraf metrics as CloudEvents in JSON format.
+menu:
+  telegraf_v1_ref:
+    name: CloudEvents
+    weight: 10
+    parent: Output data formats
+    identifier: output-data-format-cloudevents
+---
+
+Use the `cloudevents` output data format (serializer) to format Telegraf metrics as [CloudEvents](https://cloudevents.io) in [JSON format](https://github.com/cloudevents/spec/blob/v1.0/json-format.md).
+Versions v1.0 and v0.3 of the CloudEvents specification are supported, with v1.0 as the default.
+
+## Configuration
+
+```toml
+[[outputs.file]]
+  files = ["stdout", "/tmp/metrics.out"]
+
+  ## Data format to output.
+  data_format = "cloudevents"
+
+  ## Specification version to use for events.
+  ## Supported versions: "0.3" and "1.0" (default).
+  # cloudevents_version = "1.0"
+
+  ## Event source specifier.
+  ## Overwrites the source header field with the given value.
+  # cloudevents_source = "telegraf"
+
+  ## Tag to use as event source specifier.
+  ## Overwrites the source header field with the value of the specified tag.
+  ## Takes precedence over 'cloudevents_source' if both are set.
+  ## Falls back to 'cloudevents_source' if the tag doesn't exist for a metric.
+  # cloudevents_source_tag = ""
+
+  ## Event-type specifier to overwrite the default value.
+  ## Default for single metric: 'com.influxdata.telegraf.metric'
+  ## Default for batch: 'com.influxdata.telegraf.metrics' (plural)
+  # cloudevents_event_type = ""
+
+  ## Set time header of the event.
+  ## Supported values:
+  ##   none     -- do not set event time
+  ##   earliest -- use timestamp of the earliest metric
+  ##   latest   -- use timestamp of the latest metric
+  # cloudevents_time = "earliest"
+```
+
+### Configuration options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `cloudevents_version` | string | `"1.0"` | CloudEvents specification version (`"0.3"` or `"1.0"`) |
+| `cloudevents_source` | string | `"telegraf"` | Event source identifier |
+| `cloudevents_source_tag` | string | `""` | Tag to use as source (overrides `cloudevents_source`) |
+| `cloudevents_event_type` | string | auto | Event type (auto-detected based on single/batch) |
+| `cloudevents_time` | string | `"earliest"` | Event timestamp: `"none"`, `"earliest"`, or `"latest"` |
+
+## Event types
+
+By default, the serializer sets the event type based on the content:
+- Single metric: `com.influxdata.telegraf.metric`
+- Batch of metrics: `com.influxdata.telegraf.metrics`
+
+Use `cloudevents_event_type` to override this behavior.