azureblobexporter

Azure Blob Exporter for OpenTelemetry Collector

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code coverage
Code Owners	@hgaol, @MovieStoreGuy

Configuration

The following settings are required:

• url: Must be specified if auth type is not connection_string. If auth type is connection_string, it's optional or will be override by the auth.connection_string. Azure storage account endpoint. This setting might be replaced with endpoint for future. e.g. https://.blob.core.windows.net/
• auth (no default): Authentication method for exporter to ingest data.
  • type (no default): Authentication type for exporter. supported values are: connection_string, service_principal, system_managed_identity, user_managed_identity and workload_identity.
  • tenand_id: Tenand Id for the client, only needed when type is service_principal and workload_identity.
  • client_id: Client Id for the auth, only needed when type is service_principal, user_managed_identity and workload_identity.
  • client_secret: Secret for the client, only needed when type is service_principal.
  • connection_string: Connection string to the endpoint. Only needed for connection_string auth type. Once provided, it'll override the url parameter to the storage account.
  • federated_token_file: The path of the projected service account token file, only needed when type is workload_identity.

The following settings can be optionally configured and have default values:

• container: container for metrics, logs and traces. A container organizes a set of blobs, similar to a directory in a file system. More details can refer this.
  • metrics (default metrics): container to store metrics. default value is metrics.
  • logs (default logs): container to store logs. default value is logs.
  • traces (default traces): container to store traces. default value is traces.
• blob_name_format: the final blob name will be blob_name
  • template_enabled (default false): enables Go template parsing for blob name formats. If parsing fails, it will not throw an error but will log a warning and continue formatting the blob name using other rules.
  • metrics_format (default 2006/01/02/metrics_15_04_05.json): blob name format. The date format follows constants in Golang, refer here.
  • logs_format (default 2006/01/02/logs_15_04_05.json): blob name format.
  • traces_format (default 2006/01/02/traces_15_04_05.json): blob name format.
  • serial_num_enabled (default true): toggles whether a random serial number is appended to the blob name.
  • serial_num_range (default 10000): a range of random number to be appended after blob_name. e.g. blob_name_{serial_num}. the number will be in [0, serial_num_range).
  • serial_num_before_extension (default false): places the serial number before the file extension if there is one. e.g blob_name_{serial_num}.json instead of blob_name.json_{serial_num}
  • time_parser_enabled (default true): controls whether the exporter interprets the format string as a Go time layout. When false, values such as 2006 or 15_04_05 are treated as literal text.
• format (default json): json or proto. which present otel json or otel protobuf format, the file extension will be json or pb.
• encodings (default using encoding specified in format, which is json): if specified, uses the encoding extension to encode telemetry data. Overrides format.
  • logs (default nil): encoding component id.
  • metrics (default nil): encoding component id.
  • traces (default nil): encoding component id.
• append_blob: configures append blob behavior. When enabled, telemetry data is appended to a single blob instead of creating new blobs. This can be useful for aggregating data or reducing the number of blobs created.
  • enabled (default false): determines whether to use append blob mode.
  • separator (default \n): string to insert between appended data blocks.
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
  • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false

Blob Name Templates

When template_enabled is true, you can use Go templates in metrics_format, logs_format, and traces_format to create dynamic blob names based on telemetry data. The root object for the template is the telemetry data itself (pmetric.Metrics, plog.Logs, or ptrace.Traces).

The following template functions are available:

An example configuration is provided as follows:

extensions:
  zpages:
    endpoint: localhost:55679
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"

exporter:
  azureblob/1:
    url: "https://<your-account>.blob.core.windows.net/"
    container:
      logs: "logs"
      metrics: "metrics"
      traces: "traces"
    blob_name_format:
      template_enabled: true
      metrics_format: `{{ getResourceMetricAttr . 0 "service.name" }}/2006/01/02/metrics.json`
      logs_format: `{{ getScopeLogAttr . 0 0 "scope.name" }}/2006/01/02/logs.json`
      traces_format: `{{ (getSpan . 0 0 0).Name }}/2006/01/02/traces.json`
      serial_num_enabled: true
      time_parser_enabled: true
    auth:
      type: "connection_string"
      connection_string: "DefaultEndpointsProtocol=https;AccountName=<your-acount>;AccountKey=<account-key>;EndpointSuffix=core.windows.net"
    encodings:
      logs: text_encoding
    append_blob:
      enabled: true
      separator: "\n"

Append Blob

When append_blob is enabled:

• The exporter will create append blobs instead of block blobs
• New data will be appended to existing blobs rather than creating new ones
• The configured separator will be inserted between data blocks
• If the blob doesn't exist, it will be created automatically