Log Forwarding
Log Forwarding lets you send the logs collected in Arvancloud Cloud Logs to destinations outside Arvancloud's infrastructure. This feature is designed for integrating data with third-party analysis, monitoring, and storage tools. Unlike a log space, which keeps logs in the Arvancloud Cloud Logs infrastructure, a log forwarder sends them to a destination outside this infrastructure.
With Arvancloud log forwarding, you can:
- Run advanced analysis and deep searches in Splunk
- Move logs to internal services or dedicated monitoring tools
- Integrate Cloud Logs with your organization's existing observability ecosystem
The following destinations are currently supported for sending logs:
- Splunk: send data to the Splunk log analysis platform
- Custom HTTP Endpoint: send data to any URL you want over HTTP
- Arvancloud Storage: long-term log archiving in an Arvancloud Object Storage bucket
Creating a Log Forwarder
To create a log destination, go to Log Forwarding in the Cloud Logs menu and click the Create Log Forwarder button.
On the new page, enter the following details:
1. Basic Information
In this section, enter the general details of the destination:
- Name (required): choose a unique, meaningful name to easily identify this forwarder (for example,
Production-logs-to-Splunk). This name can only contain English letters, numbers, and hyphens (-), and can have up to 100 characters. - Description (optional): write a short description of how these logs are used or where they go. Up to 2000 characters are allowed.
Note that after the log forwarder is created, the name and destination type can't be changed; you can only edit the description and destination settings.
2. Choosing the Destination Type
Set the type of destination service that receives the logs:
- Splunk: to send logs to Splunk using the HTTP Event Collector (HEC).
- HTTP: to send logs to any endpoint you want with HTTP requests.
- Arvancloud Storage: for long-term log archiving in an Arvancloud Object Storage bucket.
3. Destination Settings
Depending on the destination type you chose, enter the related information:
-
Splunk settings:
- Destination address: the HEC address of the Splunk service (for example,
https://your-splunk-domain:8088/services/collector). - HEC Token: the dedicated token you got from the Splunk panel.
Note that before doing this, you must enable an HTTP Event Collector in Splunk and create a token with permission to send logs, and the endpoint address must be reachable over HTTPS.
- Destination address: the HEC address of the Splunk service (for example,
-
HTTP Endpoint settings:
- Endpoint URL: the full destination URL for receiving the data.
Note that only HTTPS is accepted, and if you don't enter the protocol,
https://is added to the beginning of the address automatically. - Username and password (optional): used for Basic Authentication. If you fill in one of these two fields, the other also becomes required.
- Headers (optional): you can define up to 10 custom HTTP headers. Duplicate keys aren't allowed in this list.
For example, to specify the source of the logs and add an authentication header, enter the following values:
Key Value arvancloud-logs Custom-Source authorization Bearer YOUR_TOKEN - Endpoint URL: the full destination URL for receiving the data.
-
Arvancloud Storage settings:
- Bucket: select one of the buckets in your Arvancloud Object Storage from the list.
- Gzip compression (optional): if you enable this option, logs are compressed before storage, which reduces file size, storage usage, and data transfer volume. If you don't enable it, logs are stored in the bucket as uncompressed JSON.
- Hive-compatible prefix (optional): if you enable this option, logs are stored with a prefix structure compatible with the Hive, Spark, and Trino analytics tools, so these tools can detect and load new data automatically.
After checking that the information is correct, click the Create button to add the log forwarder to your list.
If you notice after creating the sink that logs don't reach the destination, check the destination settings again. The HTTPS address, HEC token, or bucket access must be correct and available.
Managing and Using Forwarders
-
Managing the list: in the Log Forwarding section, you can see the list of all created destinations and search by log forwarder name.
-
Viewing details: click the info icon next to each log forwarder to open its details panel and see its destination settings (address, headers, bucket settings, and so on). Sensitive information such as the HEC token and password is masked in this panel.
-
Editing: click Edit to change the description and destination settings; the name and destination type are read-only on this page. The editable fields depend on the destination type:
Destination type Editable fields Splunk Description, destination address, HEC token HTTP Description, destination address, basic authentication, headers Arvancloud Storage Description, bucket, Gzip compression, Hive prefix -
Deleting: to delete a log forwarder, click Delete and enter its exact name in the confirmation window. Before deleting, make sure no active sink uses this log forwarder as its destination, because routing the logs of that sink stops.
-
Using in sinks: after creating a log forwarder, to start sending logs, you need to select it as the destination when creating or editing a sink. In the sink creation form, you can also create a new log forwarder without leaving the process and select it as the destination right away.
Note that creating a log forwarder alone isn't enough. If you haven't created and enabled a sink with a log forwarder as its destination, no logs are routed.
The Log Routing guide helps you with this.
To limit forwarding to the logs you want, you can define an ARQL expression in the sink's inclusion step, for example:
logType == "cdn.accessLog" AND severity >= "WARNING"
This rule sends only CDN access logs with a level of WARNING or higher to the log forwarder. When you define an inclusion rule, an exclude-all rule is automatically applied to other logs.
If the sink's inclusion and exclusion rules don't match the logs you want, those logs never reach the log forwarder.
Note that:
- Before finalizing the settings, make sure the destination is available and the entered address is correct.
- If you enter incorrect information (such as a wrong HEC token or URL), log forwarding fails and no data is transferred.
- There is no limit on the number of log forwarders, and you can define different destinations for each sink.
- The external destination must be reachable from the Arvancloud network. If the destination is behind a firewall or in a private network, give it the necessary access.
The cost of storing logs is calculated based on data volume (GB). For exact details, see the Cloud Logs pricing page.
To manage your costs, keep the following in mind:
- Stopping storage costs: disabling a Log Space stops the costs of keeping and storing its data.
- Stopping collection costs: disabling a log space doesn't stop the activity of its sink. The cost of collecting and transferring logs by the sink is still calculated. To stop the process and its costs completely, disable or delete the sink in the Log Routing section.