Skip to main content

Log Forwarding

Using our CDN service for your website or application means that Arvancloud will connect your visitors to your main servers. Therefore, the requests are first sent to Arvancloud; then, in case it is needed, they will be redirected to your main servers.

Visitors’ requests include valuable information and can be accessed in your Arvancloud user panel. Some of this information is:

  • Which of the website’s routes is the request for?
  • What are the user’s browser and operating system?
  • What is the IP address of the user sending a request?
  • What was Arvancloud’s response to the response? (Hit-Miss)

These logs help you analyze your visitor’s behavior and to identify malicious ones such as:

  • Identifying malicious bots
  • Identifying harmful web crawlers
  • Analyzing web server’s behavior
  • Identifying attacks in realtime

Arvancloud Log Forwarding

Log aggregation and processing system is a vital part of Arvancloud CDN; In such a way that different parts of Arvancloud services are directly dependent on this infrastructure service. Arvancloud CDN users can receive the logs of their requests in Near Real Time by making few settings in the Arvancloud panel. By creating a new "Log Forwarder", you can send the logs of your domain's HTTP requests, DNS and WAF services to log management services such as syslog, Loggly, Datadog, as well as Arvancloud and other S3-based storage services.

To activate this feature, first enter the user panel and select the domain you want from the CDN menu, enter the "Monitoring and Reports" section and then "Log Forwarding".

Click on "Create Log Forwarder".

Then choose the type of log you need to receive.

The fields that are sent in JSON format are shown below. You can remove them from the inbox log by disabling each one. For example, the values that can be sent on the HTTP log can be seen in the image:

The following table shows the information of each parameter:

  • WAF Events

    FieldDescription
    ProductCloud security product name
    TimestampTimestamp of log
    Remote AddressClient real IP address
    DomainHTTP request host name
    DataCloud security raw log
  • DNS Requests

    FieldDescription
    TimestampTimestamp of DNS log
    UUIDUnique ID of DNS request
    RecordDNS record of query
    TypeDNS record type in request query
    IPClient real IP address
    CountryClient real country name
    AS NumberClient IP AS number
    Response Status CodeStatus code user received in response
    Process TimeProcess time of request
  • HTTP Requests

    FieldDescription
    MethodHTTP Request Method (GET, POST, ...)
    SchemeHTTP request scheme (HTTP or HTTPS)
    Domain NameHTTP request host header
    RefererHTTP request referrer header
    IP AddressTrue IP address of visitor
    User AgentHTTP request user agent header
    CountryCountry of visitor's IP based on maxmind geoIP database
    AS NumberAS number of visitor's IP
    Content TypeContent type of HTTP request
    Response Status CodeResponse status code for HTTP request
    Server PortServer port that request was sent to
    Bytes SentSum of bytes send to visitor
    Bytes ReceivedByte received from upstream
    Upstream TimeTime of read request from upstream
    Cache StatusCache status of response (HIT, BYPASS, MISS, ...)
    Request IDUnique identifier of HTTP request
    URIURI address in HTTP request
    Query StringQuery string key-values in url
  • Errors

    FieldDescription
    Client IPIP of the client
    Upstream ProtocolThe protocol of the upstream
    Upstream URIThe URI of the upstream
    Upstream PortThe port of the upstream
    Upstream IPThe IP of the upstream
    Domain NameHTTP request host header
    HTTP VersionThe version of the HTTP protocol
    Request MethodHTTP request method (GET, POST, ...)
    Request URIURI Address in the HTTP request
    Real TimestampReal timestamp of the log
    Error messageThe error message
    Pop SiteThe ID of the pop site
    Request IDThe ID of the request
  • Event Logs

    FieldDescription
    Domain NameHTTP Request Host Header
    JA3 FingerprintJA3 Fingerprint hash is a fingerprinting algorithm to identify SSL/TLS client applications based on their handshake characteristics
    TimestampThe timestamp of the original request
    MethodHTTP Request method (GET, POST, ...)
    SchemeHTTP Request Schema (HTTP or HTTPS)
    IP addressVisitor's true IP Address
    CountryCountry of visitor's IP based on MaxMind GeoIP database
    Response Status CodeResponse status code for HTTP request
    ServerIPUpstream Server IP
    Server PortServer port that sent to
    URIURI address in HTTP request
    Query StringQuery String key-values in URI
    FirewallFirewall Detailed Log (Action, BypassFeatures, RuleMatched, KnownBotDetected, HostnameAndTLSMismatch, GlobalAllowlistMatched, GlobalDenylistMatched, UserFirewallIsDisabled)
    ProxyProxy Detailed Log (InvalidHost, CDNLoopBlocked, OperationalEndpointMatched, ReadBody, Redirect, SecureLinkVerified, PageRuleMatched, LoadConfigResult, ErrorPage, ErrorPageStatus)
    DNSDNS Detailed Log (ResolverCacheFirstTimeUpdate, ResolverCacheBackgroundUpdate)
    RateLimitRateLimit Detailed Log (Enabled, UserRlimitIsDisabled, RunningCheckPhase, RunningUpdatePhase, IsNearThreshold, RateLimitRuleMatched, RateLimitAction)
    DDoS ChallengeDDoS Challenge Detailed Log (UserChallengeIsDisabled, IgnoreRuleMatched, ChallengeMode, Action, FinalChallengeDecision)
    WAFWAF Detailed Log (WAFDeclined, WAFLog, WAFStatus, WAFRuleMatched)

In the next step, select the destination where you want to receive your logs:

For example, we use Arvancloud's Object Storage.

In this step, you must first enter a name for your log forwarding profile, then set the flush interval and sample rate according to your needs.

  • The sample rate tells you how many logs will be sent to you. For example, if a sample rate of 50% is set for HTTP requests, out of every 1000 requests, only 500 request logs will be sent to you.

In the settings section, you must enter the access information to the Arvancloud API and finally the buffer value according to your needs.

  • The buffer value determines how much of the log sent to you will be stored in the Object Storage service. For example, suppose you set your buffer value to 100. Now, if request number 101 is logged, first the first request is deleted and replaced by request number 101 to preserve the buffer value, and finally you will have requests from the second to 101 in your log.

Finally, click the save button.

Create a Log Forwarder Using API

  1. HTTP Requests with all parameters on Arvancloud S3
curl --location --request POST 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/log-forwarders' \  --header 'authority: napi.arvancloud.ir' \  --header 'accept: application/json, text/plain, */*' \  --header 'authorization: APIKEY 1 2 3 4' \  --header 'cache-control: no-cache' \  --header 'content-type: application/json;charset=UTF-8' \  --header 'pragma: no-cache' \  --data-raw '{"name":"API Log","description":"","type":"access","connection_type":"arvan_s3","data_format":{"method":true,"scheme":true,"domain":true,"referer":true,"ip":true,"ua":true,"country":true,"asn":true,"content_type":true,"status":true,"server_port":true,"bytes_sent":true,"bytes_received":true,"upstream_time":true,"cache":true,"request_id":true,"uri":true,"query_string":true},"settings":{"sample_rate":"100","s3_endpoint":"https://s3.ir-thr-at1.arvanstorage.ir","access_key":"Arvan_Access_KEY","secret_key":"Arvan_Secret_KEY","bucket_name":"exampleBucket","object_size":"100000","flush_interval":"60"},"status":true}'     
  1. WAF Events with all parameters on Arvancloud S3
curl --location --request POST 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/log-forwarders' \  --header 'authority: napi.arvancloud.ir' \  --header 'accept: application/json, text/plain, */*' \  --header 'authorization: APIKEY 1 2 3 4' \  --header 'cache-control: no-cache' \  --header 'content-type: application/json;charset=UTF-8' \  --header 'dnt: 1' \  --header 'pragma: no-cache' \  --data-raw '{"name":"WAF Events Test","description":"","type":"waf","connection_type":"arvan_s3","data_format":{"product":true,"timestamp":true,"remote_address":true,"domain":true,"data":true},"settings":{"sample_rate":"100","s3_endpoint":"https://s3.ir-thr-at1.arvanstorage.ir","access_key":"Arvan_Access_KEY","secret_key":"Arvan_Secret_KEY","bucket_name":"arvanpro","object_size":"100000","flush_interval":"60"},"status":true}'     


  1. DNS Requests with all parameters on Arvancloud S3
curl --location --request POST 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/log-forwarders' \  --header 'authority: napi.arvancloud.ir' \  --header 'accept: application/json, text/plain, */*' \  --header 'accept-language: fa' \  --header 'authorization: APIKEY 1 2 3 4' \  --header 'cache-control: no-cache' \  --header 'content-type: application/json;charset=UTF-8' \  --header 'dnt: 1' \  --header 'pragma: no-cache' \  --data-raw '{"name":"DNS Request Test","description":"","type":"dns","connection_type":"arvan_s3","data_format":{"timestamp":true,"uuid":true,"record":true,"type":true,"ip":true,"country":true,"asn":true,"response_code":true,"process_time":true},"settings":{"sample_rate":"100","s3_endpoint":"https://s3.ir-thr-at1.arvanstorage.ir  ","access_key":"Arvan_Access_KEY","secret_key":"Arvan_Secret_KEY","bucket_name",object_size":"100000","flush_interval":"60"},"status":true}'  

Main Web Server Configuration

You can use tools such as Logstash and Rsyslog to receive, convert and transfer the log data on your web server dynamically.

Installing Logstash

Logstash is the most well-known tool for log processing through which you can collect and process data. The installation is different for each operating system. This article focuses on installing it on Ubuntu v 16.04. The installation is done by the Apt file manager.

Make sure your Java is updated to v 8 or 11 before installation.

To install Java, use the instructions below:

sudo apt-get update

sudo apt-get install default-jre

You will need an elastic signing key to authenticate the downloaded package (if you have already installed one of the Elastic packages, you can skip this step):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Now you need to define the system repository:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo

tee -a /etc/apt/sources.list.d/elastic-7.x.list

Update the repository and install Logstash:

sudo apt-get update

sudo apt-get install logstash

Configuring Logstash

The configuration has three sections: input, filters, and output. The structure of the Logstash configuration is as follows:

#/etc/logstash/conf.d/

- apache.conf

- haproxy.conf

- syslog.conf

You can create a separate configuration file for each section. Each of these sections has inputs, filters, and outputs.

input {

tcp {
port => 5140
codec => json
}

}

filter {

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:Junk}: %{GREEDYDATA:request}"}
}

json { source => "request" }

}

output {

stdout { codec => rubydebug }

elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "logs-%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}

}

Introducing Rsyslog

This is an open-source and cross-platform tool used for log management. It receives inputs from various sources, converts them, and sends them to different destinations. Rsyslog can deliver more than one million logs to the local destinations each second.