Log Forwarding
Using our CDN service for your website or application means that Arvancloud will connect your visitors to your main servers. Therefore, the requests are first sent to Arvancloud; then, in case it is needed, they will be redirected to your main servers.
Visitors’ requests include valuable information and can be accessed in your Arvancloud user panel. Some of this information is:
- Which of the website’s routes is the request for?
- What are the user’s browser and operating system?
- What is the IP address of the user sending a request?
- What was Arvancloud’s response to the response? (Hit-Miss)
These logs help you analyze your visitor’s behavior and to identify malicious ones such as:
- Identifying malicious bots
- Identifying harmful web crawlers
- Analyzing web server’s behavior
- Identifying attacks in realtime
Arvancloud Log Forwarding
Log aggregation and processing system is a vital part of Arvancloud CDN; In such a way that different parts of Arvancloud services are directly dependent on this infrastructure service. Arvancloud CDN users can receive the logs of their requests in Near Real Time by making few settings in the Arvancloud panel. By creating a new "Log Forwarder", you can send the logs of your domain's HTTP requests, DNS and WAF services to log management services such as syslog, Loggly, Datadog, as well as Arvancloud and other S3-based storage services.
To activate this feature, first enter the user panel and select the domain you want from the CDN menu, enter the "Monitoring and Reports" section and then "Log Forwarding".
Click on "Create Log Forwarder".
Then choose the type of log you need to receive.
The fields that are sent in JSON format are shown below. You can remove them from the inbox log by disabling each one. For example, the values that can be sent on the HTTP log can be seen in the image:
The following table shows the information of each parameter:
WAF Events
Field Description Product Cloud security product name Timestamp Timestamp of log Remote Address Client real IP address Domain HTTP request host name Data Cloud security raw log DNS Requests
Field Description Timestamp Timestamp of DNS log UUID Unique ID of DNS request Record DNS record of query Type DNS record type in request query IP Client real IP address Country Client real country name AS Number Client IP AS number Response Status Code Status code user received in response Process Time Process time of request HTTP Requests
Field Description Method HTTP Request Method (GET, POST, ...) Scheme HTTP request scheme (HTTP or HTTPS) Domain Name HTTP request host header Referer HTTP request referrer header IP Address True IP address of visitor User Agent HTTP request user agent header Country Country of visitor's IP based on maxmind geoIP database AS Number AS number of visitor's IP Content Type Content type of HTTP request Response Status Code Response status code for HTTP request Server Port Server port that request was sent to Bytes Sent Sum of bytes send to visitor Bytes Received Byte received from upstream Upstream Time Time of read request from upstream Cache Status Cache status of response (HIT, BYPASS, MISS, ...) Request ID Unique identifier of HTTP request URI URI address in HTTP request Query String Query string key-values in url
Errors
Field Description Client IP IP of the client Upstream Protocol The protocol of the upstream Upstream URI The URI of the upstream Upstream Port The port of the upstream Upstream IP The IP of the upstream Domain Name HTTP request host header HTTP Version The version of the HTTP protocol Request Method HTTP request method (GET, POST, ...) Request URI URI Address in the HTTP request Real Timestamp Real timestamp of the log Error message The error message Pop Site The ID of the pop site Request ID The ID of the request Event Logs
Field Description Domain Name HTTP Request Host Header JA3 Fingerprint JA3 Fingerprint hash is a fingerprinting algorithm to identify SSL/TLS client applications based on their handshake characteristics Timestamp The timestamp of the original request Method HTTP Request method (GET, POST, ...) Scheme HTTP Request Schema (HTTP or HTTPS) IP address Visitor's true IP Address Country Country of visitor's IP based on MaxMind GeoIP database Response Status Code Response status code for HTTP request ServerIP Upstream Server IP Server Port Server port that sent to URI URI address in HTTP request Query String Query String key-values in URI Firewall Firewall Detailed Log (Action, BypassFeatures, RuleMatched, KnownBotDetected, HostnameAndTLSMismatch, GlobalAllowlistMatched, GlobalDenylistMatched, UserFirewallIsDisabled) Proxy Proxy Detailed Log (InvalidHost, CDNLoopBlocked, OperationalEndpointMatched, ReadBody, Redirect, SecureLinkVerified, PageRuleMatched, LoadConfigResult, ErrorPage, ErrorPageStatus) DNS DNS Detailed Log (ResolverCacheFirstTimeUpdate, ResolverCacheBackgroundUpdate) RateLimit RateLimit Detailed Log (Enabled, UserRlimitIsDisabled, RunningCheckPhase, RunningUpdatePhase, IsNearThreshold, RateLimitRuleMatched, RateLimitAction) DDoS Challenge DDoS Challenge Detailed Log (UserChallengeIsDisabled, IgnoreRuleMatched, ChallengeMode, Action, FinalChallengeDecision) WAF WAF Detailed Log (WAFDeclined, WAFLog, WAFStatus, WAFRuleMatched)
In the next step, select the destination where you want to receive your logs:
For example, we use Arvancloud's Object Storage.
In this step, you must first enter a name for your log forwarding profile, then set the flush interval and sample rate according to your needs.
- The sample rate tells you how many logs will be sent to you. For example, if a sample rate of 50% is set for HTTP requests, out of every 1000 requests, only 500 request logs will be sent to you.
In the settings section, you must enter the access information to the Arvancloud API and finally the buffer value according to your needs.
- The buffer value determines how much of the log sent to you will be stored in the Object Storage service. For example, suppose you set your buffer value to 100. Now, if request number 101 is logged, first the first request is deleted and replaced by request number 101 to preserve the buffer value, and finally you will have requests from the second to 101 in your log.
Finally, click the save button.
Create a Log Forwarder Using API
- HTTP Requests with all parameters on Arvancloud S3
curl --location --request POST 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/log-forwarders' \ --header 'authority: napi.arvancloud.ir' \ --header 'accept: application/json, text/plain, */*' \ --header 'authorization: APIKEY 1 2 3 4' \ --header 'cache-control: no-cache' \ --header 'content-type: application/json;charset=UTF-8' \ --header 'pragma: no-cache' \ --data-raw '{"name":"API Log","description":"","type":"access","connection_type":"arvan_s3","data_format":{"method":true,"scheme":true,"domain":true,"referer":true,"ip":true,"ua":true,"country":true,"asn":true,"content_type":true,"status":true,"server_port":true,"bytes_sent":true,"bytes_received":true,"upstream_time":true,"cache":true,"request_id":true,"uri":true,"query_string":true},"settings":{"sample_rate":"100","s3_endpoint":"https://s3.ir-thr-at1.arvanstorage.ir","access_key":"Arvan_Access_KEY","secret_key":"Arvan_Secret_KEY","bucket_name":"exampleBucket","object_size":"100000","flush_interval":"60"},"status":true}'
- WAF Events with all parameters on Arvancloud S3
curl --location --request POST 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/log-forwarders' \ --header 'authority: napi.arvancloud.ir' \ --header 'accept: application/json, text/plain, */*' \ --header 'authorization: APIKEY 1 2 3 4' \ --header 'cache-control: no-cache' \ --header 'content-type: application/json;charset=UTF-8' \ --header 'dnt: 1' \ --header 'pragma: no-cache' \ --data-raw '{"name":"WAF Events Test","description":"","type":"waf","connection_type":"arvan_s3","data_format":{"product":true,"timestamp":true,"remote_address":true,"domain":true,"data":true},"settings":{"sample_rate":"100","s3_endpoint":"https://s3.ir-thr-at1.arvanstorage.ir","access_key":"Arvan_Access_KEY","secret_key":"Arvan_Secret_KEY","bucket_name":"arvanpro","object_size":"100000","flush_interval":"60"},"status":true}'
- DNS Requests with all parameters on Arvancloud S3
curl --location --request POST 'https://napi.arvancloud.ir/cdn/4.0/domains/example.com/log-forwarders' \ --header 'authority: napi.arvancloud.ir' \ --header 'accept: application/json, text/plain, */*' \ --header 'accept-language: fa' \ --header 'authorization: APIKEY 1 2 3 4' \ --header 'cache-control: no-cache' \ --header 'content-type: application/json;charset=UTF-8' \ --header 'dnt: 1' \ --header 'pragma: no-cache' \ --data-raw '{"name":"DNS Request Test","description":"","type":"dns","connection_type":"arvan_s3","data_format":{"timestamp":true,"uuid":true,"record":true,"type":true,"ip":true,"country":true,"asn":true,"response_code":true,"process_time":true},"settings":{"sample_rate":"100","s3_endpoint":"https://s3.ir-thr-at1.arvanstorage.ir ","access_key":"Arvan_Access_KEY","secret_key":"Arvan_Secret_KEY","bucket_name",object_size":"100000","flush_interval":"60"},"status":true}'
Main Web Server Configuration
You can use tools such as Logstash and Rsyslog to receive, convert and transfer the log data on your web server dynamically.
Installing Logstash
Logstash is the most well-known tool for log processing through which you can collect and process data. The installation is different for each operating system. This article focuses on installing it on Ubuntu v 16.04. The installation is done by the Apt file manager.
Make sure your Java is updated to v 8 or 11 before installation.
To install Java, use the instructions below:
sudo apt-get update
sudo apt-get install default-jre
You will need an elastic signing key to authenticate the downloaded package (if you have already installed one of the Elastic packages, you can skip this step):
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Now you need to define the system repository:
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo
tee -a /etc/apt/sources.list.d/elastic-7.x.list
Update the repository and install Logstash:
sudo apt-get update
sudo apt-get install logstash
Configuring Logstash
The configuration has three sections: input, filters, and output. The structure of the Logstash configuration is as follows:
#/etc/logstash/conf.d/
- apache.conf
- haproxy.conf
- syslog.conf
You can create a separate configuration file for each section. Each of these sections has inputs, filters, and outputs.
input {
tcp {
port => 5140
codec => json
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:Junk}: %{GREEDYDATA:request}"}
}
json { source => "request" }
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "logs-%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
Introducing Rsyslog
This is an open-source and cross-platform tool used for log management. It receives inputs from various sources, converts them, and sends them to different destinations. Rsyslog can deliver more than one million logs to the local destinations each second.