This little blog got quite a few DDoS attacks in the past months. Most of them were Layer7 so I decided to gather some statistics.
Tool of choice was Grafana. HAProxy has native syslog capabilites. Easiest choice was to forward syslog, process+enrich and build a dashboard in Grafana. To store syslog I checked out promtail+loki.
Since I failed quite badly at parsing HAproxy “httplog” log format using regex see below my config-snippets using logfmt log_format in HAProxy for the export and native logfmt support in loki.
HAProxy config (makes most sense for frontend, only works for http frontends (not TCP or UDP).
1
2
3
|
- request header User-Agent len 400
- request header Host len 150
log_format: "client_ip=%ci client_port=%cp date_time=%tr frontend_name=%f backend_name=%b server_name=%s performance_metrics=%TR/%Tw/%Tc/%Tr/%Ta status_code=%ST bytes_read=%B termination_state=%tsc haproxy_metrics=%ac/%fc/%bc/%sc/%rc srv_queue=%sq backend_queue=%bq user_agent=%{+Q}[capture.req.hdr(0)] http_hostname=%{+Q}[capture.req.hdr(1)] http_version=%HV http_method=%HM http_request_uri=%HU http_request=%{+Q}r"
|
Promtail
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
|
promtail_config_server:
http_listen_port: 9080
grpc_listen_port: 9081
promtail_config_positions:
filename: "{{ promtail_positions_directory }}/positions.yaml"
sync_period: "60s"
promtail_config_clients:
- url: "http://localhost:3100/loki/api/v1/push"
promtail_config_scrape_configs:
- job_name: rsyslog
syslog:
listen_address: 0.0.0.0:8514
labels:
job: syslog
relabel_configs:
- source_labels: [__syslog_message_hostname]
target_label: host
- source_labels: [__syslog_message_hostname]
target_label: hostname
- source_labels: [__syslog_message_severity]
target_label: level
- source_labels: [__syslog_message_app_name]
target_label: application
- source_labels: [__syslog_message_facility]
target_label: facility
- source_labels: [__syslog_connection_hostname]
target_label: connection_hostname
pipeline_stages:
- logfmt:
mapping:
client_ip: client_ip
- geoip:
db: "/etc/maxmind/GeoLite2-ASN.mmdb"
source: client_ip
db_type: "asn"
- geoip:
db: "/etc/maxmind/GeoLite2-City.mmdb"
source: client_ip
db_type: "city"
|
Loki:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
auth_enabled: false
server:
http_listen_port: 3100
common:
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
replication_factor: 1
path_prefix: /loki
schema_config:
configs:
- from: 2020-05-15
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
|
Now add loki as a datasource to your grafana installation and use a parsing filter like below
1
|
{hostname="$hostname"} |= `$filter` | logfmt | __error__= ""
|
Also check out the possible HAProxy log codes if you want to gather additional / different metrics. Source: haproxy.com
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
|
+---+------+-----------------------------------------------+-------------+
| R | var | field name (8.2.2 and 8.2.3 for description) | type |
+---+------+-----------------------------------------------+-------------+
| | %o | special variable, apply flags on all next var | |
+---+------+-----------------------------------------------+-------------+
| | %B | bytes_read (from server to client) | numeric |
| H | %CC | captured_request_cookie | string |
| H | %CS | captured_response_cookie | string |
| | %H | hostname | string |
| H | %HM | HTTP method (ex: POST) | string |
| H | %HP | HTTP request URI without query string | string |
| H | %HPO | HTTP path only (without host nor query string)| string |
| H | %HQ | HTTP request URI query string (ex: ?bar=baz) | string |
| H | %HU | HTTP request URI (ex: /foo?bar=baz) | string |
| H | %HV | HTTP version (ex: HTTP/1.0) | string |
| | %ID | unique-id | string |
| | %ST | status_code | numeric |
| | %T | gmt_date_time | date |
| H | %Ta | Active time of the request (from TR to end) | numeric |
| | %Tc | Tc | numeric |
| | %Td | Td = Tt - (Tq + Tw + Tc + Tr) | numeric |
| | %Tl | local_date_time | date |
| | %Th | connection handshake time (SSL, PROXY proto) | numeric |
| H | %Ti | idle time before the HTTP request | numeric |
| H | %Tq | Th + Ti + TR | numeric |
| H | %TR | time to receive the full request from 1st byte| numeric |
| H | %Tr | Tr (response time) | numeric |
| | %Ts | timestamp | numeric |
| | %Tt | Tt | numeric |
| | %Tu | Tu | numeric |
| | %Tw | Tw | numeric |
| | %U | bytes_uploaded (from client to server) | numeric |
| | %ac | actconn | numeric |
| | %b | backend_name | string |
| | %bc | beconn (backend concurrent connections) | numeric |
| | %bi | backend_source_ip (connecting address) | IP |
| | %bp | backend_source_port (connecting address) | numeric |
| | %bq | backend_queue | numeric |
| | %ci | client_ip (accepted address) | IP |
| | %cp | client_port (accepted address) | numeric |
| | %f | frontend_name | string |
| | %fc | feconn (frontend concurrent connections) | numeric |
| | %fi | frontend_ip (accepting address) | IP |
| | %fp | frontend_port (accepting address) | numeric |
| | %ft | frontend_name_transport ('~' suffix for SSL) | string |
| | %lc | frontend_log_counter | numeric |
| | %hr | captured_request_headers default style | string |
| | %hrl | captured_request_headers CLF style | string list |
| | %hs | captured_response_headers default style | string |
| | %hsl | captured_response_headers CLF style | string list |
| | %ms | accept date milliseconds (left-padded with 0) | numeric |
| | %pid | PID | numeric |
| H | %r | http_request | string |
| | %rc | retries | numeric |
| | %rt | request_counter (HTTP req or TCP session) | numeric |
| | %s | server_name | string |
| | %sc | srv_conn (server concurrent connections) | numeric |
| | %si | server_IP (target address) | IP |
| | %sp | server_port (target address) | numeric |
| | %sq | srv_queue | numeric |
| S | %sslc| ssl_ciphers (ex: AES-SHA) | string |
| S | %sslv| ssl_version (ex: TLSv1) | string |
| | %t | date_time (with millisecond resolution) | date |
| H | %tr | date_time of HTTP request | date |
| H | %trg | gmt_date_time of start of HTTP request | date |
| H | %trl | local_date_time of start of HTTP request | date |
| | %ts | termination_state | string |
| H | %tsc | termination_state with cookie status | string |
+---+------+-----------------------------------------------+-------------+
|