Archy's Blog

So if you are saving data in elastic search and this data contains IP Addresses, you might want to consider enriching your data with GeoIP. I'll take my router logging data (see part 2) and enrich them with GeoIP data. First of all, let's adjust the Index Template and add the GeoIP Longitude, Latitude and Location fields. I'll push the index template using the dev tools in Kibana. PUT _template/mikrotik-log { "index_patterns" : [ "mikrotik-log-*" ], "settings" : { "index" : { "codec" : "best_compression", "refresh_interval" : "5s", "number_of_shards" : "1", "number_of_replicas" : "1" } }, "mappings" : { "numeric_detection" : true, "dynamic_templates" : [ { "string_fields" : { "mapping" :...

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana. Start by getting the Log Data you want to structure parsed correctly. Mikrotik Logs are a bit difficult since they show you Data in the interface which is already enriched with Time / Date. That means a message that the remote logging will send to Logstash will look like this: firewall,info forward: in:lan out:wan, src-mac aa:bb:cc:dd:ee:ff, proto UDP, 172.31.100.154:57061->109.164.113.231:443, len 76 You can check them in the grok debugger and create your own filters and mapping. The following is my example which might not fit your needs. Here are some custom patterns I wrote for my pattern matching: MIKROTIK_DATE \b(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?...

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana and an installation of Filebeat on the Host(s) where you get your nginx logs from. Start by getting the Log Data you want to structure parsed correctly. The nginx logs are pretty straight forward, so after checking them out in the grok debugger, I'll have the following structure mapped: %{IP:ClientIP} - %{DATA:username} \[%{NGINXTIMESTAMP:timestamp}%{GREEDYDATA}\] \"%{WORD:method} %{DATA:request_uri} %{DATA:http-version}\" %{RETURNCODE:http_return_code} %{GREEDYDATA} \"%{DATA:server_name}\" \"%{GREEDYDATA}\" Also, I've written some custom patterns: NGINXTIMESTAMP (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\/\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|rua...

Elasticsearch along with Logstash and Kibana is a great combination for aggregating and enriching log files, splitting them into different fields and visualizing them. For this setup, I will set up a 3 Node Cluster with every node operating every role. The recommended Setup would be separating master-eligible nodes and data nodes, as well as ingest nodes (depending on your workload).  Quick Note on the software components: Elasticsearch: Elasticsearch is a distributed search backend using the lucene engine for searching its shards and saved data. The Data indexed into this will be in the json-format. Logstash: Logstash takes your data, passes it through the grok filters you wrote, enriches them if you've configured to do so and indexes them into elasticsearch. Kibana: With Kibana you can control and manage your cluster as well as configure pipelines. Kibana will also give you a fancy frontend to search your data and build graphs. Since I'm only going to go...