Log Search and Analysis Using ELK Stack ( Elastic Search, Logstash, and Kibana)

12 min readOct 18, 2023

Elasticsearch, Kibana, Beats, and Logstash are known as the ELK Stack. It allows us to reliably and securely take data from any source, in any format, and then search, analyze, and visualize. We will use them for searching, analyzing, and visualizing the log data of our app in real time. We will learn the ELK Stack in detail in this article. We will install “Elasticsearch, Logstash, and Kibana” using the Docker commands and Docker Compose. We also will run Kibana with a sample Data Set. We will do them practically and step by step.

Topics we will cover:

1. What is The ELK Stack?

1. a. Elasticsearch

1. b. Kibana

1. c. Integrations

1. d. Logstash

1. e. Beats

2. Elasticsearch Platform and Its Solutions

3. Installing Elasticsearch with Docker

4. Installing Kibana with Docker and Running with Sample Data Set

5. Installing and Configuring Logstash

6. Installing “Elasticsearch, Logstash, and Kibana” Using Docker Compose

7. As a result

8. Next post

9. References

If you like the article, I will be happy if you click the Medium Follow button to encourage me to write more, and not miss future articles.

Your clap, follow, or subscribe, they help my articles to reach the broader audience. Thank you in advance for them.

System, security, network, and applications (with logging configured) constantly produce logs. The log is the recording of actions in the digital environment. For example, the service started, stopped, crashed, etc. The log is generated for each of these. Log files grow in size, and logs continue to be produced line by line. In case of a problem, it may take time to find and detect the relevant log. This is exactly where, The ELK stack helps us collect logs from all our systems and applications, analyze these logs, monitor our application and infrastructure, troubleshoot faster, and perform security analysis. In this article, we will use them for searching, analyzing, and visualizing the log data of our app in real-time.

1. What is The ELK Stack?

What exactly is the Elastic Stack? It’s a fast and highly scalable set of components “Elasticsearch, Kibana, Beats, Logstash, and others” that together enable you to securely take data from any source, in any format, and then search, analyze, and visualize it. (1)

1. a. Elasticsearch

Elasticsearch is a search and analytics engine at the heart of the Elastic Stack. Logstash and Beats facilitate collecting, aggregating, and enriching your data and storing it in Elasticsearch. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Elasticsearch is where the indexing, search, and analysis happen, as shown in Figure 1. Elasticsearch provides near real-time search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data, Elasticsearch can efficiently store and index it in a way that supports fast searches. (2)

1. b. Kibana

Kibana is a window into the Elastic Stack and the user interface for the Elastic Search Platform. It allows you to visualize and explore data as well as manage and monitor the entire Elastic Stack, as shown in Figure 1. (3)

1. c. Integrations

Integrations enable you to collect and connect your data with the Elastic Stack. You can ship metrics, logs, traces, content, and events from your apps and infrastructure with Logstash, Elastic Agent, web crawler, connectors, API, and Cloud provider integrations. Thus, Elastic gives you the flexibility to collect, store, and visualize any data from any source. Let’s take a brief look at these now; (4)

Logstash
If you want to collect and transform your data, check out the data processing pipeline that can ingest data regardless of format with Logstash.

Elastic Agent
If you’d like to centrally manage data ingestion with easy one-click integrations and scalability, check out the Elastic Agent, the preferred ingest method for observability and security use cases.

Web crawler
If you want to make external or even authenticated web content searchable, use the native Elastic web crawler to index your website and create a modernized search experience.

Data connectors
If you’d like to create a unified search experience across your SaaS productivity and collaboration tools, use content source connectors for Enterprise Search.

API
If you’d prefer to use APIs to ingest your data check out the Document APIs as well as the Ingest APIs for ingest pipelines and processors.

Cloud provider integrations

You can send data to Elastic Cloud with native integrations on Amazon Web Services (AWS), Microsoft Azure, and Google Cloud.

1. d. Logstash

Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash” like ElasticSearch. Offering pre-built filters and support for over 200 plugins, Logstash allows us to easily retrieve data regardless of data source or type. You can mix, match, and orchestrate different inputs, filters, and outputs to work in pipeline harmony. (5) Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. Click on the link to see the
Input plugins list.

Logstash’s working logic is shown in Figure 2.

Inputs: Easily ingest from your logs, metrics, web applications, data stores, and various AWS services, all in a continuous, streaming fashion.

Filters: As data travels from source to store, Logstash filters parse each event, identify named fields to build structure, and transform them to converge on a common format for more powerful analysis and business value. Logstash dynamically transforms and prepares your data regardless of format or complexity.

Decipher geo coordinates from IP addresses
Anonymize PII data, exclude sensitive fields completely
Ease overall processing, independent of the data source, format, or schema.

Outputs: Choose your stash, and transport your data. Logstash has a variety of outputs that let you route data where you want, giving you the flexibility to unlock a slew of downstream use cases.

1. e. Beats

Beats is a free and open platform for single-purpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch. (6)

The Beats family consists of;

Filebeat: Lightweight shipper for logs and other data. Whether you’re collecting from security devices, cloud, containers, hosts, or OT, Filebeat helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files. (7)

Metricbeat: Lightweight shipper for metric data. Collect metrics from your systems and services. From CPU to memory, Redis to NGINX, and much more, Metricbeat is a lightweight way to send system and service statistics. (8)

Packetbeat: Lightweight shipper for network data.

Winlogbeat: Lightweight shipper for Windows event logs.

Auditbeat: Lightweight shipper for audit data.

Heartbeat: Lightweight shipper for uptime monitoring. You can automate anomaly detection and accelerate root cause analysis with AIOps. (9)

Filebeat and Metricbeat include modules that simplify collecting, parsing, and visualizing information from key data sources such as cloud platforms, containers and systems, and network technologies. Run a single command and explore away. (6)

2. Elasticsearch Platform and its Solutions

Also, Elastic provides flexible search, monitoring, and security solutions below; (10)

Search

You can build an application on top of Elasticsearch with Elastic Enterprise Search. Elastic Enterprise Search enables developers and teams to build search-powered applications using the Elastic search platform. You can build a search for ecommerce, customer support, workplace content, websites, or any other application using a curated collection of tools.

Observability

Monitor hosts by collecting logs and metrics with Elastic Observability, as shown in Figure 3.

Security

You can detect, investigate, and respond to threats with Elastic Security for SIEM. Elastic Security combines Elastic SIEM, whose detection engine automates threat detection so you can quickly investigate and respond to threats, and Endpoint Security into a single solution that unifies prevention, detection, and response across your entire network, as shown in Figure 4.

Also, you can protect hosts with endpoint threat intelligence from Elastic Security.

Observe Kubernetes

Monitor your Kubernetes infrastructure with Elastic Observability, as shown in Figure 5.

Custom solution

You can deploy your own platform to store, search, and visualize any data, as shown in Figure 6.

3. Installing Elasticsearch with Docker

Prerequisite: You must install Docker. If using Docker Desktop, make sure to allocate at least 4GB of memory. You can adjust memory usage in Docker Desktop by going to Settings > Resources. I used Docker Desktop in this article’s hands-on.

We will run Elasticsearch locally. (11)

Create a new docker network, Pull the Elasticsearch Docker image, and Start an Elasticsearch container, as shown in Figure 7.

docker network create elastic
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.10.4
docker run --name elasticsearch --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -t docker.elastic.co/elasticsearch/elasticsearch:8.10.4

Note: If you want, you can use the "-m" flag to set a memory limit for the container. This removes the need to manually set the JVM size.
docker run --name es01 --net elastic -p 9200:9200 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:8.10.4

Note: When you start Elasticsearch for the first time, the generated elastic user password and Kibana enrollment token are output to the terminal, as shown in Figure 8. Copy the generated password and enrollment token and save them in a secure location. These values are shown only when you start Elasticsearch for the first time. You’ll use these to enroll Kibana with your Elasticsearch cluster and log in.

4. Installing Kibana with Docker and Running with Sample Data Set

In a new terminal session, start Kibana and connect it to your Elasticsearch container, as shown in Figure 9.

docker pull docker.elastic.co/kibana/kibana:8.10.4
docker run --name kibana --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.10.4

To access Kibana, open the generated URL in your browser. Paste the enrollment token (see Figure 8) that you copied when starting Elasticsearch and click the button to connect your Kibana instance with Elasticsearch, as shown in Figures 11–12.

Note: If you have not saved the token, you can obtain it again, as shown in Figure 13.

Kibana begins to complete its installation, as shown in Figure 14.

Log in to Kibana as the elastic user with the password that was generated when you started Elasticsearch, as shown in Figure 15.

Click on the “Add integration” button to integrate into a data set, as shown in Figure 16.

Logstash does not currently appear in the integration section, as shown in Figure 17.

Select the “Try sample data” to run with a Sample Data Set, as shown in Figure 18.

Select the “Start exploring”, as shown in Figure 19.

Kibana connected to the sample data and started to show us the logs visually, as shown in Figures 20–21.

5. Installing and Configuring Logstash

Obtaining Logstash for Docker is as simple as issuing a docker pull command against the Elastic Docker registry, as shown in Figure 22. (12)

docker pull docker.elastic.co/logstash/logstash:8.10.4

Note: Alternatively, you can download other Docker images that contain only features available under the Apache 2.0 license. To download the images, go to www.docker.elastic.co.

Configuring Logstash for Docker:

Logstash differentiates between two types of configuration: Settings and Pipeline Configuration, as shown in Figure 18. (13)

Pipeline Configuration: It is essential to place your pipeline configuration where it can be found by Logstash, as shown in Figure 23. By default, the container will look in /usr/share/logstash/pipeline/ for pipeline configuration files.

In this example, we use a bind-mounted volume to provide the configuration via the docker run command, as shown in Figure 24. (14)

docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:8.10.4

Settings: The image provides several methods for configuring settings. The conventional approach is to provide a custom logstash.yml file, but it’s also possible to use environment variables to define settings. Settings files can also be provided through bind-mounts. Logstash expects to find them at /usr/share/logstash/config/. It’s possible to provide an entire directory containing all needed files;

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg

Note -1: We can install the Logstash from Package Repositories into the Ubuntu server by using the following Commands, as shown in Figure 25.

# Download and install the Public Signing Key:
docker run --rm -it -v ~/settings/:/usr/share/logstash/config/ docker.elastic.co/logstash/logstash:8.10.4

# You may need to install the apt-transport-https package on Debian before proceeding:
sudo apt-get install apt-transport-https

# Save the repository definition to /etc/apt/sources.list.d/elastic-8.x.list:
echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

# Run sudo apt-get update and the repository is ready for use. You can install it with:
sudo apt-get update && sudo apt-get install logstash

When the installation process is completed, we can see “Elastic Search, Logstash, and Kibana” in Dockers Desktop, as shown in Figure 26.

Similarly, Longstash will now appear in the integration menu, as shown in Figure 27.

6. Installing “Elasticsearch, Logstash, and Kibana” Using Docker Compose.

Installing ELK is an easy process using Docker Compose. You can use the following “yaml file” for Docker compose. (15)

version: '3.7'
services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.6.2
    environment:
      - cluster.name=docker-cluster
      - network.host=0.0.0.0
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"  
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    networks:
      - elastic  

  logstash:
    image: docker.elastic.co/logstash/logstash:8.6.2 
    depends_on:
      - elasticsearch
    ports:
      - "5000:5000"
      - "9600:9600"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    volumes:
      - logstash_config:/etc/logstash/conf.d/
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf   
    networks:
      - elastic

  kibana:
    image: docker.elastic.co/kibana/kibana:8.6.2
    environment:
      - ELASTICSEARCH_URL=https://elasticsearch:9200
      - server.name=kibana
      - server.host=0.0.0.0      
    ports:
      - "5601:5601"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    networks:
      - elastic
    depends_on:
      - elasticsearch

volumes:
  esdata1:
    driver: local
  logstash_config:
    driver: local

networks:
  elastic:
    driver: bridge

Now, we can run the command below to start the installation;

docker-compose up

If you want it to run in the background, you can run the command below.

docker-compose up -d

Now we can check ELK’s runnig by going to “http://server_ip:5601” from our browser.

7. As a result

We learned what ELK consists of and its working logic. We installed “Elasticsearch, Logstash, and Kibana” using the Docker commands and Docker Compose. We also ran Kibana with a sample Data Set.

If you liked the article, I would be happy if you click on the Medium Following button to encourage me to write and not miss future articles.

Your clap, follow, or subscribe, they help my articles to reach the broader audience. Thank you in advance for them.

For more info and questions, please contact me on Linkedin or Medium.

8. Next post

In the next post, “Monitoring System Logs Using Logstash in Kibana”. By completing Logstash’s configuration, we will connect to a Log Data Set via the pipeline and visually observe the results in Kibana. We will do all the procedures step by step.

Happy Clouding…

Don’t forget to follow my Medium account to be informed about new articles.

9. References

Log Search and Analysis Using ELK Stack ( Elastic Search, Logstash, and Kibana)

1. What is The ELK Stack?

1. a. Elasticsearch

1. b. Kibana

1. c. Integrations

1. d. Logstash

1. e. Beats

2. Elasticsearch Platform and its Solutions

Search

Observability

Security

Observe Kubernetes

Custom solution

3. Installing Elasticsearch with Docker

4. Installing Kibana with Docker and Running with Sample Data Set

5. Installing and Configuring Logstash

6. Installing “Elasticsearch, Logstash, and Kibana” Using Docker Compose.

7. As a result

8. Next post

9. References

Written by Cumhur Akkaya