R136A1
ELK Stack ํ๊ฒฝ ๊ตฌ์ฑ ๋ณธ๋ฌธ
์คํ์์ค ํ๋ก์ ํธ ElasticSearch + LogStash + Kibana์ ์ ๊ธ์๋ง ๋ฐ์ ๋ง๋ค์ด์ง ์ฉ์ด
LogStash: ๋ฐ์ดํฐ๋ฅผ ์ฒ๋ฆฌํ๋ ํ์ดํ๋ผ์ธ, ๋ก๊ทธ๋ฅผ ์์งํ์ฌ ElasticSearch์ ์ ์ก
ElasticSearch: LogStash๋ฅผ ํตํด์ ์ ์ก๋ฐ์ ๋ฐ์ดํฐ ๋ถ์ ๋ฐ ์ ์ฅ
Kibana: ElasticSearch์ ์ ์ฅ๋์ด ์๋ ๋ฐ์ดํฐ๋ฅผ ์๊ฐํํ๊ณ , ์ค์๊ฐ์ผ๋ก ๋ถ์
์์ ELK์์ Beat๊น์ง ์ถ๊ฐ๋๋ฉด์ ELK Stack์ด๋ผ๊ณ ๋ถ๋ฆผ
Beat: ๋์ ์๋ฒ์์ ๋ฐ์ดํฐ๋ฅผ ์์งํ๋ ์ญํ ๋ด๋น
https://potato-yong.tistory.com/140
๊ณต์ ์ค์น ๋ฐฉ๋ฒ https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
์ค์น ๋ฐฉ๋ฒ - OS์ ์ข ๋ฅ์ ๋ฐ๋ผ ๋๋์ด์ ธ์์
1) Linux, MacOS => tar.gz
2) Windows => .zip
3) Debian-based System(Debian, Ubuntu) => deb
4) RPM-based System(RedHat, CentOS, SLES, OpenSuSE) => rpm
+) Docker
Support Matrix๋ฅผ ํตํ OS ์ ์ https://www.elastic.co/kr/support/matrix
=> ๊ฐ์ฅ ํธํ์ฑ์ด ์ข์ CentOS 7 ์ ํ
๊ณต์ ๋ค์ด๋ก๋ ๋งํฌ http://isoredirect.centos.org/centos/7/isos/x86_64/
์ฌ๋ฌ๊ฐ์ง ๋ฒ์ ์กด์ฌ
Minimal - ๋ฆฌ๋ ์ค ์์คํ ์ ์๊ตฌ๋๋ ์ต์ ํจํค์ง. (GUI ๋ฏธํฌํจ)
DVD - Minimal ํจํค์ง๋ฅผ ํฌํจํ๊ณ ๋ช๊ฐ์ ์ ํธ๋ฆฌํฐ ํจํค์ง๋ฅผ ํฌํจํจ.(๊ธฐ๋ณธ ๊ฐ๋ฐํจํค์ง ๋ฐ GUI ํจํค์ง)
Everything - DVD ํจํค์ง๋ฅผ ํฌํจํ๊ณ ํฅ์๋ ์ ์ ํจํค์ง, ๊ฐ๋ฐ ํจํค์ง ์ ๋ชจ๋ ํฌํจ.(Smart card support ๋ฑ)
Netinstall - ๋คํธ์ํฌ ์ค์น๋ฅผ ์ํ ์ต์์ CD ์ด๋ฏธ์ง.
๋ฐํ์ ๊ตฌ์ฑํ ๋๋ DVD๋ก ํ๊ณ ๋ผ์ฆ๋ฒ ๋ฆฌํ์ด์ ๊ตฌ์ฑํ ๋ ๋ฌด๊ฑฐ์ด ๊ฒ ๊ฐ์ผ๋ฉด Minimal ํ๋ฉด ๋ ๋ฏ
ElasticSearch
CentOS ๊ณต์ ElasticSearch ์ค์น ๋ฐฉ๋ฒ https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
์ค๊ฐ์ sudo ๋ช ๋ น์ด ์๋ผ์ ๊ณ ์นจ
๋ฃจํธ ๊ณ์ ์ผ๋ก ์ ํ su - root
echo '๊ณ์ ๋ช ALL=(ALL) ALL' >> /etc/sudoers ์ ๋ ฅ
๋ค์ su ๊ณ์ ๋ช ์ผ๋ก ์ ํํ์ฌ sudo ์ฌ์ฉํ๋ฉด ์ ๋จ
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# /etc/yum.repos.d/ ๋๋ ํ ๋ฆฌ์ elasticsearch.repo ์์ฑ
$ sudo vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
centOS์ด๋ฏ๋ก yum ์ ํ
sudo yum install --enablerepo=elasticsearch elasticsearch
sudo vim /etc/elasticsearch/elasticsearch.yml
network.host: 192.168.0.1 ์ 127.0.0.1๋ก ๋ณ๊ฒฝ & ์ฃผ์ ์ ๊ฑฐ
http.port:9200 ์ฃผ์ ์ ๊ฑฐ
curl http://127.0.0.1:9200
curl๋ก elasticsearch ํต์ ๋๋์ง ํ์ธ
์ฒ์์ ๊ณต์๋ฌธ์ ํ ๋๋ก ๋ชจ๋ ๊ณผ์ ์ ๋ํด์ ์ ๊ทธ๋ฐ์ง ์ ์ผ๋ ค๊ณ ํ๋๋ฐ
๋ญ๊ฐ ์์ฒญ ๋ง์์ ๊ทธ๋ฅ ํ๋ฉด์ ๋ฐฐ์ฐ๊ธฐ๋ก ํจ
Kibana
https://www.elastic.co/guide/en/kibana/8.3/rpm.html#rpm-repo
# /etc/yum.repos.d/ ๋๋ ํ ๋ฆฌ์ kibana.repo ์์ฑ
$ sudo vim /etc/yum.repos.d/kibana.repo
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
centos์ด๋ฏ๋ก yum ์ ํ
yum install kibana -y
kibana ์คํ
systemctl start kibana
systemctl enable kibana
์ธ๋ถ์์ ์ ๊ทผํ ์ ์๋๋ก /etc/kibana/kibana.yml์์ server.host๋ฅผ ์ฃผ์ ํด์ ํ๊ณ 0.0.0.0 ์ผ๋ก ๋ณ๊ฒฝ
file:///usr/share/doc/HTML/index.html
ifconfig๋ก ip์ฃผ์ ํ์ธํด์ฃผ๊ณ , 192.168.19.131:5601๋ก ์ ์ํ๋ฉด ๋๋ค.(๋๋ localhost:5601)
๋ญ๊ฐ kibana ์ค์นํ๊ณ ๋ถํฐ ๋ ์ด ๋๋ฌด ๊ฑธ๋ ค์...๋จ ๋๋ ค์คฌ๋ค. (1GB -> 2GB)
sudo /usr/share/kibana/bin/kibana
LogStash
๊ณต์ https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
java -version ์ผ๋ก ๊น๋ ค์๋์ง ํ์ธ ๋จผ์
# /etc/yum.repos.d/์ logstash.repo ์์ฑ
[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
yum install logstash -y
systemctl start logstash
systemctl enable logstash
logstash๋ config ํ์ผ์ด ํ์ํจ(/usr/share/logstash/bin/logstash.conf)
ํ์ดํ๋ผ์ธ 3๊ฐ์ง๋ก ๊ตฌ์ฑ๋จ (input/filter/output)
# ๊ฐ์ฅ ๊ฐ๋จํ ํํ
input {
stdin { }
}
output {
stdout { }
}
stdin = ํค๋ณด๋๋ก input(์ ๋ ฅ)์ ๋ฐ๊ฒ ๋ค
stdout = ๋ชจ๋ํฐ๋ก output(์ถ๋ ฅ) ํ๊ฒ ๋ค
ํ๊ณ ./logstash -f logstash.conf ํ๋๋ ์ด๋ด ์ค๋ฅ ๋ธ...
https://www.reddit.com/r/elasticsearch/comments/9d04av/how_do_i_solve_this_problem/
๋ฉ๋ชจ๋ฆฌ ๋ถ์กฑํ๋ค๊ธธ๋ 2GB์์ 3GB๋ก ๋๋ ค์ค.
[root@localhost bin]# ./logstash -f logstash.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000ca660000, 899284992, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 899284992 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/share/logstash/bin/hs_err_pid18636.log
๋ค์ ํ๋๋ ๋ญ๊ฐ ์ฑ๊ณตํ๊ฒ๊ฐ๊ธด ํจ
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
hello
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2022-07-16 13:43:34.801 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.
[INFO ] 2022-07-16 13:43:34.833 [main] runner - Starting Logstash {"logstash.version"=>"8.3.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[INFO ] 2022-07-16 13:43:34.834 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[INFO ] 2022-07-16 13:43:34.899 [main] settings - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2022-07-16 13:43:34.906 [main] settings - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2022-07-16 13:43:35.515 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-07-16 13:43:35.571 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"d7e5f1d9-778f-436b-a965-b3a32e0280d2", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2022-07-16 13:43:37.559 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/logstash-simple.conf"}
[ERROR] 2022-07-16 13:43:37.563 [Agent thread] sourceloader - No configuration found in the configured sources.
[INFO ] 2022-07-16 13:43:37.787 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2022-07-16 13:43:37.881 [LogStash::Runner] runner - Logstash shut down.
๊ทผ๋ฐ ์ด๋ฐ ๊ณผ์ ์ ๋ชปํ์...
# logstash script ์ ์ฉ
$ cd /usr/share/logstash/bin
$ ./system-install
Successfully created system startup script for Logstash ๊ฐ ๋จ๋ฉด ์ฑ๊ณต์ด๋ค
'PROJECT > ๐ํญ๋ง' ์นดํ ๊ณ ๋ฆฌ์ ๋ค๋ฅธ ๊ธ
ELK with IoT ์ค๊ฐ ์ ์ฐ (0) | 2022.07.24 |
---|---|
[inflearn] ELK ์คํ์ผ๋ก ๋ฐ์ดํฐ ๋ถ์ - ์น์ 5. ์ค๋ฐ์ดํฐ ๋ถ์ (0) | 2022.07.23 |
[inflearn] ELK ์คํ์ผ๋ก ๋ฐ์ดํฐ ๋ถ์ - ์น์ 3. Kibana / 4. Logstash (0) | 2022.07.17 |
[inflearn] ELK ์คํ์ผ๋ก ๋ฐ์ดํฐ ๋ถ์ - ์น์ 1~2. ElasticSearch (0) | 2022.07.06 |
220623 ์กฐ์ฌ (0) | 2022.06.23 |