R136A1
[inflearn] ELK ์คํ์ผ๋ก ๋ฐ์ดํฐ ๋ถ์ - ์น์ 5. ์ค๋ฐ์ดํฐ ๋ถ์ ๋ณธ๋ฌธ
[inflearn] ELK ์คํ์ผ๋ก ๋ฐ์ดํฐ ๋ถ์ - ์น์ 5. ์ค๋ฐ์ดํฐ ๋ถ์
r136a1x27 2022. 7. 23. 23:341. ์ค์ ์ธ๊ตฌ๋ถ์
logstash
https://catalog.data.gov/dataset ์์ population by country ์ณค๋๋ ์๋์ด
https://data.world/doe/population-bycountry-1980-2010 ์์ ๋์ ๋ค์ด๋ฐ์ ์ ์๋ค
๋๋... ๊ฐ์์ Github ch06 ๋๋ ํ ๋ฆฌ ์๋์ csv ํ์ผ ์์(NIA, - ์ ๊ฐ์ ๋ฐ์ดํฐ ์ญ์ )
ElasticSearch, Kibana๊ฐ ์ ๋์๊ฐ๋์ง ํ์ธํ๊ธฐ
ps -ef | grep kibana
ps -ef | grep elasticsearch
cd /usr/share/logstash/bin/
logstash.conf
input {
file {
path => "csv ํ์ผ์ด ์๋ ์ ๋๊ฒฝ๋ก"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Country","1980","1981","1982","1983","1984","1985","1986","1987","1988","1989","1990","1991","1992","1993","1994","1995","1996","1997","1998","1999","2000","2001","2002","2003","2004","2005","2006","2007","2008","2009","2010"]
}
mutate {convert => ["1980", "float"]}
mutate {convert => ["1981", "float"]}
mutate {convert => ["1982", "float"]}
...
mutate {convert => ["2009", "float"]}
mutate {convert => ["2010", "float"]}
}
output {
elasticsearch {
hosts => "localhost"
index => "population"
}
stdout {}
}
start_position => ์ต์ด์ ํ์ผ์ ์ฝ์ด๋ค์ด๋ ์์น
"beginning" ์ฒ์๋ถํฐ ์ฝ์ด๋ค์ธ๋ค๋ ์ค์ (ํ์ผ์ ์ฌ์ฉํ ๋ ์ฃผ๋ก ์ค์ )
"end" ๋ง์ง๋ง์ผ๋ก ์ฝ์ ์์น๋ถํฐ
(streaming data์ ๊ฒฝ์ฐ ๊ธฐ์กด์ ๋ฐ์ดํฐ๋ฅผ ์ ์ธ, ๊ฐฑ์ ๋ ์๋ก์ด ๋ฐ์ดํฐ๋ง์ ๋ฐ์์ผ ํ๊ธฐ ๋๋ฌธ)
sincedb_path => ํ์ผ์ ์ฌ์ฉํ์ ๋, offset์ ์ ์ฅํ๋ ์ฉ๋
logstash ์ฌ์์ํ๋ฉด start_position ์๊ด ์์ด ํด๋น offset์์ ์์
์ด๊ฑฐ ์ง์ ์ํด์ฃผ๋ฉด ์ฒซ ๋ฒ์งธ๋ ๋ฐ์ดํฐ๊ฐ ์ ๋ค์ด๊ฐ๋๋ฐ
๋ ๋ฒ์งธ ์ค์ตํ ๋...ํ ๋ฒ ๋ค์ด๊ฐ data๋ logstash๊ฐ ๋ฃ์ง ์์
์ฐธ๊ณ ) https://koocci-dev.tistory.com/20
์ฐ์ฐ์ ํ๊ธฐ ์ํด text์ธ ์๋ฃํ์ ์์๋ก float์ผ๋ก ๋ณ๊ฒฝํ์ฌ ์ ๋ ฅํ๋๋ก
๋ก์ปฌ ElasticSearch์ ๋ฐ๋ก ๋ฃ๋๊ฑฐ๋๊น hosts๋ localhost๋ก ์ถฉ๋ถ
index๋ช ์ population
----------
ํ์ฌ์์น: /usr/share/logstash/bin/
sudo /usr/share/logstash/bin/logstash -f {logstash.conf ์ ๋ ๊ฒฝ๋ก}
์ฌ์ค logstash๋ ./logstash์ ๊ฐ์ด ์๋ ๊ฒฝ๋ก๋ ์๊ด ์๋ค
์ด๊ฒ ์...๋ค์ด๊ฐ๊ฑด๊ฐ?
[INFO ] 2022-07-23 04:17:02.200 [Agent thread] configpathloader - No config files found in path
{:path=>"/usr/share/logstash/logstash.conf"}
[ERROR] 2022-07-23 04:17:02.205 [Agent thread] sourceloader - No configuration found in the configured sources.
์๋ฌด๋๋ ์๋ ๊ฒ ๊ฐ๋ค
sudo ./logstash -f logstash.conf๋ก ํ๋๋ ์ค๋ฅ
๋ด ์์น๋ /usr/share/logstash/bin ์ธ๋ฐ /usr/share/logstash๋ก ๋จ๋ ๊ฑธ ๋ณด๋ฉด ./logstash๊ฐ ์ ๊ธฐ์ ์คํ๋๋ ๊ฒ ๊ฐ๋ค
์ ๋๊ฒฝ๋ก๋ก ๋ฃ์ด๋ณด์...
sudo ./logstash -f /usr/share/logstash/bin/logstash.conf
์ ์์ ์ผ๋ก ์ ๋ ฅ๋๋ฉด ์ด์๊ฒ ๋ธ
kibana
ํ์ธํด๋ณด์. localhost:5601
์ธ๋ฑ์ค ํจํด ๋งค์นญํ๊ธฐ
http://localhost:5601/app/management/kibana/indexPatterns
์ ๋ฌ๋ค!
๋์ค์ปค๋ฒ Discover
๋ค์ด๊ฐ๋ณด์
http://localhost:5601/app/discover
๋ฐ์ดํฐ ํ๋ ๋๋ฌ์ Country, 1980, 2010๋ง column์ ๋จ๊ฒํ๊ธฐ
search์ korea ์ณ์ ํ์ธ ๊ฐ๋ฅ
์๊ฐํ visualize
y์ถ์ ๋ฐ๊ฟ๊ฐ๋ฉฐ ๊ด์ฐฐํด๋ณด๊ธฐ
ํ๋ฑ csv์ ๋ถ๊ณผํ๋ ๋ฐ์ดํฐ๋ฅผ ์๊ฐํํ์ฌ ๋ถ์ํ ์ ์๋ ๊ฒ์ด Kibana์ ํ
2. ์ค์ ์ฃผ์ ๋ถ์
http://blog.webkid.io/visualize-datasets-with-elk ์์ ์์ด๋์ด๋ฅผ ๋ฐ์๋ค๊ณ ๋ง์
finance YAHOO์์ ํน์ ์ข ๋ชฉ์ ๋ฐ์ดํฐ ๋ค์ด๋ฐ์ ์ ์์
Facebook(=Meta) 5๋ ๋ค์ด
ps -ef | grep kibana
ps -ef | grep elasticsearch
์ฌ์ค wgetํ๋ ๊ฒฝ๋ก๋ ๋ฑํ ์๊ด ์๋ค
wget https://raw.githubusercontent.com/minsuk-heo/BigData/master/ch06/logstash_stock.conf
vim์ ํตํด์ path ์ ๋๋ง ๋ฐ๊ฟ์ฃผ๊ธฐ
input {
file {
path => "csv์ ๋๊ฒฝ๋ก"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
}
mutate {convert => ["Open", "float"]}
mutate {convert => ["High", "float"]}
mutate {convert => ["Low", "float"]}
mutate {convert => ["Close", "float"]}
}
output {
elasticsearch {
hosts => "localhost"
index => "stock"
}
stdout {}
}
cd /usr/share/logstash/bin/
sudo ./logstash -f /usr/share/logstash/bin/logstash_stock.conf
[FATAL] 2022-07-23 05:06:28.811 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
์๋ฒ ํ๊ฒฝ์์๋ ํจ๋ถ๋ก ์๋ฒ๋ฅผ ๋๋ฉด ์๋๊ธฐ ๋๋ฌธ์
โ
systemctl status logstash
systemctl stop logstash
systemctl restart logstash
๋ก logstash๋ฅผ ์ข
๋ฃํ๊ณ ์ฌ์์ํ๋๋ก ํ์
โก
ps -ef | grep logstash
kill -9 [PID]
=> ์ด๊ฑฐ ์ ์๋จ
์ค๋ฅ ๊ทธ๋๋ก ๊ฒ์ํ๊ฑฐ๋ "Logstash ์ฌ๋ฌ๊ฐ ์คํ"์ผ๋ก ๊ฒ์ํ๋ฉด ์ฌ๋ฌ๊ฐ ๋์ค๋๋ฐ ์ด๊ฑฐ ๋ฐ๋ผํ๋ฉด ๋๋ค
๋๋ ์ฝ๊ฒ์ฝ๊ฒ..๊ทธ๋ฅ ์ฌ์์ ํ๊ธฐ๋ก
๋ง์ฝ ๊ทธ๋ฅ ์ฌ์์ํ๊ณ curl http://127.0.0.1:9200 ํด์ ์๋ต์ด ์๋ค๋ฉด...
elasticsearch๋ kibana๋ ์ฌ์์ํด์ค์ผ๋จ
[INFO ] 2022-07-23 05:18:56.503 [Ruby-0-Thread-9: :1] elasticsearch - Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)}
[WARN ] 2022-07-23 05:18:56.503 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)"}
----------
su
systemctl stop elasticsearch.service
systemctl start elasticsearch.service
systemctl enable elasticsearch.service
systemctl stop kibana
systemctl start kibana
systemctl enable kibana
๋ค์ ์ปค๋งจ๋ ์ ๋ ฅํ๋ฉด ์ด์๊ฒ ๋ค์ด๊ฐ
Stack Management - Kibana - Index Patterns์์ stock Index ๋ฑ๋กํ๊ธฐ
- ์ด ๋, Timestamp๋ฅผ Date๋ก ์ง์ ํด์ค์ผํ๋๋ฐ ์๋จ
Analytics - Dicover ์์ ๋ฐ์ดํฐ ์ฉ์ผ๋ก ๊ด์ฐฐํด๋ณด๊ธฐ (column: Date, Open, Close)
- Date ์์ผ๋ก ์ ๋ ฌ์ ๋ชปํ๋..? ๋ฐ์ดํฐ ์ ๋ ฅ ์๊ฐ์ธ TimeStamp๋ ์๋ฏธ๊ฐ ์๋๋ฐ...
- stock_toggled๋ก ์ ์ฅ
Analytics - Visualized Library - Create Visualization
y์ถ: Average, Close
x์ถ: Date ํ๋ ค๋๋ฐ ๊ฐ์์์๋ Date Histogram ์๋ Date ํ๋๋ฅผ ์ ํํ ์ ์์์ง๋ง ์ค์ต์์๋ ์๋จ
์์ธ์ ์ฐพ์๋ฉด ๋ฐ์ดํฐ๋ฅผ ๋ฃ์ ๋ filter์์ Date๋ฅผ ๋ฐ๋ก ์ง์ ์ ์ํด์ค์ text๋ก ๋ค์ด๊ฐ ๊ฒ ๊ฐ์
๊ทธ๋ฌ๋ ๊ณต์ ๋ฌธ์์ ๋ฐ๋ฅด๋ฉด convert์๋ int, float, string, boolean๊ณผ ๊ฐ์๊ฒ๋ฐ์ ์์
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
์ด๊ฑธ ์ฐ๋ฉด ๋์ง ์์๊น?
filter {
date {
match => [ "์นผ๋ผ๋ช
", "๋ฐ์ดํฐ์ ๋ ์ง ์์" ]
timezone => "Asia/Seoul"
locale => "ko"
}
}
filter {
date {
match => [ "Date", "yyyy-MM-dd" ]
timezone => "Asia/Seoul"
locale => "ko"
}
}
๋งคํ ํ์ธํด๋ณด๊ธฐ curl -XGET http://localhost:9200/stock2/?pretty
(๋๊ฒ ๋ค)
์ฌ์ ํ Date๊ฐ text, ๋์ @timestamp๊ฐ ๋ฐ์ดํฐ ์ ๋ ฅ ์๊ฐ์์ Date๋ก ๋ฐ๋ ๊ฒ์ ํ์ธํ ์ ์์
๊ทผ๋ฐ Kibana์์ ํ์ธํ๋ฉด ๋ฐ์ดํฐ๊ฐ ํ ๊ฐ๋ฐ์ ์์
๋ชป๊ณ ์น๊ฒ ๋ค~
'PROJECT > ๐ํญ๋ง' ์นดํ ๊ณ ๋ฆฌ์ ๋ค๋ฅธ ๊ธ
ELK - Beats (0) | 2022.08.04 |
---|---|
ELK with IoT ์ค๊ฐ ์ ์ฐ (0) | 2022.07.24 |
[inflearn] ELK ์คํ์ผ๋ก ๋ฐ์ดํฐ ๋ถ์ - ์น์ 3. Kibana / 4. Logstash (0) | 2022.07.17 |
[inflearn] ELK ์คํ์ผ๋ก ๋ฐ์ดํฐ ๋ถ์ - ์น์ 1~2. ElasticSearch (0) | 2022.07.06 |
ELK Stack ํ๊ฒฝ ๊ตฌ์ฑ (0) | 2022.07.05 |