본문 바로가기
MLOps 개발자 양성과정/bigdata

[Day-84]

숲로 2023. 4. 24. 17:06

Chap02.

 

[root@server01 ~]# hostname
server01.hadoop.com
[root@server01 ~]# service cloudera-scm-server-db status
pg_ctl: server is running (PID: 1876)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"
[root@server01 ~]# ssh server01.hadoop.com
root@server01.hadoop.com's password:
Last login: Mon Apr 24 09:53:04 2023 from server01.hadoop.com
[root@server01 ~]# service cloudera-scm-server status
cloudera-scm-server (pid  2081) is running...

 

종료할 때 중지 꼭 누르고!!!

 

역할 인스턴스 추가할 경우

재시작 눌러줘야 해

(우리는 저사양이라 추가x)

 

 

기본값 3인데 저사양이라 2로 만들어 둠

 

 

하둡은 저장할 때  블록 사이즈가 있어 기본적으로 128

YARN 설정

Fair가 더 좋은 설정이지만 컴퓨터 저사양이므로 First In First Out 선행선출로 변경

 

 

 

VM (Server01, Server02  구동

Moba 코드를 입력

File Zila (파일을 업로드

전체적인 작업을 하기 위해 Cloudera Manager 사용

 

 

#2. HDFS에 파일 저장 : Moba
[root@server02 ~]# cd /home/bigdata
[root@server02 bigdata]# ls
Desktop  Documents  Downloads  Music  Pictures  Public  Sample.txt  Templates  Videos

# Sample.txt 파일이 HDFS의 /tmp 디렉터리로 저장
[root@server02 bigdata]# hdfs dfs -put Sample.txt /tmp


3. HDFS에 저장한 파일 확인
[root@server02 bigdata]# hdfs dfs -ls /tmp
Found 3 items
d---------   - hdfs   supergroup          0 2023-04-24 12:07 /tmp/.cloudera_health_monitoring_canary_files
-rw-r--r--   3 root   supergroup     124322 2023-04-24 12:07 /tmp/Sample.txt
drwxrwxrwt   - mapred hadoop              0 2023-04-24 11:09 /tmp/logs

4. HDFS에 저장한 파일 내용 보기
[root@server02 bigdata]# hdfs dfs -cat /tmp/Sample.txt

 

$ hdfs dfs -stat '%b %o %r %u %n‘ /tmp/Sample.txt 파일 크기(%b). 파일 블록 크기(%o). 복제 수(%r), 소유자명(%u), 파일명(%n) 정보를 보여준다.

[root@server02 bigdata]# hdfs dfs -stat '%b %o %r %u %n' /tmp/Sample.txt
124322 134217728 3 root Sample.txt
[root@server02 bigdata]# hdfs dfs -mv /tmp/Sample.txt /tmp/Sample2.txt
[root@server02 bigdata]# hdfs fsck /
Connecting to namenode via http://server01.hadoop.com:9870/fsck?ugi=root&path=%2F
FSCK started by root (auth:SIMPLE) from /192.168.56.102 for path / at Mon Apr 24 12:17:05 KST 2023

/tmp/Sample2.txt:  Under replicated BP-400058537-192.168.56.101-1682302000907:blk_1073741883_1059. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

Status: HEALTHY
 Number of data-nodes:  1
 Number of racks:               1
 Total dirs:                    11
 Total symlinks:                0

Replicated Blocks:
 Total size:    235169738 B
 Total files:   2
 Total blocks (validated):      3 (avg. block size 78389912 B)
 Minimally replicated blocks:   3 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       1 (33.333332 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     1.0
 Missing blocks:                0
 Corrupt blocks:                0
 Missing replicas:              2 (40.0 %)
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Mon Apr 24 12:17:05 KST 2023 in 15 milliseconds


The filesystem under path '/' is HEALTHY
[root@server02 bigdata]# hdfs dfsadmin -report
Configured Capacity: 25093350605 (23.37 GB)
Present Capacity: 14811955200 (13.79 GB)
DFS Remaining: 14574891008 (13.57 GB)
DFS Used: 237064192 (226.08 MB)
DFS Used%: 1.60%
Replicated Blocks:
        Under replicated blocks: 1
        Blocks with corrupt replicas: 0
        Missing blocks: 0
        Missing blocks (with replication factor 1): 0
        Low redundancy blocks with highest priority to recover: 1
        Pending deletion blocks: 0
Erasure Coded Block Groups:
        Low redundancy block groups: 0
        Block groups with corrupt internal blocks: 0
        Missing block groups: 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (1):

Name: 192.168.56.102:9866 (server02.hadoop.com)
Hostname: server02.hadoop.com
Rack: /default
Decommission Status : Normal
Configured Capacity: 25093350605 (23.37 GB)
DFS Used: 237064192 (226.08 MB)
Non DFS Used: 9996801229 (9.31 GB)
DFS Remaining: 14574891008 (13.57 GB)
DFS Used%: 0.94%
DFS Remaining%: 58.08%
Configured Cache Capacity: 363855872 (347 MB)
Cache Used: 0 (0 B)
Cache Remaining: 363855872 (347 MB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 2
Last contact: Mon Apr 24 12:17:42 KST 2023
Last Block Report: Mon Apr 24 11:28:00 KST 2023


[root@server02 bigdata]#
[root@server02 bigdata]# hdfs dfs -get /tmp/Sample2.txt
[root@server02 bigdata]# ls
Desktop  Documents  Downloads  Music  Pictures  Public  Sample2.txt  Sample.txt  Templates  Videos
[root@server02 bigdata]# hdfs dfs -rm /tmp/Sample2.txt
23/04/24 12:39:24 INFO fs.TrashPolicyDefault: Moved: 'hdfs://server01.hadoop.com:8020/tmp/Sample2.txt' to trash at: hdfs://server01.hadoop.com:8020/user/root/.Trash/Current/tmp/Sample2.txt
[root@server02 bigdata]# ls
Desktop  Documents  Downloads  Music  Pictures  Public  Sample2.txt  Sample.txt  Templates  Videos
[root@server02 bigdata]# hdfs dfs -rm -skipTrash /tmp/Sample2.txt
ls^Hrm: `/tmp/Sample2.txt': No such file or directory
[root@server02 bigdata]# ls
Desktop  Documents  Downloads  Music  Pictures  Public  Sample2.txt  Sample.txt  Templates  Videos
[root@server02 bigdata]# hdfs dfs -ls /tmp
Found 2 items
d---------   - hdfs   supergroup          0 2023-04-24 14:05 /tmp/.cloudera_health_monitoring_canary_files
drwxrwxrwt   - mapred hadoop              0 2023-04-24 11:09 /tmp/logs
[root@server02 bigdata]#

zookeeper 실행

[root@server02 ~]# zookeeper-client
Connecting to localhost:2181
2023-04-24 14:07:56,203 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh6.3.2--1, built on 11/08/2019 13:17 GMT
2023-04-24 14:07:56,207 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=server02.hadoop.com
2023-04-24 14:07:56,207 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_181
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_181-cloudera/jre
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../build/classes:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../build/lib/*.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/slf4j-api-1.7.25.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/netty-3.10.6.Final.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../zookeeper-3.4.5-cdh6.3.2.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/zookeeper-3.4.5-cdh6.3.2.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/zookeeper.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/netty-3.10.6.Final.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/slf4j-api-1.7.25.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/slf4j-log4j12-1.7.25.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/zookeeper/lib/audience-annotations-0.5.0.jar
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.32-754.el6.x86_64
2023-04-24 14:07:56,209 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2023-04-24 14:07:56,210 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2023-04-24 14:07:56,210 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
2023-04-24 14:07:56,210 [myid:] - INFO  [main:ZooKeeper@619] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@63c12fb0
Welcome to ZooKeeper!
2023-04-24 14:07:56,232 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1118] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2023-04-24 14:07:56,336 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@962] - Socket connection established, initiating session, client: /127.0.0.1:36290, server: localhost/127.0.0.1:2181
2023-04-24 14:07:56,354 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1378] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0xff87b115dcfc00da, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
# zookeeper내부에 z노드 생성, 조회, 삭제 확인
[zk: localhost:2181(CONNECTED) 0] create /pilot-pjt bigdata
Created /pilot-pjt
[zk: localhost:2181(CONNECTED) 1] ls /
[pilot-pjt, zookeeper]
[zk: localhost:2181(CONNECTED) 2] get /pilot-pjt
bigdata
[zk: localhost:2181(CONNECTED) 3] delete /pilot-pjt
[zk: localhost:2181(CONNECTED) 4]
[root@server02 ~]#  cd /home
[root@server02 home]# mkdir -p /home/pilot-pjt/working/car-batch-log
[root@server02 home]# mkdir -p /home/pilot-pjt/working/driver-realtime-log
[root@server02 home]# chmod 777 -R /home/pilot-pjt
[root@server02 home]# cd /home
[root@server02 home]# ls
bigdata  pilot-pjt
[root@server02 home]# cd pilot-pjt
[root@server02 pilot-pjt]# ls
working
[root@server02 pilot-pjt]# cd working
[root@server02 working]# ls
car-batch-log  driver-realtime-log
[root@server02 working]# cd /home
[root@server02 home]# java -version
java version "1.7.0_181"
OpenJDK Runtime Environment (rhel-2.6.14.10.el6-x86_64 u181-b00)
OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
[root@server02 home]#

기존의 자바를 클라우데라걸로 바꿔치기 할거야

클라우데라 설치하면서 자바 설치됐어

[root@server02 home]# rm /usr/bin/java
rm: remove symbolic link `/usr/bin/java'? yes
[root@server02 home]# cd /usr/bin
# java, javac 삭제됐는지 확인
[root@server02 bin]# ls java*
ls: cannot access java*: No such file or directory

 

 

 

<운전자의 운행정보>

실시간 운행정보

# 파일 들어왔는지 확인
[root@server02 bin]# cd /home/pilot-pjt/working
[root@server02 working]# ls
bigdata.smartcar.loggen-1.0.jar  car-batch-log  driver-realtime-log

# 데이터 요청
# java 명령어 / 자료파일 안에 com안에 ~~~~~ DriverLogMain 몇년도부터 몇 개의 데이터
[root@server02 working]# java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.DriverLogMain 20160101 10

# -cp  참조하려는 클래스가 위치한 경로를 설정하는 옵션
# -d는 생성한 클래스의 위치를 성정하는 옵션
[root@server02 driver-realtime-log]# tail -f SmartCarDriverInfo.log
20160101003202,D0004,1,0,F,N,36,B10
20160101003202,Y0005,3,0,F,N,66,A03
20160101003202,P0007,3,0,F,N,77,C06
20160101003202,C0006,1,0,F,N,127,D09
20160101003202,M0009,1,0,F,N,104,E10
20160101003202,I0008,0,1,F,N,55,B01
20160101003204,D0001,0,1,F,N,7,B08
20160101003202,L0010,0,0,F,N,29,C03
20160101003204,V0003,3,0,F,N,63,B02
.
.
.
실시간 정보 계속

 

 

<스마트카에 대한 상태정보>

10개 차량에 대한 상태정보

[root@server02 working]# java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.CarLogMain 20160101 10
[root@server02 driver-realtime-log]# cd /home/pilot-pjt/working/SmartCar
[root@server02 SmartCar]# tail -f SmartCarStatusInfo_20160101.txt
20160101235924,F0010,99,100,80,84,1,1,1,1,A,A,99
20160101235928,F0010,98,71,88,94,1,1,1,1,A,A,68
20160101235932,F0010,83,93,83,98,1,1,1,1,A,A,97
20160101235936,F0010,97,85,80,80,1,1,1,1,A,A,54
20160101235940,F0010,94,72,83,100,1,1,1,1,A,A,80
20160101235944,F0010,100,76,99,72,1,1,1,1,A,A,88
20160101235948,F0010,81,79,99,85,1,1,1,1,A,B,97
20160101235952,F0010,92,93,85,84,1,1,1,1,A,A,98
20160101235956,F0010,80,73,88,91,1,1,1,1,A,A,84
20160101240000,F0010,90,81,98,95,1,1,1,1,A,A,58

 

파일럿 환경 시작 순서
1. 버추얼 박스 Server01, Server02 실행
2. moba 실행
서버 db, server, agent 가 구동중인지 status 값 확인
3. cloudera 웹브라우저 접속
* 종료 시 cloudera 중지하기 (장기간 미접속 시 강제종료되면 오류날 수 있음)

 

파일럿 환경 종료 순서
1. Cluster 중지, 클라우데라 관리 서비스 중지
2. Moba에서 시스템 종료 명령 실행
$  halt
3. vm에서 전원끄기
4. vm 관리자 닫기

Chap03. 빅데이터 수집

 

3-2 빅데이터 수집에 활용할 기술

vmffja

 

 

 

50 -> 100으로 수정

 

 

 

kafka 

 

3.5 수집 파일럿 실행 3단계

플럼 수집 기능 구현

 

SmartCar_Agent.sources  = SmartCarInfo_SpoolSource DriverCarInfo_TailSource
SmartCar_Agent.channels = SmartCarInfo_Channel DriverCarInfo_Channel
SmartCar_Agent.sinks    = SmartCarInfo_LoggerSink DriverCarInfo_KafkaSink

SmartCar_Agent.sources.SmartCarInfo_SpoolSource.type = spooldir
SmartCar_Agent.sources.SmartCarInfo_SpoolSource.spoolDir = /home/pilot-pjt/working/car-batch-log
SmartCar_Agent.sources.SmartCarInfo_SpoolSource.deletePolicy = immediate
SmartCar_Agent.sources.SmartCarInfo_SpoolSource.batchSize = 1000

SmartCar_Agent.sources.SmartCarInfo_SpoolSource.interceptors = filterInterceptor

SmartCar_Agent.sources.SmartCarInfo_SpoolSource.interceptors.filterInterceptor.type = regex_filter

SmartCar_Agent.sources.SmartCarInfo_SpoolSource.interceptors.filterInterceptor.regex = ^\\d{14}

SmartCar_Agent.sources.SmartCarInfo_SpoolSource.interceptors.filterInterceptor.excludeEvents = false

SmartCar_Agent.channels.SmartCarInfo_Channel.type = memory
SmartCar_Agent.channels.SmartCarInfo_Channel.capacity  = 100000
SmartCar_Agent.channels.SmartCarInfo_Channel.transactionCapacity  = 10000

SmartCar_Agent.sinks.SmartCarInfo_LoggerSink.type = logger

SmartCar_Agent.sources.SmartCarInfo_SpoolSource.channels = SmartCarInfo_Channel
SmartCar_Agent.sinks.SmartCarInfo_LoggerSink.channel = SmartCarInfo_Channel



SmartCar_Agent.sources.DriverCarInfo_TailSource.type = exec

SmartCar_Agent.sources.DriverCarInfo_TailSource.command = tail -F /home/pilot-pjt/working/driver-realtime-log/SmartCarDriverInfo.log
SmartCar_Agent.sources.DriverCarInfo_TailSource.restart = true
SmartCar_Agent.sources.DriverCarInfo_TailSource.batchSize = 1000

SmartCar_Agent.sources.DriverCarInfo_TailSource.interceptors = filterInterceptor2

SmartCar_Agent.sources.DriverCarInfo_TailSource.interceptors.filterInterceptor2.type = regex_filter
SmartCar_Agent.sources.DriverCarInfo_TailSource.interceptors.filterInterceptor2.regex = ^\\d{14}
SmartCar_Agent.sources.DriverCarInfo_TailSource.interceptors.filterInterceptor2.excludeEvents = false



SmartCar_Agent.sinks.DriverCarInfo_KafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
SmartCar_Agent.sinks.DriverCarInfo_KafkaSink.topic = SmartCar-Topic

SmartCar_Agent.sinks.DriverCarInfo_KafkaSink.brokerList = server02.hadoop.com:9092
SmartCar_Agent.sinks.DriverCarInfo_KafkaSink.requiredAcks = 1

SmartCar_Agent.sinks.DriverCarInfo_KafkaSink.batchSize = 1000




SmartCar_Agent.channels.DriverCarInfo_Channel.type = memory
SmartCar_Agent.channels.DriverCarInfo_Channel.capacity= 100000
SmartCar_Agent.channels.DriverCarInfo_Channel.transactionCapacity = 10000

SmartCar_Agent.sources.DriverCarInfo_TailSource.channels = DriverCarInfo_Channel
SmartCar_Agent.sinks.DriverCarInfo_KafkaSink.channel = DriverCarInfo_Channel

 

# 1. 카프카 Topic 생성
kafka-topics --create --zookeeper server02.hadoop.com:2181 --replication-factor 1 --partitions 1 --topic SmartCar-Topic

 

# Server02 세션 하나 더 열어서 카프카 Producer 사용
kafka-console-producer --broker-list server02.hadoop.com:9092 -topic SmartCar-Topic

 

카프카 통신

# 3. 카프카 Consumer 사용
kafka-console-consumer --bootstrap-server server02.hadoop.com:9092 -topic SmartCar-Topic --partition 0 --from-beginning

 

수집 기능 테스트

# 수집 기능 테스트

# 1. SmartCar 로그 시뮬레이터 작동
cd /home/pilot-pjt/working

# 2개의 스마트카 로그 시뮬레이터를 백그라운드 방식으로 실행
java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.CarLogMain 20160101 3 &
# 차량정보
java -cp bigdata.smartcar.loggen-1.0.jar com.wikibook.bigdata.smartcar.loggen.DriverLogMain 20160101 3 &
# 사용자 운행정보 백그라운드 방식으로 세 개만 실행

 

확인해보기

[root@server02 ~]# cd /home/pilot-pjt/working/SmartCar
[root@server02 SmartCar]# tail -f SmartCarStatusInfo_20160101.txt
20160101235924,N0003,89,79,96,81,1,1,1,1,A,B,86
20160101235928,N0003,91,88,74,74,1,1,1,1,A,A,81
20160101235932,N0003,92,88,85,92,1,1,1,1,B,A,81
20160101235936,N0003,83,84,78,83,1,1,1,1,B,B,53
20160101235940,N0003,71,88,70,73,1,1,1,1,A,A,52
20160101235944,N0003,85,98,73,91,1,1,1,1,A,A,91
20160101235948,N0003,93,84,88,80,1,1,1,1,A,B,92
20160101235952,N0003,87,74,90,80,1,1,1,1,A,A,85
20160101235956,N0003,85,86,85,72,1,1,1,1,A,A,93
20160101240000,N0003,79,100,81,88,1,1,1,1,A,A,51
[root@server02 SmartCar]# cd /home/pilot-pjt/working/driver-realtime-log
[root@server02 driver-realtime-log]# tail -f SmartCarDriverInfo.log
20160101011552,R0003,0,0,F,N,27,A09
20160101011556,V0002,5,0,F,N,40,B05
20160101011556,Y0001,3,0,F,N,18,A02
20160101011554,R0003,1,0,L1,L,9,A04
20160101011558,V0002,1,0,F,N,45,B05
20160101011556,R0003,5,0,F,N,34,A01
20160101011558,Y0001,1,0,F,N,23,A09
20160101011600,Y0001,1,0,L3,L,28,A10
20160101011558,R0003,4,0,F,N,54,A07
20160101011600,V0002,1,0,R2,R,50,B09
20160101011602,Y0001,2,0,F,N,11,A03
.
.
.
# 3. 플럼 SmartInfo 에이전트의 SpoolDir 경로로 이동.
mv /home/pilot-pjt/working/SmartCar/SmartCarStatusInfo_20160101.txt /home/pilot-pjt/working/car-batch-log/

 

플럼 재시작

[root@server02 SmartCar]# tail -f /var/log/flume-ng/flume-cmf-flume-AGENT-server02.hadoop.com.log
2023-04-24 16:48:55,279 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 32 34 2C 50 20160101235924,P }
2023-04-24 16:48:55,279 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 32 38 2C 50 20160101235928,P }
2023-04-24 16:48:55,279 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 33 32 2C 50 20160101235932,P }
2023-04-24 16:48:55,280 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 33 36 2C 50 20160101235936,P }
2023-04-24 16:48:55,280 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 34 30 2C 50 20160101235940,P }
2023-04-24 16:48:55,280 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 34 34 2C 50 20160101235944,P }
2023-04-24 16:48:55,280 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 34 38 2C 50 20160101235948,P }
2023-04-24 16:48:55,280 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 35 32 2C 50 20160101235952,P }
2023-04-24 16:48:55,280 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 33 35 39 35 36 2C 50 20160101235956,P }
2023-04-24 16:48:55,280 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{} body: 32 30 31 36 30 31 30 31 32 34 30 30 30 30 2C 50 20160101240000,P }

 

'MLOps 개발자 양성과정 > bigdata' 카테고리의 다른 글

[Day-86]  (0) 2023.04.26
[Day-83]  (0) 2023.04.21
[Day-81] 빅데이터 이해하기/파일럿 프로젝트  (0) 2023.04.18

'MLOps 개발자 양성과정/bigdata' Related Articles

티스토리툴바