Event Debug Server Status

Version: 6.3.1 (#1466458 built by jenkins on 20190925-0746 git: 56b387edf7c27c5d67c3391b7fa709c082f2114c)
Time now: October 14, 2024 5:20:08 AM +07:00

 |  Configure Logging  |  Threads  |  Thread CPU Time  |  Poor Man's Profiler  |  JMX  |  GC Pause Information

EventCatcherService (com.cloudera.cmf.eventcatcher.server.EventCatcherService)
There are 5000012 documents in the index


Lucene index manager (com.cloudera.cmf.eventcatcher.server.SingleIndexManager)
There are 5000012 events in the index with 10962 deleted events waiting to be expunged.
Last searcher refresh occurred 10.31 secs ago and took 0.000 secs.
Last commit occurred 59.89 secs ago and took 0.003 secs.
Last cleanup occurred 174.08 secs ago and took 3.153 secs.


Event Ingester (com.cloudera.cmf.eventcatcher.server.EventIngester)
Seen 69341 events
Last events: 
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-dec125281080803bb2b9288dd9aff9c2], HOSTS=[namenode2.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[14d8f5e0-ecc4-4d14-bbe1-89fa8cfde10d], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857771441}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-f3e08d3125e90218d8495af250c2e2a0], HOSTS=[datanode3.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[ee0391a7-88ed-4fed-ac39-f95e9a63899c], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857772424}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-0fe285ba69be7b2dc5815de987ad25fa], HOSTS=[datanode2.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], EXCEPTION_TYPES=[java.net.ConnectException, java.net.ConnectException], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[4700eb51-24f5-4c11-924d-4b99f2ae8954], STACKTRACE=[java.net.ConnectException: Call From datanode2.moe.bigdata.go.th/192.168.239.37 to namenode1.moe.bigdata.go.th:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
	at org.apache.hadoop.ipc.Client.call(Client.java:1445)
	at org.apache.hadoop.ipc.Client.call(Client.java:1355)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy20.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:847)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:686)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:789)
	at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
	at org.apache.hadoop.ipc.Client.call(Client.java:1391)
	... 9 more
], SEVERITY=[IMPORTANT]}, content=IOException in offerService, timestamp=1728857773452}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-228a1f56d12863c198b2f2fcbc5d1f78], HOSTS=[datanode5.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], EXCEPTION_TYPES=[java.net.ConnectException, java.net.ConnectException], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[b844e16e-38a1-4dc3-bac9-892675e8a5f8], STACKTRACE=[java.net.ConnectException: Call From datanode5.moe.bigdata.go.th/192.168.239.40 to namenode1.moe.bigdata.go.th:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
	at org.apache.hadoop.ipc.Client.call(Client.java:1445)
	at org.apache.hadoop.ipc.Client.call(Client.java:1355)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy20.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:847)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:686)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:789)
	at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
	at org.apache.hadoop.ipc.Client.call(Client.java:1391)
	... 9 more
], SEVERITY=[IMPORTANT]}, content=IOException in offerService, timestamp=1728857773453}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-dec125281080803bb2b9288dd9aff9c2], HOSTS=[namenode2.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[14d8f5e0-ecc4-4d14-bbe1-89fa8cfde10d], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857832237}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-f3e08d3125e90218d8495af250c2e2a0], HOSTS=[datanode3.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[ee0391a7-88ed-4fed-ac39-f95e9a63899c], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857833958}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-12c9e4112cea25acd51602f1019f29c4], HOSTS=[datanode1.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[f8bc3e01-bdae-4428-8858-572d5e23dd92], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857835428}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-9fc5d34330705b6607ef0069e1c77f0c], HOSTS=[datanode4.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], EXCEPTION_TYPES=[java.net.ConnectException, java.net.ConnectException], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[30986536-d77d-48d8-8fd5-6917445bf616], STACKTRACE=[java.net.ConnectException: Call From datanode4.moe.bigdata.go.th/192.168.239.39 to namenode1.moe.bigdata.go.th:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
	at org.apache.hadoop.ipc.Client.call(Client.java:1445)
	at org.apache.hadoop.ipc.Client.call(Client.java:1355)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy20.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:847)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:686)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:789)
	at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
	at org.apache.hadoop.ipc.Client.call(Client.java:1391)
	... 9 more
], SEVERITY=[IMPORTANT]}, content=IOException in offerService, timestamp=1728857884961}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-0fe285ba69be7b2dc5815de987ad25fa], HOSTS=[datanode2.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], EXCEPTION_TYPES=[java.net.ConnectException, java.net.ConnectException], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[4700eb51-24f5-4c11-924d-4b99f2ae8954], STACKTRACE=[java.net.ConnectException: Call From datanode2.moe.bigdata.go.th/192.168.239.37 to namenode1.moe.bigdata.go.th:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
	at org.apache.hadoop.ipc.Client.call(Client.java:1445)
	at org.apache.hadoop.ipc.Client.call(Client.java:1355)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy20.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:847)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:686)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:789)
	at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
	at org.apache.hadoop.ipc.Client.call(Client.java:1391)
	... 9 more
], SEVERITY=[IMPORTANT]}, content=IOException in offerService, timestamp=1728857886591}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-228a1f56d12863c198b2f2fcbc5d1f78], HOSTS=[datanode5.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], EXCEPTION_TYPES=[java.net.ConnectException, java.net.ConnectException], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[b844e16e-38a1-4dc3-bac9-892675e8a5f8], STACKTRACE=[java.net.ConnectException: Call From datanode5.moe.bigdata.go.th/192.168.239.40 to namenode1.moe.bigdata.go.th:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
	at org.apache.hadoop.ipc.Client.call(Client.java:1445)
	at org.apache.hadoop.ipc.Client.call(Client.java:1355)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy20.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:847)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:686)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:789)
	at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
	at org.apache.hadoop.ipc.Client.call(Client.java:1391)
	... 9 more
], SEVERITY=[IMPORTANT]}, content=IOException in offerService, timestamp=1728857887367}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-12c9e4112cea25acd51602f1019f29c4], HOSTS=[datanode1.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[f8bc3e01-bdae-4428-8858-572d5e23dd92], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857888118}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-dec125281080803bb2b9288dd9aff9c2], HOSTS=[namenode2.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[14d8f5e0-ecc4-4d14-bbe1-89fa8cfde10d], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857893998}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-f3e08d3125e90218d8495af250c2e2a0], HOSTS=[datanode3.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[ee0391a7-88ed-4fed-ac39-f95e9a63899c], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857894893}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-f3e08d3125e90218d8495af250c2e2a0], HOSTS=[datanode3.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[ee0391a7-88ed-4fed-ac39-f95e9a63899c], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857940225}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-12c9e4112cea25acd51602f1019f29c4], HOSTS=[datanode1.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[f8bc3e01-bdae-4428-8858-572d5e23dd92], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857953788}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-dec125281080803bb2b9288dd9aff9c2], HOSTS=[namenode2.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[14d8f5e0-ecc4-4d14-bbe1-89fa8cfde10d], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728857954492}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-dec125281080803bb2b9288dd9aff9c2], HOSTS=[namenode2.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[14d8f5e0-ecc4-4d14-bbe1-89fa8cfde10d], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728858000967}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-f3e08d3125e90218d8495af250c2e2a0], HOSTS=[datanode3.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[ee0391a7-88ed-4fed-ac39-f95e9a63899c], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728858001018}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-12c9e4112cea25acd51602f1019f29c4], HOSTS=[datanode1.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[f8bc3e01-bdae-4428-8858-572d5e23dd92], SEVERITY=[IMPORTANT]}, content=Problem connecting to server: namenode1.moe.bigdata.go.th/192.168.239.34:8022, timestamp=1728858008079}
  • AvroEventWrapper{attributes={ROLE=[hdfs-DATANODE-9fc5d34330705b6607ef0069e1c77f0c], HOSTS=[datanode4.moe.bigdata.go.th], ROLE_TYPE=[DATANODE], EXCEPTION_TYPES=[java.net.ConnectException, java.net.ConnectException], CATEGORY=[LOG_MESSAGE], EVENTCODE=[EV_LOG_EVENT], SERVICE=[hdfs], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOST_IDS=[30986536-d77d-48d8-8fd5-6917445bf616], STACKTRACE=[java.net.ConnectException: Call From datanode4.moe.bigdata.go.th/192.168.239.39 to namenode1.moe.bigdata.go.th:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
	at org.apache.hadoop.ipc.Client.call(Client.java:1445)
	at org.apache.hadoop.ipc.Client.call(Client.java:1355)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
	at com.sun.proxy.$Proxy20.sendHeartbeat(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:166)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:516)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:646)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:847)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:686)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:789)
	at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
	at org.apache.hadoop.ipc.Client.call(Client.java:1391)
	... 9 more
], SEVERITY=[IMPORTANT]}, content=IOException in offerService, timestamp=1728858008069}


EventMetricsPublisher (com.cloudera.cmf.eventcatcher.server.EventMetricsPublisher)

Currently not running.
12456 runs so far (0 slow), of them 0 reported exceptions.
Last duration: 3 ms. Total duration: 93852 ms.
Last start: October 14, 2024 5:19:55 AM +07:00. Last end: October 14, 2024 5:19:55 AM +07:00.

Event Schema Checker (com.cloudera.cmf.eventcatcher.server.EventSchemaChecker)
Num events without event codes: 0
Num events with unknown event codes: 0
Num events not matching schema: 0

Events not matching schema:


    EventIngester (com.cloudera.cmon.pipeline.Pipeline)
    Pipeline stage summary:
    namedroppedskippedforwardedprocessedsize
    Pipeline Stage tagger-writer000693410
    A total of 69341 messages received. Current rate: 0.06 msg/sec. Max rate: 19.09 msg/sec.

    Pipeline Stage tagger-writer (com.cloudera.cmon.pipeline.PipelineStage)
    Input queue size is 0 (of 10000).
    69341 events processed.
    0 events skipped.
    0 events forwarded.
    0 events dropped.
    Current rate: 0.06 msg/sec. Max rate: 19.09 msg/sec.
    Thread pool size is 2 (max 4).
    First element in queue: null


    Events queryer (com.cloudera.cmf.eventcatcher.server.EventsQueryer)
    Seen 74380 queries
    Last query took 0ms
    Last 20 queries:
    • +__persist_timestamp:[1728857773456 TO 1728858412131]
    • +__persist_timestamp:[1728857773456 TO 1728858422133]
    • +__persist_timestamp:[1728857773456 TO 1728858432142]
    • +__persist_timestamp:[1728857773456 TO 1728858442144]
    • +__persist_timestamp:[1728857773456 TO 1728858452145]
    • +__persist_timestamp:[1728857835429 TO 1728858462147]
    • +__persist_timestamp:[1728857835429 TO 1728858472149]
    • +__persist_timestamp:[1728857835429 TO 1728858482151]
    • +__persist_timestamp:[1728857835429 TO 1728858492152]
    • +__persist_timestamp:[1728857886593 TO 1728858502154]
    • +__persist_timestamp:[1728857894894 TO 1728858512156]
    • +__persist_timestamp:[1728857894894 TO 1728858522158]
    • +__persist_timestamp:[1728857894894 TO 1728858532159]
    • +__persist_timestamp:[1728857894894 TO 1728858542161]
    • +__persist_timestamp:[1728857940226 TO 1728858552162]
    • +__persist_timestamp:[1728857940226 TO 1728858562164]
    • +__persist_timestamp:[1728857940226 TO 1728858572165]
    • +__persist_timestamp:[1728857954493 TO 1728858582167]
    • +__persist_timestamp:[1728857954493 TO 1728858592168]
    • +__persist_timestamp:[1728857954493 TO 1728858602170]


    Lucene index manager (com.cloudera.cmf.eventcatcher.server.SingleIndexManager)
    There are 5000012 events in the index with 10962 deleted events waiting to be expunged.
    Last searcher refresh occurred 10.31 secs ago and took 0.000 secs.
    Last commit occurred 59.89 secs ago and took 0.003 secs.
    Last cleanup occurred 174.08 secs ago and took 3.153 secs.


    Event Debug Server (com.cloudera.enterprise.DebugServer)
    Web server is running? true

    AvroEventStoreServer (com.cloudera.cmf.eventcatcher.server.AvroEventStoreServer)
    Avro NettyServer is running on port 7184
    20 max worker threads


    AvroEventStoreHttpService (com.cloudera.cmf.eventcatcher.server.AvroEventStoreHttpService)
    Avro HttpServer is running on port 7185