Tip: Zookeeper Recipes

A great link to Zookeeper recipes for locking across multiple agents.

Links

Apache Zookeeper: Recipes

Solution: Kafka Kerberos Debug

If you are debugging kerberos and kafka, try this before starting Kafka add -Dsun.security.krb5.debug=true to the KAFKA_HEAP_OPTS.

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G -Djava.security.auth.login.config=/opt/kafka/security/kafka_server_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Dlog4j.properties=file:///opt/kafka/config/log4j.properties -Dsun.security.krb5.debug=true"

And update /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

You get a rich log file

Output

root@broker:/# tail -f zookeeper.log
[2017-05-20 20:25:33,482] INFO Reading configuration from: /opt/kafka/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-05-20 20:25:33,495] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2017-05-20 20:25:33,495] ERROR Invalid configuration, only one server specified (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-05-20 20:25:33,497] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-05-20 20:25:33,497] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-05-20 20:25:33,497] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-05-20 20:25:33,497] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2017-05-20 20:25:33,512] INFO Reading configuration from: /opt/kafka/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-05-20 20:25:33,512] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2017-05-20 20:25:33,513] ERROR Invalid configuration, only one server specified (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-05-20 20:25:33,513] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2017-05-20 20:25:33,520] INFO Server environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:host.name=broker.example.local (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:java.version=1.8.0_131 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:java.class.path=:/opt/kafka/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/opt/kafka/bin/../libs/argparse4j-0.5.0.jar:/opt/kafka/bin/../libs/connect-api-0.10.1.0.jar:/opt/kafka/bin/../libs/connect-file-0.10.1.0.jar:/opt/kafka/bin/../libs/connect-json-0.10.1.0.jar:/opt/kafka/bin/../libs/connect-runtime-0.10.1.0.jar:/opt/kafka/bin/../libs/guava-18.0.jar:/opt/kafka/bin/../libs/hk2-api-2.4.0-b34.jar:/opt/kafka/bin/../libs/hk2-locator-2.4.0-b34.jar:/opt/kafka/bin/../libs/hk2-utils-2.4.0-b34.jar:/opt/kafka/bin/../libs/jackson-annotations-2.6.0.jar:/opt/kafka/bin/../libs/jackson-core-2.6.3.jar:/opt/kafka/bin/../libs/jackson-databind-2.6.3.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/opt/kafka/bin/../libs/javassist-3.18.2-GA.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/bin/../libs/javax.inject-1.jar:/opt/kafka/bin/../libs/javax.inject-2.4.0-b34.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/kafka/bin/../libs/jersey-client-2.22.2.jar:/opt/kafka/bin/../libs/jersey-common-2.22.2.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.22.2.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/opt/kafka/bin/../libs/jersey-guava-2.22.2.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.22.2.jar:/opt/kafka/bin/../libs/jersey-server-2.22.2.jar:/opt/kafka/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-http-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-io-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-security-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-server-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-util-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jopt-simple-4.9.jar:/opt/kafka/bin/../libs/kafka-clients-0.10.1.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-0.10.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-0.10.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-0.10.1.0.jar:/opt/kafka/bin/../libs/kafka-tools-0.10.1.0.jar:/opt/kafka/bin/../libs/kafka_2.11-0.10.1.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.11-0.10.1.0-test-sources.jar:/opt/kafka/bin/../libs/kafka_2.11-0.10.1.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-1.3.0.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/reflections-0.9.10.jar:/opt/kafka/bin/../libs/rocksdbjni-4.9.0.jar:/opt/kafka/bin/../libs/scala-library-2.11.8.jar:/opt/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.21.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.21.jar:/opt/kafka/bin/../libs/snappy-java-1.1.2.6.jar:/opt/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/bin/../libs/zkclient-0.9.jar:/opt/kafka/bin/../libs/zookeeper-3.4.8.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:java.compiler= (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:os.version=4.9.27-moby (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,520] INFO Server environment:user.dir=/ (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,527] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,527] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-05-20 20:25:33,527] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt true ticketCache is null isInitiator true KeyTab is /etc/security/keytabs/zookeeper.keytab refreshKrb5Config is true principal is zookeeper/broker.example.local@example.LOCAL tryFirstPass is false useFirstPass is true storePass is false clearPass is false
Refreshing Kerberos configuration
Java config name: /etc/krb5.conf
Loaded from Java config

KdcAccessibility: reset
KdcAccessibility: reset
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 107; type: 18
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 91; type: 17
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 99; type: 16
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 91; type: 23
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 107; type: 18
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 91; type: 17
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 99; type: 16
KeyTabInputStream, readName(): example.LOCAL
KeyTabInputStream, readName(): zookeeper
KeyTabInputStream, readName(): broker.example.local
KeyTab: load() entry length: 91; type: 23
Looking for keys for: zookeeper/broker.example.local@example.LOCAL
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
Added key: 23version: 1
Added key: 16version: 1
Added key: 17version: 1
Added key: 18version: 1
Looking for keys for: zookeeper/broker.example.local@example.LOCAL
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
Added key: 23version: 1
Added key: 16version: 1
Added key: 17version: 1
Added key: 18version: 1
default etypes for default_tkt_enctypes: 18 17 16.
KrbAsReq creating message
KrbKdcReq send: kdc=broker.example.local UDP:88, timeout=30000, number of retries =3, #bytes=188
KDCCommunication: kdc=broker.example.local UDP:88, timeout=30000,Attempt =1, #bytes=188
KrbKdcReq send: #bytes read=804
KdcAccessibility: remove broker.example.local
Looking for keys for: zookeeper/broker.example.local@example.LOCAL
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
Added key: 23version: 1
Added key: 16version: 1
Added key: 17version: 1
Added key: 18version: 1
EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
KrbAsRep cons in KrbAsReq.getReply zookeeper/broker.example.local
principal is zookeeper/broker.example.local@example.LOCAL
Will use keytab
Commit Succeeded

[2017-05-20 20:25:33,649] INFO successfully logged in. (org.apache.zookeeper.Login)
[2017-05-20 20:25:33,651] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
[2017-05-20 20:25:33,656] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-05-20 20:25:33,666] INFO TGT valid starting at: Sat May 20 20:25:33 UTC 2017 (org.apache.zookeeper.Login)
[2017-05-20 20:25:33,666] INFO TGT expires: Sun May 21 20:25:33 UTC 2017 (org.apache.zookeeper.Login)
[2017-05-20 20:25:33,666] INFO TGT refresh sleeping until: Sun May 21 16:02:45 UTC 2017 (org.apache.zookeeper.Login)
root@broker:/#

Check Partition Count and Replication Factor

  bin/kafka-topics.sh --zookeeper `hostname --long`:2181 --describe --topic replication
 Topic:replication PartitionCount:20 ReplicationFactor:3 Configs:
  Topic: replication Partition: 0 Leader: 1005 Replicas: 1005,1001,1002 Isr: 1001,1002,1005
  Topic: replication Partition: 1 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1001,1003,1002
  Topic: replication Partition: 2 Leader: 1002 Replicas: 1002,1003,1004 Isr: 1002,1003,1004
  Topic: replication Partition: 3 Leader: 1003 Replicas: 1003,1004,1005 Isr: 1003,1004,1005
  Topic: replication Partition: 4 Leader: 1004 Replicas: 1004,1005,1001 Isr: 1001,1004,1005
  Topic: replication Partition: 5 Leader: 1005 Replicas: 1005,1002,1003 Isr: 1003,1002,1005

Alter Partitions

 bin/kafka-topics.sh --zookeeper `hostname --long`:2181 --alter --topic file-local --partitions 20
 WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
 Adding partitions succeeded!

The partitions can only be increased

Links

https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
https://stackoverflow.com/questions/37960767/how-to-change-the-replicas-of-kafka-topic
https://stackoverflow.com/questions/33677871/is-it-possible-to-add-partitions-to-an-existing-topic-in-kafka-0-8-2

Add ACL

[root@kafka-1 kafka-broker]# /usr/iop/current/kafka-broker/bin/kafka-acls.sh --add --allow-host '*' --allow-principal 'User:CN=kafka-1.local,C=US' --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=`hostname --long`:2181 --topic fhir-local --group '*' --operation 'All'
 [2017-07-20 11:30:49,713] WARN read null data from /kafka-acl-changes/acl_changes_0000000420 when processing notification acl_changes_0000000420 (kafka.common.ZkNodeChangeNotificationListener)
 [2017-07-20 11:30:49,717] WARN read null data from /kafka-acl-changes/acl_changes_0000000421 when processing notification acl_changes_0000000421 (kafka.common.ZkNodeChangeNotificationListener)
 [2017-07-20 11:30:49,736] WARN read null data from /kafka-acl-changes/acl_changes_0000000422 when processing notification acl_changes_0000000422 (kafka.common.ZkNodeChangeNotificationListener)
 Adding ACLs for resource `Topic:fhir-local`:
  User:CN=kafka-1.local,C=US has Allow permission for operations: All from hosts: *

Adding ACLs for resource `Group:*`:
  User:CN=kafka-1.local,C=US has Allow permission for operations: All from hosts: *

Solution: Change Heap Options

TO change the heap options in KAFKA, Sset on the commandline or edit kafka-env.sh on CDP or IOP Add KAFKA_HEAP_OPTS="-Xmx64G -XMs64G"

Solution: Running IBM

To run Kafka with IBM JVM

Edit the <kafka_home>/bin-kafka-run-class.sh file.

Locate the following line:

KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps"

Solution: Update Logging for GC

To enable verbose logging, change it to:

KAFKA_GC_LOG_OPTS="-Xverbosegclog:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps"

Save the file.

Quick Method to see Kafka-Broker uptime

Here is a quick method to see Kafka-Broker uptime. lstart and etime are the actual start time and the actual elapsed time since start

[userid@kafka-server ~]$ ps -eo pid,comm,lstart,etime,time,args | \
    grep -i kafka | grep -v grep
9863 java Thu Nov 3 16:19:38 2017 05:25:55 00:09:55 
      /usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64/bin/j

Kafka, Zookeeper… and Kerberos

My team runs a Kafka service for data ingestion, we ran across a rare timeout when our main Key Distribution Center (KDC) went down. When the zookeeper service restarted, zookeeper worked flawlessly. I checked the services with the zookeeper four-letter commands. However, the kafka-broker/zookeeper startup and authentication failed and the brokers went down.

We checked each system with the following:

echo ruok | nc localhost 2181
iamok

echo stat | nc localhost 2181
Zookeeper version: 3.4.6-1569965, built on XXXXXX
Clients:
/10.10.10.10:3888[1](queued=0,recved=95261,sent=95261)

Latency min/avg/max: 0/0/316
Received: 1
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x2100000000
Mode: follower
Node count: 200

We checked zookeeper with Kerberos/JAAS using the shell.

export JVMFLAGS="-Djava.security.auth.login.config=/usr/iop/current/kafka-broker/config/kafka_jaas.conf"

zookeeper-client -server `hostname --long`:2181 ls /

You’ll see a failover to the secondary server after 90 seconds and the final list. This indicates that the server finally fails over the secondary KDC.

[root@kafka-1 ~]# export JVMFLAGS="-Djava.security.auth.login.config=/usr/iop/current/kafka-broker/config/kafka_jaas.conf"
[root@kafka-1 ~]# time zookeeper-client -server `hostname --long`:2181 ls /
Connecting to kafka-1:2181
2019-03-12 17:31:44,765 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-IBM_4--1, built on 06/17/2016 01:58 GMT
2019-03-12 17:31:44,767 - INFO [main:Environment@100] - Client environment:host.name=kafka-1
2019-03-12 17:31:44,767 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_77
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.home=/usr/jdk64/java-1.8.0-openjdk/jre
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/iop/4.2.0.0/zookeeper/bin/../build/classes:/usr/iop/4.2.0.0/zookeeper/bin/../build/lib/*.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/log4j-1.2.17.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/iop/4.2.0.0/zookeeper/bin/../zookeeper-3.4.6_IBM_4.jar:/usr/iop/4.2.0.0/zookeeper/bin/../src/java/lib/*.jar:/usr/iop/4.2.0.0/zookeeper/conf::/usr/iop/4.2.0.0/zookeeper/conf:/usr/iop/4.2.0.0/zookeeper/zookeeper-3.4.6_IBM_4.jar:/usr/iop/4.2.0.0/zookeeper/zookeeper.jar:/usr/iop/4.2.0.0/zookeeper/lib/jline-0.9.94.jar:/usr/iop/4.2.0.0/zookeeper/lib/log4j-1.2.17.jar:/usr/iop/4.2.0.0/zookeeper/lib/netty-3.7.0.Final.jar:/usr/iop/4.2.0.0/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/iop/4.2.0.0/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/share/zookeeper/*
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:os.version=3.10.0-514.21.1.el7.x86_64
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:user.name=root
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:user.home=/root
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:user.dir=/root
2019-03-12 17:31:44,771 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
2019-03-12 17:31:44,875 - INFO [main-SendThread(kafka-1:2181):Login@327] - successfully logged in.
2019-03-12 17:31:44,878 - INFO [Thread-0:Login$1@156] - TGT refresh thread started.
2019-03-12 17:31:44,882 - INFO [main-SendThread(kafka-1:2181):ZooKeeperSaslClient$1@285] - Client will use GSSAPI as SASL mechanism.
2019-03-12 17:31:44,892 - INFO [Thread-0:Login@335] - TGT valid starting at: Tue Mar 12 17:31:44 UTC 2019
2019-03-12 17:31:44,892 - INFO [Thread-0:Login@336] - TGT expires: Wed Mar 13 17:31:44 UTC 2019
2019-03-12 17:31:44,892 - INFO [Thread-0:Login$1@210] - TGT refresh sleeping until: Wed Mar 13 12:45:17 UTC 2019
2019-03-12 17:31:44,894 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@975] - Opening socket connection to server kafka-1/192.168.1.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-03-12 17:31:44,952 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@852] - Socket connection established to kafka-1/192.168.1.1:2181, initiating session
2019-03-12 17:31:44,966 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server kafka-1/192.168.1.1:2181, sessionid = 0x16972ce2b3f002b, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

WATCHER::

WatchedEvent state:SaslAuthenticated type:None path:null
[controller_epoch, controller, brokers, zookeeper, kafka-acl, kafka-acl-changes, admin, isr_change_notification, consumers, config]

real 0m1.328s
user 0m0.573s
sys 0m0.102s

We removed the down KDC in our /etc/krb5.conf file. (We eventually added it back when the server was restarted). When I executed the same command as above, we were able to return the system to operation and reduce the time to get a ticket and authorize our services on startup.

019-03-12 17:33:09,364 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
2019-03-12 17:33:09,460 - INFO [main-SendThread(kafka-1:2181):Login@327] - successfully logged in.
2019-03-12 17:33:09,462 - INFO [Thread-0:Login$1@156] - TGT refresh thread started.
2019-03-12 17:33:09,465 - INFO [main-SendThread(kafka-1:2181):ZooKeeperSaslClient$1@285] - Client will use GSSAPI as SASL mechanism.
2019-03-12 17:33:09,477 - INFO [Thread-0:Login@335] - TGT valid starting at: Tue Mar 12 17:33:09 UTC 2019
2019-03-12 17:33:09,478 - INFO [Thread-0:Login@336] - TGT expires: Wed Mar 13 17:33:09 UTC 2019
2019-03-12 17:33:09,478 - INFO [Thread-0:Login$1@210] - TGT refresh sleeping until: Wed Mar 13 13:27:51 UTC 2019
2019-03-12 17:33:09,479 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@975] - Opening socket connection to server kafka-1/192.168.1.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-03-12 17:33:09,536 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@852] - Socket connection established to kafka-1/192.168.1.1:2181, initiating session
2019-03-12 17:33:09,554 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server kafka-1/192.168.1.1:2181, sessionid = 0x16972ce2b3f002c, negotiated timeout = 30000

real 0m0.718s
user 0m0.573s
sys 0m0.102s

I hope this helps you get your service back up and working.

Kafka TIps