Two New Power User Hacks

https://serverfault.com/questions/283129/why-do-consoles-sometimes-hang-forever-when-ssh-connection-breaks
~. – terminate connection (and any multiplexed sessions)
~B – send a BREAK to the remote system
~C – open a command line
~R – request rekey
~V/v – decrease/increase verbosity (LogLevel)
~^Z – suspend ssh
~# – list forwarded connections
~& – background ssh (when waiting for connections to terminate)
~? – this message
~~ – send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

Tip #2. The other a really handy technique to regain control of my system. The Kernel was hung something severe, turned out to be a problem with the network storage. I ran the magic key for Immediately reboot the system, without unmounting or syncing filesystems 

https://major.io/2009/01/29/linux-emergency-reboot-or-shutdown-with-magic-commands/ 

https://en.wikipedia.org/wiki/Magic_SysRq_key

Cups and Raspberry Pi – How to setup your print server

Print Server is one of those things that makes life easier.  I hooked up my Raspberry Pi and a Docker image in order to have a CUPS server and the Bonjour configuration.

Login to the Raspberry Pi and switch to root (sudo -s or sudo su –)

Check lsusb to see if you can see the USB device.

 root@raspberrypi:~# lsusb 
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@raspberryp:~#

If you don’t see it, check to see that the device is properly plugged in.

Check again, and you’ll see your manufacture for your printer listed. (highlighted in red)

 root@raspberrypi:~# lsusb 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@raspberrypi:~#

Further, I am able to see the exact model that is connected:

 root@raspberrypi:~# lsusb -v | grep -A 20 Hewlett-Packard 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x03f0 Hewlett-Packard
  idProduct          0x7411
  bcdDevice            1.00
  iManufacturer           1 HP
  iProduct                2 Photosmart C4600 series
  iSerial                 3 |||MASKED|||MASKED|||MASKED|||

I’ll use these details downstream in order to install and use the HP driver.

If you still don’t see the USB device, you should check dmesg and look at the pattern for USB.

 [3077006.701281] usb 1-1.2: new high-speed USB device number 7 using dwc_otg
[3077006.831891] usb 1-1.2: New USB device found, idVendor=03f0, idProduct=7411
[3077006.831908] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[3077006.831917] usb 1-1.2: Product: Photosmart C4600 series
[3077006.831925] usb 1-1.2: Manufacturer: HP

If you don’t see it, unplug and plug the device back in.

You can also use usb-devices

 T:  Bus=01 Lev=02 Prnt=02 Port=01 Cnt=02 Dev#=  7 Spd=480 MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=03f0 ProdID=7411 Rev=01.00
S:  Manufacturer=HP
S:  Product=Photosmart C4600 series
S:  SerialNumber=<<>><<>>
C:  #Ifs= 4 Cfg#= 1 Atr=c0 MxPwr=2mA
I:  If#= 0 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=cc Prot=00 Driver=(none)
I:  If#= 1 Alt= 0 #EPs= 2 Cls=07(print) Sub=01 Prot=02 Driver=usblp
I:  If#= 2 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)
I:  If#= 3 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=usb-storage

Copy the Dockerfile and the cupsd file locally (in the folder etc-cups/cupsd.conf). Cupsd is configured to listen on all interfaces: Listen 0.0.0.0:631

Build the Docker image (based off of this image https://github.com/a0js/rpi-cups and uses a different base image https://hub.docker.com/r/arm32v7/debian/ ) docker build -t cups-pi/cups-pi .

 root@raspberrypi:/data/cups-printer# docker build -t cups-pi/cups-pi . 
Sending build context to Docker daemon  8.192kB
Step 1/9 : FROM arm32v7/debian:latest
 ---> 64b4748d266b
Step 2/9 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 38ebdf2f5bb6
Step 3/9 : RUN apt-get update && apt-get install -y   sudo   locales   whois   cups   cups-client   cups-bsd   printer-driver-all   hpijs-ppds   hp-ppd   hplip
 ---> Using cache
 ---> 0b5b9eac6cef
Step 4/9 : RUN sed -i "s/^#\ \+\(en_US.UTF-8\)/\1/" /etc/locale.gen && locale-gen en_US en_US.UTF-8
 ---> Using cache
 ---> fd55737901b8
Step 5/9 : ENV LANG=en_US.UTF-8   LC_ALL=en_US.UTF-8   LANGUAGE=en_US:en
 ---> Using cache
 ---> d6cd104b40ec
Step 6/9 : RUN useradd   --groups=sudo,lp,lpadmin   --create-home   --home-dir=/home/print   --shell=/bin/bash   --password=$(mkpasswd print)   print   && sed -i '/%sudo[[:space:]]/ s/ALL[[:space:]]*$/NOPASSWD:ALL/' /etc/sudoers   && apt-get clean   && rm -rf /var/lib/apt/lists/*   && mkdir /var/lib/apt/lists/partial
 ---> Using cache
 ---> 156dcd02e397
Step 7/9 : COPY etc-cups/cupsd.conf /etc/cups/cupsd.conf
 ---> e1624a96970e
Step 8/9 : EXPOSE 631
 ---> Running in a16705e0f3d9
Removing intermediate container a16705e0f3d9
 ---> 60758af63011
Step 9/9 : ENTRYPOINT ["/usr/sbin/cupsd", "-f"]
 ---> Running in add02d421ea9
Removing intermediate container add02d421ea9
 ---> 8c6fe42423c8
Successfully built 8c6fe42423c8
Successfully tagged cups-pi/cups-pi:ltest

Run the Docker image to start the print server

root@raspberrypi:/data/cups-printer# docker run -d -p 631:631 --privileged -v /var/run/dbus:/var/run/dbus -v /dev/bus/usb:/dev/bus/usb -h myhostm.mydomain.org --name cupsm cups-pi/cups-pi:latest
fe6d9bc34c66911f05b011ef185fce95947efb965e90ef2b4ecdd0f1c3a32d68

It’s important to use the -h if you want to remotely access via hostname.

Login to the console http://myserver.mydomain.org:631/admin using print:print

Click Add Printer

Select a local printer “HP Photosmart C4600 series (HP Photosmart C4600 series)”

Click Continue

Click Share This Printer

Click Continue

Click Add Printer

Select Media Type – Letter

Click Set Default Options

Click Add

When you find the Printer, you can add the printer

You now have an image ready for use with a CUPS printer.  (If you rebuild, you may have to re-setup the default printer…. Just warning you ahead of time).  You can always look at http://myhost.mydomain.org:631/printers/HP_Photosmart_C4600_series

If you need to connect to the console, you can look at the docker ps

root@raspberrypi:/data/cups-printer# docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                                    NAMES
fe6d9bc34c66        cups-pi/cups-pi:latest   "/usr/sbin/cupsd -f"   5 minutes ago       Up 5 minutes        0.0.0.0:631->631/tcp                     cups

Extract the CONTAINER ID, and look at the /bin/sh

root@raspberrypi:/data/cups-printer# docker exec
-i -t fe6d9bc34c66 /bin/sh

Then look at the /var/log/cups files, specifically the error_log

Reference

List USB Devices Linux https://linuxhint.com/list-usb-devices-linux/

Dockerfile https://github.com/a0js/rpi-cups https://github.com/ryansch/docker-cups-rpi https://github.com/aadl/docker-cups/blob/master/2.2.1/Dockerfile

Arm 7 https://hub.docker.com/r/arm32v7/debian/

Cupsd.conf https://raw.githubusercontent.com/a0js/rpi-cups/master/etc-cups/cupsd.conf https://wiki.archlinux.org/index.php/CUPS/Printer_sharing

SSL / Advanced Configuration http://chschneider.eu/linux/server/cups.shtml http://192.168.86.176:631/admin (use IP if receiving a bad request indicator to the admin interface)

Advanced Ubuntu Configuration https://help.ubuntu.com/lts/serverguide/cups.html.en

FirewallD

Fast commands for enabling firewalld on RHEL7 and CENTOS7. I had to enable these for a secondary control on a project I am on.

  1. Check the Status of the Firewall on Each Machine and confirm firewalld is started.
systemctl status firewalld
● firewalld.service - firewalld -
dynamic firewall daemon
  
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled;
vendor preset: enabled)
  
Active: active (running) since Wed 2019-04-10 09:40:23 EDT; 3h 38min ago
     Docs: man:firewalld(1)
 Main PID: 2878 (firewalld)
   
Tasks: 2
  
CGroup: /system.slice/firewalld.service
           └─2878 /usr/bin/python -Es
/usr/sbin/firewalld --nofork --nopid
  1. If the firewalld is stopped, enable and start the firewall.
$ systemctl enable firewalld
  1. Start firewalld
$ systemctl start firewalld
  1. Check the machines to see each have a public zone
$ firewall-cmd --get-active-zones

public

interfaces: <>
  1. Add SSH
firewall-cmd --add-service=ssh --timeout 15m

Warning: ALREADY_ENABLED: 'ssh' already in 'public'

Success
  1. Transition the Changes to Permanent
$ firewall-cmd
--runtime-to-permanent
Success
  1. Reload the Rules
$ firewall-cmd --reload
success
  1. Check to see if the list of rules is saved
$ firewall-cmd --permanent
--zone=public --list-rich-rules
rule family="ipv4"
source address="10.173.166.48" accept
rule family="ipv4"
source address="10.173.166.2" accept

References

https://firewalld.org/documentation/the-daemon-firewalld.html

VirtualBox Additions missing

If you see Kernel Headers not found for target kernel level when installing the VBoxLinuxAdditions.run “Please install them and execute”. This issue was found on centos. You’ll see the commands below: yum install “kernel-devel-uname-r ==$(uname -r)”

References

https://unix.stackexchange.com/questions/232147/your-kernel-headers-for-kernel-3-10-0-229-el7-x86-64-cannot-be-found

Check Process File Handles

I needed to check process file handles efficiently and compare against the settings in /etc/limits.conf and /etc/limits.d

db2inst1 13607 13605 0 Mar28 ? 02:47:56 db2sysc 0
Max open files 65534 65534 files
141

db2inst1 13620 13605 0 Mar28 ? 00:00:00 db2vend (PD Vendor Process - 1) 0
Max open files 4096 4096 files
9
db2inst1 13626 13605 0 Mar28 ? 00:07:38 db2acd 0 ,0,0,0,1,0,0,0,0000,1,0,995cf0,14,1e014,2,0,1,41fc0,0x210000000,0x210000000,1600000,683b8008,2,10bb8018
Max open files 4096 4096 files
7

References

https://stackoverflow.com/questions/21752067/counting-open-files-per-process

https://superuser.com/questions/810951/how-do-i-check-the-ulimit-for-another-user-and-change-open-files

Fix DB_RUNRECOVERY

Ran into this issue today, and a fast fix.

[root@xyz ~]# yum
mut_tas:172, pid: 16043, flag: 19
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 – (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
[root@xyz ~]#

rpm –rebuilddb

[root@xyz~]# rpm –rebuilddb
[root@xyz ~]# yum
Loaded plugins: search-disabled-repos
You need to give some command
Usage: yum [options] COMMAND

Kerberos and Java

I have worked on a kerberos smoke test for my team. I learned a few tips in the process.

The useTicketCache is a preferred use in case the java process dies while the KDC is down.

HBase Canary Testing runs on a kerberos enabled cluster using hbase canaryhttp://hbase.apache.org/book.html#trouble.client

If you are port forwarding over SSH, you’ll want to switch to tcp using this trick in your krb5.conf file. Thanks to IBM’s site, it’s an easy fix… https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/tsec_kerb_create_conf.html

A working example for Kerberos is as follows:

The site Kerberos Java site describes in detail how to build a kerberos client.

Forwarding DGram in node.js

For a project I am working on I needed to rewrite a DGram port. I moved the ports around and found a few quick tests.

Testing with NC

my-machine:~$ echo -n “data-message” | nc -v -4u -w1 localhost 88
found 0 associations
found 1 connections:
1: flags=82<CONNECTED,PREFERRED>
outif (null)
src 127.0.0.1 port 53862
dst 127.0.0.1 port 88
rank info not available
Connection to localhost port 88 [udp/radan-http] succeeded!

Rewriting incoming datagrams to another port

You can run the sample, and get the results as follows

server listening 0.0.0.0:88
server got: j��0����

COS and Hadoop FS issue

I ran into this issue with Python and IBM Cloud Object Storage.

Py4JJavaError: An error occurred while calling o34.parquet.
: java.io.IOException: No FileSystem for scheme: cos
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:643)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

I applied a quick fix: pyspark –packages com.ibm.stocator:stocator:1.0.24

https://github.com/ibm-watson-data-lab/ibmos2spark/tree/master/python

https://github.com/ibm-watson-data-lab/ibmos2spark/tree/master/python
https://blog.sicara.com/get-started-pyspark-jupyter-guide-tutorial-ae2fe84f594f
https://stackoverflow.com/questions/46011671/no-filesystem-for-scheme-cos



Kafka, Zookeeper… and Kerberos

My team runs a Kafka service for data ingestion, we ran across a rare timeout when our main Key Distribution Center (KDC) went down. When the zookeeper service restarted, zookeeper worked flawlessly. I checked the services with the zookeeper four-letter commands. However, the kafka-broker/zookeeper startup and authentication failed and the brokers went down.

We checked each system with the following:

echo ruok | nc localhost 2181
iamok

echo stat | nc localhost 2181
Zookeeper version: 3.4.6-1569965, built on XXXXXX
Clients:
/10.10.10.10:3888[1](queued=0,recved=95261,sent=95261)

Latency min/avg/max: 0/0/316
Received: 1
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x2100000000
Mode: follower
Node count: 200

We checked zookeeper with Kerberos/JAAS using the shell.

export JVMFLAGS="-Djava.security.auth.login.config=/usr/iop/current/kafka-broker/config/kafka_jaas.conf"

zookeeper-client -server `hostname --long`:2181 ls /

You’ll see a failover to the secondary server after 90 seconds and the final list. This indicates that the server finally fails over the secondary KDC.

[root@kafka-1 ~]# export JVMFLAGS="-Djava.security.auth.login.config=/usr/iop/current/kafka-broker/config/kafka_jaas.conf"
[root@kafka-1 ~]# time zookeeper-client -server `hostname --long`:2181 ls /
Connecting to kafka-1:2181
2019-03-12 17:31:44,765 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-IBM_4--1, built on 06/17/2016 01:58 GMT
2019-03-12 17:31:44,767 - INFO [main:Environment@100] - Client environment:host.name=kafka-1
2019-03-12 17:31:44,767 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_77
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.home=/usr/jdk64/java-1.8.0-openjdk/jre
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/iop/4.2.0.0/zookeeper/bin/../build/classes:/usr/iop/4.2.0.0/zookeeper/bin/../build/lib/*.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/log4j-1.2.17.jar:/usr/iop/4.2.0.0/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/iop/4.2.0.0/zookeeper/bin/../zookeeper-3.4.6_IBM_4.jar:/usr/iop/4.2.0.0/zookeeper/bin/../src/java/lib/*.jar:/usr/iop/4.2.0.0/zookeeper/conf::/usr/iop/4.2.0.0/zookeeper/conf:/usr/iop/4.2.0.0/zookeeper/zookeeper-3.4.6_IBM_4.jar:/usr/iop/4.2.0.0/zookeeper/zookeeper.jar:/usr/iop/4.2.0.0/zookeeper/lib/jline-0.9.94.jar:/usr/iop/4.2.0.0/zookeeper/lib/log4j-1.2.17.jar:/usr/iop/4.2.0.0/zookeeper/lib/netty-3.7.0.Final.jar:/usr/iop/4.2.0.0/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/iop/4.2.0.0/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/share/zookeeper/*
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-03-12 17:31:44,769 - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:os.version=3.10.0-514.21.1.el7.x86_64
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:user.name=root
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:user.home=/root
2019-03-12 17:31:44,770 - INFO [main:Environment@100] - Client environment:user.dir=/root
2019-03-12 17:31:44,771 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
2019-03-12 17:31:44,875 - INFO [main-SendThread(kafka-1:2181):Login@327] - successfully logged in.
2019-03-12 17:31:44,878 - INFO [Thread-0:Login$1@156] - TGT refresh thread started.
2019-03-12 17:31:44,882 - INFO [main-SendThread(kafka-1:2181):ZooKeeperSaslClient$1@285] - Client will use GSSAPI as SASL mechanism.
2019-03-12 17:31:44,892 - INFO [Thread-0:Login@335] - TGT valid starting at: Tue Mar 12 17:31:44 UTC 2019
2019-03-12 17:31:44,892 - INFO [Thread-0:Login@336] - TGT expires: Wed Mar 13 17:31:44 UTC 2019
2019-03-12 17:31:44,892 - INFO [Thread-0:Login$1@210] - TGT refresh sleeping until: Wed Mar 13 12:45:17 UTC 2019
2019-03-12 17:31:44,894 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@975] - Opening socket connection to server kafka-1/192.168.1.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-03-12 17:31:44,952 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@852] - Socket connection established to kafka-1/192.168.1.1:2181, initiating session
2019-03-12 17:31:44,966 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server kafka-1/192.168.1.1:2181, sessionid = 0x16972ce2b3f002b, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

WATCHER::

WatchedEvent state:SaslAuthenticated type:None path:null
[controller_epoch, controller, brokers, zookeeper, kafka-acl, kafka-acl-changes, admin, isr_change_notification, consumers, config]

real 0m1.328s
user 0m0.573s
sys 0m0.102s

We removed the down KDC in our /etc/krb5.conf file. (We eventually added it back when the server was restarted). When I executed the same command as above, we were able to return the system to operation and reduce the time to get a ticket and authorize our services on startup.

019-03-12 17:33:09,364 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
2019-03-12 17:33:09,460 - INFO [main-SendThread(kafka-1:2181):Login@327] - successfully logged in.
2019-03-12 17:33:09,462 - INFO [Thread-0:Login$1@156] - TGT refresh thread started.
2019-03-12 17:33:09,465 - INFO [main-SendThread(kafka-1:2181):ZooKeeperSaslClient$1@285] - Client will use GSSAPI as SASL mechanism.
2019-03-12 17:33:09,477 - INFO [Thread-0:Login@335] - TGT valid starting at: Tue Mar 12 17:33:09 UTC 2019
2019-03-12 17:33:09,478 - INFO [Thread-0:Login@336] - TGT expires: Wed Mar 13 17:33:09 UTC 2019
2019-03-12 17:33:09,478 - INFO [Thread-0:Login$1@210] - TGT refresh sleeping until: Wed Mar 13 13:27:51 UTC 2019
2019-03-12 17:33:09,479 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@975] - Opening socket connection to server kafka-1/192.168.1.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-03-12 17:33:09,536 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@852] - Socket connection established to kafka-1/192.168.1.1:2181, initiating session
2019-03-12 17:33:09,554 - INFO [main-SendThread(kafka-1:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server kafka-1/192.168.1.1:2181, sessionid = 0x16972ce2b3f002c, negotiated timeout = 30000

real 0m0.718s
user 0m0.573s
sys 0m0.102s

I hope this helps you get your service back up and working.