Two New Power User Hacks

https://serverfault.com/questions/283129/why-do-consoles-sometimes-hang-forever-when-ssh-connection-breaks
~. – terminate connection (and any multiplexed sessions)
~B – send a BREAK to the remote system
~C – open a command line
~R – request rekey
~V/v – decrease/increase verbosity (LogLevel)
~^Z – suspend ssh
~# – list forwarded connections
~& – background ssh (when waiting for connections to terminate)
~? – this message
~~ – send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

Tip #2. The other a really handy technique to regain control of my system. The Kernel was hung something severe, turned out to be a problem with the network storage. I ran the magic key for Immediately reboot the system, without unmounting or syncing filesystems 

https://major.io/2009/01/29/linux-emergency-reboot-or-shutdown-with-magic-commands/ 

https://en.wikipedia.org/wiki/Magic_SysRq_key

Cups and Raspberry Pi – How to setup your print server

Print Server is one of those things that makes life easier.  I hooked up my Raspberry Pi and a Docker image in order to have a CUPS server and the Bonjour configuration.

Login to the Raspberry Pi and switch to root (sudo -s or sudo su –)

Check lsusb to see if you can see the USB device.

 root@raspberrypi:~# lsusb 
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@raspberryp:~#

If you don’t see it, check to see that the device is properly plugged in.

Check again, and you’ll see your manufacture for your printer listed. (highlighted in red)

 root@raspberrypi:~# lsusb 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@raspberrypi:~#

Further, I am able to see the exact model that is connected:

 root@raspberrypi:~# lsusb -v | grep -A 20 Hewlett-Packard 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x03f0 Hewlett-Packard
  idProduct          0x7411
  bcdDevice            1.00
  iManufacturer           1 HP
  iProduct                2 Photosmart C4600 series
  iSerial                 3 |||MASKED|||MASKED|||MASKED|||

I’ll use these details downstream in order to install and use the HP driver.

If you still don’t see the USB device, you should check dmesg and look at the pattern for USB.

 [3077006.701281] usb 1-1.2: new high-speed USB device number 7 using dwc_otg
[3077006.831891] usb 1-1.2: New USB device found, idVendor=03f0, idProduct=7411
[3077006.831908] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[3077006.831917] usb 1-1.2: Product: Photosmart C4600 series
[3077006.831925] usb 1-1.2: Manufacturer: HP

If you don’t see it, unplug and plug the device back in.

You can also use usb-devices

 T:  Bus=01 Lev=02 Prnt=02 Port=01 Cnt=02 Dev#=  7 Spd=480 MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=03f0 ProdID=7411 Rev=01.00
S:  Manufacturer=HP
S:  Product=Photosmart C4600 series
S:  SerialNumber=<<>><<>>
C:  #Ifs= 4 Cfg#= 1 Atr=c0 MxPwr=2mA
I:  If#= 0 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=cc Prot=00 Driver=(none)
I:  If#= 1 Alt= 0 #EPs= 2 Cls=07(print) Sub=01 Prot=02 Driver=usblp
I:  If#= 2 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)
I:  If#= 3 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=usb-storage

Copy the Dockerfile and the cupsd file locally (in the folder etc-cups/cupsd.conf). Cupsd is configured to listen on all interfaces: Listen 0.0.0.0:631

Build the Docker image (based off of this image https://github.com/a0js/rpi-cups and uses a different base image https://hub.docker.com/r/arm32v7/debian/ ) docker build -t cups-pi/cups-pi .

 root@raspberrypi:/data/cups-printer# docker build -t cups-pi/cups-pi . 
Sending build context to Docker daemon  8.192kB
Step 1/9 : FROM arm32v7/debian:latest
 ---> 64b4748d266b
Step 2/9 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 38ebdf2f5bb6
Step 3/9 : RUN apt-get update && apt-get install -y   sudo   locales   whois   cups   cups-client   cups-bsd   printer-driver-all   hpijs-ppds   hp-ppd   hplip
 ---> Using cache
 ---> 0b5b9eac6cef
Step 4/9 : RUN sed -i "s/^#\ \+\(en_US.UTF-8\)/\1/" /etc/locale.gen && locale-gen en_US en_US.UTF-8
 ---> Using cache
 ---> fd55737901b8
Step 5/9 : ENV LANG=en_US.UTF-8   LC_ALL=en_US.UTF-8   LANGUAGE=en_US:en
 ---> Using cache
 ---> d6cd104b40ec
Step 6/9 : RUN useradd   --groups=sudo,lp,lpadmin   --create-home   --home-dir=/home/print   --shell=/bin/bash   --password=$(mkpasswd print)   print   && sed -i '/%sudo[[:space:]]/ s/ALL[[:space:]]*$/NOPASSWD:ALL/' /etc/sudoers   && apt-get clean   && rm -rf /var/lib/apt/lists/*   && mkdir /var/lib/apt/lists/partial
 ---> Using cache
 ---> 156dcd02e397
Step 7/9 : COPY etc-cups/cupsd.conf /etc/cups/cupsd.conf
 ---> e1624a96970e
Step 8/9 : EXPOSE 631
 ---> Running in a16705e0f3d9
Removing intermediate container a16705e0f3d9
 ---> 60758af63011
Step 9/9 : ENTRYPOINT ["/usr/sbin/cupsd", "-f"]
 ---> Running in add02d421ea9
Removing intermediate container add02d421ea9
 ---> 8c6fe42423c8
Successfully built 8c6fe42423c8
Successfully tagged cups-pi/cups-pi:ltest

Run the Docker image to start the print server

root@raspberrypi:/data/cups-printer# docker run -d -p 631:631 --privileged -v /var/run/dbus:/var/run/dbus -v /dev/bus/usb:/dev/bus/usb -h myhostm.mydomain.org --name cupsm cups-pi/cups-pi:latest
fe6d9bc34c66911f05b011ef185fce95947efb965e90ef2b4ecdd0f1c3a32d68

It’s important to use the -h if you want to remotely access via hostname.

Login to the console http://myserver.mydomain.org:631/admin using print:print

Click Add Printer

Select a local printer “HP Photosmart C4600 series (HP Photosmart C4600 series)”

Click Continue

Click Share This Printer

Click Continue

Click Add Printer

Select Media Type – Letter

Click Set Default Options

Click Add

When you find the Printer, you can add the printer

You now have an image ready for use with a CUPS printer.  (If you rebuild, you may have to re-setup the default printer…. Just warning you ahead of time).  You can always look at http://myhost.mydomain.org:631/printers/HP_Photosmart_C4600_series

If you need to connect to the console, you can look at the docker ps

root@raspberrypi:/data/cups-printer# docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                                    NAMES
fe6d9bc34c66        cups-pi/cups-pi:latest   "/usr/sbin/cupsd -f"   5 minutes ago       Up 5 minutes        0.0.0.0:631->631/tcp                     cups

Extract the CONTAINER ID, and look at the /bin/sh

root@raspberrypi:/data/cups-printer# docker exec
-i -t fe6d9bc34c66 /bin/sh

Then look at the /var/log/cups files, specifically the error_log

Reference

List USB Devices Linux https://linuxhint.com/list-usb-devices-linux/

Dockerfile https://github.com/a0js/rpi-cups https://github.com/ryansch/docker-cups-rpi https://github.com/aadl/docker-cups/blob/master/2.2.1/Dockerfile

Arm 7 https://hub.docker.com/r/arm32v7/debian/

Cupsd.conf https://raw.githubusercontent.com/a0js/rpi-cups/master/etc-cups/cupsd.conf https://wiki.archlinux.org/index.php/CUPS/Printer_sharing

SSL / Advanced Configuration http://chschneider.eu/linux/server/cups.shtml http://192.168.86.176:631/admin (use IP if receiving a bad request indicator to the admin interface)

Advanced Ubuntu Configuration https://help.ubuntu.com/lts/serverguide/cups.html.en

FirewallD

Fast commands for enabling firewalld on RHEL7 and CENTOS7. I had to enable these for a secondary control on a project I am on.

  1. Check the Status of the Firewall on Each Machine and confirm firewalld is started.
systemctl status firewalld
● firewalld.service - firewalld -
dynamic firewall daemon
  
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled;
vendor preset: enabled)
  
Active: active (running) since Wed 2019-04-10 09:40:23 EDT; 3h 38min ago
     Docs: man:firewalld(1)
 Main PID: 2878 (firewalld)
   
Tasks: 2
  
CGroup: /system.slice/firewalld.service
           └─2878 /usr/bin/python -Es
/usr/sbin/firewalld --nofork --nopid
  1. If the firewalld is stopped, enable and start the firewall.
$ systemctl enable firewalld
  1. Start firewalld
$ systemctl start firewalld
  1. Check the machines to see each have a public zone
$ firewall-cmd --get-active-zones

public

interfaces: <>
  1. Add SSH
firewall-cmd --add-service=ssh --timeout 15m

Warning: ALREADY_ENABLED: 'ssh' already in 'public'

Success
  1. Transition the Changes to Permanent
$ firewall-cmd
--runtime-to-permanent
Success
  1. Reload the Rules
$ firewall-cmd --reload
success
  1. Check to see if the list of rules is saved
$ firewall-cmd --permanent
--zone=public --list-rich-rules
rule family="ipv4"
source address="10.173.166.48" accept
rule family="ipv4"
source address="10.173.166.2" accept

References

https://firewalld.org/documentation/the-daemon-firewalld.html

VirtualBox Additions missing

If you see Kernel Headers not found for target kernel level when installing the VBoxLinuxAdditions.run “Please install them and execute”. This issue was found on centos. You’ll see the commands below: yum install “kernel-devel-uname-r ==$(uname -r)”

References

https://unix.stackexchange.com/questions/232147/your-kernel-headers-for-kernel-3-10-0-229-el7-x86-64-cannot-be-found

Check Process File Handles

I needed to check process file handles efficiently and compare against the settings in /etc/limits.conf and /etc/limits.d

db2inst1 13607 13605 0 Mar28 ? 02:47:56 db2sysc 0
Max open files 65534 65534 files
141

db2inst1 13620 13605 0 Mar28 ? 00:00:00 db2vend (PD Vendor Process - 1) 0
Max open files 4096 4096 files
9
db2inst1 13626 13605 0 Mar28 ? 00:07:38 db2acd 0 ,0,0,0,1,0,0,0,0000,1,0,995cf0,14,1e014,2,0,1,41fc0,0x210000000,0x210000000,1600000,683b8008,2,10bb8018
Max open files 4096 4096 files
7

References

https://stackoverflow.com/questions/21752067/counting-open-files-per-process

https://superuser.com/questions/810951/how-do-i-check-the-ulimit-for-another-user-and-change-open-files

Fix DB_RUNRECOVERY

Ran into this issue today, and a fast fix.

[root@xyz ~]# yum
mut_tas:172, pid: 16043, flag: 19
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 – (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
[root@xyz ~]#

rpm –rebuilddb

[root@xyz~]# rpm –rebuilddb
[root@xyz ~]# yum
Loaded plugins: search-disabled-repos
You need to give some command
Usage: yum [options] COMMAND

Kerberos and Java

I have worked on a kerberos smoke test for my team. I learned a few tips in the process.

The useTicketCache is a preferred use in case the java process dies while the KDC is down.

HBase Canary Testing runs on a kerberos enabled cluster using hbase canaryhttp://hbase.apache.org/book.html#trouble.client

If you are port forwarding over SSH, you’ll want to switch to tcp using this trick in your krb5.conf file. Thanks to IBM’s site, it’s an easy fix… https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/tsec_kerb_create_conf.html

A working example for Kerberos is as follows:

The site Kerberos Java site describes in detail how to build a kerberos client.

Forwarding DGram in node.js

For a project I am working on I needed to rewrite a DGram port. I moved the ports around and found a few quick tests.

Testing with NC

my-machine:~$ echo -n “data-message” | nc -v -4u -w1 localhost 88
found 0 associations
found 1 connections:
1: flags=82<CONNECTED,PREFERRED>
outif (null)
src 127.0.0.1 port 53862
dst 127.0.0.1 port 88
rank info not available
Connection to localhost port 88 [udp/radan-http] succeeded!

Rewriting incoming datagrams to another port

You can run the sample, and get the results as follows

server listening 0.0.0.0:88
server got: j��0����

COS and Hadoop FS issue

I ran into this issue with Python and IBM Cloud Object Storage.

Py4JJavaError: An error occurred while calling o34.parquet.
: java.io.IOException: No FileSystem for scheme: cos
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:643)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

I applied a quick fix: pyspark –packages com.ibm.stocator:stocator:1.0.24

https://github.com/ibm-watson-data-lab/ibmos2spark/tree/master/python

https://github.com/ibm-watson-data-lab/ibmos2spark/tree/master/python
https://blog.sicara.com/get-started-pyspark-jupyter-guide-tutorial-ae2fe84f594f
https://stackoverflow.com/questions/46011671/no-filesystem-for-scheme-cos