Regaining Control of a System (Danger Will Robinson)

This technique was used to regain control of my system. The Kernel was hung something severe, turned out to be a problem with the network storage. I ran the magic key for the Immediate reboot the system, without unmounting or syncing filesystems.

cat 1 > /proc/sys/kernel/sysrq

Links

• Magic SysReq Key
• Wikipedia: Magic SysRq Key

Solution: Fix DB_RUNRECOVERY

Ran into this issue today, and a fast fix.

[root@xyz ~]# yum
mut_tas:172, pid: 16043, flag: 19
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
[root@xyz ~]#

Run the rpm --rebuilddb command

[root@xyz~]# rpm --rebuilddb
[root@xyz ~]# 

Confirm yum works

[root@xyz ~]# yum
Loaded plugins: search-disabled-repos
You need to give some command
Usage: yum [options] COMMAND

Solution: LSLogins Quick Summary

I needed a summary of my datastage admin id for analysis of a login problem. lslogins was super helpful.

[root@myserver ~]# lslogins dsadm --time-format=iso
Username: dsadm
UID: 1120
Gecos field:
Home directory: /home/dsadm
Shell: /bin/bash
No login: no
Password is locked: no
Password not required: no
Login by password disabled: no
Primary group: dstage
GID: 1119
Supplementary groups: db2iadm
Supplementary group IDs: 1117
Hushed: no
Password expiration warn interval: 7
Password changed: 2018-12-31
Minimum change time: 7
Maximum change time: 90
Running processes: 3

Last logs:
2018-12-11T16:51:03+0000 su[7327]: pam_unix(su-l:session): session opened for user dsadm by myuserid(uid=1120)
2018-12-11T16:51:06+0000 su[7327]: pam_unix(su-l:session): session closed for user dsadm
2018-12-21T18:30:02+0000 crontab[25267]: (dsadm) AUTH (crontab command not allowed)

Solution: VIM and viewing Unicode and ASCII

I cannot stress the dangers of copying data from Excel or HTML and assuming that it’s ASCII. For example U+0040 is the unicode version of @. We ingested the unicode version and couldn’t see why a JOIN condition on the data table wasn’t working.

I looked at the source FHIR DSTU2 JSON source and loaded in VIM and used the following trick:

set encoding=latin1

We ended up showing that our data table’s contents were different using:

SELECT HEX(RESOURCE_VALUE) FROM FHIR.DIM_GROUP
0A40 vs 40
References

Links

• VIM Hex

Solution: Check Process File Handles

I needed to check process file handles efficiently and compare against the settings in /etc/limits.conf and /etc/limits.d

for ID in $(ps -u db2inst1 --no-header | awk '{print $1}')
do
echo ""
ps -ef | grep -i $ID | grep -v grep
grep 'open files' /proc/${ID}/limits
ls /proc/$ID/fd/ | wc -l
done

Output

db2inst1 13607 13605 0 Mar28 ? 02:47:56 db2sysc 0
 Max open files 65534 65534 files
 141

db2inst1 13620 13605 0 Mar28 ? 00:00:00 db2vend (PD Vendor Process - 1) 0
 Max open files 4096 4096 files
 9
db2inst1 13626 13605 0 Mar28 ? 00:07:38 db2acd 0 ,0,0,0,1,0,0,0,0000,1,0,995cf0,14,1e014,2,0,1,41fc0,0x210000000,0x210000000,1600000,683b8008,2,10bb8018
 Max open files 4096 4096 files
 7

Links

• StackOverflow: Count Proc
• StackOverflow: Check ulimit

Tracking down RPM install dates/reasons/who

I had to find the date of an RPM install to track the lineage of an RPM. I found two very helpful commands. The rpm -q basesystem –qf ‘%{installtime:date}\n', which I found thanks to StackExchange, was exceptionally helpful for placing the date/time of an installed RPM. The rpm -qi basesystem also provided some excellent additional details.

The RPM query returned the UTC time of the installed RPM

[root@vm ~]# rpm -q basesystem --qf '%{installtime:date}\n'
Thu 03 Mar 2016 04:22:55 PM UTC

To get further details on install times and histories, I can see the ordering with yum history list all. I am able to pinpoint who, installed and at what time. The lineage was critical to trace back to the automation which kicked off the installation. (Note: I use basesystem as an example, but it can be used to estimate the time the OS was installed).

[paul@vm ~]# yum history list all
Loaded plugins: search-disabled-repos
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
363 | System <unset> | 2019-01-09 19:53 | Install | 2
362 | System <unset> | 2019-01-07 19:10 | Erase | 2
361 | System <unset> | 2019-01-07 19:04 | Install | 1
...
3 | cloud | 2018-05-14 23:45 | Install | 1
2 | cloud | 2018-05-14 23:45 | O, U | 8
1 | cloud | 2018-05-14 23:45 | Install | 1
```

<em># Solution: Fast Firewalld Setup\nHere are some fast commands for enabling firewalld on RHEL7 and CENTOS7. I had to enable these for a secondary control on a project I am on.\n\n1 – Check the Status of the Firewall on Each Machine and confirm firewalld is started. \n“`\nsystemctl status firewalld\n\n● firewalld.service – firewalld -\ndynamic firewall daemon\n \nLoaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled;\nvendor preset: enabled)\n\nActive: active (running) since Wed 2019-04-10 09:40:23 EDT; 3h 38min ago\n Docs: man:firewalld(1)\n Main PID: 2878 (firewalld)\nTasks: 2\n \nCGroup: /system.slice/firewalld.service\n └─2878 /usr/bin/python -Es\n/usr/sbin/firewalld \u002d\u002dnofork \u002d\u002dnopid\n“`\n\n2 – If the firewalld is stopped, enable and start the firewall. \n“` \n$ systemctl enable firewalld\n“` \n\n3 – Start firewalld\n“` \n$ systemctl start firewalld\n“` \n\n4 – Check the machines to see each have a public zone\n\n“` \n$ firewall-cmd \u002d\u002dget-active-zones\npublic\ninterfaces: \u003c\u003e\n“` \n \n5 – Add SSH \n“` \n$ firewall-cmd \u002d\u002dadd-service=ssh \u002d\u002dtimeout 15m \nWarning: ALREADY_ENABLED: ‘ssh’ already in ‘public’\nSuccess\n“` \n\n6 – Transition the Changes to Permanent \n\n“` \n$ firewall-cmd \u002d\u002druntime-to-permanent\nSuccess\n“` \n\n7 – Reload the Rules \n“` \n$ firewall-cmd \u002d\u002dreload\nsuccess\n“` \n\n8 – Check to see if the list of rules is saved\n\n“`\n$ firewall-cmd \u002d\u002dpermanent \u002d\u002dzone=public \u002d\u002dlist-rich-rules\nrule family=\u0022ipv4\u0022\nsource address=\u002210.173.166.48\u0022 accept\n\nrule family=\u0022ipv4\u0022\nsource address=\u002210.173.166.2\u0022 accept\n“` \n\nFirewall enabled. \n\n**Links**\n-

FirewallD https://firewalld.org/documentation/the-daemon-firewalld.html

Solution: VirtualBox Additions missing

If you see Kernel Headers not found for target kernel level when installing the VBoxLinuxAdditions.run “Please install them and execute”. This issue was found on centos. You’ll see the commands below:

yum install "kernel-devel-uname-r ==$(uname -r)"

Links

• Cannot Be Found

Download your Windows 7 Install ISO from your Microsoft Software Developer Network. The Download, in my case, is en_windows_7_ultimate_with_sp1_x64_dvd_u_677332.iso

Plugin a USB key you care nothing about (it’s write-once)

Type ls /dev/disk/by-label/ I see Transcend

Type findfs LABEL=Transcend I see /dev/sdb1

Type

dd \
if=/home/paul/Downloads/en_windows_7_ultimate_with_sp1_x64_dvd_u_677332.iso \ 
of=/dev/sdb

Note, if you see input/output error, you should redo the command

I then unplugged the USB Key and put it in my Windows Desktop machine to recover it.

Links

• Making Minimal Boot Media

top - 12:23:18 up 14 days,  1:25,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 270 total,   2 running, 268 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.3 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15980.3 total,   1289.8 free,    681.0 used,  14009000.5 buff/cache
MiB Swap:   4096.0 total,   3894.0 free,    202.0 used.  256.0 avail Mem