Paul Bastide

Tracking down RPM install dates/reasons/who

I had to find the date of an RPM install to track the lineage of an RPM. I found two very helpful commands. The “rpm -q basesystem –qf ‘%{installtime:date}\n'”, which I found thanks to StackExchange, was exceptionally helpful for placing the date/time of an installed RPM. The rpm -qi basesystem also provided some excellent additional details.

The RPM query returned the UTC time of the installed RPM

[root@vm ~]# rpm -q basesystem --qf '%{installtime:date}\n'
Thu 03 Mar 2016 04:22:55 PM UTC

To get further details on install times and histories, I can see the ordering with yum history list all. I am able to pinpoint who, installed and at what time. The lineage was critical to trace back to the automation which kicked off the installation. (Note: I use basesystem as an example, but it can be used to estimate the time the OS was installed).

[paul@vm ~]# yum history list all
Loaded plugins: search-disabled-repos
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
363 | System <unset> | 2019-01-09 19:53 | Install | 2
362 | System <unset> | 2019-01-07 19:10 | Erase | 2
361 | System <unset> | 2019-01-07 19:04 | Install | 1
...
3 | cloud | 2018-05-14 23:45 | Install | 1
2 | cloud | 2018-05-14 23:45 | O, U | 8
1 | cloud | 2018-05-14 23:45 | Install | 1

Jupyter Notebook: Email Analysis to a Lotus Notes View

I wanted to do an analysis of my emails since I joined IBM, and see the flow of messages in-and-out of my inbox.

With my preferences for Jupyter Notebooks, I built a small notebook for analysis.

Steps
Open IBM Lotus Notes Rich Client

Open the Notes Database with the View you want to analyze.

Select the View you are interested in ‘All Documents’. For instance the All Documents view, like my inbox *obfuscated* with a purpose.

Click File > Export

Enter a file name – email.csv

Select Format “Comma Separate Value”

Click Export

Upload the Notebook to your Jupyter server

Email Analysis.ipynb Download

The notebook is describes the flow through my process. If you encounter ValueError: (‘Unknown string format:’, ’12/10/2018ï»¿ 08:34 AM’), you can refer to https://stackoverflow.com/a/8562577/1873438

iconv -c -f utf-8 -t ascii email.csv > email.csv.clean

You can break the data into month-year-day analysis with the following, and peek the results with df_emailA.head()

When you run the final cell, the code generates a Year-Month-Day count as a bar graph.

    # Title: Volume in Months when emails are sent.
    # Plots volume based on known year-mm-dd
    # to be included in the list, one must have data in those years.
    # Kind is a bar graph, so that the (Y - YYYY,MM can be read)
    y_m_df = df_emailA.groupby(['year','month','day']).year.count()
    y_m_df.plot(kind="bar")

    plt.title('Numbers submitted By YYYY-MM-DD')
    plt.xlabel('Email Flow')
    plt.ylabel('Year-Month-Day')
    plt.autoscale(enable=True, axis='both', tight=False)
    plt.rcParams['figure.figsize'] = [20, 200]

You’ll see the trend of emails I receive over the years.

LSLogins Quick Summary

I needed a summary of my datastage admin id for analysis of a login problem. lslogins was super helpful. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-displaying_comprehensive_user_information

[root@myserver ~]# lslogins dsadm --time-format=iso
Username: dsadm
UID: 1120
Gecos field:
Home directory: /home/dsadm
Shell: /bin/bash
No login: no
Password is locked: no
Password not required: no
Login by password disabled: no
Primary group: dstage
GID: 1119
Supplementary groups: db2iadm
Supplementary group IDs: 1117
Hushed: no
Password expiration warn interval: 7
Password changed: 2018-12-31
Minimum change time: 7
Maximum change time: 90
Running processes: 3

Last logs:
2018-12-11T16:51:03+0000 su[7327]: pam_unix(su-l:session): session opened for user dsadm by myuserid(uid=1120)
2018-12-11T16:51:06+0000 su[7327]: pam_unix(su-l:session): session closed for user dsadm
2018-12-21T18:30:02+0000 crontab[25267]: (dsadm) AUTH (crontab command not allowed)

Datastage Randomly Locked out

Suddenly, my datastage pipeline stopped working. I hit this error:

DB2_Connector_2: [Input link 0] SQLConnect reported: SQLSTATE = 42724: Native Error Code = -10,013: Msg = [IBM][CLI Driver] SQL10013N The specified library "GSKit Error: 408" could not be loaded. SQLSTATE=42724 (CC_DB2Connection::connect, file CC_DB2Connection.cpp, line 856)

The error was due to permission change on our SSL credentials (p12/jks)

DSEngine Status Code 81016

I hit the following issue with the dsadmin

[dsadm@server-1 DSEngine]$ bin/dsadmin -listprojects
ERROR: Connection to the specified engine tier host failed or was refused. Check that the RPC daemon service is running on the host and that no firewall is blocking the connection
Status code = 81016

As I don’t own the server implementation, I had to check the services setup

cat /etc/services | grep -i dsrpc

You should see:

dsrpc 31538/tcp # RPCdaemon DSEngine@/opt/IBM/InformationServer/Server/DSEngine

netstat -plutn | grep -i 31538

the process wasn’t started and I restarted DSEngine

Stop the InfoSphere DataStage engine services by using the following command:

su - dsadm 
cd /opt/IBM/InformationServer/Server/DSEngine
source dsenv
bin/uv -admin -stop

Stopping JobMonApp
No running PX Yarn Client for dsadm found to stop.
JobMonApp has been shut down.
Stopping DSAppWatcher
AppWatcher:ALREADY_STOPPED
ResMonApp:STOPPING
ResMonApp:ALREADY_STOPPED
ODBQueryApp:STOPPING
ODBQueryApp:ALREADY_STOPPED
EngMonApp:STOPPING
EngMonApp:ALREADY_STOPPED
DataStage Engine <<VERSION>> instance "ade" has been brought down.

Wait 30 seconds so that the engine services stop.

Start the engine services by using the following command:

su - dsadm 
cd /opt/IBM/InformationServer/Server/DSEngine
source dsenv
bin/uv -admin -start

Checking NLS locale OFF
Checking NLS locale DEFAULT
Checking NLS locale CA-ENGLISH
Checking NLS locale GB-ENGLISH
Checking NLS locale IE-ENGLISH
Checking NLS locale AU-ENGLISH
Checking NLS locale ZA-ENGLISH
Checking NLS locale US-ENGLISH
Checking NLS locale NZ-ENGLISH
Loading NLS map file UNICODE
Loading NLS map file UTF8
Loading NLS map file MS1256-WIN2K-CS
Loading NLS map file ISO8859-2-CS
Loading NLS map file MNEMONICS
Loading NLS map file ISO8859-3
Loading NLS map file ISO8859-7+MARKS
Loading NLS map file MS949
Loading NLS map file ASCII
Loading NLS map file MS1255
Loading NLS map file PC437
Loading NLS map file ISO8859-5
Loading NLS map file ISO8859-4
Loading NLS map file MS949-CS
Loading NLS map file PC874-CS
Loading NLS map file ISO8859-10-CS
Loading NLS map file ISO8859-10+MARKS
Loading NLS map file ISO8859-9-CS
Loading NLS map file ISO8859-6+MARKS
Loading NLS map file ISO8859-4-CS
Loading NLS map file MS1252+MARKS
Loading NLS map file ISO8859-2+MARKS
Loading NLS map file ISO8859-10
Loading NLS map file MS1256-CS
Loading NLS map file ISO8859-8
Loading NLS map file MS1251
Loading NLS map file MS936
Loading NLS map file ISO8859-3+MARKS
Loading NLS map file MS936-CS
Loading NLS map file MS1256
Loading NLS map file ISO8859-5+MARKS
Loading NLS map file ISO8859-6-CS
Loading NLS map file MS932-CS
Loading NLS map file ISO8859-8+MARKS
Loading NLS map file MS1250-CS
Loading NLS map file MS1253
Loading NLS map file MS1256-WIN2K
Loading NLS map file ISO8859-9
Loading NLS map file TIS620-CS
Loading NLS map file MS932
Loading NLS map file MS1253-CS
Loading NLS map file ISO8859-2
Loading NLS map file MS1254
Loading NLS map file ISO8859-6
Loading NLS map file MS1252-CS
Loading NLS map file ISO8859-1-CS
Loading NLS map file MS1251-CS
Loading NLS map file PC850
Loading NLS map file MS1250
Loading NLS map file ISO8859-1+MARKS
Loading NLS map file ISO8859-1
Loading NLS map file MS1254-CS
Loading NLS map file MS950-CS
Loading NLS map file ISO8859-5-CS
Loading NLS map file ISO8859-3-CS
Loading NLS map file ISO8859-8-CS
Loading NLS map file MS1255-CS
Loading NLS map file ISO8859-15-CS
Loading NLS map file ISO8859-7
Loading NLS map file UTF8-CS
Loading NLS map file ISO8859-4+MARKS
Loading NLS map file MS1252
Loading NLS map file ISO8859-15
Loading NLS map file ISO8859-9+MARKS
Loading NLS map file ASCII+MARKS
Loading NLS map file ISO8859-7-CS
Loading NLS map file MS950
Loading NLS map file ISO8859-15+MARKS
68 NLS Character Set Maps loaded in 4055120 bytes.
Loading 9 NLS Locales
9 NLS Locales loaded in 793824 bytes
DataStage Engine <<VERSION>> instance "ade" has been brought up.
Starting JobMonApp
JobMonApp has been started.
resource_tracker has been started.
Starting DSAppWatcher

It was restarted after the stop/start.

Reference
http://www-01.ibm.com/support/docview.wss?uid=swg21402853
http://www-01.ibm.com/support/docview.wss?uid=swg21452589
https://www.ibm.com/support/knowledgecenter/en/SSZJPZ_8.7.0/com.ibm.swg.im.iis.productization.iisinfsv.migrate.doc/topics/a_start_stop_ds_server.html

Spark and Data Tips for November 2018

Full Hadoop / HBase data platform for testing spark

I found the following docker very handy for testing hadoop.
https://hub.docker.com/r/bigdatauniversity/spark2/
docker pull bigdatauniversity/spark2
docker run -it –name bdu_spark2 -P -p 4040:4040 -p 4041:4041 -p 8080:8080 -p 8081:8081 bigdatauniversity/spark2:latest /etc/bootstrap.sh -bash

Spark Notebooks

I found these sites useful – http://spark-notebook.io/ and https://github.com/spark-notebook/spark-notebook and https://github.com/IBM?language=jupyter+notebook

Version Mismatch

If you hit this error

Exception: Python in worker has different version 2.6 than that in driver 2.7, PySpark cannot run with different minor versions

It’s a quick fix

Update spark-env.sh 
export PYSPARK_PYTHON=/usr/local/bin/python3

Thanks to http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Python-in-worker-has-different-version-than-that-in-driver/td-p/60620

Kernels

A handy list of kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

I used the Python 3 kernel.

pip3 install spylon-kernel

I did also play with Toreee
https://github.com/apache/incubator-toree

pip3 install toree
jupyter toree install

UCD: Application Processes Branching

Urban Code Deploy (UCD) is a tool we use to manage the deployment of our healthcare platform. I needed to branch between two different processes, and the setup and steps to get branching done between Application processes was not clearly documented.

I built some custom bash logic to switch based on results:
#!/bin/bash
if [ -f /etc/SHORT_CIRCUIT ]
then
echo "result=YES"
else
echo "result=NO"
fi

I created a Post Processing Step

var exit = properties.get('exitCode');

scanner.register("result=", function(lineNumber, line) {
var value=line.replace("result=","");
properties.put("ShortCircuit",value);
});

if (exit == 0) {
properties.put('Status', 'Success');
}
else {
properties.put('Status', 'Failure');
}

scanner.scan();

the scanner matches the text output for a value that has result= in it. I highly suggest having this in your scanner, otherwise you might see ucd propertyValue=script content: which indicates that it did not find any result to set.

Once you get the value in the step, it’ll be available downstream once you set Process Properties. In this case, I pickup the property from the <STEP NAME><SLASH><PROPERTY NAME> and set it into ShortCircuitStatus.

I then use it to make a decision downstream in a switch statement. I do suggest not using a default path in the switch statement, it ensures there is a deterministic and expected outcome of the prior process. I used YES/NO in the switch statement.

References

https://developer.ibm.com/urbancode/2016/11/21/passing-a-property-from-one-step-to-another-with-groovy/
https://www.ibm.com/support/knowledgecenter/en/SS4GSP_7.0.1/com.ibm.udeploy.doc/topics/comp_postprocess_examples.html

Paul Bastide

Tracking down RPM install dates/reasons/who

Jupyter Notebook: Email Analysis to a Lotus Notes View

LSLogins Quick Summary

Datastage Randomly Locked out

DSEngine Status Code 81016

Spark and Data Tips for November 2018

Full Hadoop / HBase data platform for testing spark

Spark Notebooks

Version Mismatch

Kernels

UCD: Application Processes Branching

References

Maven Repository – Go Offline with dependencies

Helpful Tips – October 2018

XCode Tools shortcut

Solid OO Design

Extreme Presentation

Telemetry Reporting in Visual Studio Code

Maven Thread Speed Up

February 2019
M	T	W	T	F	S	S
« Jan
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28