Kafka, Zookeeper… and Kerberos

My team runs a Kafka service for data ingestion, we ran across a rare timeout when our main Key Distribution Center (KDC) went down. When the zookeeper service restarted, zookeeper worked flawlessly. I checked the services with the zookeeper four-letter commands. However, the kafka-broker/zookeeper startup and authentication failed and the brokers went down. We checked […]

Datastage Randomly Locked out

Suddenly, my datastage pipeline stopped working. I hit this error: DB2_Connector_2: [Input link 0] SQLConnect reported: SQLSTATE = 42724: Native Error Code = -10,013: Msg = [IBM][CLI Driver] SQL10013N The specified library “GSKit Error: 408” could not be loaded. SQLSTATE=42724 (CC_DB2Connection::connect, file CC_DB2Connection.cpp, line 856) The error was due to permission change on our SSL […]

DSEngine Status Code 81016

I hit the following issue with the dsadmin [dsadm@server-1 DSEngine]$ bin/dsadmin -listprojects ERROR: Connection to the specified engine tier host failed or was refused. Check that the RPC daemon service is running on the host and that no firewall is blocking the connection Status code = 81016 As I don’t own the server implementation, I […]

Spark and Data Tips for November 2018

Full Hadoop / HBase data platform for testing spark I found the following docker very handy for testing hadoop. https://hub.docker.com/r/bigdatauniversity/spark2/ docker pull bigdatauniversity/spark2 docker run -it –name bdu_spark2 -P -p 4040:4040 -p 4041:4041 -p 8080:8080 -p 8081:8081 bigdatauniversity/spark2:latest /etc/bootstrap.sh -bash Spark Notebooks I found these sites useful – http://spark-notebook.io/ and https://github.com/spark-notebook/spark-notebook and https://github.com/IBM?language=jupyter+notebook Version Mismatch […]

UCD: Application Processes Branching

Urban Code Deploy (UCD) is a tool we use to manage the deployment of our healthcare platform. I needed to branch between two different processes, and the setup and steps to get branching done between Application processes was not clearly documented. I built some custom bash logic to switch based on results: #!/bin/bash if [ […]

Using jjs to confirm issue with DatatypeConverter in WebSphere Liberty

I kept running into a funky ‘java.lang.NullPointerException’ with the WebSphere Liberty included DataValidator. To debug the issue, I used the jjs – nashorn engine If you need, to figure out where the class is located try { Class c = Class.forName(“javax.xml.bind.DatatypeConverter”); System.out.println(“Location ” + c.getProtectionDomain().getCodeSource().getLocation()); } catch (ClassNotFoundException e1) { TODO Auto-generated catch block e1.printStackTrace(); […]

Deleting a bad changelog in Liquibase

1 – Identify the change set you need to remove (to run again) <changeSet author=“me” id=“2.1.0-x-table”> <sqlFile path=“/mycustomsql.sql” relativeToChangelogFile=“false” stripComments=“true” /> </changeSet> 2 – As your db instance owner, remove the change log entry [db2inst1@db myfolder]$ db2 “DELETE from myschema.DATABASECHANGELOG WHERE ID = ‘2.1.0-x-table’” DB20000I The SQL command completed successfully. 3 – Confirm the change […]

Logging using PQS lower than 4.6

To logging PQS, you can log using NGINX as a proxy that records the HTTP Request Body and the timing. On your Centos/RHEL machine, switch to root privileges sudo -s Add nginx.repo Install nginx packages, and you can setup a proxy yum install nginx.x86_64 -y Create the nginx configuration so you have a log_format that […]