Maven Animal Sniffer Plugin

For the past few years, most of my personal and professional projects are built using Maven.  The dependency management and corresponding build lifecycle enable me to do some complex builds (for instance HBase 1.2.5 Client Jars).

Command

 mvn dependency:tree -f demo-app/pom.xml

Result

[INFO] demo.app:demo-app:jar:1.0-SNAPSHOT
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] \- org.apache.hbase:hbase:pom:1.2.5:compile
[INFO]    +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO]    \- log4j:log4j:jar:1.2.17:compile

When these dependencies change underlying the build change, the build and debugging is a pain. I ran into this plugin Maven Animal Sniffer Plugin. I did have to enable a specific configuration <configuration><includeJavaHome>false</includeJavaHome></configuration>

Command

mvn animal-sniffer:build -f demo-app/pom.xml 

Result

[INFO] --- animal-sniffer-maven-plugin:1.16:build (default-cli) @ demo-app ---
[INFO] Parsing signatures from com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1
[INFO] Parsing signatures from log4j:log4j:jar:1.2.17
[INFO] Parsing signatures from /Users/paulbastide/Downloads/animal/demo-app/target/classes
[INFO] Wrote signatures for 337 classes.

I can now store this along with my build. The generated file demo-app/target/demo-app-1.0-SNAPSHOT.signature can then be used to check plugin

Jenkinsfile Triggers

It tooks me far too long to get Jenkinsfile to stop overwriting my Triggers as noted in https://github.com/jenkinsci/gitlab-plugin/issues/692 . You’ll see

WARNING: The properties step will remove all JobPropertys currently configured in this job, either from the UI or from an earlier properties step.

References
https://github.com/jenkinsci/gitlab-plugin
https://github.com/jenkinsci/gitlab-plugin/blob/master/src/main/java/com/dabsquared/gitlabjenkins/GitLabPushTrigger.java
https://github.com/jenkinsci/gitlab-plugin/blob/master/src/main/resources/com/dabsquared/gitlabjenkins/GitLabPushTrigger/config.jelly
https://github.com/jenkinsci/job-dsl-plugin/blob/master/job-dsl-core/src/main/groovy/javaposse/jobdsl/dsl/helpers/triggers/GitLabTriggerContext.groovy
https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Trigger-runs
https://dev.to/pencillr/jenkins-pipelines-and-their-dirty-secrets-2
https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Parametrized-pipelines
https://issues.jenkins-ci.org/browse/JENKINS-45053
https://jenkins.io/doc/book/pipeline/syntax/#declarative-pipeline
https://jenkins.io/doc/pipeline/steps/
https://gitlab.switch.ch/etienne.dysli-metref/idpv3-mfa/commit/6d2974b1199fb2101f5f4299a974cac66a220080
https://github.com/jenkinsci/gitlab-plugin/issues/417

Using jjs to confirm issue with DatatypeConverter in WebSphere Liberty

I kept running into a funky ‘java.lang.NullPointerException’ with the WebSphere Liberty included DataValidator. To debug the issue, I used the jjs – nashorn engine

If you need, to figure out where the class is located

try {
Class c = Class.forName("javax.xml.bind.DatatypeConverter");
System.out.println("Location " + c.getProtectionDomain().getCodeSource().getLocation());
} catch (ClassNotFoundException e1) {
TODO Auto-generated catch block
e1.printStackTrace();
}

You’ll get output like this:

 
Location /opt/ibm/wlp/dev/api/spec/com.ibm.ws.javaee.jaxb.2.2_1.0.12.jar

Then you can look at the default JDK – Java SDK – 1.8

 
/opt/ibm/ibm-java-sdk-8.0-4.5/jre/bin/jjs 
jjs> javax.xml.bind.DatatypeConverter.printHexBinary("get".getBytes())
676574
jjs>

Then you can look at the specific jar with jdk Java SDK – 1.8 + Liberty Jaxb

 
/opt/ibm/ibm-java-sdk-8.0-4.5/jre/bin/jjs --dump-on-error -classpath "/opt/ibm/wlp/dev/api/spec/com.ibm.ws.javaee.jaxb.2.2_1.0.12.jar:." 
jjs> javax.xml.bind.DatatypeConverter.printHexBinary("get".getBytes())
java.lang.NullPointerException
jjs>

From this, I determined that I needed to update in “com.ibm.websphere.javaee.jaxb.2.2_1.0.20.jar”

Formatting JSON with VIM

I am working on an analytics project where we generate very complicated medical analysis and put it in a hierarchical data model.

{ "test" : { "test1" : "val1" } }

Open the JSON in vim and use python -m json.tool

:%!python -m json.tool

Results

{
    "test": {
        "test1": "val1"
    }
}

References
https://coderwall.com/p/faceag/format-json-in-vim
https://til.hashrocket.com/posts/ha0ci0pvkj-format-json-in-vim-with-jq (jq is another option, but… not always available on every system, python tends to work everywhere).

Co-Inventing Model with Gephi

I was curious who my frequent co-inventors were, so I downloaded Gephi. I downloaded my patent data from Google Patents and built a fun little model from CSV data (similar to the attached).

SOURCE WEIGHT TARGET
Matthew E. Broomhall 121 PAUL Bastide
Robert E. Loredo 119 PAUL Bastide
Fang Lu 60 PAUL Bastide
Alaa Abou Mahmoud 43 PAUL Bastide
Lisa Seacat Deluca 23 PAUL Bastide
Lydia M. Do 17 PAUL Bastide
Dale M. Schultz 13 PAUL Bastide
Andrew E. Davis 7 PAUL Bastide
Ralph E. LeBlanc 7 PAUL Bastide
Sean Callanan 7 PAUL Bastide
Donna K. Byron 5 PAUL Bastide
Sandra L. Kogan 5 PAUL Bastide
Asima Silva 4 PAUL Bastide
Aaron J. Quirk 3 PAUL Bastide
Aaron M. Cohen 3 PAUL Bastide
Daniel B. Harris 3 PAUL Bastide
Eric S. Portner 3 PAUL Bastide
John M. Boyer 3 PAUL Bastide
Michael L. Taylor 3 PAUL Bastide
Alexander Pikovsky 2 PAUL Bastide
Corville O. Allen 2 PAUL Bastide
Dana L. Price 2 PAUL Bastide
Eric M. Wilcox 2 PAUL Bastide
Jeffrey R. Hoy 2 PAUL Bastide
John A. Jacobson 2 PAUL Bastide
Kulvir S. Bhogal 2 PAUL Bastide
Liam Harpur 2 PAUL Bastide
Marco A. Vicente 2 PAUL Bastide
Patrick J. O’Sullivan 2 PAUL Bastide
Scott J. Martin 2 PAUL Bastide
Shane M. Kilmon 2 PAUL Bastide
Stephen Crawford 2 PAUL Bastide
Thomas J. Evans IV 2 PAUL Bastide
Vijay Francis 2 PAUL Bastide
Weisong Wang 2 PAUL Bastide
Adam L. Cutler 1 PAUL Bastide
Amanda N. Savitzky 1 PAUL Bastide
Andrew L. Schirmer 1 PAUL Bastide
Arun Vishwanath 1 PAUL Bastide
Bernadette A. Carter 1 PAUL Bastide
Beth Anne M. Collopy 1 PAUL Bastide
Beth L. Hoffman 1 PAUL Bastide
Bradley W. Hurley 1 PAUL Bastide
Brenton P. Chasse 1 PAUL Bastide
Brian M. Walsh 1 PAUL Bastide
Carl J. Kraenzel 1 PAUL Bastide
Christopher W. Desforges 1 PAUL Bastide
Damian E.A. Garcia 1 PAUL Bastide
Dan DUMONT 1 PAUL Bastide
Dwarikanath Mahapatra 1 PAUL Bastide
Fred Raguillat 1 PAUL Bastide
Isabell Kiral-Kornek 1 PAUL Bastide
Jaime M. Stockton 1 PAUL Bastide
James A. Hart 1 PAUL Bastide
Jennifer L. Vargus 1 PAUL Bastide
Jodi RAJANIEMI 1 PAUL Bastide
Jose L. Lopez 1 PAUL Bastide
Juliana M. Leong 1 PAUL Bastide
Katherine M. Parsons 1 PAUL Bastide
Kelley L. ANDERS 1 PAUL Bastide
King Shing K. Lui 1 PAUL Bastide
Leah A. Lawrence 1 PAUL Bastide
Leho Nigul 1 PAUL Bastide
Lei Wang 1 PAUL Bastide
Lorelei M. McCollum 1 PAUL Bastide
Margo L. Ezekiel 1 PAUL Bastide
Mark Gargan 1 PAUL Bastide
Mary E. Miller 1 PAUL Bastide
Matthew Stephen Rosno 1 PAUL Bastide
Melissa A. Lord 1 PAUL Bastide
Michael G. Alexander 1 PAUL Bastide
Na Pei 1 PAUL Bastide
Neal Fishman 1 PAUL Bastide
Pei Sun 1 PAUL Bastide
Richard Gorzela 1 PAUL Bastide
Richard T. Bassemir 1 PAUL Bastide
Shelbee D. Smith-Eigenbrode 1 PAUL Bastide
Shu Qiang Li 1 PAUL Bastide
Shunguo Yan 1 PAUL Bastide
Stacy M. Cannon 1 PAUL Bastide
Stanley K. Jerrard-Dunne 1 PAUL Bastide
Stefan von Cavallar 1 PAUL Bastide
Susmita Saha 1 PAUL Bastide
Tamer E. Abuelsaad 1 PAUL Bastide
Thomas J. Evans 1 PAUL Bastide
Trudy L. Hewitt 1 PAUL Bastide
Xujin Liu 1 PAUL Bastide
Ying Mo 1 PAUL Bastide
PAUL Bastide 0 PAUL Bastide
Graph of Co-Inventors

 

 

Urban Code Deploy: When a value doesn’t exist?

Urban Code Deploy: When a value doesn’t exist?

Recently, I ran into an issue with a resource referece in an Urban Code Deploy (UCD) resource that did not yet exist. When the value, that doesn’t yet exist I found this a great tip (UCD Documentation).

Change from
${p:resource/value-not-yet-populated}
to
${p?:resource/value-not-yet-populated}

UCD replaces the missing value with an empty string. It saved my sanity and ease of use.

BufferPool Error, Debug and Relevant Solution

My Java code kept hitting a buffer pool issue.

com.ibm.db2.jcc.am.SqlException: There are no pages currently available in bufferpool "4099".. SQLCODE=-1218, SQLSTATE=57011, DRIVER=4.19.49
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.ResultSet.completeSqlca(Unknown Source)
at com.ibm.db2.jcc.am.ResultSet.earlyCloseComplete(Unknown Source)
at com.ibm.db2.jcc.t4.bb.a(Unknown Source)
at com.ibm.db2.jcc.t4.bb.n(Unknown Source)
at com.ibm.db2.jcc.t4.bb.j(Unknown Source)
at com.ibm.db2.jcc.t4.bb.d(Unknown Source)
at com.ibm.db2.jcc.t4.p.c(Unknown Source)
at com.ibm.db2.jcc.t4.vb.j(Unknown Source)
at com.ibm.db2.jcc.am.fp.mb(Unknown Source)
at com.ibm.db2.jcc.am.gp.b(Unknown Source)
at com.ibm.db2.jcc.am.gp.kc(Unknown Source)
at com.ibm.db2.jcc.t4.j.kc(Unknown Source)
at com.ibm.db2.jcc.am.gp.executeQuery(Unknown Source)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeQuery(WSJdbcPreparedStatement.java:528)

I checked the db2diag file. The query was using the hidden bufferpools – Buffer Pool Documentation.  4099 is one of the hidden / default bufferpools.

2018-03-21-20.22.16.750579+000 E55613036E857 LEVEL: Warning
 PID : 23072 TID : 139860291610368 PROC : db2sysc 0
 INSTANCE: db2inst1 NODE : 000 DB : MYDB
 APPHDL : 0-9747 APPID: *LOCAL.db2inst1.180321202200
 AUTHID : DB2INST1 HOSTNAME: test.local
 EDUID : 158 EDUNAME: db2agent (MYDB) 0
 FUNCTION: DB2 UDB, buffer pool services, sqlbAssignBufferPool, probe:2
 MESSAGE : ADM6073W The table space "TBS_LDE" (ID "7") is configured to use
  buffer pool ID "2", but this buffer pool is not active at this time.
  In the interim the table space will use buffer pool ID "4099". The
  inactive buffer pool should become available at next database startup
  provided that the required memory is available.
2018-03-21-20.22.26.859427+000 E55613894E861 LEVEL: Warning
 PID : 23072 TID : 139860291610368 PROC : db2sysc 0
 INSTANCE: db2inst1 NODE : 000 DB : MYDB
 APPHDL : 0-9747 APPID: *LOCAL.db2inst1.180321202200
 AUTHID : DB2INST1 HOSTNAME: test.local
 EDUID : 158 EDUNAME: db2agent (MYDB) 0
 FUNCTION: DB2 UDB, buffer pool services, sqlbAssignBufferPool, probe:2
 MESSAGE : ADM6073W The table space "TBS_MY_IDX" (ID "8") is configured to use
  buffer pool ID "2", but this buffer pool is not active at this time.
  In the interim the table space will use buffer pool ID "4099". The
  inactive buffer pool should become available at next database startup
  provided that the required memory is available.

I ran a script to determine the status of the buffer pools, minimal hits in the bufferpools. IBMSYSTEMBP32K

 
A silly solution presented itself… restart the database instance, and the buffer pool was activated
 
https://www.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052485.html
https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10.5.0/com.ibm.db2.luw.sql.rtn.doc/doc/r0022013.html
https://www.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.perf.doc/doc/c0005395.html
https://www.ibm.com/support/knowledgecenter/en/SSEPGG_9.7.0/com.ibm.db2.luw.sql.rtn.doc/doc/r0053942.html
https://www.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052485.html

Generating Swagger as part of a Maven Build

Most of my projects use Maven to build and coordinate dependencies and run unit and build integration tests. I’ve found it a real pain that I could not generate my swagger docs as part of the build.

I have found a really easy way. In Jenkins, I have a downstream job that runs specific maven goals. The WebSphere Liberty maven goals are documented WebSphere Liberty Maven Documentation. Please note my downstream job is an experimental downstream project; I am a bit concerned about edge cases where the process dies.

The commands of the downstream job are:

mvn dependency:copy-dependencies liberty:install-apps liberty:test-start-server

The commands execute three key elements:

  1. Dummy Server.xml – attached at the bottom
  2. Execution for Goal for pre-integration-test
  3. Execution must follow the copy-dependencies so the war file is copied on in

You can check your server is up and running:

tail -f target/liberty/wlp/usr/servers/defaultServer/logs/messages.log

Download the Swagger (YAML)

curl http://localhost:9080/swagger/api/swagger.yaml -o myswagger.yaml

The full code is below