VIM – JOIN Conditions with Unicode and ASCII

JOIN Conditions with Unicode and ASCII

I cannot stress the dangers of copying data from Excel or HTML and assuming that it’s ASCII. For example U+0040 is the unicode version of @. We ingested the unicode version and couldn’t see why a JOIN condition on the data table wasn’t working.

I looked at the source JSON ( a FHIR DSTU2 Group ) and loaded in VIM and used the following trick:

set encoding=latin1

We ended up showing that our data table’s contents were different using:

0A40 vs 40


Remove Duplicates in DB2 Columnar Format

I had dupe data in my OLAP table, where the columnar data can be duplicated based on event id. (I loaded data 2x). I had to differentiate the data and remove the duplicates, so I assigned row_numbers over a partition ordered by.

I hope this helps you.
db2 "update (select OME.*, row_number() over(partition by IDN_EVENT_ID order by IDN_EVENT_ID) as rnk from X.OLAP OME) set APP_NM = rnk"

Then I removed using this.

I recommend the two-phase, as you can in theory run this en batch, or async, and double check, versus hope it works.

Advanced HBase Shell

Recently, I’ve had to do some advanced HBase Shell scripting to check data consistency.

Tip # 1 – You can easily establish min/max times in nanoseconds from BASH and feed them into your script.

  $(date +%s%6N) 

gets you 6 digits of precision 3 for millis 3 for nanos.

Tip # 2 – Use the ‘include Java’ line to get access to all JARs that HBase has access to.

Tip # 3 – Forget about Bytes…. convert to Base64 to make it human readable (and you don’t want to break lines – the number 8 keeps it from wrapping).

import org.apache.hadoop.hbase.util.Base64;
content = Bytes.toString(tableName.getValue(Bytes.toBytes("m"),Bytes.toBytes("d")))
x = Base64.encodeBytes(r.getRow(), 8)
puts "#{x}"

Tip # 4 – use GSON to parse JSON efficiently across a scan.

parser =
jsonBody = Bytes.toString(tableName.getValue(Bytes.toBytes("d"),Bytes.toBytes("b")))

json = parser.parse(jsonBody)
object = json.getAsJsonObject()
metaObj = object.get('mx')
objVer = metaObj.get('vid').getAsString()
objId = object.get('id').getAsString()

Tip #5 – use it as a script

time /usr/iop/current/hbase-client/bin/hbase org.jruby.Main /tmp/check.rb > check.log

Maven Animal Sniffer Plugin

For the past few years, most of my personal and professional projects are built using Maven.  The dependency management and corresponding build lifecycle enable me to do some complex builds (for instance HBase 1.2.5 Client Jars).


 mvn dependency:tree -f demo-app/pom.xml


[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] \- org.apache.hbase:hbase:pom:1.2.5:compile
[INFO]    +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO]    \- log4j:log4j:jar:1.2.17:compile

When these dependencies change underlying the build change, the build and debugging is a pain. I ran into this plugin Maven Animal Sniffer Plugin. I did have to enable a specific configuration <configuration><includeJavaHome>false</includeJavaHome></configuration>


mvn animal-sniffer:build -f demo-app/pom.xml 


[INFO] --- animal-sniffer-maven-plugin:1.16:build (default-cli) @ demo-app ---
[INFO] Parsing signatures from com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1
[INFO] Parsing signatures from log4j:log4j:jar:1.2.17
[INFO] Parsing signatures from /Users/paulbastide/Downloads/animal/demo-app/target/classes
[INFO] Wrote signatures for 337 classes.

I can now store this along with my build. The generated file demo-app/target/demo-app-1.0-SNAPSHOT.signature can then be used to check plugin

Jenkinsfile Triggers

It tooks me far too long to get Jenkinsfile to stop overwriting my Triggers as noted in . You’ll see

WARNING: The properties step will remove all JobPropertys currently configured in this job, either from the UI or from an earlier properties step.


Using jjs to confirm issue with DatatypeConverter in WebSphere Liberty

I kept running into a funky ‘java.lang.NullPointerException’ with the WebSphere Liberty included DataValidator. To debug the issue, I used the jjs – nashorn engine

If you need, to figure out where the class is located

try {
Class c = Class.forName("javax.xml.bind.DatatypeConverter");
System.out.println("Location " + c.getProtectionDomain().getCodeSource().getLocation());
} catch (ClassNotFoundException e1) {
TODO Auto-generated catch block

You’ll get output like this:

Location /opt/ibm/wlp/dev/api/spec/

Then you can look at the default JDK – Java SDK – 1.8

jjs> javax.xml.bind.DatatypeConverter.printHexBinary("get".getBytes())

Then you can look at the specific jar with jdk Java SDK – 1.8 + Liberty Jaxb

/opt/ibm/ibm-java-sdk-8.0-4.5/jre/bin/jjs --dump-on-error -classpath "/opt/ibm/wlp/dev/api/spec/" 
jjs> javax.xml.bind.DatatypeConverter.printHexBinary("get".getBytes())

From this, I determined that I needed to update in “”