As many developers know, HBase’s default client has everything, netting 10s of Megabytes of size. Lilyproject reduces this to a more manageable and useful size.
https://github.com/NGDATA/lilyproject/blob/master/global/hbase-client/pom.xml
<dependency>
<groupId>org.lilyproject</groupId>
<artifactId>lily-hbase-client</artifactId>
<version>2.6.1</version>
</dependency>
To Avoid thrashing with Lots and Lots of Files on the Mac, I had to refer to http://blog.hostilefork.com/trashes-fseventsd-and-spotlight-v100/
defaults write com.apple.desktopservices DSDontWriteNetworkStores true
My team and I run Ambari and Ambari agents which controls our HDFS/HBase and general HADOOP/Apache ecosystem machines. Our bare metal machines hung, and we could not get anything restarted.
In the logs, we had:
{'msg': 'Unable to read structured output from /var/lib/ambari-agent/data/structured-out-status.json'}
We found a link at https://community.hortonworks.com/content/supportkb/49517/services-are-running-but-ambari-reports-them-faile.html and the fix.
Our ambari and setup now works.
I cannot stress the dangers of copying data from Excel or HTML and assuming that it’s ASCII. For example U+0040 is the unicode version of @. We ingested the unicode version and couldn’t see why a JOIN condition on the data table wasn’t working.
I looked at the source JSON ( a FHIR DSTU2 Group ) and loaded in VIM and used the following trick:
set encoding=latin1We ended up showing that our data table’s contents were different using:
SELECT HEX(RESOURCE_VALUE) FROM FHIR.DIM_GROUP 0A40 vs 40
How to remove CTRL-M (^M) characters from a file in Linux
Removing the ^M character in VIM / VI
:%s/^M//g
Coalesce
When you have an unexpected bad data coalesce it. IF NULL, THEN the alternative in COALESCE
I had dupe data in my OLAP table, where the columnar data can be duplicated based on event id. (I loaded data 2x). I had to differentiate the data and remove the duplicates, so I assigned row_numbers over a partition ordered by.
I hope this helps you.
db2 "update (select OME.*, row_number() over(partition by IDN_EVENT_ID order by IDN_EVENT_ID) as rnk from X.OLAP OME) set APP_NM = rnk"
Then I removed using this.
db2 "DELETE X.OLAP OME WHERE APP_NM = 2"
I recommend the two-phase, as you can in theory run this en batch, or async, and double check, versus hope it works.
I lost my changelog tables and the lock table for Liquibase
I was able to Dump Change Log in Liquibase, and then load the resulting inserts into the changelog.
I ran into this issue with CISCO VPN and VirtualBox where the VBox is unable to resolve DNS on the VPN of the host
$ VBoxManage list vms
⁃ “Data Stage Server” {e94cc386-9ab9-476c-8dfa-fa877ffa0c8c}
$ VBoxManage modifyvm “Data Stage Server” –natdnshostresolver1 on
Recently, I’ve had to do some advanced HBase Shell scripting to check data consistency.
Tip # 1 – You can easily establish min/max times in nanoseconds from BASH and feed them into your script.
$(date +%s%6N)
gets you 6 digits of precision 3 for millis 3 for nanos.
Tip # 2 – Use the ‘include Java’ line to get access to all JARs that HBase has access to.
Tip # 3 – Forget about Bytes…. convert to Base64 to make it human readable (and you don’t want to break lines – the number 8 keeps it from wrapping). https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/util/Base64.html#DONT_BREAK_LINES
import org.apache.hadoop.hbase.util.Base64;
content = Bytes.toString(tableName.getValue(Bytes.toBytes("m"),Bytes.toBytes("d")))
x = Base64.encodeBytes(r.getRow(), 8)
puts "#{x}"
Tip # 4 – use GSON to parse JSON efficiently across a scan.
import com.google.gson.JsonParser;
parser = JsonParser.new
jsonBody = Bytes.toString(tableName.getValue(Bytes.toBytes("d"),Bytes.toBytes("b")))
json = parser.parse(jsonBody)
object = json.getAsJsonObject()
metaObj = object.get('mx')
objVer = metaObj.get('vid').getAsString()
objId = object.get('id').getAsString()
Tip #5 – use it as a script
time /usr/iop/current/hbase-client/bin/hbase org.jruby.Main /tmp/check.rb > check.log