Hadoop KMS Ranger API – Tips and cURLs

I use Hadoop KMS Ranger in one environment. Some sample rest api calls are below, along with two tips. versionName is used in multiple queries. When not using kerberos – set ?user.name=hdfs on the URL   References https://hadoop.apache.org/docs/current/hadoop-kms/index.html#KMS_HTTP_REST_API https://hadoop.apache.org/docs/current/hadoop-kms/index.html#Get_Key_Names https://stackoverflow.com/questions/37601763/authentication-issue-with-kms-hadoop

Lightweight HBase Client

As many developers know, HBase’s default client has everything, netting 10s of Megabytes of size. Lilyproject reduces this to a more manageable and useful size. https://github.com/NGDATA/lilyproject/blob/master/global/hbase-client/pom.xml <dependency> <groupId>org.lilyproject</groupId> <artifactId>lily-hbase-client</artifactId> <version>2.6.1</version> </dependency>

Ambari All Sorts of Messed Up

My team and I run Ambari and Ambari agents which controls our HDFS/HBase and general HADOOP/Apache ecosystem machines.  Our bare metal machines hung, and we could not get anything restarted. In the logs, we had: {‘msg’: ‘Unable to read structured output from /var/lib/ambari-agent/data/structured-out-status.json’} We found a link at https://community.hortonworks.com/content/supportkb/49517/services-are-running-but-ambari-reports-them-faile.html and the fix. Remove /var/lib/ambari-agent/data/structured-out-status.json Restart […]

VIM – JOIN Conditions with Unicode and ASCII

JOIN Conditions with Unicode and ASCII I cannot stress the dangers of copying data from Excel or HTML and assuming that it’s ASCII. For example U+0040 is the unicode version of @. We ingested the unicode version and couldn’t see why a JOIN condition on the data table wasn’t working. I looked at the source […]

Remove Duplicates in DB2 Columnar Format

I had dupe data in my OLAP table, where the columnar data can be duplicated based on event id. (I loaded data 2x). I had to differentiate the data and remove the duplicates, so I assigned row_numbers over a partition ordered by. I hope this helps you. db2 “update (select OME.*, row_number() over(partition by IDN_EVENT_ID […]