COS and Hadoop FS issue

I ran into this issue with Python and IBM Cloud Object Storage. I applied a quick fix: pyspark –packages com.ibm.stocator:stocator:1.0.24 https://github.com/ibm-watson-data-lab/ibmos2spark/tree/master/python https://github.com/ibm-watson-data-lab/ibmos2spark/tree/master/pythonhttps://blog.sicara.com/get-started-pyspark-jupyter-guide-tutorial-ae2fe84f594f https://stackoverflow.com/questions/46011671/no-filesystem-for-scheme-cos

Spark and Data Tips for November 2018

Full Hadoop / HBase data platform for testing spark I found the following docker very handy for testing hadoop. https://hub.docker.com/r/bigdatauniversity/spark2/ docker pull bigdatauniversity/spark2 docker run -it –name bdu_spark2 -P -p 4040:4040 -p 4041:4041 -p 8080:8080 -p 8081:8081 bigdatauniversity/spark2:latest /etc/bootstrap.sh -bash Spark Notebooks I found these sites useful – http://spark-notebook.io/ and https://github.com/spark-notebook/spark-notebook and https://github.com/IBM?language=jupyter+notebook Version Mismatch […]

Hadoop KMS Ranger API – Tips and cURLs

I use Hadoop KMS Ranger in one environment. Some sample rest api calls are below, along with two tips. versionName is used in multiple queries. When not using kerberos – set ?user.name=hdfs on the URL   References https://hadoop.apache.org/docs/current/hadoop-kms/index.html#KMS_HTTP_REST_API https://hadoop.apache.org/docs/current/hadoop-kms/index.html#Get_Key_Names https://stackoverflow.com/questions/37601763/authentication-issue-with-kms-hadoop