app dev

Gatsby & Carbon: Build with Github Action

As some of you know, I work on the IBM FHIR Server and with my colleagues, I have started automating some of the actions we take – Build, Test, Deploy, Deploy our website.

More specific to the “Deploy our website” automation, our website uses technologies, such as Gatsby, Carbon, Gatsby Carbon Theme. Fundamentally, a static site generation technology, like Jekyll, Gatsby uses Node, Yarn and some nice React code.

To build our site with GitHub actions, I built out a site workflow.  The key elements to this workflow are:

  • Triggers
  • Node.js and Ubuntu Images
  • Build
  • Add, Commit and Push to GH Pages
  • Debugging and Replicating Locally


For the Triggers, I recommend limiting the site generation to master branches.  The master branch filter and on push, limits the re-deployment, also keep your site building on on docs/** changes.

– “docs/**”
– master
There is a subtlety the websites are cached for 10 minutes, confirmed on the site – Caching assets in website served from GitHub pages

Node.js and Ubuntu Images

I opted to use Ubuntu with Node.js
runs-on: ubuntu-latest

node-version: [12.x]
The important thing is ubuntu-latest which has some incompatibility with Gatsby Carbon’s build. 


I build the system as follows:

Checkout the repo to a folder

-name: Grab the Master Branch
uses: actions/checkout@v1
working-directory: fhir
ref: refs/heads/master
fetch-depth: 1
path: fhir
Activate Node
name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
node-version: ${{ matrix.node-version }}
Setup the build
echo “Check on Path”
cd docs/
npm install -g gatsby-cli
gatsby telemetry –disable
Install the packages, note, fsevents is not used on linux images, so use–no-optional (these plugins are suspect).
npm install –no-optional –save react react-copy-to-clipboard react-dom react-ga classnames carbon @carbon/addons-website carbon-components carbon-components-react carbon-addons-cloud carbon-icons gatsby gatsby-theme-carbon-starter markdown-it gatsby-plugin-manifest gatsby-plugin-slug gatsby-plugin-sitemap gatsby-plugin-sharp
With ubuntu, you can’t use gatsby build directly per, 
so I use the suggestion as a workaround due to path/issues in the gatsby component dependency of fsevents.
npm –prefix-paths run build
cp -R public/ ../../public/
Grab the GH-Pages branch
– name: Grab the GH Pages Branch
uses: actions/checkout@v1
working-directory: gh-pages
ref: refs/heads/gh-pages
fetch-depth: 1
path: docs
token: ${{ secrets.GITHUB_TOKEN }}

Per Bypassing Jekyll on GitHub Pages, be sure to add the .nojekll to the root of the gh-pages. I added a guard in the shell script to check if the file is there, and create the file if it does not exist.

If you need Environment variables, you should add the environment variables to the step.

Add, Commit and Push to GH Pages

I add the gitignore and nojekll files while removing any cached files, before moving in the new files.

I also like to make sure when this runs there is a build.txt file to trace when the site is built. (This file contains the build time Thu Nov 21 19:39:49 UTC 2019

I then use the github environment variables passed in to push the contents to the repo the branch is from. 

-name: Commit and Add GH Pages

run: |

echo "cleaning up the prior files on the branch"

if [ ! -f .nojekyll ]


touch .nojekyll

rm -f _config.yml


rm -f *.js webpack.stats.json styles-*.js styles-* webpack-runtime-* webpack-runtime-*.js manifest.webmanifest component---*.js* app-*.js*

rm -rf docs/node_modules docs/public docs/.cache

echo "Moving the files around for gh-pages"

cp -Rf ../public/* ./

find .

date > build.txt

git config --global "${{ secrets.GITHUB_ACTOR }}"

git config --global "Git Hub Site Automation"

git add .

git commit -m "Update to GH-Pages"

- name: Push changes to GH Pages

run: |

echo "Push Changes"

git branch


git push "${remote_repo}" HEAD:gh-pages





CI: true

Debugging and Replicating Locally

If you are troubleshooting, you can use a couple of approaches: 

1 – Create a Docker Image

Create the Image

docker run -itd –name gatsby-node -v docs:/data node:latest

Copy the Files

docker cp ~/FHIR/docs 6d810efb3b586739932166d424641003ee9b238de506543fcdd47eb7e7d41699:/data

Launch the shell and try the build

npm install –no-optional –save react react-copy-to-clipboard react-dom react-ga classnames carbon @carbon/addons-website carbon-components carbon-components-react carbon-addons-cloud carbon-icons gatsby gatsby-theme-carbon-starter markdown-it gatsby-plugin-manifest gatsby-plugin-slug gatsby-plugin-sitemap gatsby-plugin-sharp

Run the gatsby build

npm –prefix-paths run build

2. If you want complicated navigation, refer to however… gatsby-carbon-theme’s sidebar links only uses the to value not the href value.

3.  If you have an issue with your deployment check a couple of things:

Check your Deployed Environment. You should see a deployment in the last few seconds.

Check your Settings You should see no issues, else investigate the site locally on the gh-pages branch, and check Troubleshooting Jekyll build errors for GitHub Pages sites

Best of luck with your build!

app dev

Migrating Source Code Git-to-Git

Migrating source code is a pain in the butt, I know.  There are about 9 million variations, and one of interest to me – git to 

There are a number of tools to clean up your git history and prepare to move.

  • Git and Scripting
  • BFG Repo Cleaner
  • Git-Python
  • JGit

I found Git-Python a bit cumbersome, BFG Repo Cleaner more than I needed/wanted, and Git / Scripting too much work. After some prototyping, I opted for JGit from Eclipse and some Git knowhow.

First, I switched to the source Git Repo branch I wanted to migrate and exported the commit list.

git rev-list HEAD > commits.txt

which results in




This commits.txt is useful down the line.

I am a Maven disciple so, I created a maven java project with Java 1.8 and the following dependencies:
















I used the JGit to check the list of commits (note the REPO here must have .git at end).

try (Git git = File(SOURCE_GIT_REPO))) {


            System.out.println("Starting Branch is " + git.getRepository().getBranch());

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit =;

                String binSha =;




I flip it around, so I can process OLDEST to NEWEST


I used the git log (LogCommand in JGit) to find out all the times a FILE was modified, and do custom processing:

try (Git git = File(REPO))) {

            LogCommand logCommand = git.log().add(git.getRepository().resolve(Constants.HEAD)).addPath(fileName.replace(REPO, ""));

            Set<String> years = new HashSet<>();

            for (RevCommit revCommit : {

                Instant instant = Instant.ofEpochSecond(revCommit.getCommitTime());





To find out the files specifically in the HEAD of the repo, gets the files and paths, and puts it in a List

try (Git git = File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            if (iter.hasNext()) {

                RevCommit commit =;

                try (RevWalk walk = new RevWalk(git.getRepository());) {

                    RevTree tree = walk.parseTree(commit.getId());

                    try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {



                        while ( {








I built a history of changes.

try (Git git = File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit =;

                try (DiffFormatter df = new DiffFormatter(DisabledOutputStream.INSTANCE);) {

                    df.setRepository(git.getRepository());                    df.setDiffComparator(RawTextComparator.DEFAULT);


                    CommitHistoryEntry.Builder builder =                            CommitHistoryEntry.builder().binsha(;

                    RevCommit[] parents = commit.getParents();

                    if (parents != null && parents.length > 0) {

                        List<DiffEntry> diffs = df.scan(commit.getTree(), parents[0]);


                    } else {


                        try (RevWalk walk = new RevWalk(git.getRepository());) {

                            RevTree tree = walk.parseTree(commit.getId());

                            try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {



                                while ( {                                   









I did implement the Visitor pattern to optimize the modifications to the commit details and cleanup any bad Identity mappings (folks had many emails and names which I unified) and cleanedup the email addresses.

Next, I created a destination git (all fresh and ready to go):

try (Git thisGit = Git.init().setDirectory(new File(REPO_DIR)).call()) {

   git = thisGit;


One should make sure the path exists, and it doesn’t matter if you have files in it.

Commit the files in the git directory… you can commit without FILES!

CommitCommand commitCommand = git.commit();

        // Setup the Identity and date

        Date aWhen = new Date(entry.getCommitTime() * 1000);

        PersonIdent authorIdent =

                new PersonIdent(entry.getAuthorName(), entry.getAuthorEmail(), aWhen, TimeZone.getDefault());







Note, you can set to almost any point in time.  As long as you don’t sign it, it’ll be OK.  I don’t recommend this as a general practice.

To grab the file, you can do a tree walk, and resolve to the object ID.

try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {



                            int localCount = 0;

                            while ( {

                                String fileName = treeWalk.getPathString();

ObjectId objectId = treeWalk.getObjectId(0);

            ObjectLoader loader = git.getRepository().open(objectId);

            String fileOutput = GIT_OUTPUT + "/" + binSha + "/" + fileNameWithRelativePath;

            int last = fileOutput.lastIndexOf('/');

            String fileOutputDir = fileOutput.substring(0, last);

            File dir = new File(fileOutputDir);


            // and then one can the loader to read the file

            try (FileOutputStream out =

                    new FileOutputStream(GIT_OUTPUT + "/" + binSha + "/"

                            + fileNameWithRelativePath);) {

                // System.out

                byte[] bytes = loader.getBytes();

                if (hasBeenModified(bytes, fileNameWithRelativePath)) {



                    result = true;



Note, I did check if the file was duplicate, it saved a couple of steps.

If you want to add files, you can set:



Git in the background builds the DIFFs for any file that is not specially treated as binary in the .gitattributes file.

For each commit, I loaded the file – checking for stop-words, checked the copyright header, check the file type, and compared it against the head . 

Tip, if you need to reset from a bad test.

git reset --hard origin # reset the branch

rm -rf .git # reset the repo (also be sure to remove the files)

The moving of the branch, you can execute


git checkout <BRANCH_TO_MIGRATE>

git reset --hard origin

git pull

git gc --aggressive --prune=now

git push<MyOrg>/<DEST_REPO>.git <BRANCH_TO_MIGRATE>:master

Note, I did rename the master branch in the repo prior.  Voila.  550M+ Repo moved and cleaned up.

The repo is now migrated, and up-to-date.  I hope this helps you.


Rename Branch in Git


Rewrite History


BFG Repo Cleaner




app dev development

Code Graph showing the Layout of the Code base

I’ve been mixing data analysis and Java programming recently.  I wrote a tool to do the analysis (Maven/Python).

Here is the obfuscated output of the analysis, showing the hotspots.  I opted to show a thumbnail of the image here to protect the confidentiality of the project.  The generated image was also 78 Megabytes.  (a bit much, but you can zoom right in).

Complicated Graph

If you use a smaller set of classes and imports, the Maven plugin generates a reasonable diagram.csv file using

mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml

You then see the output diagram.csv.

To generate the layout dependencies of classes in your project.  Use the snippets, and the Jupyter Notebook – and view at

Simple Graph


Code Graph on Git

CSV Sample Data diagram.csv


<project xmlns=""





				<!-- Embeds the dependencies in fhir-tools into the jar. -->

package demo;

import java.util.Properties;

import org.apache.maven.execution.MavenSession;
import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugin.MojoFailureException;
import org.apache.maven.plugins.annotations.Execute;
import org.apache.maven.plugins.annotations.LifecyclePhase;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.plugins.annotations.Parameter;
import org.apache.maven.plugins.annotations.ResolutionScope;
import org.apache.maven.project.MavenProject;


 * This class coordinates the calls to the Diagram generation plugin
 * The phase is initialize. To find a list of phases -
 * Run the following to setup the plugin: <code>
 * mvn clean package install -f generate-diagram/pom.xml
 * </code>
 * Run the following to setup the classes in fhir-model: <code> 
 * mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml
 * </code>
 * @author PBastide
 * @requiresDependencyResolution runtime
@Mojo(name = "generate-diagram", //$NON-NLS-1$
        requiresProject = true, requiresDependencyResolution = ResolutionScope.RUNTIME_PLUS_SYSTEM, requiresDependencyCollection = ResolutionScope.RUNTIME_PLUS_SYSTEM, defaultPhase = LifecyclePhase.GENERATE_SOURCES, requiresOnline = false, threadSafe = false, aggregator = true)
@Execute(phase = LifecyclePhase.GENERATE_SOURCES)
public class DiagramPlugin extends AbstractMojo {

    @Parameter(defaultValue = "${project}", required = true, readonly = true) //$NON-NLS-1$
    protected MavenProject mavenProject;

    @Parameter(defaultValue = "${session}")
    private MavenSession session;

    @Parameter(defaultValue = "${project.basedir}", required = true, readonly = true) //$NON-NLS-1$
    private File baseDir;

    public void execute() throws MojoExecutionException, MojoFailureException {
        if (baseDir == null || !baseDir.exists()) {
            throw new MojoFailureException("The Base Directory is not found.  Throwing failure. ");

        // Grab the Properties (the correct way)
        Properties userProps = session.getUserProperties();
        String useTestsDirectoryStr = userProps.getProperty("useTestsDirectory", "false");

        // Converts Limit value to boolean value.
        boolean useTestsDirectory = Boolean.parseBoolean(useTestsDirectoryStr);

        // Grab the right generator and set it up.
        IDiagramGenerator generator = DiagramFactory.getDiagramGenerator();

        // Set the use of tests directory

        // Get the base directory .
        generator.setTargetProjectBaseDirectory(baseDir.getAbsolutePath() + "/target");

        // Passes the Log to the implementation code.

        // Add the project

        // Builds the Diagram
package example.impl;

import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.StringJoiner;

import org.apache.maven.plugin.logging.Log;
import org.apache.maven.project.MavenProject;
import org.jboss.forge.roaster.Roaster;
import org.jboss.forge.roaster.model.JavaType;
import org.jboss.forge.roaster.model.source.Import;
import org.jboss.forge.roaster.model.source.JavaAnnotationSource;
import org.jboss.forge.roaster.model.source.JavaClassSource;
import org.jboss.forge.roaster.model.source.JavaEnumSource;
import org.jboss.forge.roaster.model.source.JavaInterfaceSource;
import org.jboss.forge.roaster.model.source.JavaSource;
import org.w3c.dom.DOMImplementation;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

public class DiagramImpl implements IDiagramGenerator {

    private String absolutePath = null;
    private Log log = null;
    private Boolean useTestFiles = false;

    private List<MavenProject> projects = new ArrayList<>();

    private List<String> sourceDirectories = new ArrayList<>();

    private List<String> sourceFiles = new ArrayList<>();

    private List<String> countWithNested = new ArrayList<>();

    private Map<String, List<String>> sourceFileImports = new HashMap<>();

    public void add(MavenProject mavenProject) {
        if (mavenProject == null) {
            throw new IllegalArgumentException("no access to the maven project's plugin object");
        }"Projects added...");
        projects = mavenProject.getCollectedProjects();


    public void use(Boolean useTestFiles) {
        this.useTestFiles = useTestFiles;


    public void setLog(Log log) {
        this.log = log;

    public void generateDiagram() {
        if (absolutePath == null) {
            throw new IllegalArgumentException("Bad Path " + absolutePath);

        if (log == null) {
            throw new IllegalArgumentException("Unexpected no log passed in");

        for (MavenProject project : projects) {

            List<String> locations = project.getCompileSourceRoots();
            for (String location : locations) {
      "Location of Directory -> " + location);


            if (useTestFiles) {
      "Adding the Test Files");


        // Find the Files in each directory.
        // Don't Follow links.
        for (String directory : sourceDirectories) {


        printOutCardinality();"Total Number of Java Files in the Project are: " + sourceFiles.size());"Total Number of Classes/Interfaces/Enums in the Project are: " + countWithNested.size());



    private void generateImageFile() {
        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        try(FileOutputStream fos = new FileOutputStream("diagram.csv");) {
            for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
                StringJoiner joiner = new StringJoiner(",");
                for(String val : sourceFileImports.get(key)) {
                String line = key + ",\"" + joiner.toString() + "\"";

        } catch (Exception e) {
            log.warn("Issue processing", e);


    public void printOutCardinality() {

        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        //"Cardinality count - imports into other classes");
        for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
   + " -> " + sourceFileImports.get(key).size());


    public void processCardinalityMap() {
        // Import > List<Classes>
        // Stored in -> sourceFileImports

        for (String source : sourceFiles) {
            File srcFile = new File(source);
            try {
                JavaType<?> jtFile = Roaster.parse(srcFile);

                String parentJavaClass = jtFile.getQualifiedName();

                if (jtFile instanceof JavaClassSource) {
          "[C] -> " + parentJavaClass);
                    JavaClassSource jcs = (JavaClassSource) jtFile;

                    helperImports(parentJavaClass, jcs.getImports());

                    for (JavaSource<?> child : jcs.getNestedTypes()) {

                        String childLoc = child.getQualifiedName();
              "  [CC] -> " + childLoc);

                else if (jtFile instanceof JavaEnumSource) {
          "[E] -> " + parentJavaClass);
                    JavaEnumSource jes = (JavaEnumSource) jtFile;

                    helperImports(parentJavaClass, jes.getImports());

                    for (Object child : jes.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
              "  [EC] -> " + childLoc);

                } else if (jtFile instanceof JavaInterfaceSource) {

          "[I] -> " + parentJavaClass);
                    JavaInterfaceSource jis = (JavaInterfaceSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());

                    for (Object child : jis.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
              "  [IC] -> " + childLoc);
                } else if (jtFile instanceof JavaAnnotationSource) {

          "[A] -> " + parentJavaClass);
                    JavaAnnotationSource jis = (JavaAnnotationSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());

                else {
          "[O] -> " + parentJavaClass);

            } catch (IOException e) {
      "unable to parse file " + srcFile);

        }"Parsed the Cardinality Map:");


    private void helperImports(String parentJavaClass, List<Import> imports) {
        // sourceFileImports
        List<String> importOut = sourceFileImports.get(parentJavaClass);
        if (importOut == null) {
            sourceFileImports.put(parentJavaClass, new ArrayList<String>());

        for (Import importX : imports) {
            String importXStr = importX.getQualifiedName();
            importOut = sourceFileImports.get(importXStr);
            if (importOut == null) {
                importOut = new ArrayList<>();
                sourceFileImports.put(importXStr, importOut);




    public static void main(String... args) {
        SvgDiagramImpl impl = new SvgDiagramImpl();
        String proc = "";
        Log log = new Log() {

            public boolean isDebugEnabled() {
                return false;

            public void debug(CharSequence content) {


            public void debug(CharSequence content, Throwable error) {


            public void debug(Throwable error) {


            public boolean isInfoEnabled() {
                return false;

            public void info(CharSequence content) {


            public void info(CharSequence content, Throwable error) {


            public void info(Throwable error) {


            public boolean isWarnEnabled() {
                return false;

            public void warn(CharSequence content) {


            public void warn(CharSequence content, Throwable error) {


            public void warn(Throwable error) {


            public boolean isErrorEnabled() {
                return false;

            public void error(CharSequence content) {


            public void error(CharSequence content, Throwable error) {


            public void error(Throwable error) {


    private void addSourceFile(String proc) {


    public void findSourceFiles(String directory) {
        File dir = new File(directory);
        if (dir.exists()) {

            File[] listFiles = dir.listFiles((file, name) -> {
                return name.endsWith(".java");

            // Add to source directory
            if (listFiles != null) {
                for (File file : listFiles) {
          " File Added to Processing: " + file.getAbsolutePath());

            File[] listFilesFolders = dir.listFiles((file, name) -> {
                return file.isDirectory();

            if (listFilesFolders != null) {
                for (File tmpDir : listFilesFolders) {

        } else {
            log.warn("Directory does not exist " + directory);

    public void setTargetProjectBaseDirectory(String absolutePath) {
        this.absolutePath = absolutePath;


app dev raspberry pi

Raspberry Pi: Setting up backup

I have a Raspberry Pi providing household automation and productivity services – WebDav, Backups and Calendar. I always worry about a jolt of power, a failed byte and something that is unrecoverable. Time for a Backup solution.

I plugged in a USB stick – 64GB, and immediately checked the file system is there and visible as SDA (unmounted).

pi@raspberrypi:~# sudo su - 

root@raspberrypi:~# lsblk 
 sda           8:0    1 58.2G  1 disk 
 └─sda1        8:1    1 58.2G  1 part 
 mmcblk0     179:0    0 14.9G  0 disk 
 ├─mmcblk0p1 179:1    0 43.9M  0 part /boot
 └─mmcblk0p2 179:2    0 14.8G  0 part /

I check to see which one is assigned to the SD card slot (mmc), I really don’t want to reformat Raspbian.  I see the USB stick is on /dev/sda. All of my subsequent commands use /dev/sda as part of the command.

root@raspberrypi:~# parted -ls
Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.
 Model: JetFlash Transcend 64GB (scsi)
 Disk /dev/sda: 62.5GB
 Sector size (logical/physical): 512B/512B
 Partition Table: unknown
 Disk Flags: 

 Model: SD SC16G (sd/mmc)
 Disk /dev/mmcblk0: 15.9GB
 Sector size (logical/physical): 512B/512B
 Partition Table: msdos
 Disk Flags: 
 Number  Start   End     Size    Type     File system  Flags
  1      4194kB  50.2MB  46.0MB  primary  fat32        lba
  2      50.3MB  15.9GB  15.9GB  primary  ext4

If you don’t see the relevant information on your HD or run into issues formatting the hard-drive, install hdparm.  and check with hdparm -r0 /dev/sda  .

TIP: I did run into an issue with an ISO written to a USB drive which locks the partition table and makes it unwriteable.

root@raspberrypi:~# apt-get install hdparm
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 The following package was automatically installed and is no longer required:
 Use 'apt autoremove' to remove it.
 The following additional packages will be installed:
 Suggested packages:
 The following NEW packages will be installed:
   hdparm powermgmt-base
 0 upgraded, 2 newly installed, 0 to remove and 148 not upgraded.
 Need to get 114 kB of archives.
 After this operation, 278 kB of additional disk space will be used.
 Do you want to continue? [Y/n] y
 Get:1 stretch/main armhf hdparm armhf 9.51+ds-1+deb9u1 [105 kB]
 Get:2 stretch/main armhf powermgmt-base all 1.31+nmu1 [9,240 B]
 Fetched 114 kB in 0s (120 kB/s)           
 Selecting previously unselected package hdparm.
 (Reading database … 135688 files and directories currently installed.)
 Preparing to unpack …/hdparm_9.51+ds-1+deb9u1_armhf.deb …
 Unpacking hdparm (9.51+ds-1+deb9u1) …
 Selecting previously unselected package powermgmt-base.
 Preparing to unpack …/powermgmt-base_1.31+nmu1_all.deb …
 Unpacking powermgmt-base (1.31+nmu1) …
 Setting up powermgmt-base (1.31+nmu1) …
 Setting up hdparm (9.51+ds-1+deb9u1) …
 Processing triggers for man-db ( …
Preparing to unpack …/hdparm_9.51+ds-1+deb9u1_armhf.deb …

Unpacking hdparm (9.51+ds-1+deb9u1) …

Selecting previously unselected package powermgmt-base.

Preparing to unpack …/powermgmt-base_1.31+nmu1_all.deb …

Unpacking powermgmt-base (1.31+nmu1) …

Setting up powermgmt-base (1.31+nmu1) …

Setting up hdparm (9.51+ds-1+deb9u1) …

Processing triggers for man-db ( …
root@raspberrypi:~# hdparm -r0 /dev/sda
  setting readonly to 0 (off)
  readonly      =  0 (off)

Now that I know the drive is writeable, I need to create the partition. I used


Navigate through the menu and select the maximum size
                                                                                        Disk: /dev/sda
Size: 58.2 GiB, 62495129600 bytes, 122060800 sectors
Label: dos, identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
>> /dev/sda1 2048 122060799 122058752 58.2G 83 Linux

Once you see “Syncing disks.”, you can format the disk. I formatted the partition sda1 with ext4 (I may want to encrypt in the future). Unmount and then format.

root@raspberrypi:~# umount /dev/sda1
root@raspberrypi:~# mkfs.ext4 /dev/sda1
mke2fs 1.43.4 (31-Jan-2017)
Found a dos partition table in /dev/sda1
Proceed anyway? (y,N) y
Creating filesystem with 2828032 4k blocks and 707136 inodes
Filesystem UUID: 363f1b4a-b0f5-4c7b-bf91-66f3823032d6
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done


Make the backup directory, edit fstab and mount the directory.  Insert into fstab with your uuid “UUID=363f1b4a-b0f5-4c7b-bf91-66f3823032d6 /backups auto nosuid,nodev,nofail 0 0
”   The second to last avoids backup and the last one enables fsck on reboot.

root@raspberrypi:~# blkid 
/dev/mmcblk0p1: LABEL="boot" UUID="DDAB-3A15" TYPE="vfat" PARTUUID="b53687e8-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="5fa1ec37-3719-4b25-be14-1f7d29135a13" TYPE="ext4" PARTUUID="b53687e8-02"
/dev/mmcblk0: PTUUID="b53687e8" PTTYPE="dos"
/dev/sdb: UUID="363f1b4a-b0f5-4c7b-bf91-66f3823032d6" TYPE="ext4"
root@raspberrypi:~# mkdir /backups 
root@raspberrypi:~# vim /etc/fstab
root@raspberrypi:~# mount -a
root@raspberrypi:~# mount
UUID=363f1b4a-b0f5-4c7b-bf91-66f3823032d6 /backups auto nosuid,nodev,nofail 0 0

You should see backups listed. (Note: I bricked my Raspberry Pi with a bad FSTAB entry, and mounted it on my Mac using Paragon and removed the bad fstab entry. )

Update Crontab with daily backups.

crontab -e

Setup an editor for crontab.

root@raspberrypi:~# crontab -e
no crontab for root - using an empty one

Select an editor. To change later, run 'select-editor'.
1. /bin/ed
2. /bin/nano <---- easiest
3. /usr/bin/vim.basic
4. /usr/bin/vim.tiny

Choose 1-4 [2]: 3
crontab: installing new crontab

I added this line and copied it to /etc/cron.daily/

0 1 * * * /usr/bin/rsync -r /data/ /backups/`date +%w-%A`

crontab -l > pi-backup
mv /root/pi-backup /etc/cron.daily
run-parts /etc/cron.daily

Note, I had to add #!/bin/bash after I copied and remove the timing of the job.

Also, check to see if rsync is installed with which rsync and apt-get install rsync.

This enables backups on a daily basis rotating every 7 days.

Check back on the following day to see your backups

root@raspberrypi:~# /usr/bin/rsync -r /data/ /backups/`date +%w-%A`
root@raspberrypi:~# find /backups

Good luck, I hope this helps you with your Raspberry Pi.


raspberry pi

Cups and Raspberry Pi – How to setup your print server

Print Server is one of those things that makes life easier.  I hooked up my Raspberry Pi and a Docker image in order to have a CUPS server and the Bonjour configuration.

Login to the Raspberry Pi and switch to root (sudo -s or sudo su –)

Check lsusb to see if you can see the USB device.

 root@raspberrypi:~# lsusb 
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

If you don’t see it, check to see that the device is properly plugged in.

Check again, and you’ll see your manufacture for your printer listed. (highlighted in red)

 root@raspberrypi:~# lsusb 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Further, I am able to see the exact model that is connected:

 root@raspberrypi:~# lsusb -v | grep -A 20 Hewlett-Packard 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x03f0 Hewlett-Packard
  idProduct          0x7411
  bcdDevice            1.00
  iManufacturer           1 HP
  iProduct                2 Photosmart C4600 series
  iSerial                 3 |||MASKED|||MASKED|||MASKED|||

I’ll use these details downstream in order to install and use the HP driver.

If you still don’t see the USB device, you should check dmesg and look at the pattern for USB.

 [3077006.701281] usb 1-1.2: new high-speed USB device number 7 using dwc_otg
[3077006.831891] usb 1-1.2: New USB device found, idVendor=03f0, idProduct=7411
[3077006.831908] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[3077006.831917] usb 1-1.2: Product: Photosmart C4600 series
[3077006.831925] usb 1-1.2: Manufacturer: HP

If you don’t see it, unplug and plug the device back in.

You can also use usb-devices

 T:  Bus=01 Lev=02 Prnt=02 Port=01 Cnt=02 Dev#=  7 Spd=480 MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=03f0 ProdID=7411 Rev=01.00
S:  Manufacturer=HP
S:  Product=Photosmart C4600 series
S:  SerialNumber=<<>><<>>
C:  #Ifs= 4 Cfg#= 1 Atr=c0 MxPwr=2mA
I:  If#= 0 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=cc Prot=00 Driver=(none)
I:  If#= 1 Alt= 0 #EPs= 2 Cls=07(print) Sub=01 Prot=02 Driver=usblp
I:  If#= 2 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)
I:  If#= 3 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=usb-storage

Copy the Dockerfile and the cupsd file locally (in the folder etc-cups/cupsd.conf). Cupsd is configured to listen on all interfaces: Listen

Build the Docker image (based off of this image and uses a different base image ) docker build -t cups-pi/cups-pi .

 root@raspberrypi:/data/cups-printer# docker build -t cups-pi/cups-pi . 
Sending build context to Docker daemon  8.192kB
Step 1/9 : FROM arm32v7/debian:latest
 ---> 64b4748d266b
Step 2/9 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 38ebdf2f5bb6
Step 3/9 : RUN apt-get update && apt-get install -y   sudo   locales   whois   cups   cups-client   cups-bsd   printer-driver-all   hpijs-ppds   hp-ppd   hplip
 ---> Using cache
 ---> 0b5b9eac6cef
Step 4/9 : RUN sed -i "s/^#\ \+\(en_US.UTF-8\)/\1/" /etc/locale.gen && locale-gen en_US en_US.UTF-8
 ---> Using cache
 ---> fd55737901b8
Step 5/9 : ENV LANG=en_US.UTF-8   LC_ALL=en_US.UTF-8   LANGUAGE=en_US:en
 ---> Using cache
 ---> d6cd104b40ec
Step 6/9 : RUN useradd   --groups=sudo,lp,lpadmin   --create-home   --home-dir=/home/print   --shell=/bin/bash   --password=$(mkpasswd print)   print   && sed -i '/%sudo[[:space:]]/ s/ALL[[:space:]]*$/NOPASSWD:ALL/' /etc/sudoers   && apt-get clean   && rm -rf /var/lib/apt/lists/*   && mkdir /var/lib/apt/lists/partial
 ---> Using cache
 ---> 156dcd02e397
Step 7/9 : COPY etc-cups/cupsd.conf /etc/cups/cupsd.conf
 ---> e1624a96970e
Step 8/9 : EXPOSE 631
 ---> Running in a16705e0f3d9
Removing intermediate container a16705e0f3d9
 ---> 60758af63011
Step 9/9 : ENTRYPOINT ["/usr/sbin/cupsd", "-f"]
 ---> Running in add02d421ea9
Removing intermediate container add02d421ea9
 ---> 8c6fe42423c8
Successfully built 8c6fe42423c8
Successfully tagged cups-pi/cups-pi:ltest

Run the Docker image to start the print server

root@raspberrypi:/data/cups-printer# docker run -d -p 631:631 --privileged -v /var/run/dbus:/var/run/dbus -v /dev/bus/usb:/dev/bus/usb -h --name cupsm cups-pi/cups-pi:latest

It’s important to use the -h if you want to remotely access via hostname.

Login to the console using print:print

Click Add Printer

Select a local printer “HP Photosmart C4600 series (HP Photosmart C4600 series)”

Click Continue

Click Share This Printer

Click Continue

Click Add Printer

Select Media Type – Letter

Click Set Default Options

Click Add

When you find the Printer, you can add the printer

You now have an image ready for use with a CUPS printer.  (If you rebuild, you may have to re-setup the default printer…. Just warning you ahead of time).  You can always look at

If you need to connect to the console, you can look at the docker ps

root@raspberrypi:/data/cups-printer# docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                                    NAMES
fe6d9bc34c66        cups-pi/cups-pi:latest   "/usr/sbin/cupsd -f"   5 minutes ago       Up 5 minutes>631/tcp                     cups

Extract the CONTAINER ID, and look at the /bin/sh

root@raspberrypi:/data/cups-printer# docker exec
-i -t fe6d9bc34c66 /bin/sh

Then look at the /var/log/cups files, specifically the error_log


List USB Devices Linux


Arm 7


SSL / Advanced Configuration (use IP if receiving a bad request indicator to the admin interface)

Advanced Ubuntu Configuration

app dev jupyter

Jupyter Notebook: Email Analysis to a Lotus Notes View

I wanted to do an analysis of my emails since I joined IBM, and see the flow of messages in-and-out of my inbox.

With my preferences for Jupyter Notebooks, I built a small notebook for analysis.

Open IBM Lotus Notes Rich Client

Open the Notes Database with the View you want to analyze.

Select the View you are interested in ‘All Documents’. For instance the All Documents view, like my inbox *obfuscated* with a purpose.

Click File > Export

Enter a file name – email.csv

Select Format “Comma Separate Value”

Click Export

Upload the Notebook to your Jupyter server

The notebook is describes the flow through my process. If you encounter ValueError: (‘Unknown string format:’, ’12/10/2018 08:34 AM’), you can refer to

iconv -c -f utf-8 -t ascii email.csv > email.csv.clean

You can break the data into month-year-day analysis with the following, and peek the results with df_emailA.head()

When you run the final cell, the code generates a Year-Month-Day count as a bar graph.

    # Title: Volume in Months when emails are sent.
    # Plots volume based on known year-mm-dd
    # to be included in the list, one must have data in those years.
    # Kind is a bar graph, so that the (Y - YYYY,MM can be read)
    y_m_df = df_emailA.groupby(['year','month','day']).year.count()

    plt.title('Numbers submitted By YYYY-MM-DD')
    plt.xlabel('Email Flow')
    plt.autoscale(enable=True, axis='both', tight=False)
    plt.rcParams['figure.figsize'] = [20, 200]

You’ll see the trend of emails I receive over the years.

Trends of Email

Co-Inventing Model with Gephi

I was curious who my frequent co-inventors were, so I downloaded Gephi. I downloaded my patent data from Google Patents and built a fun little model from CSV data (similar to the attached).

Matthew E. Broomhall 121 PAUL Bastide
Robert E. Loredo 119 PAUL Bastide
Fang Lu 60 PAUL Bastide
Alaa Abou Mahmoud 43 PAUL Bastide
Lisa Seacat Deluca 23 PAUL Bastide
Lydia M. Do 17 PAUL Bastide
Dale M. Schultz 13 PAUL Bastide
Andrew E. Davis 7 PAUL Bastide
Ralph E. LeBlanc 7 PAUL Bastide
Sean Callanan 7 PAUL Bastide
Donna K. Byron 5 PAUL Bastide
Sandra L. Kogan 5 PAUL Bastide
Asima Silva 4 PAUL Bastide
Aaron J. Quirk 3 PAUL Bastide
Aaron M. Cohen 3 PAUL Bastide
Daniel B. Harris 3 PAUL Bastide
Eric S. Portner 3 PAUL Bastide
John M. Boyer 3 PAUL Bastide
Michael L. Taylor 3 PAUL Bastide
Alexander Pikovsky 2 PAUL Bastide
Corville O. Allen 2 PAUL Bastide
Dana L. Price 2 PAUL Bastide
Eric M. Wilcox 2 PAUL Bastide
Jeffrey R. Hoy 2 PAUL Bastide
John A. Jacobson 2 PAUL Bastide
Kulvir S. Bhogal 2 PAUL Bastide
Liam Harpur 2 PAUL Bastide
Marco A. Vicente 2 PAUL Bastide
Patrick J. O’Sullivan 2 PAUL Bastide
Scott J. Martin 2 PAUL Bastide
Shane M. Kilmon 2 PAUL Bastide
Stephen Crawford 2 PAUL Bastide
Thomas J. Evans IV 2 PAUL Bastide
Vijay Francis 2 PAUL Bastide
Weisong Wang 2 PAUL Bastide
Adam L. Cutler 1 PAUL Bastide
Amanda N. Savitzky 1 PAUL Bastide
Andrew L. Schirmer 1 PAUL Bastide
Arun Vishwanath 1 PAUL Bastide
Bernadette A. Carter 1 PAUL Bastide
Beth Anne M. Collopy 1 PAUL Bastide
Beth L. Hoffman 1 PAUL Bastide
Bradley W. Hurley 1 PAUL Bastide
Brenton P. Chasse 1 PAUL Bastide
Brian M. Walsh 1 PAUL Bastide
Carl J. Kraenzel 1 PAUL Bastide
Christopher W. Desforges 1 PAUL Bastide
Damian E.A. Garcia 1 PAUL Bastide
Dan DUMONT 1 PAUL Bastide
Dwarikanath Mahapatra 1 PAUL Bastide
Fred Raguillat 1 PAUL Bastide
Isabell Kiral-Kornek 1 PAUL Bastide
Jaime M. Stockton 1 PAUL Bastide
James A. Hart 1 PAUL Bastide
Jennifer L. Vargus 1 PAUL Bastide
Jose L. Lopez 1 PAUL Bastide
Juliana M. Leong 1 PAUL Bastide
Katherine M. Parsons 1 PAUL Bastide
Kelley L. ANDERS 1 PAUL Bastide
King Shing K. Lui 1 PAUL Bastide
Leah A. Lawrence 1 PAUL Bastide
Leho Nigul 1 PAUL Bastide
Lei Wang 1 PAUL Bastide
Lorelei M. McCollum 1 PAUL Bastide
Margo L. Ezekiel 1 PAUL Bastide
Mark Gargan 1 PAUL Bastide
Mary E. Miller 1 PAUL Bastide
Matthew Stephen Rosno 1 PAUL Bastide
Melissa A. Lord 1 PAUL Bastide
Michael G. Alexander 1 PAUL Bastide
Na Pei 1 PAUL Bastide
Neal Fishman 1 PAUL Bastide
Pei Sun 1 PAUL Bastide
Richard Gorzela 1 PAUL Bastide
Richard T. Bassemir 1 PAUL Bastide
Shelbee D. Smith-Eigenbrode 1 PAUL Bastide
Shu Qiang Li 1 PAUL Bastide
Shunguo Yan 1 PAUL Bastide
Stacy M. Cannon 1 PAUL Bastide
Stanley K. Jerrard-Dunne 1 PAUL Bastide
Stefan von Cavallar 1 PAUL Bastide
Susmita Saha 1 PAUL Bastide
Tamer E. Abuelsaad 1 PAUL Bastide
Thomas J. Evans 1 PAUL Bastide
Trudy L. Hewitt 1 PAUL Bastide
Xujin Liu 1 PAUL Bastide
Ying Mo 1 PAUL Bastide
PAUL Bastide 0 PAUL Bastide

Graph of Co-Inventors



troubleshooting WHC Development

CURL and LDAPS – How to Search and Debug

I hit an issue where I needed to Search LDAP from a machine I didn’t have access to install new RPMs on. I found this cool article on CURL and LDAP Search. I had to make some minor modifications to get it to work with a secure connection (–insecure ldaps:// and 636). I also added -v to diagnosis some connection problems.

curl "ldaps://,objectClass?sub?(objectClass=)" -u "cn=user1,ou=test_org3,o=dr,DC=IBM.COM" --insecure -v
Enter host password for user 'cn=user1,ou=test_org3,o=dr,DC=IBM.COM':
* Trying
* Connected to ( port 636 (#0)
* LDAP local: LDAP Vendor = OpenLDAP ; LDAP Version = 20428
* LDAP local: ldaps://,objectClass?sub?(objectClass=
* LDAP local: trying to establish encrypted connection
objectClass: domain
objectClass: top

DN: o=dr,
objectClass: organization
objectClass: top

DN: ou=test_org3,o=dr,
objectClass: organizationalunit
objectClass: top

You can then find the userids you need quickly. I left them off the output intentionally.

If you see connected, but no results, I suggest changing to the top level of the ldap, and using this string – ldaps://,objectClass?sub?(objectClass=*)

raspberry pi

Raspberry Pi – Part IV – Simple Wiring Test

I am plugging away working with my Pi. I went back to the Ada Fruit Site, and started working on one of the tutorials from the part I purchased –   I want to be able to demonstrate a simple wiring with my Pi works.

I selected Email Notifier. I read through it, and realized I needed to convert it to work for the B+ model. The pi4j website has a nice diagram for the pins and I simplified the test a bit.

I launched the simple python file

pi@seconds ~ $ sudo python

Pi with a Green Light - Using Python
Pi with a Green Light – Using Python

I added the pi4j dependency to my maven pom


I create a servlet which replicated the same code from before (GPIO).

I copied the war file to the local webapp directory, and restarted jetty

pi@seconds /opt/jetty/web/bbq/webapps $ sudo cp ~/pi.webapp.war ./
pi@seconds /opt/jetty/web/bbq/webapps $ sudo chown jetty:jetty pi.webapp.war
pi@seconds /opt/jetty/web/bbq/webapps $ sudo /etc/init.d/jetty restart

Refer to  and I did have to add –module=jsp to the start.ini for my web configuration.

I hit the servlet –


Which makes sense, since “Software using the Pi4J library must be run with ROOT level permissions.” (actually it doesn’t since I am running as root)…

Issues remain.  I’ll tackle them on a different day 🙂   … Part 5 …

raspberry pi

Raspberry Pi – Part III – Setting up Jetty

The Raspberry Pi is a device that has so many options open: You can extend the hardware. You can extend the software. I choose to tackle extending the software to start, so I can get the experience that I want setup, and running on the device.  The experience I am after is the web interface to control my Raspberry Pi.

For a Java developer, I conclude that a lightweight interface hosted on a Jetty server is probably easiest.  The Jetty server is a 12.9M download and 30M expanded. It’s also used frequently in devices, and why reinvent the wheel, use the approach that Industry is using.  Jetty is designed for a small memory footprint.   (I did consider using Tomcat Embedded, and came to the conclusion I’d probably want to many features.)

I downloaded the Jetty archive to my local user’s directory. Note, since I want to just use I’m renaming the output file with wget -O.

wget -O

I make a directory for the application server, and move over to that directory.

sudo mkdir -p /opt/jetty
sudo mv /opt/jetty

Next I launched into the sudoers root shell, and extract the Jetty archive.  I decided to move it into the runtime folder, so I can have an easy place to backup from.  (Remove the extra space from the .zip)

sudo -s 
cd /opt/jetty
mv jetty-distribution-9.2.10.v20150310/ runtime

Next, I wanted to setup Jetty as a service, and check that Jetty starts

cp runtime/bin/ /etc/init.d/jetty
echo JETTY_HOME=`pwd`/runtime > /etc/default/jetty
service jetty start
Starting Jetty: . . . OK Sun Apr 12 15:38:40 UTC 2015
service jetty stop
Stopping Jetty: OK

Next, I want to automatically start Jetty with the right runlevels

update-rc.d jetty defaults

I want to configure Jetty to run with a set user jetty.

mkdir -p /opt/jetty/web/bbq
mkdir -p /opt/jetty/temp
useradd --user-group --shell /bin/false --home-dir /opt/jetty/temp jetty

There is a base configuration that needs to be setup the base site.

cd /opt/jetty/web/bbq
java -jar /opt/jetty/runtime/start.jar --add-to-start=deploy,http,logging

Next, edit the default port value.

vi start.ini

## HTTP port to listen on

Finally, I wrap the configuration of the jetty service and app.

chown -R jetty:jetty /opt/jetty
echo "JETTY_HOME=/opt/jetty/runtime" > /etc/default/jetty
echo "JETTY_BASE=/opt/jetty/web/bbq" >> /etc/default/jetty
echo "TMPDIR=/opt/jetty/temp" >> /etc/default/jetty

A good check is to look at service jetty status, and confirm the settings, and then restart your Raspberry Pi. I did find that the startup time was significantly effected by the additional service.  ( +25 seconds from the original 15)

Navigate to (or whatever IP you have used) and confirm the page loads Jetty.  If you see a 404, you’re off and ready for the next phase. (as am I) 🙂

Jetty Starting Point
Jetty Starting Point