Gatsby & Carbon: Build with Github Action

As some of you know, I work on the IBM FHIR Server and with my colleagues, I have started automating some of the actions we take – Build, Test, Deploy, Deploy our website.

More specific to the “Deploy our website” automation, our website uses technologies, such as Gatsby, Carbon, Gatsby Carbon Theme. Fundamentally, a static site generation technology, like Jekyll, Gatsby uses Node, Yarn and some nice React code.

To build our site with GitHub actions, I built out a site workflow.  The key elements to this workflow are:

  • Triggers
  • Node.js and Ubuntu Images
  • Build
  • Add, Commit and Push to GH Pages
  • Debugging and Replicating Locally

Triggers

For the Triggers, I recommend limiting the site generation to master branches.  The master branch filter and on push, limits the re-deployment, also keep your site building on on docs/** changes.

on:
push:
paths:
– “docs/**”
branches:
– master
 
There is a subtlety the websites are cached for 10 minutes, confirmed on the site – Caching assets in website served from GitHub pages

Node.js and Ubuntu Images

I opted to use Ubuntu with Node.js
 
jobs:
build:
runs-on: ubuntu-latest

strategy:
matrix:
node-version: [12.x]
 
The important thing is ubuntu-latest which has some incompatibility with Gatsby Carbon’s build. 

Build

I build the system as follows:

Checkout the repo to a folder

steps:
-name: Grab the Master Branch
uses: actions/checkout@v1
with:
working-directory: fhir
ref: refs/heads/master
fetch-depth: 1
path: fhir
 
Activate Node
name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
 
Setup the build
 
echo “Check on Path”
pwd
cd docs/
npm install -g gatsby-cli
gatsby telemetry –disable
 
Install the packages, note, fsevents is not used on linux images, so use–no-optional (these plugins are suspect).
 
npm install –no-optional –save react react-copy-to-clipboard react-dom react-ga classnames carbon @carbon/addons-website carbon-components carbon-components-react carbon-addons-cloud carbon-icons gatsby gatsby-theme-carbon-starter markdown-it gatsby-plugin-manifest gatsby-plugin-slug gatsby-plugin-sitemap gatsby-plugin-sharp
 
With ubuntu, you can’t use gatsby build directly per https://github.com/gatsbyjs/gatsby/issues/17557, 
so I use the suggestion as a workaround due to path/issues in the gatsby component dependency of fsevents.
 
npm –prefix-paths run build
cp -R public/ ../../public/
 
Grab the GH-Pages branch
 
– name: Grab the GH Pages Branch
uses: actions/checkout@v1
with:
working-directory: gh-pages
ref: refs/heads/gh-pages
fetch-depth: 1
path: docs
token: ${{ secrets.GITHUB_TOKEN }}

Per Bypassing Jekyll on GitHub Pages, be sure to add the .nojekll to the root of the gh-pages. I added a guard in the shell script to check if the file is there, and create the file if it does not exist.

If you need Environment variables, you should add the environment variables to the step.

Add, Commit and Push to GH Pages

I add the gitignore and nojekll files while removing any cached files, before moving in the new files.

I also like to make sure when this runs there is a build.txt file to trace when the site is built. (This file contains the build time Thu Nov 21 19:39:49 UTC 2019

I then use the github environment variables passed in to push the contents to the repo the branch is from. 

-name: Commit and Add GH Pages

run: |

echo "cleaning up the prior files on the branch"

if [ ! -f .nojekyll ]

then

touch .nojekyll

rm -f _config.yml

fi

rm -f *.js webpack.stats.json styles-*.js styles-*.js.map webpack-runtime-*.js.map webpack-runtime-*.js manifest.webmanifest component---*.js* app-*.js*

rm -rf docs/node_modules docs/public docs/.cache

echo "Moving the files around for gh-pages"

cp -Rf ../public/* ./

find .

date > build.txt

git config --global user.email "${{ secrets.GITHUB_ACTOR }}@users.noreply.github.com"

git config --global user.name "Git Hub Site Automation"

git add .

git commit -m "Update to GH-Pages"

- name: Push changes to GH Pages

run: |

echo "Push Changes"

git branch

remote_repo="https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"

git push "${remote_repo}" HEAD:gh-pages

env:

GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

GITHUB_REPOSITORY: ${{ secrets.GITHUB_REPOSITORY }}

GITHUB_ACTOR: ${{ secrets.GITHUB_ACTOR }}

CI: true

Debugging and Replicating Locally

If you are troubleshooting, you can use a couple of approaches: 

1 – Create a Docker Image

Create the Image

docker run -itd –name gatsby-node -v docs:/data node:latest

Copy the Files

docker cp ~/FHIR/docs 6d810efb3b586739932166d424641003ee9b238de506543fcdd47eb7e7d41699:/data

Launch the shell and try the build

npm install –no-optional –save react react-copy-to-clipboard react-dom react-ga classnames carbon @carbon/addons-website carbon-components carbon-components-react carbon-addons-cloud carbon-icons gatsby gatsby-theme-carbon-starter markdown-it gatsby-plugin-manifest gatsby-plugin-slug gatsby-plugin-sitemap gatsby-plugin-sharp

Run the gatsby build

npm –prefix-paths run build

2. If you want complicated navigation, refer to https://github.com/gatsbyjs/gatsby/blob/master/www/src/data/sidebars/doc-links.yaml however… gatsby-carbon-theme’s sidebar links only uses the to value not the href value.

3.  If you have an issue with your deployment check a couple of things:

Check your Deployed Environment. You should see a deployment in the last few seconds.

Check your Settings You should see no issues, else investigate the site locally on the gh-pages branch, and check Troubleshooting Jekyll build errors for GitHub Pages sites

Best of luck with your build!

Application Development Notes: Tips and Tricks

LC_ALL

Exception = javax.resource.spi.ResourceAllocationException
Source = com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection
probeid = 299
Stack Dump = javax.resource.spi.ResourceAllocationExceptionDSRA8100E: Unable to get a XAConnection from the DataSource. with SQL State : XJ041 SQL Code : 40000
    at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1578)

LC_ALL is the way to fix.

Before executing the Java application.

export LC_ALL=en_US.UTF-8

https://stackoverflow.com/a/9894836/1873438
http://db.apache.org/derby/docs/10.5/tools/ctools1004764.html#ctools1004764__rtoolsijtools91283
https://stackoverflow.com/questions/55673886/what-is-the-difference-between-c-utf-8-and-en-us-utf-8-locales

Multi-module Javadocs where Search doesn’t resolve the context properly

<additionalJOption>--no-module-directories</additionalJOption>
or
<additionalOption>--no-module-directories</additionalOption>

Search will generate which avoid the <undefined>, and use the root folder.

https://stackoverflow.com/questions/52326318/maven-javadoc-search-redirects-to-undefined-url

Gatsby with the Carbon Theme

The IBM project I work on uses Carbon Theme with Gatsby to generate our HTML pages. The Carbon Design system has a nice git repository at GIT: gatsby-theme-carbon. I recommend cloning these repository.  The repository has an example and a place to figure out where and how to Shadow a feature – “provide their own components for the theme to use over the defaults”.

A couple of plugins I recommend are:
gatsby-plugin-slug – https://www.gatsbyjs.org/packages/gatsby-plugin-slug/ resolves inter-page links
gatsby-plugin-manifesthttps://www.gatsbyjs.org/packages/gatsby-plugin-manifest/?=gatsby-plugin-manifest favicon, general theme details, metadata, supports mdx and md formatted files.
gatsby-plugin-sitemap – https://www.gatsbyjs.org/packages/gatsby-plugin-sitemap/ makes the site searchable

To update the components, use

– Navigation (on left) – Update the src/data/nav-items.yaml
– Footer – Update the src/gatsby-theme-carbon/components/Footer.js Only one column and content area used. (RIGHT column not used)
– Header – The custom header src/gatsby-theme-carbon/components/Header.js is in this file. This is the only one that should be edited.  The other files are theme.

To see team’s configuration, you can view https://github.com/IBM/FHIR/tree/master/docs

Building a Carbon Site
0 – clone git: IBM/FHIR and change to /docs
1 – npm install
– if prompted, install these plugins:
npm install –save gatsby-plugin-manifest
npm install –save gatsby-plugin-slug
npm install –save gatsby-plugin-manifest
2 – gatsby build –prefix-paths
The build should output:
08:02:18-pb@pauls-mbp:~/git/wffh/oct27/FHIR/docs$ gatsby build –prefix-paths
success open and validate gatsby-configs – 0.090 s
success load plugins – 1.701 s
success onPreInit – 0.017 s
success delete html and css files from previous builds – 0.019 s
success initialize cache – 0.031 s
success copy gatsby files – 0.080 s
success onPreBootstrap – 1.931 s
success source and transform nodes – 4.337 s
success building schema – 0.272 s
success createPages – 0.014 s
success createPagesStatefully – 1.165 s
success onPreExtractQueries – 0.014 s
success update schema – 0.071 s
success extract queries from components – 0.243 s
success write out requires – 0.017 s
success write out redirect data – 0.015 s
success Build manifest and related icons – 0.110 s
success Build manifest and related icons – 0.129 s
success onPostBootstrap – 0.268 s

info bootstrap finished – 15.674 s

success run static queries – 0.049 s — 6/6 187.18 queries/second
success Building production JavaScript and CSS bundles – 105.557 s
success Rewriting compilation hashes – 0.029 s
success run page queries – 0.087 s — 19/19 312.56 queries/second
success Building static HTML for pages – 21.400 s — 19/19 14.81 pages/second
info Done building in 142.923 sec

3 – gatsby serve (to check)
4 – on a different repo, copy the files from `public/` over to the gh-pages branch at the root
5 – git add .
6 – git push

I hope this helps you with Carbon/Gatsby.

Additional Links
Gatsby Theme: Carbon https://github.com/carbon-design-system/gatsby-theme-carbon
Gatsby Theme Carbon: Guides https://gatsby-theme-carbon.now.sh/guides/
Carbon Design https://www.carbondesignsystem.com/components/link/code/
Carbon Theme Details http://react.carbondesignsystem.com/?path=/story/link–default
MDX https://mdxjs.com/

IBM Db2 with IBM FHIR Server – FHIR Schema

My colleague Albert has also merged a Dockerfile for testing, it’s a good place to start.


docker run -itd –name fhirdby –privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=password -e DBNAME=fhirdb -v database:/database ibmcom/db2


docker exec -ti fhirdby bash -c “su – db2inst1”
source .bashrc

# Recreate the DB with the right page size the tablespace is implictly created by the tool
[db2inst1@c58991a6a1a2 ~]$ db2 terminate
DB20000I The TERMINATE command completed successfully.
[db2inst1@c58991a6a1a2 ~]$ db2 drop db fhirdb
DB20000I The DROP DATABASE command completed successfully.
[db2inst1@c58991a6a1a2 ~]$ db2 CREATE DB FHIRDB using codeset UTF-8 territory us PAGESIZE 32768
DB20000I The CREATE DATABASE command completed successfully.

# Create the schemas
[db2inst1@c58991a6a1a2 ~]$ db2 connect to fhirdb
Database Connection Information
Database server = DB2/LINUXX8664 11.5.0.0
SQL authorization ID = DB2INST1
Local database alias = FHIRDB

[db2inst1@c58991a6a1a2 ~]$ db2 “create schema fhirdata”
DB20000I The SQL command completed successfully.
[db2inst1@c58991a6a1a2 ~]$ db2 “create schema fhir_admin”
DB20000I The SQL command completed successfully.

# Run the Schema creation with a small pool
–prop-file /Users/paulbastide/git/wffh/FHIR/fhir-persistence-schema/db.properties
–schema-name FHIRDATA
–pool-size 2
–update-schema

# Create a new tenant
–prop-file /Users/paulbastide/git/wffh/FHIR/fhir-persistence-schema/db.properties
–schema-name FHIRDATA
–allocate-tenant TNT1

Contents of db.properties are
db.host=localhost
db.port=50000
db.database=fhirdb
user=db2inst1
password=change-me
sslConnection=false

Good luck with Docker/IBM FHIR Server

Finding the Pesky Todo Javadoc Tags

The common pattern while developing code is to drop a TODO  in the code. Other tropes are FIXME, XXX markers in the code to indicate that something has to be done to connect the pieces or finish an implementation.

package test;
public Test {
// TODO add an implementation.
}

At some point, you lose the number, the todos, and those pesky tags are lost. I found the mvn plugin taglist finds those pesky tags.

I ran the command

mvn taglist:taglist -f fhir-parent/pom.xml -Daggregate=true

I wanted to transform the report, so I added to the module’s target’s taglist.xml.

<?xml-stylesheet type="text/xml" href="#stylesheet"?>
<!DOCTYPE catelog [
<!ATTLIST xsl:stylesheet
id ID #REQUIRED>
]>
<report>
<xsl:stylesheet id="stylesheet" version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
<html>
<body>
<table>
<tr>
<th>TYPE</th>
<th>FILE</th>
<th>LINE</th>
<th>Comment</th>
</tr>
<xsl:for-each select="report/tags/tag/files/file/comments/comment">
<tr>
<td><xsl:value-of select="../../../../@name"/></td>
<td><xsl:value-of select="../../@name"/></td>
<td><xsl:value-of select="lineNumber"/></td>
<td><xsl:value-of select="comment"/></td>
</tr>
</xsl:for-each>
</table>
</body>
</html>
</xsl:template>
</xsl:stylesheet>

When loaded in Firefox, it’ll generate a table output, and you can open an issue to address or fix the TODOs. 

I hope this helps you out. (GIST below)

Reference

Random EAR Projects

I had a similar experience and ran across https://wiki.eclipse.org/M2E-WTP/New_and_Noteworthy/1.1.0 .

m2e-wtp can also be disabled via the m2e.wtp.activation>false in the Maven section of your project pom;xml.

I create a new workspace and cloned a fresh copy of my code in a temp directory. I then updated the parent pom.xml with the following:

<properties>
<m2e.wtp.activation>false</m2e.wtp.activation>
<m2e.jaxrs.activation>false</m2e.jaxrs.activation>
<m2e.jpa.activation>false</m2e.jpa.activation>
<m2e.jsf.activation>false</m2e.jsf.activation>
<m2e.cdi.activation>true</m2e.cdi.activation>
</properties>

I then imported Existing Maven Projects and no extra EAR projects.

The  JAX-RS, JPA and JSF configurators can be enabled or disabled at a workspace level from Window > Preferences > Maven > Java EE Integration.

Lessons Learned for 20-SEPT-2019

XStream Illegal Reflective Access

If you are compiling with AdoptOpenJDK 11, you might hit –  “Illegal reflective access” com.thoughtworks.xstream.converters.collections.TreeMapConverter

If your build and your dependencies don’t show xstream in it, check your plugins:
mvn dependency:resolve-plugins -f fhir-parent/pom.xml | grep -B20 -i xst
[INFO] Plugin Resolved: maven-war-plugin-3.2.3.jar
[INFO] Plugin Dependency Resolved: maven-plugin-api-3.0.jar
[INFO] Plugin Dependency Resolved: maven-core-3.0.jar
[INFO] Plugin Dependency Resolved: maven-archiver-3.4.0.jar
[INFO] Plugin Dependency Resolved: commons-io-2.5.jar
[INFO] Plugin Dependency Resolved: plexus-archiver-4.1.0.jar
[INFO] Plugin Dependency Resolved: plexus-interpolation-1.25.jar
[INFO] Plugin Dependency Resolved: xstream-1.4.10.jar

I upgraded to the latest maven-war-plugin (3.1.0) and it was solved

Depgraph Maven Plugin

https://ferstl.github.io/depgraph-maven-plugin/aggregate-by-groupid-mojo.html

Use the depgraph-maven-plugin to aggregate the output as an image.  (I used this in the FHIR project to see if there were any unknown dependencies – https://github.com/IBM/FHIR/issues/87 )

 mvn com.github.ferstl:depgraph-maven-plugin:3.3.0:aggregate -f fhir-parent/pom.xml -DshowAllAttributesForJson=true -DcreateImage=true -DreduceEdges=true -DmergeClassifiers=true -DmergeTypes=true -Dexcludes=testng:: -DgraphFormat=json

Extract the Certs (All of Them)

Quick way to extract the main cert, and the intermediate CA and ROOT ca from a host.

echo “” | openssl s_client -showcerts -prexit -connect HOSTNAME:443 2> /dev/null | sed -n -e ‘/BEGIN CERTIFICATE/,/END CERTIFICATE/ p’

You’ll get a PEM as output (just capture into a file you can use)

Travis APIs

LINTING the Travis Yaml

curl -X POST https://api.travis-ci.com/lint -H “Accept: application/vnd.travis-ci.2+json” -H “Authorization: token <MYTOKEN>” –data-binary “@./.travis.yml” -v

 

Example response

{“lint”:{“warnings”:[{“key”:[“deploy”],”message”:”dropping \”file\” section: unexpected sequence”},{“key”:[],”message”:”value for \”addons\” section is empty, dropping”},{“key”:[“addons”],”message”:”unexpected key \”apt\”, dropping”}]}}

{
“lint”: {
“warnings”: [
{
“key”: [
“deploy”
],
“message”: “dropping \”file\” section: unexpected sequence”
}
]
}
}

{
“lint”: {
“warnings”: []
}
}

More detail at https://developer.travis-ci.com/resource/lint#Lint

Sonarcube – Maven and Docker analysis

SonarLint is one tool I gravitate towards – inline analysis of my code in Eclipse.  I have finally broken down and investigated using Sonarcube with maven – the heavy weight tool for evaluating code.  It’s exciting.

You need to pull your sonarqube docker. You can find more details at https://hub.docker.com/_/sonarqube/?tab=description

:~/my-repo$ docker pull sonarqube
Using default tag: latest
latest: Pulling from library/sonarqube
b8f262c62ec6: Pull complete
377e264464dd: Pull complete
bde67c1ff89f: Pull complete
6ba84ddbf1b2: Pull complete
ee22adb378a6: Pull complete
41d339c20e4f: Pull complete
25c2c6b7a1f3: Pull complete
4b36ae3e85ab: Pull complete
1062305937e9: Pull complete
Digest: sha256:032ae6e1021533a3731d5c6c0547615dea8d888dcec58802f8db3a9bd6f26237
Status: Downloaded newer image for sonarqube:latest
docker.io/library/sonarqube:latest

Start the container with a localhost hostname using

--hostname localhost.
$ docker run --hostname localhost -d --name sonarqube -p 9000:9000 sonarqube
d2c698884d4d01a527afd8f2231fcb6bbd514c5ed7c56d2dc3f7f7a758b4977d

Now, that sonarcube ist started, you can execute maven to generate the report.

mvn clean verify jacoco:report-aggregate sonar:sonar -Dsonar.host.url=http://localhost:9000 -f my-parent/pom.xml -DskipTests -pl '!../my-big-model/'

Once you execute the maven goals, you don’t want to see any ‘SKIPPED’.  If you do, you should add clean and verify to the goals you send to maven.

[INFO] Analysis total time: 59.857 s
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for My-Project:
[INFO]
[INFO] my-maven-parent ...................... SUCCESS [01:01 min]
[INFO] my-maven-project .......................................... SUCCESS [ 1.193 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:34 min
[INFO] Finished at: 2019-09-15T13:24:10-04:00
[INFO] ------------------------------------------------------------------------

For each project in the output, you see details about the execution, check to see that there are no WARNING or ERROR. If there are, you should check out some of the troubleshooting I did (at the end.)

[INFO] ------------- Run sensors on module my-maven-project
[INFO] Sensor JavaSquidSensor [java]
[INFO] Configured Java source version (sonar.java.source): 8
[INFO] JavaClasspath initialization
[INFO] JavaClasspath initialization (done) | time=1ms
[INFO] JavaTestClasspath initialization
[INFO] JavaTestClasspath initialization (done) | time=0ms
[INFO] Java Main Files AST scan
[INFO] 1 source files to be analyzed
[INFO] Java Main Files AST scan (done) | time=51ms
[INFO] Java Test Files AST scan
[INFO] 1/1 source files have been analyzed
[INFO] 0 source files to be analyzed
[INFO] 0/0 source files have been analyzed
[INFO] Java Test Files AST scan (done) | time=1ms
[INFO] Sensor JavaSquidSensor [java] (done) | time=65ms
[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
[INFO] Sensor SurefireSensor [java]
[INFO] parsing [/repo-folder/my-maven-project/target/surefire-reports]
[INFO] Sensor SurefireSensor [java] (done) | time=1ms
[INFO] Sensor JaCoCoSensor [java]
[INFO] Sensor JaCoCoSensor [java] (done) | time=0ms
[INFO] Sensor JavaXmlSensor [java]
[INFO] 1 source files to be analyzed
[INFO] Sensor JavaXmlSensor [java] (done) | time=5ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor HTML [web]
[INFO] Sensor HTML [web] (done) | time=0ms
[INFO] Sensor XML Sensor [xml]
[INFO] 1 source files to be analyzed
[INFO] Sensor XML Sensor [xml] (done) | time=4ms
[INFO] 1/1 source files have been analyzed

Towards the end of the execution you see:

[INFO] ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard?id=my.group%3Amy-parent
[INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
[INFO] More about the report processing at http://localhost:9000/api/ce/task?id=AW01-ot7J-mI0tW_q5b5

Checking the Report Processing I can see the successful result:

I can dig into the report tosee various recommendations and errors.

I can re-run and see the differences on demand.  This tool is awesome.

Finally stop the container.

docker stop container d2c698884d4d01a527afd8f2231fcb6bbd514c5ed7c56d2dc3f7f7a758b4977d

Good luck, I hope this helps.

Troubleshooting

AST Out of Memory

If you see
Exception in thread “Report about progress of Java AST analyzer” java.lang.OutOfMemoryError: Java heap space“, then
Set the memory boundaries https://stackoverflow.com/questions/12967427/sonar-outofmemoryerror-java-heap-space
export SONAR_SCANNER_OPTS="-Xmx3062m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=128m"


Exclude a Project
Out of Memory issue for specific projects, you can exclude them.
If you still see an issue use – -pl '!../my-big-model/' to skip the offending project (specifically if you have a parent in a different folder.

Missing Class

If you see

[WARNING] Classes not found during the analysis : [com.mypackage.MyClass]
[INFO] Java Test Files AST scan (done) | time=25ms

make sure you have clean and verify in the the goal list (the byte code should exist) you can also use package and install

Alternatively, if you see
[WARNING] The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
[WARNING] o com.mygroup:my-jar:jar:4.0.0-SNAPSHOT (compile)
[WARNING] Try running the build up to the lifecycle phase “package”
[WARNING] The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
Then make sure you could packagemvn clean package -f my-parent/pom.xml -DskipTests

Reference

https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-maven/
https://github.com/SonarSource/sonar-scanning-examples/tree/master/sonarqube-scanner-maven

Migrating Source Code Git-to-Git

Migrating source code is a pain in the butt, I know.  There are about 9 million variations, and one of interest to me – git to github.com. 

There are a number of tools to clean up your git history and prepare to move.

  • Git and Scripting
  • BFG Repo Cleaner
  • Git-Python
  • JGit

I found Git-Python a bit cumbersome, BFG Repo Cleaner more than I needed/wanted, and Git / Scripting too much work. After some prototyping, I opted for JGit from Eclipse and some Git knowhow.

First, I switched to the source Git Repo branch I wanted to migrate and exported the commit list.

git rev-list HEAD > commits.txt

which results in

7452e8eb1f287e2ad2d8c2d005455197ba4183f2

baac5e4d0ce999d983c016d67175a898f50444b3

2a8e2ec7507e05555e277f214bf79119cda4f025

This commits.txt is useful down the line.

I am a Maven disciple so, I created a maven java project with Java 1.8 and the following dependencies:

        <dependency>

            <groupId>org.eclipse.jgit</groupId>

            <artifactId>org.eclipse.jgit</artifactId>

            <version>${jgit.version}</version>

        </dependency>

        <dependency>

            <groupId>com.google.guava</groupId>

            <artifactId>guava</artifactId>

            <version>20.0</version>

        </dependency>

        <dependency>

            <groupId>org.slf4j</groupId>

            <artifactId>slf4j-nop</artifactId>

            <version>1.7.25</version>

        </dependency>

I used the JGit to check the list of commits (note the REPO here must have .git at end).

try (Git git = Git.open(new File(SOURCE_GIT_REPO))) {

            printHeaderLine();

            System.out.println("Starting Branch is " + git.getRepository().getBranch());

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                String binSha = commit.name();

                commits.add(binSha);

            }

        }

I flip it around, so I can process OLDEST to NEWEST

        Collections.reverse(commits);

I used the git log (LogCommand in JGit) to find out all the times a FILE was modified, and do custom processing:

try (Git git = Git.open(new File(REPO))) {

            LogCommand logCommand = git.log().add(git.getRepository().resolve(Constants.HEAD)).addPath(fileName.replace(REPO, ""));

            Set<String> years = new HashSet<>();

            for (RevCommit revCommit : logCommand.call()) {

                Instant instant = Instant.ofEpochSecond(revCommit.getCommitTime());

// YOUR PROCESSING

            }

        }

}

To find out the files specifically in the HEAD of the repo, gets the files and paths, and puts it in a List

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            if (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (RevWalk walk = new RevWalk(git.getRepository());) {

                    RevTree tree = walk.parseTree(commit.getId());

                    try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                        treeWalk.addTree(tree);

                        treeWalk.setRecursive(true);

                        while (treeWalk.next()) {

                            headFiles.add(treeWalk.getPathString());

                        }

                    }

                }

            } 

        }

}

I built a history of changes.

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (DiffFormatter df = new DiffFormatter(DisabledOutputStream.INSTANCE);) {

                    df.setRepository(git.getRepository());                    df.setDiffComparator(RawTextComparator.DEFAULT);

                    df.setDetectRenames(true);


                    CommitHistoryEntry.Builder builder =                            CommitHistoryEntry.builder().binsha(commit.name()).commitTime(commit.getCommitTime()).authorEmail(commit.getAuthorIdent().getEmailAddress()).shortMessage(commit.getShortMessage()).fullMessage(commit.getFullMessage());

                    RevCommit[] parents = commit.getParents();

                    if (parents != null && parents.length > 0) {

                        List<DiffEntry> diffs = df.scan(commit.getTree(), parents[0]);

                        builder.add(diffs);

                    } else {

                        builder.root(true);

                        try (RevWalk walk = new RevWalk(git.getRepository());) {

                            RevTree tree = walk.parseTree(commit.getId());

                            try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                                treeWalk.addTree(tree);

                                treeWalk.setRecursive(true);

                                while (treeWalk.next()) {                                   

builder.file(treeWalk.getPathString());

                                }

                            }

                        }

                    }

                    entries.add(entry);

                }

            }

I did implement the Visitor pattern to optimize the modifications to the commit details and cleanup any bad Identity mappings (folks had many emails and names which I unified) and cleanedup the email addresses.

Next, I created a destination git (all fresh and ready to go):

try (Git thisGit = Git.init().setDirectory(new File(REPO_DIR)).call()) {

   git = thisGit;

     }

One should make sure the path exists, and it doesn’t matter if you have files in it.

Commit the files in the git directory… you can commit without FILES!

CommitCommand commitCommand = git.commit();

        // Setup the Identity and date

        Date aWhen = new Date(entry.getCommitTime() * 1000);

        PersonIdent authorIdent =

                new PersonIdent(entry.getAuthorName(), entry.getAuthorEmail(), aWhen, TimeZone.getDefault());

        commitCommand.setCommitter(authorIdent);

        commitCommand.setAllowEmpty(true);

        commitCommand.setAuthor(authorIdent);

        commitCommand.setMessage(entry.getShortMessage());

        commitCommand.setNoVerify(true);

        commitCommand.setSign(false);

        commitCommand.call();

Note, you can set to almost any point in time.  As long as you don’t sign it, it’ll be OK.  I don’t recommend this as a general practice.

To grab the file, you can do a tree walk, and resolve to the object ID.

try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                            treeWalk.addTree(tree);

                            treeWalk.setRecursive(true);

                            int localCount = 0;

                            while (treeWalk.next()) {

                                String fileName = treeWalk.getPathString();

ObjectId objectId = treeWalk.getObjectId(0);

            ObjectLoader loader = git.getRepository().open(objectId);

            String fileOutput = GIT_OUTPUT + "/" + binSha + "/" + fileNameWithRelativePath;

            int last = fileOutput.lastIndexOf('/');

            String fileOutputDir = fileOutput.substring(0, last);

            File dir = new File(fileOutputDir);

            dir.mkdirs();

            // and then one can the loader to read the file

            try (FileOutputStream out =

                    new FileOutputStream(GIT_OUTPUT + "/" + binSha + "/"

                            + fileNameWithRelativePath);) {

                // System.out

                byte[] bytes = loader.getBytes();

                if (hasBeenModified(bytes, fileNameWithRelativePath)) {

                    loader.copyTo(out);

                    count++;

                    result = true;

                }

            }

Note, I did check if the file was duplicate, it saved a couple of steps.

If you want to add files, you can set:

commitCommand.setAll(addFiles);

git.add().addFilepattern(file).call();

Git in the background builds the DIFFs for any file that is not specially treated as binary in the .gitattributes file.

For each commit, I loaded the file – checking for stop-words, checked the copyright header, check the file type, and compared it against the head . 

Tip, if you need to reset from a bad test.

git reset --hard origin # reset the branch

rm -rf .git # reset the repo (also be sure to remove the files)

The moving of the branch, you can execute

cd <GIT_REPO>

git checkout <BRANCH_TO_MIGRATE>

git reset --hard origin

git pull

git gc --aggressive --prune=now

git push git@github.com:<MyOrg>/<DEST_REPO>.git <BRANCH_TO_MIGRATE>:master

Note, I did rename the master branch in the repo prior.  Voila.  550M+ Repo moved and cleaned up.

The repo is now migrated, and up-to-date.  I hope this helps you.

References

Rename Branch in Git

https://multiplestates.wordpress.com/2015/02/05/rename-a-local-and-remote-branch-in-git/

 

Rewrite History

https://help.github.com/en/articles/removing-sensitive-data-from-a-repository

https://stackoverflow.com/questions/tagged/git-rewrite-history

 

BFG Repo Cleaner

https://github.com/rtyley/bfg-repo-cleaner

https://rtyley.github.io/bfg-repo-cleaner/

 

JGit

https://www.vogella.com/tutorials/JGit/article.html

https://github.com/eclipse/jgit

http://wiki.eclipse.org/JGit/User_Guide#Repository

https://www.programcreek.com/java-api-examples/?class=org.eclipse.jgit.revwalk.RevWalk&method=parseCommit

https://www.eclipse.org/forums/index.php/t/213979/

https://stackoverflow.com/questions/46727610/how-to-get-the-list-of-files-as-a-part-of-commit-in-jgit

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/ListNotes.java

https://stackoverflow.com/questions/9683279/make-the-current-commit-the-only-initial-commit-in-a-git-repository

https://stackoverflow.com/questions/40590039/how-to-get-the-file-list-for-a-commit-with-jgit

https://doc.nuxeo.com/blog/jgit-example/

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/api/ReadFileFromCommit.java

https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit.test/tst/org/eclipse/jgit/api/AddCommandTest.java

https://stackoverflow.com/questions/12734760/jgit-how-to-add-all-files-to-staging-area

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/DiffFilesInCommit.java

 

Changing a Keystore and Key’s Password

Create a list of keystores

cat << EOF > keystore-list.txt
testTruststore.jks
testKeystore.jks
EOF
Iterate over the list to check status and process 
for KEYSTORE in `cat keystore-list.txt`
do
echo $KEYSTORE
[ ! -f $KEYSTORE ] && echo NOT


VAL="`cat $KEYSTORE | wc -l`"
[ ${VAL} -eq "1" ] && echo NOT_RIGHT

# show the private key / trust key
keytool -keystore $KEYSTORE -list -storepass ACTUAL_PASS 2>&1 | grep -v Warn | grep -v PKCS12 | grep -i PrivateKey
keytool -keystore $KEYSTORE -list -storepass ACTUAL_PASS 2>&1 | grep -v Warn | grep -v PKCS12 | grep -i Trust

done
Change the Passwords for the Key 

keytool -keypasswd -alias default -keypass OLDKEYPASS -new NEWpassword -keystore testKeystore.jks -storepass OLDPassword

keytool -storepasswd -keystore ./fhir-server-test/src/test/resources/fhirClientKeystore.jks -new change-password -storepass password

Reference
https://www.ibm.com/support/knowledgecenter/SS4GSP_6.2.4/com.ibm.udeploy.doc/topics/keystore_change_password.html
https://www.ibm.com/support/knowledgecenter/en/SS4GSP_7.0.3/com.ibm.udeploy.doc/topics/keystore_encryption_change_password.html