Application Development Notes: Tips and Tricks

LC_ALL

Exception = javax.resource.spi.ResourceAllocationException
Source = com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection
probeid = 299
Stack Dump = javax.resource.spi.ResourceAllocationExceptionDSRA8100E: Unable to get a XAConnection from the DataSource. with SQL State : XJ041 SQL Code : 40000
    at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1578)

LC_ALL is the way to fix.

Before executing the Java application.

export LC_ALL=en_US.UTF-8

https://stackoverflow.com/a/9894836/1873438
http://db.apache.org/derby/docs/10.5/tools/ctools1004764.html#ctools1004764__rtoolsijtools91283
https://stackoverflow.com/questions/55673886/what-is-the-difference-between-c-utf-8-and-en-us-utf-8-locales

Multi-module Javadocs where Search doesn’t resolve the context properly

<additionalJOption>--no-module-directories</additionalJOption>
or
<additionalOption>--no-module-directories</additionalOption>

Search will generate which avoid the <undefined>, and use the root folder.

https://stackoverflow.com/questions/52326318/maven-javadoc-search-redirects-to-undefined-url

Gatsby with the Carbon Theme

The IBM project I work on uses Carbon Theme with Gatsby to generate our HTML pages. The Carbon Design system has a nice git repository at GIT: gatsby-theme-carbon. I recommend cloning these repository.  The repository has an example and a place to figure out where and how to Shadow a feature – “provide their own components for the theme to use over the defaults”.

A couple of plugins I recommend are:
gatsby-plugin-slug – https://www.gatsbyjs.org/packages/gatsby-plugin-slug/ resolves inter-page links
gatsby-plugin-manifesthttps://www.gatsbyjs.org/packages/gatsby-plugin-manifest/?=gatsby-plugin-manifest favicon, general theme details, metadata, supports mdx and md formatted files.
gatsby-plugin-sitemap – https://www.gatsbyjs.org/packages/gatsby-plugin-sitemap/ makes the site searchable

To update the components, use

– Navigation (on left) – Update the src/data/nav-items.yaml
– Footer – Update the src/gatsby-theme-carbon/components/Footer.js Only one column and content area used. (RIGHT column not used)
– Header – The custom header src/gatsby-theme-carbon/components/Header.js is in this file. This is the only one that should be edited.  The other files are theme.

To see team’s configuration, you can view https://github.com/IBM/FHIR/tree/master/docs

Building a Carbon Site
0 – clone git: IBM/FHIR and change to /docs
1 – npm install
– if prompted, install these plugins:
npm install –save gatsby-plugin-manifest
npm install –save gatsby-plugin-slug
npm install –save gatsby-plugin-manifest
2 – gatsby build –prefix-paths
The build should output:
08:02:18-pb@pauls-mbp:~/git/wffh/oct27/FHIR/docs$ gatsby build –prefix-paths
success open and validate gatsby-configs – 0.090 s
success load plugins – 1.701 s
success onPreInit – 0.017 s
success delete html and css files from previous builds – 0.019 s
success initialize cache – 0.031 s
success copy gatsby files – 0.080 s
success onPreBootstrap – 1.931 s
success source and transform nodes – 4.337 s
success building schema – 0.272 s
success createPages – 0.014 s
success createPagesStatefully – 1.165 s
success onPreExtractQueries – 0.014 s
success update schema – 0.071 s
success extract queries from components – 0.243 s
success write out requires – 0.017 s
success write out redirect data – 0.015 s
success Build manifest and related icons – 0.110 s
success Build manifest and related icons – 0.129 s
success onPostBootstrap – 0.268 s

info bootstrap finished – 15.674 s

success run static queries – 0.049 s — 6/6 187.18 queries/second
success Building production JavaScript and CSS bundles – 105.557 s
success Rewriting compilation hashes – 0.029 s
success run page queries – 0.087 s — 19/19 312.56 queries/second
success Building static HTML for pages – 21.400 s — 19/19 14.81 pages/second
info Done building in 142.923 sec

3 – gatsby serve (to check)
4 – on a different repo, copy the files from `public/` over to the gh-pages branch at the root
5 – git add .
6 – git push

I hope this helps you with Carbon/Gatsby.

Additional Links
Gatsby Theme: Carbon https://github.com/carbon-design-system/gatsby-theme-carbon
Gatsby Theme Carbon: Guides https://gatsby-theme-carbon.now.sh/guides/
Carbon Design https://www.carbondesignsystem.com/components/link/code/
Carbon Theme Details http://react.carbondesignsystem.com/?path=/story/link–default
MDX https://mdxjs.com/

IBM Db2 with IBM FHIR Server – FHIR Schema

My colleague Albert has also merged a Dockerfile for testing, it’s a good place to start.


docker run -itd –name fhirdby –privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=password -e DBNAME=fhirdb -v database:/database ibmcom/db2


docker exec -ti fhirdby bash -c “su – db2inst1”
source .bashrc

# Recreate the DB with the right page size the tablespace is implictly created by the tool
[db2inst1@c58991a6a1a2 ~]$ db2 terminate
DB20000I The TERMINATE command completed successfully.
[db2inst1@c58991a6a1a2 ~]$ db2 drop db fhirdb
DB20000I The DROP DATABASE command completed successfully.
[db2inst1@c58991a6a1a2 ~]$ db2 CREATE DB FHIRDB using codeset UTF-8 territory us PAGESIZE 32768
DB20000I The CREATE DATABASE command completed successfully.

# Create the schemas
[db2inst1@c58991a6a1a2 ~]$ db2 connect to fhirdb
Database Connection Information
Database server = DB2/LINUXX8664 11.5.0.0
SQL authorization ID = DB2INST1
Local database alias = FHIRDB

[db2inst1@c58991a6a1a2 ~]$ db2 “create schema fhirdata”
DB20000I The SQL command completed successfully.
[db2inst1@c58991a6a1a2 ~]$ db2 “create schema fhir_admin”
DB20000I The SQL command completed successfully.

# Run the Schema creation with a small pool
–prop-file /Users/paulbastide/git/wffh/FHIR/fhir-persistence-schema/db.properties
–schema-name FHIRDATA
–pool-size 2
–update-schema

# Create a new tenant
–prop-file /Users/paulbastide/git/wffh/FHIR/fhir-persistence-schema/db.properties
–schema-name FHIRDATA
–allocate-tenant TNT1

Contents of db.properties are
db.host=localhost
db.port=50000
db.database=fhirdb
user=db2inst1
password=change-me
sslConnection=false

Good luck with Docker/IBM FHIR Server

Finding the Pesky Todo Javadoc Tags

The common pattern while developing code is to drop a TODO  in the code. Other tropes are FIXME, XXX markers in the code to indicate that something has to be done to connect the pieces or finish an implementation.

package test;
public Test {
// TODO add an implementation.
}

At some point, you lose the number, the todos, and those pesky tags are lost. I found the mvn plugin taglist finds those pesky tags.

I ran the command

mvn taglist:taglist -f fhir-parent/pom.xml -Daggregate=true

I wanted to transform the report, so I added to the module’s target’s taglist.xml.

<?xml-stylesheet type="text/xml" href="#stylesheet"?>
<!DOCTYPE catelog [
<!ATTLIST xsl:stylesheet
id ID #REQUIRED>
]>
<report>
<xsl:stylesheet id="stylesheet" version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
<html>
<body>
<table>
<tr>
<th>TYPE</th>
<th>FILE</th>
<th>LINE</th>
<th>Comment</th>
</tr>
<xsl:for-each select="report/tags/tag/files/file/comments/comment">
<tr>
<td><xsl:value-of select="../../../../@name"/></td>
<td><xsl:value-of select="../../@name"/></td>
<td><xsl:value-of select="lineNumber"/></td>
<td><xsl:value-of select="comment"/></td>
</tr>
</xsl:for-each>
</table>
</body>
</html>
</xsl:template>
</xsl:stylesheet>

When loaded in Firefox, it’ll generate a table output, and you can open an issue to address or fix the TODOs. 

I hope this helps you out. (GIST below)

Reference

Random EAR Projects

I had a similar experience and ran across https://wiki.eclipse.org/M2E-WTP/New_and_Noteworthy/1.1.0 .

m2e-wtp can also be disabled via the m2e.wtp.activation>false in the Maven section of your project pom;xml.

I create a new workspace and cloned a fresh copy of my code in a temp directory. I then updated the parent pom.xml with the following:

<properties>
<m2e.wtp.activation>false</m2e.wtp.activation>
<m2e.jaxrs.activation>false</m2e.jaxrs.activation>
<m2e.jpa.activation>false</m2e.jpa.activation>
<m2e.jsf.activation>false</m2e.jsf.activation>
<m2e.cdi.activation>true</m2e.cdi.activation>
</properties>

I then imported Existing Maven Projects and no extra EAR projects.

The  JAX-RS, JPA and JSF configurators can be enabled or disabled at a workspace level from Window > Preferences > Maven > Java EE Integration.

Lessons Learned for 20-SEPT-2019

XStream Illegal Reflective Access

If you are compiling with AdoptOpenJDK 11, you might hit –  “Illegal reflective access” com.thoughtworks.xstream.converters.collections.TreeMapConverter

If your build and your dependencies don’t show xstream in it, check your plugins:
mvn dependency:resolve-plugins -f fhir-parent/pom.xml | grep -B20 -i xst
[INFO] Plugin Resolved: maven-war-plugin-3.2.3.jar
[INFO] Plugin Dependency Resolved: maven-plugin-api-3.0.jar
[INFO] Plugin Dependency Resolved: maven-core-3.0.jar
[INFO] Plugin Dependency Resolved: maven-archiver-3.4.0.jar
[INFO] Plugin Dependency Resolved: commons-io-2.5.jar
[INFO] Plugin Dependency Resolved: plexus-archiver-4.1.0.jar
[INFO] Plugin Dependency Resolved: plexus-interpolation-1.25.jar
[INFO] Plugin Dependency Resolved: xstream-1.4.10.jar

I upgraded to the latest maven-war-plugin (3.1.0) and it was solved

Depgraph Maven Plugin

https://ferstl.github.io/depgraph-maven-plugin/aggregate-by-groupid-mojo.html

Use the depgraph-maven-plugin to aggregate the output as an image.  (I used this in the FHIR project to see if there were any unknown dependencies – https://github.com/IBM/FHIR/issues/87 )

 mvn com.github.ferstl:depgraph-maven-plugin:3.3.0:aggregate -f fhir-parent/pom.xml -DshowAllAttributesForJson=true -DcreateImage=true -DreduceEdges=true -DmergeClassifiers=true -DmergeTypes=true -Dexcludes=testng:: -DgraphFormat=json

Extract the Certs (All of Them)

Quick way to extract the main cert, and the intermediate CA and ROOT ca from a host.

echo “” | openssl s_client -showcerts -prexit -connect HOSTNAME:443 2> /dev/null | sed -n -e ‘/BEGIN CERTIFICATE/,/END CERTIFICATE/ p’

You’ll get a PEM as output (just capture into a file you can use)

Travis APIs

LINTING the Travis Yaml

curl -X POST https://api.travis-ci.com/lint -H “Accept: application/vnd.travis-ci.2+json” -H “Authorization: token <MYTOKEN>” –data-binary “@./.travis.yml” -v

 

Example response

{“lint”:{“warnings”:[{“key”:[“deploy”],”message”:”dropping \”file\” section: unexpected sequence”},{“key”:[],”message”:”value for \”addons\” section is empty, dropping”},{“key”:[“addons”],”message”:”unexpected key \”apt\”, dropping”}]}}

{
“lint”: {
“warnings”: [
{
“key”: [
“deploy”
],
“message”: “dropping \”file\” section: unexpected sequence”
}
]
}
}

{
“lint”: {
“warnings”: []
}
}

More detail at https://developer.travis-ci.com/resource/lint#Lint

Sonarcube – Maven and Docker analysis

SonarLint is one tool I gravitate towards – inline analysis of my code in Eclipse.  I have finally broken down and investigated using Sonarcube with maven – the heavy weight tool for evaluating code.  It’s exciting.

You need to pull your sonarqube docker. You can find more details at https://hub.docker.com/_/sonarqube/?tab=description

:~/my-repo$ docker pull sonarqube
Using default tag: latest
latest: Pulling from library/sonarqube
b8f262c62ec6: Pull complete
377e264464dd: Pull complete
bde67c1ff89f: Pull complete
6ba84ddbf1b2: Pull complete
ee22adb378a6: Pull complete
41d339c20e4f: Pull complete
25c2c6b7a1f3: Pull complete
4b36ae3e85ab: Pull complete
1062305937e9: Pull complete
Digest: sha256:032ae6e1021533a3731d5c6c0547615dea8d888dcec58802f8db3a9bd6f26237
Status: Downloaded newer image for sonarqube:latest
docker.io/library/sonarqube:latest

Start the container with a localhost hostname using

--hostname localhost.
$ docker run --hostname localhost -d --name sonarqube -p 9000:9000 sonarqube
d2c698884d4d01a527afd8f2231fcb6bbd514c5ed7c56d2dc3f7f7a758b4977d

Now, that sonarcube ist started, you can execute maven to generate the report.

mvn clean verify jacoco:report-aggregate sonar:sonar -Dsonar.host.url=http://localhost:9000 -f my-parent/pom.xml -DskipTests -pl '!../my-big-model/'

Once you execute the maven goals, you don’t want to see any ‘SKIPPED’.  If you do, you should add clean and verify to the goals you send to maven.

[INFO] Analysis total time: 59.857 s
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for My-Project:
[INFO]
[INFO] my-maven-parent ...................... SUCCESS [01:01 min]
[INFO] my-maven-project .......................................... SUCCESS [ 1.193 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:34 min
[INFO] Finished at: 2019-09-15T13:24:10-04:00
[INFO] ------------------------------------------------------------------------

For each project in the output, you see details about the execution, check to see that there are no WARNING or ERROR. If there are, you should check out some of the troubleshooting I did (at the end.)

[INFO] ------------- Run sensors on module my-maven-project
[INFO] Sensor JavaSquidSensor [java]
[INFO] Configured Java source version (sonar.java.source): 8
[INFO] JavaClasspath initialization
[INFO] JavaClasspath initialization (done) | time=1ms
[INFO] JavaTestClasspath initialization
[INFO] JavaTestClasspath initialization (done) | time=0ms
[INFO] Java Main Files AST scan
[INFO] 1 source files to be analyzed
[INFO] Java Main Files AST scan (done) | time=51ms
[INFO] Java Test Files AST scan
[INFO] 1/1 source files have been analyzed
[INFO] 0 source files to be analyzed
[INFO] 0/0 source files have been analyzed
[INFO] Java Test Files AST scan (done) | time=1ms
[INFO] Sensor JavaSquidSensor [java] (done) | time=65ms
[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
[INFO] Sensor SurefireSensor [java]
[INFO] parsing [/repo-folder/my-maven-project/target/surefire-reports]
[INFO] Sensor SurefireSensor [java] (done) | time=1ms
[INFO] Sensor JaCoCoSensor [java]
[INFO] Sensor JaCoCoSensor [java] (done) | time=0ms
[INFO] Sensor JavaXmlSensor [java]
[INFO] 1 source files to be analyzed
[INFO] Sensor JavaXmlSensor [java] (done) | time=5ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor HTML [web]
[INFO] Sensor HTML [web] (done) | time=0ms
[INFO] Sensor XML Sensor [xml]
[INFO] 1 source files to be analyzed
[INFO] Sensor XML Sensor [xml] (done) | time=4ms
[INFO] 1/1 source files have been analyzed

Towards the end of the execution you see:

[INFO] ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard?id=my.group%3Amy-parent
[INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
[INFO] More about the report processing at http://localhost:9000/api/ce/task?id=AW01-ot7J-mI0tW_q5b5

Checking the Report Processing I can see the successful result:

I can dig into the report tosee various recommendations and errors.

I can re-run and see the differences on demand.  This tool is awesome.

Finally stop the container.

docker stop container d2c698884d4d01a527afd8f2231fcb6bbd514c5ed7c56d2dc3f7f7a758b4977d

Good luck, I hope this helps.

Troubleshooting

AST Out of Memory

If you see
Exception in thread “Report about progress of Java AST analyzer” java.lang.OutOfMemoryError: Java heap space“, then
Set the memory boundaries https://stackoverflow.com/questions/12967427/sonar-outofmemoryerror-java-heap-space
export SONAR_SCANNER_OPTS="-Xmx3062m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=128m"


Exclude a Project
Out of Memory issue for specific projects, you can exclude them.
If you still see an issue use – -pl '!../my-big-model/' to skip the offending project (specifically if you have a parent in a different folder.

Missing Class

If you see

[WARNING] Classes not found during the analysis : [com.mypackage.MyClass]
[INFO] Java Test Files AST scan (done) | time=25ms

make sure you have clean and verify in the the goal list (the byte code should exist) you can also use package and install

Alternatively, if you see
[WARNING] The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
[WARNING] o com.mygroup:my-jar:jar:4.0.0-SNAPSHOT (compile)
[WARNING] Try running the build up to the lifecycle phase “package”
[WARNING] The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
Then make sure you could packagemvn clean package -f my-parent/pom.xml -DskipTests

Reference

https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-maven/
https://github.com/SonarSource/sonar-scanning-examples/tree/master/sonarqube-scanner-maven

Migrating Source Code Git-to-Git

Migrating source code is a pain in the butt, I know.  There are about 9 million variations, and one of interest to me – git to github.com. 

There are a number of tools to clean up your git history and prepare to move.

  • Git and Scripting
  • BFG Repo Cleaner
  • Git-Python
  • JGit

I found Git-Python a bit cumbersome, BFG Repo Cleaner more than I needed/wanted, and Git / Scripting too much work. After some prototyping, I opted for JGit from Eclipse and some Git knowhow.

First, I switched to the source Git Repo branch I wanted to migrate and exported the commit list.

git rev-list HEAD > commits.txt

which results in

7452e8eb1f287e2ad2d8c2d005455197ba4183f2

baac5e4d0ce999d983c016d67175a898f50444b3

2a8e2ec7507e05555e277f214bf79119cda4f025

This commits.txt is useful down the line.

I am a Maven disciple so, I created a maven java project with Java 1.8 and the following dependencies:

        <dependency>

            <groupId>org.eclipse.jgit</groupId>

            <artifactId>org.eclipse.jgit</artifactId>

            <version>${jgit.version}</version>

        </dependency>

        <dependency>

            <groupId>com.google.guava</groupId>

            <artifactId>guava</artifactId>

            <version>20.0</version>

        </dependency>

        <dependency>

            <groupId>org.slf4j</groupId>

            <artifactId>slf4j-nop</artifactId>

            <version>1.7.25</version>

        </dependency>

I used the JGit to check the list of commits (note the REPO here must have .git at end).

try (Git git = Git.open(new File(SOURCE_GIT_REPO))) {

            printHeaderLine();

            System.out.println("Starting Branch is " + git.getRepository().getBranch());

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                String binSha = commit.name();

                commits.add(binSha);

            }

        }

I flip it around, so I can process OLDEST to NEWEST

        Collections.reverse(commits);

I used the git log (LogCommand in JGit) to find out all the times a FILE was modified, and do custom processing:

try (Git git = Git.open(new File(REPO))) {

            LogCommand logCommand = git.log().add(git.getRepository().resolve(Constants.HEAD)).addPath(fileName.replace(REPO, ""));

            Set<String> years = new HashSet<>();

            for (RevCommit revCommit : logCommand.call()) {

                Instant instant = Instant.ofEpochSecond(revCommit.getCommitTime());

// YOUR PROCESSING

            }

        }

}

To find out the files specifically in the HEAD of the repo, gets the files and paths, and puts it in a List

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            if (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (RevWalk walk = new RevWalk(git.getRepository());) {

                    RevTree tree = walk.parseTree(commit.getId());

                    try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                        treeWalk.addTree(tree);

                        treeWalk.setRecursive(true);

                        while (treeWalk.next()) {

                            headFiles.add(treeWalk.getPathString());

                        }

                    }

                }

            } 

        }

}

I built a history of changes.

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (DiffFormatter df = new DiffFormatter(DisabledOutputStream.INSTANCE);) {

                    df.setRepository(git.getRepository());                    df.setDiffComparator(RawTextComparator.DEFAULT);

                    df.setDetectRenames(true);


                    CommitHistoryEntry.Builder builder =                            CommitHistoryEntry.builder().binsha(commit.name()).commitTime(commit.getCommitTime()).authorEmail(commit.getAuthorIdent().getEmailAddress()).shortMessage(commit.getShortMessage()).fullMessage(commit.getFullMessage());

                    RevCommit[] parents = commit.getParents();

                    if (parents != null && parents.length > 0) {

                        List<DiffEntry> diffs = df.scan(commit.getTree(), parents[0]);

                        builder.add(diffs);

                    } else {

                        builder.root(true);

                        try (RevWalk walk = new RevWalk(git.getRepository());) {

                            RevTree tree = walk.parseTree(commit.getId());

                            try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                                treeWalk.addTree(tree);

                                treeWalk.setRecursive(true);

                                while (treeWalk.next()) {                                   

builder.file(treeWalk.getPathString());

                                }

                            }

                        }

                    }

                    entries.add(entry);

                }

            }

I did implement the Visitor pattern to optimize the modifications to the commit details and cleanup any bad Identity mappings (folks had many emails and names which I unified) and cleanedup the email addresses.

Next, I created a destination git (all fresh and ready to go):

try (Git thisGit = Git.init().setDirectory(new File(REPO_DIR)).call()) {

   git = thisGit;

     }

One should make sure the path exists, and it doesn’t matter if you have files in it.

Commit the files in the git directory… you can commit without FILES!

CommitCommand commitCommand = git.commit();

        // Setup the Identity and date

        Date aWhen = new Date(entry.getCommitTime() * 1000);

        PersonIdent authorIdent =

                new PersonIdent(entry.getAuthorName(), entry.getAuthorEmail(), aWhen, TimeZone.getDefault());

        commitCommand.setCommitter(authorIdent);

        commitCommand.setAllowEmpty(true);

        commitCommand.setAuthor(authorIdent);

        commitCommand.setMessage(entry.getShortMessage());

        commitCommand.setNoVerify(true);

        commitCommand.setSign(false);

        commitCommand.call();

Note, you can set to almost any point in time.  As long as you don’t sign it, it’ll be OK.  I don’t recommend this as a general practice.

To grab the file, you can do a tree walk, and resolve to the object ID.

try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                            treeWalk.addTree(tree);

                            treeWalk.setRecursive(true);

                            int localCount = 0;

                            while (treeWalk.next()) {

                                String fileName = treeWalk.getPathString();

ObjectId objectId = treeWalk.getObjectId(0);

            ObjectLoader loader = git.getRepository().open(objectId);

            String fileOutput = GIT_OUTPUT + "/" + binSha + "/" + fileNameWithRelativePath;

            int last = fileOutput.lastIndexOf('/');

            String fileOutputDir = fileOutput.substring(0, last);

            File dir = new File(fileOutputDir);

            dir.mkdirs();

            // and then one can the loader to read the file

            try (FileOutputStream out =

                    new FileOutputStream(GIT_OUTPUT + "/" + binSha + "/"

                            + fileNameWithRelativePath);) {

                // System.out

                byte[] bytes = loader.getBytes();

                if (hasBeenModified(bytes, fileNameWithRelativePath)) {

                    loader.copyTo(out);

                    count++;

                    result = true;

                }

            }

Note, I did check if the file was duplicate, it saved a couple of steps.

If you want to add files, you can set:

commitCommand.setAll(addFiles);

git.add().addFilepattern(file).call();

Git in the background builds the DIFFs for any file that is not specially treated as binary in the .gitattributes file.

For each commit, I loaded the file – checking for stop-words, checked the copyright header, check the file type, and compared it against the head . 

Tip, if you need to reset from a bad test.

git reset --hard origin # reset the branch

rm -rf .git # reset the repo (also be sure to remove the files)

The moving of the branch, you can execute

cd <GIT_REPO>

git checkout <BRANCH_TO_MIGRATE>

git reset --hard origin

git pull

git gc --aggressive --prune=now

git push git@github.com:<MyOrg>/<DEST_REPO>.git <BRANCH_TO_MIGRATE>:master

Note, I did rename the master branch in the repo prior.  Voila.  550M+ Repo moved and cleaned up.

The repo is now migrated, and up-to-date.  I hope this helps you.

References

Rename Branch in Git

https://multiplestates.wordpress.com/2015/02/05/rename-a-local-and-remote-branch-in-git/

 

Rewrite History

https://help.github.com/en/articles/removing-sensitive-data-from-a-repository

https://stackoverflow.com/questions/tagged/git-rewrite-history

 

BFG Repo Cleaner

https://github.com/rtyley/bfg-repo-cleaner

https://rtyley.github.io/bfg-repo-cleaner/

 

JGit

https://www.vogella.com/tutorials/JGit/article.html

https://github.com/eclipse/jgit

http://wiki.eclipse.org/JGit/User_Guide#Repository

https://www.programcreek.com/java-api-examples/?class=org.eclipse.jgit.revwalk.RevWalk&method=parseCommit

https://www.eclipse.org/forums/index.php/t/213979/

https://stackoverflow.com/questions/46727610/how-to-get-the-list-of-files-as-a-part-of-commit-in-jgit

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/ListNotes.java

https://stackoverflow.com/questions/9683279/make-the-current-commit-the-only-initial-commit-in-a-git-repository

https://stackoverflow.com/questions/40590039/how-to-get-the-file-list-for-a-commit-with-jgit

https://doc.nuxeo.com/blog/jgit-example/

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/api/ReadFileFromCommit.java

https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit.test/tst/org/eclipse/jgit/api/AddCommandTest.java

https://stackoverflow.com/questions/12734760/jgit-how-to-add-all-files-to-staging-area

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/DiffFilesInCommit.java

 

Changing a Keystore and Key’s Password

Create a list of keystores

cat << EOF > keystore-list.txt
testTruststore.jks
testKeystore.jks
EOF
Iterate over the list to check status and process 
for KEYSTORE in `cat keystore-list.txt`
do
echo $KEYSTORE
[ ! -f $KEYSTORE ] && echo NOT


VAL="`cat $KEYSTORE | wc -l`"
[ ${VAL} -eq "1" ] && echo NOT_RIGHT

# show the private key / trust key
keytool -keystore $KEYSTORE -list -storepass ACTUAL_PASS 2>&1 | grep -v Warn | grep -v PKCS12 | grep -i PrivateKey
keytool -keystore $KEYSTORE -list -storepass ACTUAL_PASS 2>&1 | grep -v Warn | grep -v PKCS12 | grep -i Trust

done
Change the Passwords for the Key 

keytool -keypasswd -alias default -keypass OLDKEYPASS -new NEWpassword -keystore testKeystore.jks -storepass OLDPassword

keytool -storepasswd -keystore ./fhir-server-test/src/test/resources/fhirClientKeystore.jks -new change-password -storepass password

Reference
https://www.ibm.com/support/knowledgecenter/SS4GSP_6.2.4/com.ibm.udeploy.doc/topics/keystore_change_password.html
https://www.ibm.com/support/knowledgecenter/en/SS4GSP_7.0.3/com.ibm.udeploy.doc/topics/keystore_encryption_change_password.html

EAR Projects Generated for JavaEE Import

Importing my project into Eclipse, I found so many additional EAR projects were being generated. To stop this feature, I went to Eclipse > Preference > Maven > Java EE Integration and unchecked Enable Java EE Configuration. I removed the cached EAR projects (deleting from disk) and removed the regular projects, and imported again.  Voila…. it worked.

 

Reference: https://stackoverflow.com/questions/50613029/eclipse-import-existing-maven-projects-create-ear-project-automatically