Lessons Learned for 20-SEPT-2019

XStream Illegal Reflective Access

If you are compiling with AdoptOpenJDK 11, you might hit –  “Illegal reflective access” com.thoughtworks.xstream.converters.collections.TreeMapConverter

If your build and your dependencies don’t show xstream in it, check your plugins:
mvn dependency:resolve-plugins -f fhir-parent/pom.xml | grep -B20 -i xst
[INFO] Plugin Resolved: maven-war-plugin-3.2.3.jar
[INFO] Plugin Dependency Resolved: maven-plugin-api-3.0.jar
[INFO] Plugin Dependency Resolved: maven-core-3.0.jar
[INFO] Plugin Dependency Resolved: maven-archiver-3.4.0.jar
[INFO] Plugin Dependency Resolved: commons-io-2.5.jar
[INFO] Plugin Dependency Resolved: plexus-archiver-4.1.0.jar
[INFO] Plugin Dependency Resolved: plexus-interpolation-1.25.jar
[INFO] Plugin Dependency Resolved: xstream-1.4.10.jar

I upgraded to the latest maven-war-plugin (3.1.0) and it was solved

Depgraph Maven Plugin

https://ferstl.github.io/depgraph-maven-plugin/aggregate-by-groupid-mojo.html

Use the depgraph-maven-plugin to aggregate the output as an image.  (I used this in the FHIR project to see if there were any unknown dependencies – https://github.com/IBM/FHIR/issues/87 )

 mvn com.github.ferstl:depgraph-maven-plugin:3.3.0:aggregate -f fhir-parent/pom.xml -DshowAllAttributesForJson=true -DcreateImage=true -DreduceEdges=true -DmergeClassifiers=true -DmergeTypes=true -Dexcludes=testng:: -DgraphFormat=json

Extract the Certs (All of Them)

Quick way to extract the main cert, and the intermediate CA and ROOT ca from a host.

echo “” | openssl s_client -showcerts -prexit -connect HOSTNAME:443 2> /dev/null | sed -n -e ‘/BEGIN CERTIFICATE/,/END CERTIFICATE/ p’

You’ll get a PEM as output (just capture into a file you can use)

Travis APIs

LINTING the Travis Yaml

curl -X POST https://api.travis-ci.com/lint -H “Accept: application/vnd.travis-ci.2+json” -H “Authorization: token <MYTOKEN>” –data-binary “@./.travis.yml” -v

 

Example response

{“lint”:{“warnings”:[{“key”:[“deploy”],”message”:”dropping \”file\” section: unexpected sequence”},{“key”:[],”message”:”value for \”addons\” section is empty, dropping”},{“key”:[“addons”],”message”:”unexpected key \”apt\”, dropping”}]}}

{
“lint”: {
“warnings”: [
{
“key”: [
“deploy”
],
“message”: “dropping \”file\” section: unexpected sequence”
}
]
}
}

{
“lint”: {
“warnings”: []
}
}

More detail at https://developer.travis-ci.com/resource/lint#Lint

Sonarcube – Maven and Docker analysis

SonarLint is one tool I gravitate towards – inline analysis of my code in Eclipse.  I have finally broken down and investigated using Sonarcube with maven – the heavy weight tool for evaluating code.  It’s exciting.

You need to pull your sonarqube docker. You can find more details at https://hub.docker.com/_/sonarqube/?tab=description

:~/my-repo$ docker pull sonarqube
Using default tag: latest
latest: Pulling from library/sonarqube
b8f262c62ec6: Pull complete
377e264464dd: Pull complete
bde67c1ff89f: Pull complete
6ba84ddbf1b2: Pull complete
ee22adb378a6: Pull complete
41d339c20e4f: Pull complete
25c2c6b7a1f3: Pull complete
4b36ae3e85ab: Pull complete
1062305937e9: Pull complete
Digest: sha256:032ae6e1021533a3731d5c6c0547615dea8d888dcec58802f8db3a9bd6f26237
Status: Downloaded newer image for sonarqube:latest
docker.io/library/sonarqube:latest

Start the container with a localhost hostname using

--hostname localhost.
$ docker run --hostname localhost -d --name sonarqube -p 9000:9000 sonarqube
d2c698884d4d01a527afd8f2231fcb6bbd514c5ed7c56d2dc3f7f7a758b4977d

Now, that sonarcube ist started, you can execute maven to generate the report.

mvn clean verify jacoco:report-aggregate sonar:sonar -Dsonar.host.url=http://localhost:9000 -f my-parent/pom.xml -DskipTests -pl '!../my-big-model/'

Once you execute the maven goals, you don’t want to see any ‘SKIPPED’.  If you do, you should add clean and verify to the goals you send to maven.

[INFO] Analysis total time: 59.857 s
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for My-Project:
[INFO]
[INFO] my-maven-parent ...................... SUCCESS [01:01 min]
[INFO] my-maven-project .......................................... SUCCESS [ 1.193 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:34 min
[INFO] Finished at: 2019-09-15T13:24:10-04:00
[INFO] ------------------------------------------------------------------------

For each project in the output, you see details about the execution, check to see that there are no WARNING or ERROR. If there are, you should check out some of the troubleshooting I did (at the end.)

[INFO] ------------- Run sensors on module my-maven-project
[INFO] Sensor JavaSquidSensor [java]
[INFO] Configured Java source version (sonar.java.source): 8
[INFO] JavaClasspath initialization
[INFO] JavaClasspath initialization (done) | time=1ms
[INFO] JavaTestClasspath initialization
[INFO] JavaTestClasspath initialization (done) | time=0ms
[INFO] Java Main Files AST scan
[INFO] 1 source files to be analyzed
[INFO] Java Main Files AST scan (done) | time=51ms
[INFO] Java Test Files AST scan
[INFO] 1/1 source files have been analyzed
[INFO] 0 source files to be analyzed
[INFO] 0/0 source files have been analyzed
[INFO] Java Test Files AST scan (done) | time=1ms
[INFO] Sensor JavaSquidSensor [java] (done) | time=65ms
[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
[INFO] Sensor SurefireSensor [java]
[INFO] parsing [/repo-folder/my-maven-project/target/surefire-reports]
[INFO] Sensor SurefireSensor [java] (done) | time=1ms
[INFO] Sensor JaCoCoSensor [java]
[INFO] Sensor JaCoCoSensor [java] (done) | time=0ms
[INFO] Sensor JavaXmlSensor [java]
[INFO] 1 source files to be analyzed
[INFO] Sensor JavaXmlSensor [java] (done) | time=5ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor HTML [web]
[INFO] Sensor HTML [web] (done) | time=0ms
[INFO] Sensor XML Sensor [xml]
[INFO] 1 source files to be analyzed
[INFO] Sensor XML Sensor [xml] (done) | time=4ms
[INFO] 1/1 source files have been analyzed

Towards the end of the execution you see:

[INFO] ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard?id=my.group%3Amy-parent
[INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
[INFO] More about the report processing at http://localhost:9000/api/ce/task?id=AW01-ot7J-mI0tW_q5b5

Checking the Report Processing I can see the successful result:

I can dig into the report tosee various recommendations and errors.

I can re-run and see the differences on demand.  This tool is awesome.

Finally stop the container.

docker stop container d2c698884d4d01a527afd8f2231fcb6bbd514c5ed7c56d2dc3f7f7a758b4977d

Good luck, I hope this helps.

Troubleshooting

AST Out of Memory

If you see
Exception in thread “Report about progress of Java AST analyzer” java.lang.OutOfMemoryError: Java heap space“, then
Set the memory boundaries https://stackoverflow.com/questions/12967427/sonar-outofmemoryerror-java-heap-space
export SONAR_SCANNER_OPTS="-Xmx3062m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=128m"


Exclude a Project
Out of Memory issue for specific projects, you can exclude them.
If you still see an issue use – -pl '!../my-big-model/' to skip the offending project (specifically if you have a parent in a different folder.

Missing Class

If you see

[WARNING] Classes not found during the analysis : [com.mypackage.MyClass]
[INFO] Java Test Files AST scan (done) | time=25ms

make sure you have clean and verify in the the goal list (the byte code should exist) you can also use package and install

Alternatively, if you see
[WARNING] The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
[WARNING] o com.mygroup:my-jar:jar:4.0.0-SNAPSHOT (compile)
[WARNING] Try running the build up to the lifecycle phase “package”
[WARNING] The following dependencies could not be resolved at this point of the build but seem to be part of the reactor:
Then make sure you could packagemvn clean package -f my-parent/pom.xml -DskipTests

Reference

https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-maven/
https://github.com/SonarSource/sonar-scanning-examples/tree/master/sonarqube-scanner-maven

Migrating Source Code Git-to-Git

Migrating source code is a pain in the butt, I know.  There are about 9 million variations, and one of interest to me – git to github.com. 

There are a number of tools to clean up your git history and prepare to move.

  • Git and Scripting
  • BFG Repo Cleaner
  • Git-Python
  • JGit

I found Git-Python a bit cumbersome, BFG Repo Cleaner more than I needed/wanted, and Git / Scripting too much work. After some prototyping, I opted for JGit from Eclipse and some Git knowhow.

First, I switched to the source Git Repo branch I wanted to migrate and exported the commit list.

git rev-list HEAD > commits.txt

which results in

7452e8eb1f287e2ad2d8c2d005455197ba4183f2

baac5e4d0ce999d983c016d67175a898f50444b3

2a8e2ec7507e05555e277f214bf79119cda4f025

This commits.txt is useful down the line.

I am a Maven disciple so, I created a maven java project with Java 1.8 and the following dependencies:

        <dependency>

            <groupId>org.eclipse.jgit</groupId>

            <artifactId>org.eclipse.jgit</artifactId>

            <version>${jgit.version}</version>

        </dependency>

        <dependency>

            <groupId>com.google.guava</groupId>

            <artifactId>guava</artifactId>

            <version>20.0</version>

        </dependency>

        <dependency>

            <groupId>org.slf4j</groupId>

            <artifactId>slf4j-nop</artifactId>

            <version>1.7.25</version>

        </dependency>

I used the JGit to check the list of commits (note the REPO here must have .git at end).

try (Git git = Git.open(new File(SOURCE_GIT_REPO))) {

            printHeaderLine();

            System.out.println("Starting Branch is " + git.getRepository().getBranch());

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                String binSha = commit.name();

                commits.add(binSha);

            }

        }

I flip it around, so I can process OLDEST to NEWEST

        Collections.reverse(commits);

I used the git log (LogCommand in JGit) to find out all the times a FILE was modified, and do custom processing:

try (Git git = Git.open(new File(REPO))) {

            LogCommand logCommand = git.log().add(git.getRepository().resolve(Constants.HEAD)).addPath(fileName.replace(REPO, ""));

            Set<String> years = new HashSet<>();

            for (RevCommit revCommit : logCommand.call()) {

                Instant instant = Instant.ofEpochSecond(revCommit.getCommitTime());

// YOUR PROCESSING

            }

        }

}

To find out the files specifically in the HEAD of the repo, gets the files and paths, and puts it in a List

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            if (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (RevWalk walk = new RevWalk(git.getRepository());) {

                    RevTree tree = walk.parseTree(commit.getId());

                    try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                        treeWalk.addTree(tree);

                        treeWalk.setRecursive(true);

                        while (treeWalk.next()) {

                            headFiles.add(treeWalk.getPathString());

                        }

                    }

                }

            } 

        }

}

I built a history of changes.

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (DiffFormatter df = new DiffFormatter(DisabledOutputStream.INSTANCE);) {

                    df.setRepository(git.getRepository());                    df.setDiffComparator(RawTextComparator.DEFAULT);

                    df.setDetectRenames(true);


                    CommitHistoryEntry.Builder builder =                            CommitHistoryEntry.builder().binsha(commit.name()).commitTime(commit.getCommitTime()).authorEmail(commit.getAuthorIdent().getEmailAddress()).shortMessage(commit.getShortMessage()).fullMessage(commit.getFullMessage());

                    RevCommit[] parents = commit.getParents();

                    if (parents != null && parents.length > 0) {

                        List<DiffEntry> diffs = df.scan(commit.getTree(), parents[0]);

                        builder.add(diffs);

                    } else {

                        builder.root(true);

                        try (RevWalk walk = new RevWalk(git.getRepository());) {

                            RevTree tree = walk.parseTree(commit.getId());

                            try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                                treeWalk.addTree(tree);

                                treeWalk.setRecursive(true);

                                while (treeWalk.next()) {                                   

builder.file(treeWalk.getPathString());

                                }

                            }

                        }

                    }

                    entries.add(entry);

                }

            }

I did implement the Visitor pattern to optimize the modifications to the commit details and cleanup any bad Identity mappings (folks had many emails and names which I unified) and cleanedup the email addresses.

Next, I created a destination git (all fresh and ready to go):

try (Git thisGit = Git.init().setDirectory(new File(REPO_DIR)).call()) {

   git = thisGit;

     }

One should make sure the path exists, and it doesn’t matter if you have files in it.

Commit the files in the git directory… you can commit without FILES!

CommitCommand commitCommand = git.commit();

        // Setup the Identity and date

        Date aWhen = new Date(entry.getCommitTime() * 1000);

        PersonIdent authorIdent =

                new PersonIdent(entry.getAuthorName(), entry.getAuthorEmail(), aWhen, TimeZone.getDefault());

        commitCommand.setCommitter(authorIdent);

        commitCommand.setAllowEmpty(true);

        commitCommand.setAuthor(authorIdent);

        commitCommand.setMessage(entry.getShortMessage());

        commitCommand.setNoVerify(true);

        commitCommand.setSign(false);

        commitCommand.call();

Note, you can set to almost any point in time.  As long as you don’t sign it, it’ll be OK.  I don’t recommend this as a general practice.

To grab the file, you can do a tree walk, and resolve to the object ID.

try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                            treeWalk.addTree(tree);

                            treeWalk.setRecursive(true);

                            int localCount = 0;

                            while (treeWalk.next()) {

                                String fileName = treeWalk.getPathString();

ObjectId objectId = treeWalk.getObjectId(0);

            ObjectLoader loader = git.getRepository().open(objectId);

            String fileOutput = GIT_OUTPUT + "/" + binSha + "/" + fileNameWithRelativePath;

            int last = fileOutput.lastIndexOf('/');

            String fileOutputDir = fileOutput.substring(0, last);

            File dir = new File(fileOutputDir);

            dir.mkdirs();

            // and then one can the loader to read the file

            try (FileOutputStream out =

                    new FileOutputStream(GIT_OUTPUT + "/" + binSha + "/"

                            + fileNameWithRelativePath);) {

                // System.out

                byte[] bytes = loader.getBytes();

                if (hasBeenModified(bytes, fileNameWithRelativePath)) {

                    loader.copyTo(out);

                    count++;

                    result = true;

                }

            }

Note, I did check if the file was duplicate, it saved a couple of steps.

If you want to add files, you can set:

commitCommand.setAll(addFiles);

git.add().addFilepattern(file).call();

Git in the background builds the DIFFs for any file that is not specially treated as binary in the .gitattributes file.

For each commit, I loaded the file – checking for stop-words, checked the copyright header, check the file type, and compared it against the head . 

Tip, if you need to reset from a bad test.

git reset --hard origin # reset the branch

rm -rf .git # reset the repo (also be sure to remove the files)

The moving of the branch, you can execute

cd <GIT_REPO>

git checkout <BRANCH_TO_MIGRATE>

git reset --hard origin

git pull

git gc --aggressive --prune=now

git push git@github.com:<MyOrg>/<DEST_REPO>.git <BRANCH_TO_MIGRATE>:master

Note, I did rename the master branch in the repo prior.  Voila.  550M+ Repo moved and cleaned up.

The repo is now migrated, and up-to-date.  I hope this helps you.

References

Rename Branch in Git

https://multiplestates.wordpress.com/2015/02/05/rename-a-local-and-remote-branch-in-git/

 

Rewrite History

https://help.github.com/en/articles/removing-sensitive-data-from-a-repository

https://stackoverflow.com/questions/tagged/git-rewrite-history

 

BFG Repo Cleaner

https://github.com/rtyley/bfg-repo-cleaner

https://rtyley.github.io/bfg-repo-cleaner/

 

JGit

https://www.vogella.com/tutorials/JGit/article.html

https://github.com/eclipse/jgit

http://wiki.eclipse.org/JGit/User_Guide#Repository

https://www.programcreek.com/java-api-examples/?class=org.eclipse.jgit.revwalk.RevWalk&method=parseCommit

https://www.eclipse.org/forums/index.php/t/213979/

https://stackoverflow.com/questions/46727610/how-to-get-the-list-of-files-as-a-part-of-commit-in-jgit

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/ListNotes.java

https://stackoverflow.com/questions/9683279/make-the-current-commit-the-only-initial-commit-in-a-git-repository

https://stackoverflow.com/questions/40590039/how-to-get-the-file-list-for-a-commit-with-jgit

https://doc.nuxeo.com/blog/jgit-example/

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/api/ReadFileFromCommit.java

https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit.test/tst/org/eclipse/jgit/api/AddCommandTest.java

https://stackoverflow.com/questions/12734760/jgit-how-to-add-all-files-to-staging-area

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/DiffFilesInCommit.java

 

Changing a Keystore and Key’s Password

Create a list of keystores

cat << EOF > keystore-list.txt
testTruststore.jks
testKeystore.jks
EOF
Iterate over the list to check status and process 
for KEYSTORE in `cat keystore-list.txt`
do
echo $KEYSTORE
[ ! -f $KEYSTORE ] && echo NOT


VAL="`cat $KEYSTORE | wc -l`"
[ ${VAL} -eq "1" ] && echo NOT_RIGHT

# show the private key / trust key
keytool -keystore $KEYSTORE -list -storepass ACTUAL_PASS 2>&1 | grep -v Warn | grep -v PKCS12 | grep -i PrivateKey
keytool -keystore $KEYSTORE -list -storepass ACTUAL_PASS 2>&1 | grep -v Warn | grep -v PKCS12 | grep -i Trust

done
Change the Passwords for the Key 

keytool -keypasswd -alias default -keypass OLDKEYPASS -new NEWpassword -keystore testKeystore.jks -storepass OLDPassword

keytool -storepasswd -keystore ./fhir-server-test/src/test/resources/fhirClientKeystore.jks -new change-password -storepass password

Reference
https://www.ibm.com/support/knowledgecenter/SS4GSP_6.2.4/com.ibm.udeploy.doc/topics/keystore_change_password.html
https://www.ibm.com/support/knowledgecenter/en/SS4GSP_7.0.3/com.ibm.udeploy.doc/topics/keystore_encryption_change_password.html

EAR Projects Generated for JavaEE Import

Importing my project into Eclipse, I found so many additional EAR projects were being generated. To stop this feature, I went to Eclipse > Preference > Maven > Java EE Integration and unchecked Enable Java EE Configuration. I removed the cached EAR projects (deleting from disk) and removed the regular projects, and imported again.  Voila…. it worked.

 

Reference: https://stackoverflow.com/questions/50613029/eclipse-import-existing-maven-projects-create-ear-project-automatically

Using Maven to Port Multiple version of the same code

As you can tell, I am a maven power user.  I use it for most of my projects (sometimes we run into pure Eclipse, Gradle and Ant). 

I started work on porting one version of the exact version of one model to a new version of the same model (same classes). I need to use both JARs in the same classpath.  This leads to a problem – how to use both at the same time.  

I like to use the maven-shade-plugin and relocate the class files such that package.class-file is relocated to package.old.class-file.  Generally this works well (except where you use reflexsive pathing in Class.forName to look up a class). I was able to prepend a string before shading the class.

The pom.xml I used is much like the attached project.  I hope this helps you all.

Maven Surefire runs out of memory

I’m running a complicated integration test, and needed to allocate memory appropriately – MAVEN_OPTS was used to increase memory, and I still ran into an issue with MAVEN out-of-memory.  To fix the issue, I had to use the Surefire argsline: 

<plugins>
   <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
         <argLine>-Xms2G -Xmx2G</argLine>
      </configuration>
   </plugin>
</plugins>

Async JAX-RS: Quick Tips…

This article outlines the use of Jetty with JAX-RS Async.

Use the webapp archetype and create the webapp in the specific archetype version.

mvn archetype:generate -DgroupId=org.bastide -DartifactId=webapp -DarchetypeArtifactId=maven-archetype-webapp -DarchetypeVersion=1.4

When prompted, confirm details:
[INFO] Using property: groupId = org.bastide
[INFO] Using property: artifactId = webapp
Define value for property ‘version’ 1.0-SNAPSHOT: : 1.0-SNAPSHOT
[INFO] Using property: package = org.bastide
Confirm properties configuration:
groupId: org.bastide
artifactId: webapp
version: 1.0-SNAPSHOT
package: org.bastide
Y: : Y

You’ll see
[INFO] —————————————————————————-
[INFO] Using following parameters for creating project from Archetype: maven-archetype-webapp:1.4
[INFO] —————————————————————————-
[INFO] Parameter: groupId, Value: org.bastide
[INFO] Parameter: artifactId, Value: webapp
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: org.bastide
[INFO] Parameter: packageInPathFormat, Value: org/bastide
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: org.bastide
[INFO] Parameter: groupId, Value: org.bastide
[INFO] Parameter: artifactId, Value: webapp
[INFO] Project created from Archetype in dir: /Users/user/Desktop/j/project/webapp
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 01:02 min
[INFO] Finished at: 2019-08-18T16:52:03-04:00
[INFO] ————————————————————————

The typical archetype catalog is found at repo1 – http://repo1.maven.org/maven2/ . To add an archetype catalog in Eclipse use https://howtodoinjava.com/eclipse/how-to-import-maven-remote-archetype-catalogs-in-eclipse/ . To read more on the Apache archetypes, reference github https://github.com/apache/maven-archetypes/tree/maven-archetype-bundles-1.4

Open the pom.xml and update to Java 1.8.
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>

Add the Jetty Version. Find the latest version on Maven Central – https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-maven-plugin/ (in this case ) * scan carefully * the jetty versions are not in the approved versioning, and 20 comes before 9.4.3.
<jetty.version>9.4.20.v20190813</jetty.version>
<junit.version>5.5.1</junit.version>

Add a dependencies block for jetty, and replace junit.
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>${jetty.version}</version>
</dependency>

Type mvn dependency:tree to show the jar files that’ll be accessible in the classpath.

~:webapp$ mvn dependency:tree
[INFO] Scanning for projects…
[INFO]
[INFO] ————————-< org.bastide:webapp >————————-
[INFO] Building webapp Maven Webapp 1.0-SNAPSHOT
[INFO] ——————————–[ war ]———————————
[INFO]
[INFO] — maven-dependency-plugin:2.8:tree (default-cli) @ webapp —
[INFO] org.bastide:webapp:war:1.0-SNAPSHOT
[INFO] +- org.junit.jupiter:junit-jupiter-api:jar:5.5.1:test
[INFO] | +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] | +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] | \- org.junit.platform:junit-platform-commons:jar:1.5.1:test
[INFO] \- org.eclipse.jetty:jetty-server:jar:9.4.9.v20180320:compile
[INFO] +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] +- org.eclipse.jetty:jetty-http:jar:9.4.9.v20180320:compile
[INFO] | \- org.eclipse.jetty:jetty-util:jar:9.4.9.v20180320:compile
[INFO] \- org.eclipse.jetty:jetty-io:jar:9.4.9.v20180320:compile
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 2.199 s
[INFO] Finished at: 2019-08-18T17:05:38-04:00
[INFO] ————————————————————————

Check the build mvn clean package, and check to see BUILD SUCCESS.

Add the maven plugin for jetty under build > plugins
<plugins>
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>${jetty.version}</version>
<configuration>
<jettyXml></jettyXml>
</configuration>
</plugin>
</plugins>

Execute mvn jetty:run and the command pauses at the end (as the process is not forked)
[INFO] Started o.e.j.m.p.JettyWebAppContext@29ea78b1{/,file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/,AVAILABLE}{file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/}
[INFO] Started ServerConnector@74294c1a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
[INFO] Started @6480ms
[INFO] Started Jetty Server

Open a new terminal
curl http://localhost:8080/

Once you see the response, you know the project is setup correctly. (note this is from ./src/main/webapp/index.jsp)
<html>
<body>
<h2>Hello World!</h2>
</body>
</html>

You can stop the mvn command (Control + C)

^C[INFO] Stopped ServerConnector@74294c1a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
[INFO] Stopped scavenging
[INFO] Stopped o.e.j.m.p.JettyWebAppContext@29ea78b1{/,file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/,UNAVAILABLE}{file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/}
[INFO] Jetty server exiting.
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 02:31 min
[INFO] Finished at: 2019-08-18T19:04:02-04:00
[INFO] ————————————————————————

Add the jersey version to the properties element in the pom.xml
        <jersey.version>2.28</jersey.version>

Add the Jersey dependencies to the pom.xml
<dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-server</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-jetty-http</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet-core</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jackson</artifactId>
            <version>${jersey.version}</version>
        </dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet</artifactId>
<version>${jersey.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.inject</groupId>
<artifactId>jersey-hk2</artifactId>
<version>${jersey.version}</version>
</dependency>

You can test the dependencies and properties using mvn clean package and check the contents with jar -tvf target/webapp.war
webapp$ jar -tvf target/webapp.war
101 Sun Aug 18 19:09:22 EDT 2019 META-INF/MANIFEST.MF
0 Sun Aug 18 19:09:22 EDT 2019 META-INF/
0 Sun Aug 18 19:09:22 EDT 2019 WEB-INF/
0 Sun Aug 18 19:09:22 EDT 2019 WEB-INF/classes/
0 Sun Aug 18 19:09:22 EDT 2019 WEB-INF/lib/
52 Sun Aug 18 16:52:02 EDT 2019 index.jsp
83813 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-entity-filtering-2.28.jar
502985 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jetty-util-9.4.12.v20180830.jar
20235 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/osgi-resource-locator-1.0.1.jar
40254 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-container-jetty-http-2.28.jar
93107 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/validation-api-2.0.1.Final.jar
66894 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jackson-annotations-2.9.8.jar
25150 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jakarta.annotation-api-1.3.4.jar
85920 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-media-jaxb-2.28.jar
5408 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jakarta.inject-2.5.0.jar
1141593 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-common-2.28.jar
1347236 Sun Aug 18 19:09:20 EDT 2019 WEB-INF/lib/jackson-databind-2.9.8.jar
325619 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jackson-core-2.9.8.jar
16680 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jetty-continuation-9.4.12.v20180830.jar
75210 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-media-json-jackson-2.28.jar
584440 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/jetty-server-9.4.9.v20180320.jar
215 Sun Aug 18 16:52:02 EDT 2019 WEB-INF/web.xml
3425 Sun Aug 18 19:09:00 EDT 2019 META-INF/maven/org.bastide/webapp/pom.xml
90 Sun Aug 18 19:09:22 EDT 2019 META-INF/maven/org.bastide/webapp/pom.properties
196494 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-client-2.28.jar
169671 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/jetty-http-9.4.9.v20180320.jar
95806 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/javax.servlet-api-3.1.0.jar
32627 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jackson-module-jaxb-annotations-2.9.8.jar
73338 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-container-servlet-core-2.28.jar
140262 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jakarta.ws.rs-api-2.1.5.jar
134935 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/jetty-io-9.4.9.v20180320.jar
935993 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-server-2.28.jar

Update the web.xml with inner xml.
 
<servlet>
        <servlet-name>App</servlet-name>
        <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
        <init-param>
            <param-name>javax.ws.rs.Application</param-name>
            <param-value>org.bastide.jaxrs.Application</param-value>
        </init-param>
<async-supported>true</async-supported>
    </servlet>
    <servlet-mapping>
        <servlet-name>App</servlet-name>
        <url-pattern>/rest/*</url-pattern>
    </servlet-mapping>

Create the Jax-RS application java file
mkdir -p src/main/java/org/bastide/jaxrs/
touch src/main/java/org/bastide/jaxrs/Application.java

Update the Application.java with the contents to activate your JAX-RS code.
package org.bastide.jaxrs;

import javax.ws.rs.ApplicationPath;
import org.glassfish.jersey.server.ResourceConfig;

@ApplicationPath(“rest”)
public class Application extends ResourceConfig {
 
public Application() {
System.out.println(“Setting up the application”);
packages(“org.bastide.jaxrs”);
}
}

Add a new sync api SyncApi.java in the package.
package org.bastide.jaxrs;

import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;

@Path(“/sync”)
public class SyncApi {
 
@GET
    @Path(“/get”)
public String test() {
return “Example Content”;
}
}

Startup Jetty using (jetty:run) and execute a curl request, and you should see:
webapp$ curl http://localhost:8080/rest/sync/get
Example Content
webapp$

Now, create AsyncApi.java
package org.bastide.jaxrs;

import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.container.CompletionCallback;
import javax.ws.rs.container.Suspended;
import javax.ws.rs.container.AsyncResponse;

@Path(“/async”)
public class AsyncApi {
 
@GET
    @Path(“/get”)
public void asyncGetWithTimeout(@Suspended final AsyncResponse asyncResponse) {
asyncResponse.register(new CompletionCallback() {
@Override
public void onComplete(Throwable throwable) {
System.out.println(“Complete”);
}
});
 
new Thread(new Runnable() {
@Override
public void run() {
String result = veryExpensiveOperation();
asyncResponse.resume(result);
}
 
private String veryExpensiveOperation() {
try{
Thread.sleep(10000L);
System.out.println(“TEST IS COMPLETE”);
} catch (java.lang.InterruptedException ie){
ie.printStackTrace();
}
return “DONE”;
}
}).start();
}
}

The code starts a thread for 10 seconds, and runs as async.

time curl http://localhost:8080/rest/async/get
DONE
real    0m10.066s
user    0m0.008s
sys 0m0.014s

This technique is a great way to hold a connection, or dispatch work on a specific JVM and coordinate long running actions.  Eventually, I’ll be using this with my Raspberry Pi.

Reference
https://www.eclipse.org/jetty/documentation/9.4.20.v20190813/maven-and-jetty.html#jetty-maven-helloworld
https://www.eclipse.org/jetty/documentation/9.4.20.v20190813/jetty-maven-plugin.html
https://jersey.github.io/documentation/latest/index.html
https://jersey.github.io/documentation/latest/deployment.html#d0e3342
https://github.com/AlanHohn/jaxrs/blob/master/src/main/java/org/anvard/jaxrs/server/EmbeddedServer.java
https://stackoverflow.com/a/44546979
https://www.eclipse.org/jetty/documentation/9.4.x/jetty-maven-plugin.html#jetty-run-goal
https://jersey.github.io/documentation/latest/async.html#d0e9895
https://jersey.github.io/documentation/latest/async.html#d0e9895
 

Check Security Versions

As many know, the projects I work on are typically maven projects. These projects have a variety of requirements, and I started experimenting with static analysis tools. I found a cool one based on the Red Hat victims project. I ran this and found two embedded and out of date jars. Below, one sees the command runs, and highlights the vulnerabilities and CVEs that correspond. Further, it puts a report in the target directory for each module (great for reporting and seeing where/how it’s vulnerable.

$ mvn com.redhat.victims.maven:security-versions:check -f parent/pom.xml
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] parent [pom]
[INFO] api [jar]
[INFO] webapp [webapp]
[INFO]
[INFO] ---------------< group:parent >----------------
[INFO] Building parent 99-SNAPSHOT [1/3]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- security-versions:1.0.6:check (default-cli) @ parent ---
[INFO] Analyzing the dependencies for group:parent
[INFO] Syncing with the victims repository (based on the atom feed)
[INFO] Downloading: https://github.com/victims/victims-cve-db/commits.atom
[INFO] Already to the latest version.
INFO] Analyzing the dependencies for group:api
[ERROR] com.fasterxml.jackson.core:jackson-databind is vulnerable to CVE-2017-7525
[INFO] Analyzing the dependencies for group:webapp
[ERROR] commons-collections:commons-collections is vulnerable to CVE-2015-7501
------------------------------------------------------------------------
[INFO] Reactor Summary for parent 99-SNAPSHOT:
[INFO]
[INFO] parent ......... SUCCESS [ 3.170 s]
[INFO] api ...................................... SKIPPED
[INFO] webapp .............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.636 s
[INFO] Finished at: 2019-08-13T07:04:08-04:00
[INFO] ------------------------------------------------------------------------
$

Code Graph showing the Layout of the Code base

I’ve been mixing data analysis and Java programming recently.  I wrote a tool to do the analysis (Maven/Python).

Here is the obfuscated output of the analysis, showing the hotspots.  I opted to show a thumbnail of the image here to protect the confidentiality of the project.  The generated image was also 78 Megabytes.  (a bit much, but you can zoom right in).

Complicated Graph

If you use a smaller set of classes and imports, the Maven plugin generates a reasonable diagram.csv file using

mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml

You then see the output diagram.csv.

To generate the layout dependencies of classes in your project.  Use the snippets, and the Jupyter Notebook – https://github.com/prb112/examples/blob/master/code-graph/code-graph.ipynb and view at https://nbviewer.jupyter.org/github/prb112/examples/blob/master/code-graph/code-graph.ipynb

Simple Graph

Reference

Code Graph on Git https://github.com/prb112/examples/tree/master/code-graph

CSV Sample Data diagram.csv

POM

<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>example</groupId>
	<artifactId>generate-diagram</artifactId>
	<version>99-SNAPSHOT</version>
	<packaging>maven-plugin</packaging>

	<name>generate-diagram</name>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
		<version.roaster>2.21.0.Final</version.roaster>
	</properties>



	<build>
		<pluginManagement>
			<plugins>
				<plugin>
					<artifactId>maven-jar-plugin</artifactId>
					<version>2.6</version>
					<executions>
						<execution>
							<goals>
								<goal>test-jar</goal>
							</goals>
						</execution>
					</executions>
					<configuration>
						<archive>
							<manifest>
								<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
								<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
							</manifest>
						</archive>
					</configuration>
				</plugin>
				<plugin>
					<groupId>org.apache.maven.plugins</groupId>
					<artifactId>maven-plugin-plugin</artifactId>
					<version>3.6.0</version>
					<configuration>
						<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
					</configuration>
					<executions>
						<execution>
							<id>mojo-descriptor</id>
							<goals>
								<goal>descriptor</goal>
							</goals>
							<phase>process-classes</phase>
							<configuration>
								<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
							</configuration>
						</execution>
					</executions>
				</plugin>
			</plugins>
		</pluginManagement>

		<plugins>
			<plugin>
				<!-- Embeds the dependencies in fhir-tools into the jar. -->
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-shade-plugin</artifactId>
				<version>3.2.1</version>
				<executions>
					<execution>
						<phase>package</phase>
						<goals>
							<goal>shade</goal>
						</goals>
						<configuration>
							<artifactSet>
								<excludes>
									<exclude>org.testng:testng</exclude>
									<exclude>org.apache.maven:lib:tests</exclude>
									<exclude>org.apache.maven</exclude>
								</excludes>
							</artifactSet>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>

	<dependencies>
		<dependency>
			<groupId>org.jboss.forge.roaster</groupId>
			<artifactId>roaster-api</artifactId>
			<version>${version.roaster}</version>
		</dependency>
		<dependency>
			<groupId>org.jboss.forge.roaster</groupId>
			<artifactId>roaster-jdt</artifactId>
			<version>${version.roaster}</version>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-plugin-api</artifactId>
			<version>3.6.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugin-tools</groupId>
			<artifactId>maven-plugin-annotations</artifactId>
			<version>3.6.0</version>
			<optional>true</optional>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-core</artifactId>
			<version>3.6.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-artifact</artifactId>
			<version>3.6.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-model</artifactId>
			<version>3.6.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-compat</artifactId>
			<version>3.6.1</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugin-testing</groupId>
			<artifactId>maven-plugin-testing-harness</artifactId>
			<version>3.3.0</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
</project>
package demo;

import java.io.File;
import java.util.Properties;

import org.apache.maven.execution.MavenSession;
import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugin.MojoFailureException;
import org.apache.maven.plugins.annotations.Execute;
import org.apache.maven.plugins.annotations.LifecyclePhase;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.plugins.annotations.Parameter;
import org.apache.maven.plugins.annotations.ResolutionScope;
import org.apache.maven.project.MavenProject;

import com.ibm.watsonhealth.fhir.tools.plugin.diagram.DiagramFactory;
import com.ibm.watsonhealth.fhir.tools.plugin.diagram.impl.IDiagramGenerator;

/**
 * This class coordinates the calls to the Diagram generation plugin
 * 
 * The phase is initialize. To find a list of phases -
 * https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference
 * 
 * Run the following to setup the plugin: <code>
 * mvn clean package install -f generate-diagram/pom.xml
 * </code>
 * 
 * Run the following to setup the classes in fhir-model: <code> 
 * mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml
 * </code>
 * 
 * @author PBastide
 * 
 * @requiresDependencyResolution runtime
 *
 */
@Mojo(name = "generate-diagram", //$NON-NLS-1$
        requiresProject = true, requiresDependencyResolution = ResolutionScope.RUNTIME_PLUS_SYSTEM, requiresDependencyCollection = ResolutionScope.RUNTIME_PLUS_SYSTEM, defaultPhase = LifecyclePhase.GENERATE_SOURCES, requiresOnline = false, threadSafe = false, aggregator = true)
@Execute(phase = LifecyclePhase.GENERATE_SOURCES)
public class DiagramPlugin extends AbstractMojo {

    @Parameter(defaultValue = "${project}", required = true, readonly = true) //$NON-NLS-1$
    protected MavenProject mavenProject;

    @Parameter(defaultValue = "${session}")
    private MavenSession session;

    @Parameter(defaultValue = "${project.basedir}", required = true, readonly = true) //$NON-NLS-1$
    private File baseDir;

    @Override
    public void execute() throws MojoExecutionException, MojoFailureException {
        if (baseDir == null || !baseDir.exists()) {
            throw new MojoFailureException("The Base Directory is not found.  Throwing failure. ");
        }

        // Grab the Properties (the correct way)
        // https://maven.apache.org/plugin-developers/common-bugs.html#Using_System_Properties
        Properties userProps = session.getUserProperties();
        String useTestsDirectoryStr = userProps.getProperty("useTestsDirectory", "false");

        // Converts Limit value to boolean value.
        boolean useTestsDirectory = Boolean.parseBoolean(useTestsDirectoryStr);

        // Grab the right generator and set it up.
        IDiagramGenerator generator = DiagramFactory.getDiagramGenerator();

        // Set the use of tests directory
        generator.use(useTestsDirectory);

        // Get the base directory .
        generator.setTargetProjectBaseDirectory(baseDir.getAbsolutePath() + "/target");

        // Passes the Log to the implementation code.
        generator.setLog(getLog());

        // Add the project
        generator.add(mavenProject);

        // Builds the Diagram
        generator.generateDiagram();
    }
}
package example.impl;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.StringJoiner;
import java.util.stream.Collectors;

import org.apache.maven.plugin.logging.Log;
import org.apache.maven.project.MavenProject;
import org.jboss.forge.roaster.Roaster;
import org.jboss.forge.roaster.model.JavaType;
import org.jboss.forge.roaster.model.source.Import;
import org.jboss.forge.roaster.model.source.JavaAnnotationSource;
import org.jboss.forge.roaster.model.source.JavaClassSource;
import org.jboss.forge.roaster.model.source.JavaEnumSource;
import org.jboss.forge.roaster.model.source.JavaInterfaceSource;
import org.jboss.forge.roaster.model.source.JavaSource;
import org.w3c.dom.DOMImplementation;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

public class DiagramImpl implements IDiagramGenerator {

    private String absolutePath = null;
    private Log log = null;
    private Boolean useTestFiles = false;

    private List<MavenProject> projects = new ArrayList<>();

    private List<String> sourceDirectories = new ArrayList<>();

    private List<String> sourceFiles = new ArrayList<>();

    private List<String> countWithNested = new ArrayList<>();

    private Map<String, List<String>> sourceFileImports = new HashMap<>();

    @Override
    public void add(MavenProject mavenProject) {
        if (mavenProject == null) {
            throw new IllegalArgumentException("no access to the maven project's plugin object");
        }

        log.info("Projects added...");
        projects = mavenProject.getCollectedProjects();

    }

    @Override
    public void use(Boolean useTestFiles) {
        this.useTestFiles = useTestFiles;

    }

    @Override
    public void setLog(Log log) {
        this.log = log;
    }

    @Override
    public void generateDiagram() {
        if (absolutePath == null) {
            throw new IllegalArgumentException("Bad Path " + absolutePath);
        }

        if (log == null) {
            throw new IllegalArgumentException("Unexpected no log passed in");
        }

        for (MavenProject project : projects) {

            List<String> locations = project.getCompileSourceRoots();
            for (String location : locations) {
                log.info("Location of Directory -> " + location);

            }
            sourceDirectories.addAll(locations);

            if (useTestFiles) {
                log.info("Adding the Test Files");
                sourceDirectories.addAll(project.getTestCompileSourceRoots());
            }

        }

        // Find the Files in each directory.
        // Don't Follow links.
        for (String directory : sourceDirectories) {
            findSourceFiles(directory);
        }

        processCardinalityMap();

        printOutCardinality();

        log.info("Total Number of Java Files in the Project are: " + sourceFiles.size());
        log.info("Total Number of Classes/Interfaces/Enums in the Project are: " + countWithNested.size());

        generateImageFile();

    }

    private void generateImageFile() {
        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        try(FileOutputStream fos = new FileOutputStream("diagram.csv");) {
            
            for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
                
                StringJoiner joiner = new StringJoiner(",");
                for(String val : sourceFileImports.get(key)) {
                    joiner.add(val);
                }
                
                String line = key + ",\"" + joiner.toString() + "\"";
                fos.write(line.getBytes());
                fos.write("\n".getBytes());
            }

        } catch (Exception e) {
            log.warn("Issue processing", e);
        }

    }

    public void printOutCardinality() {

        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        //
        log.info("Cardinality count - imports into other classes");
        for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
            log.info(key + " -> " + sourceFileImports.get(key).size());
        }

    }

    public void processCardinalityMap() {
        // Import > List<Classes>
        // Stored in -> sourceFileImports

        for (String source : sourceFiles) {
            File srcFile = new File(source);
            try {
                JavaType<?> jtFile = Roaster.parse(srcFile);

                String parentJavaClass = jtFile.getQualifiedName();

                if (jtFile instanceof JavaClassSource) {
                    countWithNested.add(parentJavaClass);
                    log.info("[C] -> " + parentJavaClass);
                    JavaClassSource jcs = (JavaClassSource) jtFile;

                    helperImports(parentJavaClass, jcs.getImports());

                    for (JavaSource<?> child : jcs.getNestedTypes()) {

                        String childLoc = child.getQualifiedName();
                        countWithNested.add(childLoc);
                        log.info("  [CC] -> " + childLoc);
                    }
                }

                else if (jtFile instanceof JavaEnumSource) {
                    log.info("[E] -> " + parentJavaClass);
                    countWithNested.add(parentJavaClass);
                    JavaEnumSource jes = (JavaEnumSource) jtFile;

                    helperImports(parentJavaClass, jes.getImports());

                    for (Object child : jes.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
                        countWithNested.add(childLoc);
                        log.info("  [EC] -> " + childLoc);
                    }

                } else if (jtFile instanceof JavaInterfaceSource) {
                    countWithNested.add(parentJavaClass);

                    log.info("[I] -> " + parentJavaClass);
                    JavaInterfaceSource jis = (JavaInterfaceSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());

                    for (Object child : jis.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
                        countWithNested.add(childLoc);
                        log.info("  [IC] -> " + childLoc);
                    }
                } else if (jtFile instanceof JavaAnnotationSource) {
                    countWithNested.add(parentJavaClass);

                    log.info("[A] -> " + parentJavaClass);
                    JavaAnnotationSource jis = (JavaAnnotationSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());
                }

                else {
                    log.info("[O] -> " + parentJavaClass);
                }

            } catch (IOException e) {
                log.info("unable to parse file " + srcFile);
            }

        }
        log.info("Parsed the Cardinality Map:");

    }

    private void helperImports(String parentJavaClass, List<Import> imports) {
        // sourceFileImports
        List<String> importOut = sourceFileImports.get(parentJavaClass);
        if (importOut == null) {
            sourceFileImports.put(parentJavaClass, new ArrayList<String>());
        }

        for (Import importX : imports) {
            String importXStr = importX.getQualifiedName();
            importOut = sourceFileImports.get(importXStr);
            if (importOut == null) {
                importOut = new ArrayList<>();
                sourceFileImports.put(importXStr, importOut);
            }

            importOut.add(parentJavaClass);

        }

    }

    public static void main(String... args) {
        SvgDiagramImpl impl = new SvgDiagramImpl();
        String proc = "Test.java";
        Log log = new Log() {

            @Override
            public boolean isDebugEnabled() {
                return false;
            }

            @Override
            public void debug(CharSequence content) {

            }

            @Override
            public void debug(CharSequence content, Throwable error) {

            }

            @Override
            public void debug(Throwable error) {

            }

            @Override
            public boolean isInfoEnabled() {
                return false;
            }

            @Override
            public void info(CharSequence content) {

            }

            @Override
            public void info(CharSequence content, Throwable error) {

            }

            @Override
            public void info(Throwable error) {

            }

            @Override
            public boolean isWarnEnabled() {
                return false;
            }

            @Override
            public void warn(CharSequence content) {

            }

            @Override
            public void warn(CharSequence content, Throwable error) {

            }

            @Override
            public void warn(Throwable error) {

            }

            @Override
            public boolean isErrorEnabled() {
                return false;
            }

            @Override
            public void error(CharSequence content) {

            }

            @Override
            public void error(CharSequence content, Throwable error) {

            }

            @Override
            public void error(Throwable error) {

            }
        };
        impl.setLog(log);
        impl.addSourceFile(proc);
        impl.processCardinalityMap();
    }

    private void addSourceFile(String proc) {
        sourceFiles.add(proc);

    }

    public void findSourceFiles(String directory) {
        File dir = new File(directory);
        if (dir.exists()) {

            File[] listFiles = dir.listFiles((file, name) -> {
                return name.endsWith(".java");
            });

            // Add to source directory
            if (listFiles != null) {
                for (File file : listFiles) {
                    sourceFiles.add(file.getAbsolutePath());
                    log.info(" File Added to Processing: " + file.getAbsolutePath());
                }
            }

            File[] listFilesFolders = dir.listFiles((file, name) -> {
                return file.isDirectory();
            });

            if (listFilesFolders != null) {
                for (File tmpDir : listFilesFolders) {
                    findSourceFiles(tmpDir.getAbsolutePath());
                }
            }

        } else {
            log.warn("Directory does not exist " + directory);
        }
    }

    @Override
    public void setTargetProjectBaseDirectory(String absolutePath) {
        this.absolutePath = absolutePath;

    }

}