Maven Surefire runs out of memory

I’m running a complicated integration test, and needed to allocate memory appropriately – MAVEN_OPTS was used to increase memory, and I still ran into an issue with MAVEN out-of-memory.  To fix the issue, I had to use the Surefire argsline: 

<plugins>
   <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
         <argLine>-Xms2G -Xmx2G</argLine>
      </configuration>
   </plugin>
</plugins>

Async JAX-RS: Quick Tips…

This article outlines the use of Jetty with JAX-RS Async.

Use the webapp archetype and create the webapp in the specific archetype version.

mvn archetype:generate -DgroupId=org.bastide -DartifactId=webapp -DarchetypeArtifactId=maven-archetype-webapp -DarchetypeVersion=1.4

When prompted, confirm details:
[INFO] Using property: groupId = org.bastide
[INFO] Using property: artifactId = webapp
Define value for property ‘version’ 1.0-SNAPSHOT: : 1.0-SNAPSHOT
[INFO] Using property: package = org.bastide
Confirm properties configuration:
groupId: org.bastide
artifactId: webapp
version: 1.0-SNAPSHOT
package: org.bastide
Y: : Y

You’ll see
[INFO] —————————————————————————-
[INFO] Using following parameters for creating project from Archetype: maven-archetype-webapp:1.4
[INFO] —————————————————————————-
[INFO] Parameter: groupId, Value: org.bastide
[INFO] Parameter: artifactId, Value: webapp
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: org.bastide
[INFO] Parameter: packageInPathFormat, Value: org/bastide
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: org.bastide
[INFO] Parameter: groupId, Value: org.bastide
[INFO] Parameter: artifactId, Value: webapp
[INFO] Project created from Archetype in dir: /Users/user/Desktop/j/project/webapp
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 01:02 min
[INFO] Finished at: 2019-08-18T16:52:03-04:00
[INFO] ————————————————————————

The typical archetype catalog is found at repo1 – http://repo1.maven.org/maven2/ . To add an archetype catalog in Eclipse use https://howtodoinjava.com/eclipse/how-to-import-maven-remote-archetype-catalogs-in-eclipse/ . To read more on the Apache archetypes, reference github https://github.com/apache/maven-archetypes/tree/maven-archetype-bundles-1.4

Open the pom.xml and update to Java 1.8.
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>

Add the Jetty Version. Find the latest version on Maven Central – https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-maven-plugin/ (in this case ) * scan carefully * the jetty versions are not in the approved versioning, and 20 comes before 9.4.3.
<jetty.version>9.4.20.v20190813</jetty.version>
<junit.version>5.5.1</junit.version>

Add a dependencies block for jetty, and replace junit.
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>${jetty.version}</version>
</dependency>

Type mvn dependency:tree to show the jar files that’ll be accessible in the classpath.

~:webapp$ mvn dependency:tree
[INFO] Scanning for projects…
[INFO]
[INFO] ————————-< org.bastide:webapp >————————-
[INFO] Building webapp Maven Webapp 1.0-SNAPSHOT
[INFO] ——————————–[ war ]———————————
[INFO]
[INFO] — maven-dependency-plugin:2.8:tree (default-cli) @ webapp —
[INFO] org.bastide:webapp:war:1.0-SNAPSHOT
[INFO] +- org.junit.jupiter:junit-jupiter-api:jar:5.5.1:test
[INFO] | +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] | +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] | \- org.junit.platform:junit-platform-commons:jar:1.5.1:test
[INFO] \- org.eclipse.jetty:jetty-server:jar:9.4.9.v20180320:compile
[INFO] +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] +- org.eclipse.jetty:jetty-http:jar:9.4.9.v20180320:compile
[INFO] | \- org.eclipse.jetty:jetty-util:jar:9.4.9.v20180320:compile
[INFO] \- org.eclipse.jetty:jetty-io:jar:9.4.9.v20180320:compile
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 2.199 s
[INFO] Finished at: 2019-08-18T17:05:38-04:00
[INFO] ————————————————————————

Check the build mvn clean package, and check to see BUILD SUCCESS.

Add the maven plugin for jetty under build > plugins
<plugins>
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>${jetty.version}</version>
<configuration>
<jettyXml></jettyXml>
</configuration>
</plugin>
</plugins>

Execute mvn jetty:run and the command pauses at the end (as the process is not forked)
[INFO] Started o.e.j.m.p.JettyWebAppContext@29ea78b1{/,file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/,AVAILABLE}{file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/}
[INFO] Started ServerConnector@74294c1a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
[INFO] Started @6480ms
[INFO] Started Jetty Server

Open a new terminal
curl http://localhost:8080/

Once you see the response, you know the project is setup correctly. (note this is from ./src/main/webapp/index.jsp)
<html>
<body>
<h2>Hello World!</h2>
</body>
</html>

You can stop the mvn command (Control + C)

^C[INFO] Stopped ServerConnector@74294c1a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
[INFO] Stopped scavenging
[INFO] Stopped o.e.j.m.p.JettyWebAppContext@29ea78b1{/,file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/,UNAVAILABLE}{file:///Users/paulbastide/Desktop/j/project/webapp/src/main/webapp/}
[INFO] Jetty server exiting.
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 02:31 min
[INFO] Finished at: 2019-08-18T19:04:02-04:00
[INFO] ————————————————————————

Add the jersey version to the properties element in the pom.xml
        <jersey.version>2.28</jersey.version>

Add the Jersey dependencies to the pom.xml
<dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-server</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-jetty-http</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet-core</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jackson</artifactId>
            <version>${jersey.version}</version>
        </dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet</artifactId>
<version>${jersey.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.inject</groupId>
<artifactId>jersey-hk2</artifactId>
<version>${jersey.version}</version>
</dependency>

You can test the dependencies and properties using mvn clean package and check the contents with jar -tvf target/webapp.war
webapp$ jar -tvf target/webapp.war
101 Sun Aug 18 19:09:22 EDT 2019 META-INF/MANIFEST.MF
0 Sun Aug 18 19:09:22 EDT 2019 META-INF/
0 Sun Aug 18 19:09:22 EDT 2019 WEB-INF/
0 Sun Aug 18 19:09:22 EDT 2019 WEB-INF/classes/
0 Sun Aug 18 19:09:22 EDT 2019 WEB-INF/lib/
52 Sun Aug 18 16:52:02 EDT 2019 index.jsp
83813 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-entity-filtering-2.28.jar
502985 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jetty-util-9.4.12.v20180830.jar
20235 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/osgi-resource-locator-1.0.1.jar
40254 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-container-jetty-http-2.28.jar
93107 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/validation-api-2.0.1.Final.jar
66894 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jackson-annotations-2.9.8.jar
25150 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jakarta.annotation-api-1.3.4.jar
85920 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-media-jaxb-2.28.jar
5408 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jakarta.inject-2.5.0.jar
1141593 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-common-2.28.jar
1347236 Sun Aug 18 19:09:20 EDT 2019 WEB-INF/lib/jackson-databind-2.9.8.jar
325619 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jackson-core-2.9.8.jar
16680 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jetty-continuation-9.4.12.v20180830.jar
75210 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-media-json-jackson-2.28.jar
584440 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/jetty-server-9.4.9.v20180320.jar
215 Sun Aug 18 16:52:02 EDT 2019 WEB-INF/web.xml
3425 Sun Aug 18 19:09:00 EDT 2019 META-INF/maven/org.bastide/webapp/pom.xml
90 Sun Aug 18 19:09:22 EDT 2019 META-INF/maven/org.bastide/webapp/pom.properties
196494 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-client-2.28.jar
169671 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/jetty-http-9.4.9.v20180320.jar
95806 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/javax.servlet-api-3.1.0.jar
32627 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jackson-module-jaxb-annotations-2.9.8.jar
73338 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-container-servlet-core-2.28.jar
140262 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jakarta.ws.rs-api-2.1.5.jar
134935 Sun Aug 18 17:03:02 EDT 2019 WEB-INF/lib/jetty-io-9.4.9.v20180320.jar
935993 Sun Aug 18 19:09:18 EDT 2019 WEB-INF/lib/jersey-server-2.28.jar

Update the web.xml with inner xml.
 
<servlet>
        <servlet-name>App</servlet-name>
        <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
        <init-param>
            <param-name>javax.ws.rs.Application</param-name>
            <param-value>org.bastide.jaxrs.Application</param-value>
        </init-param>
<async-supported>true</async-supported>
    </servlet>
    <servlet-mapping>
        <servlet-name>App</servlet-name>
        <url-pattern>/rest/*</url-pattern>
    </servlet-mapping>

Create the Jax-RS application java file
mkdir -p src/main/java/org/bastide/jaxrs/
touch src/main/java/org/bastide/jaxrs/Application.java

Update the Application.java with the contents to activate your JAX-RS code.
package org.bastide.jaxrs;

import javax.ws.rs.ApplicationPath;
import org.glassfish.jersey.server.ResourceConfig;

@ApplicationPath(“rest”)
public class Application extends ResourceConfig {
 
public Application() {
System.out.println(“Setting up the application”);
packages(“org.bastide.jaxrs”);
}
}

Add a new sync api SyncApi.java in the package.
package org.bastide.jaxrs;

import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;

@Path(“/sync”)
public class SyncApi {
 
@GET
    @Path(“/get”)
public String test() {
return “Example Content”;
}
}

Startup Jetty using (jetty:run) and execute a curl request, and you should see:
webapp$ curl http://localhost:8080/rest/sync/get
Example Content
webapp$

Now, create AsyncApi.java
package org.bastide.jaxrs;

import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.container.CompletionCallback;
import javax.ws.rs.container.Suspended;
import javax.ws.rs.container.AsyncResponse;

@Path(“/async”)
public class AsyncApi {
 
@GET
    @Path(“/get”)
public void asyncGetWithTimeout(@Suspended final AsyncResponse asyncResponse) {
asyncResponse.register(new CompletionCallback() {
@Override
public void onComplete(Throwable throwable) {
System.out.println(“Complete”);
}
});
 
new Thread(new Runnable() {
@Override
public void run() {
String result = veryExpensiveOperation();
asyncResponse.resume(result);
}
 
private String veryExpensiveOperation() {
try{
Thread.sleep(10000L);
System.out.println(“TEST IS COMPLETE”);
} catch (java.lang.InterruptedException ie){
ie.printStackTrace();
}
return “DONE”;
}
}).start();
}
}

The code starts a thread for 10 seconds, and runs as async.

time curl http://localhost:8080/rest/async/get
DONE
real    0m10.066s
user    0m0.008s
sys 0m0.014s

This technique is a great way to hold a connection, or dispatch work on a specific JVM and coordinate long running actions.  Eventually, I’ll be using this with my Raspberry Pi.

Reference
https://www.eclipse.org/jetty/documentation/9.4.20.v20190813/maven-and-jetty.html#jetty-maven-helloworld
https://www.eclipse.org/jetty/documentation/9.4.20.v20190813/jetty-maven-plugin.html
https://jersey.github.io/documentation/latest/index.html
https://jersey.github.io/documentation/latest/deployment.html#d0e3342
https://github.com/AlanHohn/jaxrs/blob/master/src/main/java/org/anvard/jaxrs/server/EmbeddedServer.java
https://stackoverflow.com/a/44546979
https://www.eclipse.org/jetty/documentation/9.4.x/jetty-maven-plugin.html#jetty-run-goal
https://jersey.github.io/documentation/latest/async.html#d0e9895
https://jersey.github.io/documentation/latest/async.html#d0e9895
 

Check Security Versions

As many know, the projects I work on are typically maven projects. These projects have a variety of requirements, and I started experimenting with static analysis tools. I found a cool one based on the Red Hat victims project. I ran this and found two embedded and out of date jars. Below, one sees the command runs, and highlights the vulnerabilities and CVEs that correspond. Further, it puts a report in the target directory for each module (great for reporting and seeing where/how it’s vulnerable.

$ mvn com.redhat.victims.maven:security-versions:check -f parent/pom.xml
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] parent [pom]
[INFO] api [jar]
[INFO] webapp [webapp]
[INFO]
[INFO] ---------------< group:parent >----------------
[INFO] Building parent 99-SNAPSHOT [1/3]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- security-versions:1.0.6:check (default-cli) @ parent ---
[INFO] Analyzing the dependencies for group:parent
[INFO] Syncing with the victims repository (based on the atom feed)
[INFO] Downloading: https://github.com/victims/victims-cve-db/commits.atom
[INFO] Already to the latest version.
INFO] Analyzing the dependencies for group:api
[ERROR] com.fasterxml.jackson.core:jackson-databind is vulnerable to CVE-2017-7525
[INFO] Analyzing the dependencies for group:webapp
[ERROR] commons-collections:commons-collections is vulnerable to CVE-2015-7501
------------------------------------------------------------------------
[INFO] Reactor Summary for parent 99-SNAPSHOT:
[INFO]
[INFO] parent ......... SUCCESS [ 3.170 s]
[INFO] api ...................................... SKIPPED
[INFO] webapp .............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.636 s
[INFO] Finished at: 2019-08-13T07:04:08-04:00
[INFO] ------------------------------------------------------------------------
$

Code Graph showing the Layout of the Code base

I’ve been mixing data analysis and Java programming recently.  I wrote a tool to do the analysis (Maven/Python).

Here is the obfuscated output of the analysis, showing the hotspots.  I opted to show a thumbnail of the image here to protect the confidentiality of the project.  The generated image was also 78 Megabytes.  (a bit much, but you can zoom right in).

Complicated Graph

If you use a smaller set of classes and imports, the Maven plugin generates a reasonable diagram.csv file using

mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml

You then see the output diagram.csv.

To generate the layout dependencies of classes in your project.  Use the snippets, and the Jupyter Notebook – https://github.com/prb112/examples/blob/master/code-graph/code-graph.ipynb and view at https://nbviewer.jupyter.org/github/prb112/examples/blob/master/code-graph/code-graph.ipynb

Simple Graph

Reference

Code Graph on Git https://github.com/prb112/examples/tree/master/code-graph

CSV Sample Data diagram.csv

POM

<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>example</groupId>
	<artifactId>generate-diagram</artifactId>
	<version>99-SNAPSHOT</version>
	<packaging>maven-plugin</packaging>

	<name>generate-diagram</name>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
		<version.roaster>2.21.0.Final</version.roaster>
	</properties>



	<build>
		<pluginManagement>
			<plugins>
				<plugin>
					<artifactId>maven-jar-plugin</artifactId>
					<version>2.6</version>
					<executions>
						<execution>
							<goals>
								<goal>test-jar</goal>
							</goals>
						</execution>
					</executions>
					<configuration>
						<archive>
							<manifest>
								<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
								<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
							</manifest>
						</archive>
					</configuration>
				</plugin>
				<plugin>
					<groupId>org.apache.maven.plugins</groupId>
					<artifactId>maven-plugin-plugin</artifactId>
					<version>3.6.0</version>
					<configuration>
						<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
					</configuration>
					<executions>
						<execution>
							<id>mojo-descriptor</id>
							<goals>
								<goal>descriptor</goal>
							</goals>
							<phase>process-classes</phase>
							<configuration>
								<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
							</configuration>
						</execution>
					</executions>
				</plugin>
			</plugins>
		</pluginManagement>

		<plugins>
			<plugin>
				<!-- Embeds the dependencies in fhir-tools into the jar. -->
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-shade-plugin</artifactId>
				<version>3.2.1</version>
				<executions>
					<execution>
						<phase>package</phase>
						<goals>
							<goal>shade</goal>
						</goals>
						<configuration>
							<artifactSet>
								<excludes>
									<exclude>org.testng:testng</exclude>
									<exclude>org.apache.maven:lib:tests</exclude>
									<exclude>org.apache.maven</exclude>
								</excludes>
							</artifactSet>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>

	<dependencies>
		<dependency>
			<groupId>org.jboss.forge.roaster</groupId>
			<artifactId>roaster-api</artifactId>
			<version>${version.roaster}</version>
		</dependency>
		<dependency>
			<groupId>org.jboss.forge.roaster</groupId>
			<artifactId>roaster-jdt</artifactId>
			<version>${version.roaster}</version>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-plugin-api</artifactId>
			<version>3.6.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugin-tools</groupId>
			<artifactId>maven-plugin-annotations</artifactId>
			<version>3.6.0</version>
			<optional>true</optional>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-core</artifactId>
			<version>3.6.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-artifact</artifactId>
			<version>3.6.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-model</artifactId>
			<version>3.6.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-compat</artifactId>
			<version>3.6.1</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugin-testing</groupId>
			<artifactId>maven-plugin-testing-harness</artifactId>
			<version>3.3.0</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
</project>
package demo;

import java.io.File;
import java.util.Properties;

import org.apache.maven.execution.MavenSession;
import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugin.MojoFailureException;
import org.apache.maven.plugins.annotations.Execute;
import org.apache.maven.plugins.annotations.LifecyclePhase;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.plugins.annotations.Parameter;
import org.apache.maven.plugins.annotations.ResolutionScope;
import org.apache.maven.project.MavenProject;

import com.ibm.watsonhealth.fhir.tools.plugin.diagram.DiagramFactory;
import com.ibm.watsonhealth.fhir.tools.plugin.diagram.impl.IDiagramGenerator;

/**
 * This class coordinates the calls to the Diagram generation plugin
 * 
 * The phase is initialize. To find a list of phases -
 * https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference
 * 
 * Run the following to setup the plugin: <code>
 * mvn clean package install -f generate-diagram/pom.xml
 * </code>
 * 
 * Run the following to setup the classes in fhir-model: <code> 
 * mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml
 * </code>
 * 
 * @author PBastide
 * 
 * @requiresDependencyResolution runtime
 *
 */
@Mojo(name = "generate-diagram", //$NON-NLS-1$
        requiresProject = true, requiresDependencyResolution = ResolutionScope.RUNTIME_PLUS_SYSTEM, requiresDependencyCollection = ResolutionScope.RUNTIME_PLUS_SYSTEM, defaultPhase = LifecyclePhase.GENERATE_SOURCES, requiresOnline = false, threadSafe = false, aggregator = true)
@Execute(phase = LifecyclePhase.GENERATE_SOURCES)
public class DiagramPlugin extends AbstractMojo {

    @Parameter(defaultValue = "${project}", required = true, readonly = true) //$NON-NLS-1$
    protected MavenProject mavenProject;

    @Parameter(defaultValue = "${session}")
    private MavenSession session;

    @Parameter(defaultValue = "${project.basedir}", required = true, readonly = true) //$NON-NLS-1$
    private File baseDir;

    @Override
    public void execute() throws MojoExecutionException, MojoFailureException {
        if (baseDir == null || !baseDir.exists()) {
            throw new MojoFailureException("The Base Directory is not found.  Throwing failure. ");
        }

        // Grab the Properties (the correct way)
        // https://maven.apache.org/plugin-developers/common-bugs.html#Using_System_Properties
        Properties userProps = session.getUserProperties();
        String useTestsDirectoryStr = userProps.getProperty("useTestsDirectory", "false");

        // Converts Limit value to boolean value.
        boolean useTestsDirectory = Boolean.parseBoolean(useTestsDirectoryStr);

        // Grab the right generator and set it up.
        IDiagramGenerator generator = DiagramFactory.getDiagramGenerator();

        // Set the use of tests directory
        generator.use(useTestsDirectory);

        // Get the base directory .
        generator.setTargetProjectBaseDirectory(baseDir.getAbsolutePath() + "/target");

        // Passes the Log to the implementation code.
        generator.setLog(getLog());

        // Add the project
        generator.add(mavenProject);

        // Builds the Diagram
        generator.generateDiagram();
    }
}
package example.impl;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.StringJoiner;
import java.util.stream.Collectors;

import org.apache.maven.plugin.logging.Log;
import org.apache.maven.project.MavenProject;
import org.jboss.forge.roaster.Roaster;
import org.jboss.forge.roaster.model.JavaType;
import org.jboss.forge.roaster.model.source.Import;
import org.jboss.forge.roaster.model.source.JavaAnnotationSource;
import org.jboss.forge.roaster.model.source.JavaClassSource;
import org.jboss.forge.roaster.model.source.JavaEnumSource;
import org.jboss.forge.roaster.model.source.JavaInterfaceSource;
import org.jboss.forge.roaster.model.source.JavaSource;
import org.w3c.dom.DOMImplementation;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

public class DiagramImpl implements IDiagramGenerator {

    private String absolutePath = null;
    private Log log = null;
    private Boolean useTestFiles = false;

    private List<MavenProject> projects = new ArrayList<>();

    private List<String> sourceDirectories = new ArrayList<>();

    private List<String> sourceFiles = new ArrayList<>();

    private List<String> countWithNested = new ArrayList<>();

    private Map<String, List<String>> sourceFileImports = new HashMap<>();

    @Override
    public void add(MavenProject mavenProject) {
        if (mavenProject == null) {
            throw new IllegalArgumentException("no access to the maven project's plugin object");
        }

        log.info("Projects added...");
        projects = mavenProject.getCollectedProjects();

    }

    @Override
    public void use(Boolean useTestFiles) {
        this.useTestFiles = useTestFiles;

    }

    @Override
    public void setLog(Log log) {
        this.log = log;
    }

    @Override
    public void generateDiagram() {
        if (absolutePath == null) {
            throw new IllegalArgumentException("Bad Path " + absolutePath);
        }

        if (log == null) {
            throw new IllegalArgumentException("Unexpected no log passed in");
        }

        for (MavenProject project : projects) {

            List<String> locations = project.getCompileSourceRoots();
            for (String location : locations) {
                log.info("Location of Directory -> " + location);

            }
            sourceDirectories.addAll(locations);

            if (useTestFiles) {
                log.info("Adding the Test Files");
                sourceDirectories.addAll(project.getTestCompileSourceRoots());
            }

        }

        // Find the Files in each directory.
        // Don't Follow links.
        for (String directory : sourceDirectories) {
            findSourceFiles(directory);
        }

        processCardinalityMap();

        printOutCardinality();

        log.info("Total Number of Java Files in the Project are: " + sourceFiles.size());
        log.info("Total Number of Classes/Interfaces/Enums in the Project are: " + countWithNested.size());

        generateImageFile();

    }

    private void generateImageFile() {
        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        try(FileOutputStream fos = new FileOutputStream("diagram.csv");) {
            
            for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
                
                StringJoiner joiner = new StringJoiner(",");
                for(String val : sourceFileImports.get(key)) {
                    joiner.add(val);
                }
                
                String line = key + ",\"" + joiner.toString() + "\"";
                fos.write(line.getBytes());
                fos.write("\n".getBytes());
            }

        } catch (Exception e) {
            log.warn("Issue processing", e);
        }

    }

    public void printOutCardinality() {

        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        //
        log.info("Cardinality count - imports into other classes");
        for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
            log.info(key + " -> " + sourceFileImports.get(key).size());
        }

    }

    public void processCardinalityMap() {
        // Import > List<Classes>
        // Stored in -> sourceFileImports

        for (String source : sourceFiles) {
            File srcFile = new File(source);
            try {
                JavaType<?> jtFile = Roaster.parse(srcFile);

                String parentJavaClass = jtFile.getQualifiedName();

                if (jtFile instanceof JavaClassSource) {
                    countWithNested.add(parentJavaClass);
                    log.info("[C] -> " + parentJavaClass);
                    JavaClassSource jcs = (JavaClassSource) jtFile;

                    helperImports(parentJavaClass, jcs.getImports());

                    for (JavaSource<?> child : jcs.getNestedTypes()) {

                        String childLoc = child.getQualifiedName();
                        countWithNested.add(childLoc);
                        log.info("  [CC] -> " + childLoc);
                    }
                }

                else if (jtFile instanceof JavaEnumSource) {
                    log.info("[E] -> " + parentJavaClass);
                    countWithNested.add(parentJavaClass);
                    JavaEnumSource jes = (JavaEnumSource) jtFile;

                    helperImports(parentJavaClass, jes.getImports());

                    for (Object child : jes.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
                        countWithNested.add(childLoc);
                        log.info("  [EC] -> " + childLoc);
                    }

                } else if (jtFile instanceof JavaInterfaceSource) {
                    countWithNested.add(parentJavaClass);

                    log.info("[I] -> " + parentJavaClass);
                    JavaInterfaceSource jis = (JavaInterfaceSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());

                    for (Object child : jis.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
                        countWithNested.add(childLoc);
                        log.info("  [IC] -> " + childLoc);
                    }
                } else if (jtFile instanceof JavaAnnotationSource) {
                    countWithNested.add(parentJavaClass);

                    log.info("[A] -> " + parentJavaClass);
                    JavaAnnotationSource jis = (JavaAnnotationSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());
                }

                else {
                    log.info("[O] -> " + parentJavaClass);
                }

            } catch (IOException e) {
                log.info("unable to parse file " + srcFile);
            }

        }
        log.info("Parsed the Cardinality Map:");

    }

    private void helperImports(String parentJavaClass, List<Import> imports) {
        // sourceFileImports
        List<String> importOut = sourceFileImports.get(parentJavaClass);
        if (importOut == null) {
            sourceFileImports.put(parentJavaClass, new ArrayList<String>());
        }

        for (Import importX : imports) {
            String importXStr = importX.getQualifiedName();
            importOut = sourceFileImports.get(importXStr);
            if (importOut == null) {
                importOut = new ArrayList<>();
                sourceFileImports.put(importXStr, importOut);
            }

            importOut.add(parentJavaClass);

        }

    }

    public static void main(String... args) {
        SvgDiagramImpl impl = new SvgDiagramImpl();
        String proc = "Test.java";
        Log log = new Log() {

            @Override
            public boolean isDebugEnabled() {
                return false;
            }

            @Override
            public void debug(CharSequence content) {

            }

            @Override
            public void debug(CharSequence content, Throwable error) {

            }

            @Override
            public void debug(Throwable error) {

            }

            @Override
            public boolean isInfoEnabled() {
                return false;
            }

            @Override
            public void info(CharSequence content) {

            }

            @Override
            public void info(CharSequence content, Throwable error) {

            }

            @Override
            public void info(Throwable error) {

            }

            @Override
            public boolean isWarnEnabled() {
                return false;
            }

            @Override
            public void warn(CharSequence content) {

            }

            @Override
            public void warn(CharSequence content, Throwable error) {

            }

            @Override
            public void warn(Throwable error) {

            }

            @Override
            public boolean isErrorEnabled() {
                return false;
            }

            @Override
            public void error(CharSequence content) {

            }

            @Override
            public void error(CharSequence content, Throwable error) {

            }

            @Override
            public void error(Throwable error) {

            }
        };
        impl.setLog(log);
        impl.addSourceFile(proc);
        impl.processCardinalityMap();
    }

    private void addSourceFile(String proc) {
        sourceFiles.add(proc);

    }

    public void findSourceFiles(String directory) {
        File dir = new File(directory);
        if (dir.exists()) {

            File[] listFiles = dir.listFiles((file, name) -> {
                return name.endsWith(".java");
            });

            // Add to source directory
            if (listFiles != null) {
                for (File file : listFiles) {
                    sourceFiles.add(file.getAbsolutePath());
                    log.info(" File Added to Processing: " + file.getAbsolutePath());
                }
            }

            File[] listFilesFolders = dir.listFiles((file, name) -> {
                return file.isDirectory();
            });

            if (listFilesFolders != null) {
                for (File tmpDir : listFilesFolders) {
                    findSourceFiles(tmpDir.getAbsolutePath());
                }
            }

        } else {
            log.warn("Directory does not exist " + directory);
        }
    }

    @Override
    public void setTargetProjectBaseDirectory(String absolutePath) {
        this.absolutePath = absolutePath;

    }

}

Dynamic Test-NG Tests

In my last few projects, I have used Test-NG.  Uniquely in my current project, I had to generate tests programmatically.  Instead of writing one test for each element in the project, I am able to generate a bunch at-will using the following pattern:

Factory

package test;

import org.testng.annotations.Factory;

public class DynamicTestFactory {


    @Factory
    public Object[] createInstances() {
        
     Object[] result = new Object[10]; 
     for (int i = 0; i < 10; i++) {
        result[i] = new ExampleProcessorTest(Integer.toString(i * 10)+ "A",Integer.toString(i*10) + "B");
      }
      return result;
    }
    
}

Test


package test;

import static org.testng.Assert.assertTrue;

import org.testng.annotations.Test;
 
public class ExampleProcessorTest {

    private String a; 
    private String b; 
    
    public ExampleProcessorTest(String a, String b) {
        this.a = a;
        this.b = b;
        
    }
    
    @Test
    public void testServer() {
        System.out.println("TEST");
        assertTrue(true);;
    }
}

Running Code

[RemoteTestNG] detected TestNG version 6.9.10
[TestNG] Running:
/private/var/folders/07/sw3n5r3170q202d5j4tx8fhw0000gn/T/testng-eclipse--1754065412/testng-customsuite.xml

TEST
TEST
TEST
TEST
TEST
TEST
TEST
TEST
TEST
TEST
PASSED: testServer
PASSED: testServer
PASSED: testServer
PASSED: testServer
PASSED: testServer
PASSED: testServer
PASSED: testServer
PASSED: testServer
PASSED: testServer
PASSED: testServer

===============================================
Default test
Tests run: 10, Failures: 0, Skips: 0
===============================================


===============================================
Default suite
Total tests run: 10, Failures: 0, Skips: 0
===============================================

[TestNG] Time taken by org.testng.reporters.EmailableReporter2@76a4d6c: 9 ms
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 5 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@32cf48b7: 4 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@130f889: 23 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@6e2c9341: 8 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@58a90037: 10 ms
I can also trigger using testng.xml
<class name="DynamicTestFactory" />

Reference

Behavior Driven Development using Cucumber and Test NG

I have started playing with Test NG and Behavior Driven Development using Cucumber. Here are my notes from playing with Cucumber and Test NG.

In Eclipse, Click Help > Eclipse Marketplace

Enter into the Find Box – Cucumber

Click Go

You should see Eclipse plugin.

Click Install

When presented with the Review License dialog, click I Accept

Click Finish

Click Install anyway

Note, I personally clicked details and found they are the Eclipse Cucumber features

/Applications/Eclipse.app/Contents/Eclipse/plugins/cucumber.eclipse.editor_0.0.22.201806032001.jar /Applications/Eclipse.app/Contents/Eclipse/features/cucumber.eclipse.feature_0.0.22.201806032001

/Applications/Eclipse.app/Contents/Eclipse/plugins/cucumber.eclipse.steps.integration_0.0.22.201806032001.jar

/Applications/Eclipse.app/Contents/Eclipse/plugins/cucumber.eclipse.steps.jdt_0.0.22.201806032001.jar

You can always click on the job to see the status.  In the lower right, you are going to see the install progress

Click Restart Now

Be sure to save your work.

Once Eclipse has restarted, click Create > Maven Project

Click Create a simple project

Click Next

Click Advanced

Enter Group Id – cucumber

Enter Artifact Id – test

Enter Version – 99-SNAPSHOT

Click Finish

Right Click the Project – test

Select Configure -> Convert to Cucumber Project

Open the pom.xml and add a dependency for Cucumber and Test NG

     <dependencies>

          <dependency>

               <groupId>io.cucumber</groupId>

               <artifactId>cucumber-java</artifactId>

               <version>4.7.1</version>

               <scope>test</scope>

          </dependency>

          <dependency>

               <groupId>io.cucumber</groupId>

               <artifactId>cucumber-testng</artifactId>

               <version>4.7.1</version>

               <scope>test</scope>

               <exclusions>

                     <exclusion>

                          <groupId>junit</groupId>

                          <artifactId>junit</artifactId>

                     </exclusion>

               </exclusions>

          </dependency>

          <dependency>

               <groupId>org.testng</groupId>

               <artifactId>testng</artifactId>

               <version>6.14.3</version>

               <scope>test</scope>

          </dependency>

     </dependencies>

 

While in the pom.xml, make sure the version of the project is Java 1.8.

     <properties>

          <maven.compiler.source>1.8</maven.compiler.source>

          <maven.compiler.target>1.8</maven.compiler.target>

     </properties>

Click File > Save

Right Click New > Class

Enter Name – RunCucumberTest

Click Browse

Find AbstractTestNGCucumberTests

Click OK

Click OK

Annotate the class with

@CucumberOptions(plugin = "pretty", features = "src/test/resources/TestMe.feature")

It’s a good reminder of where you can go to modify Cucumber.

Right Click the Project > New > Other

Select Cucumber Step Definition File

Enter the Package Name – cucumber

Enter the Class Name – TestMe

Select a few of the annotations@Given @Then @And

Click Finish

Note, I changed the packageio.cucumber.” to the newest approaches.

Replace the content of TestMe.java

package cucumber;

import org.testng.Assert;

import io.cucumber.java.en.And;

import io.cucumber.java.en.Given;

import io.cucumber.java.en.Then;

public class TestMe {

     private int value;

     private int percentage;

     @Given("^I have \"([^\"]*)\"$")

     public void given(String start) throws Throwable {

          this.value = Integer.parseInt(start);

     }

     @And("^I want to add a \"([^\"]*)\"$")

     public void and(String increase) throws Throwable {

          this.percentage = Integer.parseInt(increase);

     }

     @Then("^I should have \"([^\"]*)\"$")

     public void then(String target) throws Throwable {

          Assert.assertEquals(value * percentage, Integer.parseInt(target));

     }

}

The class tests the given, and, then predicates with a simple Assert.  You can always use the TestNG Assert.

Right Click on the TestMe file > Run Configuration

Select Cucumber Feature

Enter Name – TestMe.feature

Replace the Contents with the following:

Feature: TestMe

  Scenario Outline: use the API

    Given I have "<start>"

    And I want to add a "<increase>"

    Then I should have "<total>"

  Examples:

    | start | increase | total |

    |    100 |   100 |    10000 |

Right Click on the Project

Select Run As

Select maven test

You should see

[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< cucumber:test>----------------------------
[INFO] Building test 99-SNAPSHOT
[INFO] --------------------------------[ jar]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ test ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ test ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ test ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ test ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ test ---
[INFO] Surefire report directory: /Users/paulbastide/eclipse-workspace/test/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

Running TestSuite

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@123772c4

Feature: TestMe
  Scenario Outline: use the API with Java7 style # src/test/resources/TestMe.feature:2

    Given I have "<start>"
    And I want to add a "<increase>"
    Then I should have "<total>"

    Examples:

  Scenario Outline: use the API with Java7 style # src/test/resources/TestMe.feature:9
    Given I have "100"                           # TestMe.given(String)
    And I want to add a "100"                    # TestMe.and(String)
    Then I should have "10000"                   # TestMe.then(String)

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.336 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.644 s
[INFO] Finished at: 2019-08-04T13:03:15-04:00
[INFO] ------------------------------------------------------------------------

This is the scenario which you can use, mix and match with other inputs to drive an optimal testing experience.

 

Next step, to figure out how I can use this approach in my healthcare API projects and increase efficacy of my products.

References

Cucumber Eclipse https://cucumber.github.io/cucumber-eclipse/

Cucumber Test NG – https://github.com/cucumber/cucumber-jvm/tree/master/testng

Auto Populated Feature file – https://github.com/cucumber/cucumber-eclipse/wiki/Auto-Generation-of-Sample-Feature-Definition-Template

Step Definition https://github.com/cucumber/cucumber-eclipse/wiki/New-Step-Definition-File-Wizard

Maven Standard Directory Layout http://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html

Test NG and Cucumber – https://medium.com/agile-vision/cucumber-bdd-part-2-creating-a-sample-java-project-with-cucumber-testng-and-maven-127a1053c180

Test https://cucumber.io/docs/guides/10-minute-tutorial/

Examples https://cucumber.io/docs/bdd/example-mapping/

Raspberry Pi: Setting up backup

I have a Raspberry Pi providing household automation and productivity services – WebDav, Backups and Calendar. I always worry about a jolt of power, a failed byte and something that is unrecoverable. Time for a Backup solution.

I plugged in a USB stick – 64GB, and immediately checked the file system is there and visible as SDA (unmounted).

pi@raspberrypi:~# sudo su - 

root@raspberrypi:~# lsblk 
 NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
 sda           8:0    1 58.2G  1 disk 
 └─sda1        8:1    1 58.2G  1 part 
 mmcblk0     179:0    0 14.9G  0 disk 
 ├─mmcblk0p1 179:1    0 43.9M  0 part /boot
 └─mmcblk0p2 179:2    0 14.8G  0 part /

I check to see which one is assigned to the SD card slot (mmc), I really don’t want to reformat Raspbian.  I see the USB stick is on /dev/sda. All of my subsequent commands use /dev/sda as part of the command.

root@raspberrypi:~# parted -ls
Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.
 Model: JetFlash Transcend 64GB (scsi)
 Disk /dev/sda: 62.5GB
 Sector size (logical/physical): 512B/512B
 Partition Table: unknown
 Disk Flags: 

 Model: SD SC16G (sd/mmc)
 Disk /dev/mmcblk0: 15.9GB
 Sector size (logical/physical): 512B/512B
 Partition Table: msdos
 Disk Flags: 
 Number  Start   End     Size    Type     File system  Flags
  1      4194kB  50.2MB  46.0MB  primary  fat32        lba
  2      50.3MB  15.9GB  15.9GB  primary  ext4

If you don’t see the relevant information on your HD or run into issues formatting the hard-drive, install hdparm.  and check with hdparm -r0 /dev/sda  .

TIP: I did run into an issue with an ISO written to a USB drive which locks the partition table and makes it unwriteable.

root@raspberrypi:~# apt-get install hdparm
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 The following package was automatically installed and is no longer required:
   realpath
 Use 'apt autoremove' to remove it.
 The following additional packages will be installed:
   powermgmt-base
 Suggested packages:
   apmd
 The following NEW packages will be installed:
   hdparm powermgmt-base
 0 upgraded, 2 newly installed, 0 to remove and 148 not upgraded.
 Need to get 114 kB of archives.
 After this operation, 278 kB of additional disk space will be used.
 Do you want to continue? [Y/n] y
 Get:1 http://raspbian.mirror.constant.com/raspbian stretch/main armhf hdparm armhf 9.51+ds-1+deb9u1 [105 kB]
 Get:2 http://raspbian-us.ngc292.space/raspbian stretch/main armhf powermgmt-base all 1.31+nmu1 [9,240 B]
 Fetched 114 kB in 0s (120 kB/s)           
 Selecting previously unselected package hdparm.
 (Reading database … 135688 files and directories currently installed.)
 Preparing to unpack …/hdparm_9.51+ds-1+deb9u1_armhf.deb …
 Unpacking hdparm (9.51+ds-1+deb9u1) …
 Selecting previously unselected package powermgmt-base.
 Preparing to unpack …/powermgmt-base_1.31+nmu1_all.deb …
 Unpacking powermgmt-base (1.31+nmu1) …
 Setting up powermgmt-base (1.31+nmu1) …
 Setting up hdparm (9.51+ds-1+deb9u1) …
 Processing triggers for man-db (2.7.6.1-2) …
Preparing to unpack …/hdparm_9.51+ds-1+deb9u1_armhf.deb …

Unpacking hdparm (9.51+ds-1+deb9u1) …

Selecting previously unselected package powermgmt-base.

Preparing to unpack …/powermgmt-base_1.31+nmu1_all.deb …

Unpacking powermgmt-base (1.31+nmu1) …

Setting up powermgmt-base (1.31+nmu1) …

Setting up hdparm (9.51+ds-1+deb9u1) …

Processing triggers for man-db (2.7.6.1-2) …
root@raspberrypi:~# hdparm -r0 /dev/sda
 /dev/sda:
  setting readonly to 0 (off)
  readonly      =  0 (off)

Now that I know the drive is writeable, I need to create the partition. I used

cfdisk

Navigate through the menu and select the maximum size
                                                                                        Disk: /dev/sda
Size: 58.2 GiB, 62495129600 bytes, 122060800 sectors
Label: dos, identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
>> /dev/sda1 2048 122060799 122058752 58.2G 83 Linux

Once you see “Syncing disks.”, you can format the disk. I formatted the partition sda1 with ext4 (I may want to encrypt in the future). Unmount and then format.

root@raspberrypi:~# umount /dev/sda1
root@raspberrypi:~# mkfs.ext4 /dev/sda1
mke2fs 1.43.4 (31-Jan-2017)
Found a dos partition table in /dev/sda1
Proceed anyway? (y,N) y
Creating filesystem with 2828032 4k blocks and 707136 inodes
Filesystem UUID: 363f1b4a-b0f5-4c7b-bf91-66f3823032d6
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

root@raspberrypi:~#

Make the backup directory, edit fstab and mount the directory.  Insert into fstab with your uuid “UUID=363f1b4a-b0f5-4c7b-bf91-66f3823032d6 /backups auto nosuid,nodev,nofail 0 0
”   The second to last avoids backup and the last one enables fsck on reboot.

root@raspberrypi:~# blkid 
/dev/mmcblk0p1: LABEL="boot" UUID="DDAB-3A15" TYPE="vfat" PARTUUID="b53687e8-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="5fa1ec37-3719-4b25-be14-1f7d29135a13" TYPE="ext4" PARTUUID="b53687e8-02"
/dev/mmcblk0: PTUUID="b53687e8" PTTYPE="dos"
/dev/sdb: UUID="363f1b4a-b0f5-4c7b-bf91-66f3823032d6" TYPE="ext4"
root@raspberrypi:~# mkdir /backups 
root@raspberrypi:~# vim /etc/fstab
root@raspberrypi:~# mount -a
root@raspberrypi:~# mount
UUID=363f1b4a-b0f5-4c7b-bf91-66f3823032d6 /backups auto nosuid,nodev,nofail 0 0


You should see backups listed. (Note: I bricked my Raspberry Pi with a bad FSTAB entry, and mounted it on my Mac using Paragon and removed the bad fstab entry. )

Update Crontab with daily backups.

crontab -e

Setup an editor for crontab.

root@raspberrypi:~# crontab -e
no crontab for root - using an empty one

Select an editor. To change later, run 'select-editor'.
1. /bin/ed
2. /bin/nano <---- easiest
3. /usr/bin/vim.basic
4. /usr/bin/vim.tiny

Choose 1-4 [2]: 3
crontab: installing new crontab

I added this line and copied it to /etc/cron.daily/

0 1 * * * /usr/bin/rsync -r /data/ /backups/`date +%w-%A`

crontab -l > pi-backup
mv /root/pi-backup /etc/cron.daily
run-parts /etc/cron.daily

Note, I had to add #!/bin/bash after I copied and remove the timing of the job.

Also, check to see if rsync is installed with which rsync and apt-get install rsync.

This enables backups on a daily basis rotating every 7 days.

Check back on the following day to see your backups

root@raspberrypi:~# /usr/bin/rsync -r /data/ /backups/`date +%w-%A`
root@raspberrypi:~# find /backups
/backups
/backups/lost+found
/backups/0-Sunday
/backups/0-Sunday/startup.sh

Good luck, I hope this helps you with your Raspberry Pi.

References

Java Micro Benchmarking Links

One of my colleagues mentioned Java MicroBenchmarking Harness (jmh), curiosity got the best of me, and I started playing with it for my project. Per a recommendation, I ran this as a new project. I created a new project using:
 mvn archetype:generate \
         -DinteractiveMode=false \
         -DarchetypeGroupId=org.openjdk.jmh \
         -DarchetypeArtifactId=jmh-
java-benchmark-archetype \
         -DgroupId=org.sample \
         -DartifactId=
test \
         -Dversion=1.0


https://javapapers.com/java/java-micro-benchmark-with-jmh/
https://openjdk.java.net/projects/code-tools/jmh/
https://www.baeldung.com/java-microbenchmark-harness
https://gist.github.com/nicoulaj/5517176
https://www.mkyong.com/java/java-jmh-benchmark-tutorial/
http://tutorials.jenkov.com/java-performance/jmh.html

Lessons Learned in July 2019

There is a Mojo in My Dojo (How to write a Maven plugin)

A handy website that outlines the creation of a maven plugin we used in a Java Healthcare project I was working on.   The Mojo I wrote used lessons learned in unit testing from the site.  Thank you to the author.

JAXP Debug Log

I am working on a Java healthcare project and needed to debug a JAXP model and how the serializers are loaded. This site outlined the use of the System property for -Djaxp.debug=1  to trigger debugging logs.

Zookeeper Port Forwarding to all servers from local machine

To simply testing with Zookeeper on a remote Kafka cluster, one must connect to the client application ports on the backend.  When the remote Kafka cluster has multiple nodes and behind a firewall and a SSH jump server, the complexity is fairly high.  Note, the SSH jump server is the permitted man in the middle. 
 
The client must allow application access to Zookeeper on Kafka – listening locally. Current techniques allow for a single port hosted on the developers machine for instance, 2181 listening on the local machine, and a single remote server.  This approach is not reliable – servers are taken out of service, added back, fail, or reroute to the master (another separate server).   
Port Description
88/tcp Kerberos
2181/tcp zookeeper.property.clientPort

A typical connection looks like: 

ssh -J jump-server kafka-1 -L 2181:kafka-1:2181 "while true; do echo "waiting"; sleep 180; done"
 
I worked to develop a small proxy. Setup hosts file.
1 – Edit /etc/hosts
2 – Add entry to hosts file
127.0.0.1 kafka-1

127.0.0.2 kafka-2

127.0.0.3 kafka-3

127.0.0.4 kafka-4

127.0.0.5 kafka-5
3 – Save the hosts file
4 – Setup Available interfaces (1 for each unique service)
1 is already up and in use (you only need to add the extras)
sudo ifconfig lo0 alias 127.0.0.2 up

sudo ifconfig lo0 alias 127.0.0.3 up

sudo ifconfig lo0 alias 127.0.0.4 up

sudo ifconfig lo0 alias 127.0.0.5 up
5 – Setup the port forwarding, forward to jump server ssh -L 30991:localhost:30991 jump-server
6 – Forward to Kafka server ssh -L 30991:localhost:2181 kafka-1
7 – Loop while on kafka server while true; do echo “waiting”; sleep 180; done
8 – Repeat for each kafka server increasing the port by 1 (refer to ports section for mapping)
9 – Setup the Terminal – node krb5-tcp.js
10 – Setup the Terminal – node proxy_socket.js

echo stats | nc kafka-1 2181
Zookeeper version: 3.4.6-IBM_4–1, built on 06/17/2016 01:58 GMT
Clients:
/192.168.12.47:50404[1](queued=0,recved=1340009,sent=1360508)
/192.168.12.46:48694[1](queued=0,recved=1348346,sent=1368936)
/192.168.12.48:39842[1](queued=0,recved=1341655,sent=1362178)
/0:0:0:0:0:0:0:1:39644[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/2205
Received: 4878752
Sent: 4944171
Connections: 4
Outstanding: 0
Zxid: 0x1830001944e
Mode: follower
Node count: 442


11 – Use your code to access Zookeeper Server

References
https://github.com/nodejitsu/node-http-proxy
sudo ifconfig lo0 alias 127.0.0.6 up

sudo ifconfig lo0 alias 127.0.0.7 up

sudo ifconfig lo0 alias 127.0.0.8 up

Configuration

{

"2181": {

"type": "socket",

"members": [

{ "hostname": "kafka-1", "port": 30991 },

{ "hostname": "kafka-2", "port": 30992 },

{ "hostname": "kafka-3", "port": 30993 },

{ "hostname": "kafka-4", "port": 30994 },

{ "hostname": "kafka-5", "port": 30995 }

]

}

}

 

Jaas Configuration
./kerberos/src/main/java/demo/kerberos/jaas.conf
TestClient {
com.sun.security.auth.module.Krb5LoginModule required
principal="ctest4@test.COM"
debug=true
useKeyTab=true
storeKey=true
doNotPrompt=false
keyTab="/Users/paulbastide/tmp/kerberos/test.headless.keytab"
useTicketCache=false;
};

Java – App.java

package demo.kerberos;

import javax.security.auth.*;
import javax.security.auth.login.*;
import javax.security.auth.callback.*;
import javax.security.auth.kerberos.*;
import java.io.*;

public class App {
public static void main(String[] args) {

System.setProperty("java.security.auth.login.config",
"/Users/paulbastide/tmp/kerberos/src/main/java/demo/kerberos/jaas.conf");
System.setProperty("java.security.krb5.conf", "/Users/paulbastide/tmp/kerberos/krb5.conf");

Subject mysubject = new Subject();
LoginContext lc;

try {

lc = new LoginContext("TestClient", mysubject, new MyCallBackHandler());
lc.login();

} catch (LoginException e) {
e.printStackTrace();
}

}

}

Java - MyCallBackHandler.java
package demo.kerberos;

import javax.security.auth.*;
import javax.security.auth.login.*;
import javax.security.auth.callback.*;
import javax.security.auth.kerberos.*;
import java.io.*;

public class MyCallBackHandler implements CallbackHandler {
public void handle(Callback[] callbacks)
throws IOException, UnsupportedCallbackException {

for (int i = 0; i < callbacks.length; i++) {
System.out.println(callbacks[i]);
}
}
}