Tag: python

  • Code Graph showing the Layout of the Code base

    I’ve been mixing data analysis and Java programming recently.  I wrote a tool to do the analysis (Maven/Python).

    Here is the obfuscated output of the analysis, showing the hotspots.  I opted to show a thumbnail of the image here to protect the confidentiality of the project.  The generated image was also 78 Megabytes.  (a bit much, but you can zoom right in).

    Complicated Graph

    If you use a smaller set of classes and imports, the Maven plugin generates a reasonable diagram.csv file using

    mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml

    You then see the output diagram.csv.

    To generate the layout dependencies of classes in your project.  Use the snippets, and the Jupyter Notebook – https://github.com/prb112/examples/blob/master/code-graph/code-graph.ipynb and view at https://nbviewer.jupyter.org/github/prb112/examples/blob/master/code-graph/code-graph.ipynb

    Simple Graph

    Reference

    Code Graph on Git https://github.com/prb112/examples/tree/master/code-graph

    CSV Sample Data diagram.csv

    POM

    <project xmlns="http://maven.apache.org/POM/4.0.0"
    	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    	<modelVersion>4.0.0</modelVersion>
    
    	<groupId>example</groupId>
    	<artifactId>generate-diagram</artifactId>
    	<version>99-SNAPSHOT</version>
    	<packaging>maven-plugin</packaging>
    
    	<name>generate-diagram</name>
    
    	<properties>
    		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    		<maven.compiler.source>1.8</maven.compiler.source>
    		<maven.compiler.target>1.8</maven.compiler.target>
    		<version.roaster>2.21.0.Final</version.roaster>
    	</properties>
    
    
    
    	<build>
    		<pluginManagement>
    			<plugins>
    				<plugin>
    					<artifactId>maven-jar-plugin</artifactId>
    					<version>2.6</version>
    					<executions>
    						<execution>
    							<goals>
    								<goal>test-jar</goal>
    							</goals>
    						</execution>
    					</executions>
    					<configuration>
    						<archive>
    							<manifest>
    								<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
    								<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
    							</manifest>
    						</archive>
    					</configuration>
    				</plugin>
    				<plugin>
    					<groupId>org.apache.maven.plugins</groupId>
    					<artifactId>maven-plugin-plugin</artifactId>
    					<version>3.6.0</version>
    					<configuration>
    						<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
    					</configuration>
    					<executions>
    						<execution>
    							<id>mojo-descriptor</id>
    							<goals>
    								<goal>descriptor</goal>
    							</goals>
    							<phase>process-classes</phase>
    							<configuration>
    								<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
    							</configuration>
    						</execution>
    					</executions>
    				</plugin>
    			</plugins>
    		</pluginManagement>
    
    		<plugins>
    			<plugin>
    				<!-- Embeds the dependencies in fhir-tools into the jar. -->
    				<groupId>org.apache.maven.plugins</groupId>
    				<artifactId>maven-shade-plugin</artifactId>
    				<version>3.2.1</version>
    				<executions>
    					<execution>
    						<phase>package</phase>
    						<goals>
    							<goal>shade</goal>
    						</goals>
    						<configuration>
    							<artifactSet>
    								<excludes>
    									<exclude>org.testng:testng</exclude>
    									<exclude>org.apache.maven:lib:tests</exclude>
    									<exclude>org.apache.maven</exclude>
    								</excludes>
    							</artifactSet>
    						</configuration>
    					</execution>
    				</executions>
    			</plugin>
    		</plugins>
    	</build>
    
    	<dependencies>
    		<dependency>
    			<groupId>org.jboss.forge.roaster</groupId>
    			<artifactId>roaster-api</artifactId>
    			<version>${version.roaster}</version>
    		</dependency>
    		<dependency>
    			<groupId>org.jboss.forge.roaster</groupId>
    			<artifactId>roaster-jdt</artifactId>
    			<version>${version.roaster}</version>
    			<scope>runtime</scope>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.maven</groupId>
    			<artifactId>maven-plugin-api</artifactId>
    			<version>3.6.1</version>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.maven.plugin-tools</groupId>
    			<artifactId>maven-plugin-annotations</artifactId>
    			<version>3.6.0</version>
    			<optional>true</optional>
    			<scope>provided</scope>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.maven</groupId>
    			<artifactId>maven-core</artifactId>
    			<version>3.6.1</version>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.maven</groupId>
    			<artifactId>maven-artifact</artifactId>
    			<version>3.6.0</version>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.maven</groupId>
    			<artifactId>maven-model</artifactId>
    			<version>3.6.0</version>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.maven</groupId>
    			<artifactId>maven-compat</artifactId>
    			<version>3.6.1</version>
    			<scope>test</scope>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.maven.plugin-testing</groupId>
    			<artifactId>maven-plugin-testing-harness</artifactId>
    			<version>3.3.0</version>
    			<scope>test</scope>
    		</dependency>
    	</dependencies>
    </project>
    
    package demo;
    
    import java.io.File;
    import java.util.Properties;
    
    import org.apache.maven.execution.MavenSession;
    import org.apache.maven.plugin.AbstractMojo;
    import org.apache.maven.plugin.MojoExecutionException;
    import org.apache.maven.plugin.MojoFailureException;
    import org.apache.maven.plugins.annotations.Execute;
    import org.apache.maven.plugins.annotations.LifecyclePhase;
    import org.apache.maven.plugins.annotations.Mojo;
    import org.apache.maven.plugins.annotations.Parameter;
    import org.apache.maven.plugins.annotations.ResolutionScope;
    import org.apache.maven.project.MavenProject;
    
    import com.ibm.watsonhealth.fhir.tools.plugin.diagram.DiagramFactory;
    import com.ibm.watsonhealth.fhir.tools.plugin.diagram.impl.IDiagramGenerator;
    
    /**
     * This class coordinates the calls to the Diagram generation plugin
     * 
     * The phase is initialize. To find a list of phases -
     * https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference
     * 
     * Run the following to setup the plugin: <code>
     * mvn clean package install -f generate-diagram/pom.xml
     * </code>
     * 
     * Run the following to setup the classes in fhir-model: <code> 
     * mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml
     * </code>
     * 
     * @author PBastide
     * 
     * @requiresDependencyResolution runtime
     *
     */
    @Mojo(name = "generate-diagram", //$NON-NLS-1$
            requiresProject = true, requiresDependencyResolution = ResolutionScope.RUNTIME_PLUS_SYSTEM, requiresDependencyCollection = ResolutionScope.RUNTIME_PLUS_SYSTEM, defaultPhase = LifecyclePhase.GENERATE_SOURCES, requiresOnline = false, threadSafe = false, aggregator = true)
    @Execute(phase = LifecyclePhase.GENERATE_SOURCES)
    public class DiagramPlugin extends AbstractMojo {
    
        @Parameter(defaultValue = "${project}", required = true, readonly = true) //$NON-NLS-1$
        protected MavenProject mavenProject;
    
        @Parameter(defaultValue = "${session}")
        private MavenSession session;
    
        @Parameter(defaultValue = "${project.basedir}", required = true, readonly = true) //$NON-NLS-1$
        private File baseDir;
    
        @Override
        public void execute() throws MojoExecutionException, MojoFailureException {
            if (baseDir == null || !baseDir.exists()) {
                throw new MojoFailureException("The Base Directory is not found.  Throwing failure. ");
            }
    
            // Grab the Properties (the correct way)
            // https://maven.apache.org/plugin-developers/common-bugs.html#Using_System_Properties
            Properties userProps = session.getUserProperties();
            String useTestsDirectoryStr = userProps.getProperty("useTestsDirectory", "false");
    
            // Converts Limit value to boolean value.
            boolean useTestsDirectory = Boolean.parseBoolean(useTestsDirectoryStr);
    
            // Grab the right generator and set it up.
            IDiagramGenerator generator = DiagramFactory.getDiagramGenerator();
    
            // Set the use of tests directory
            generator.use(useTestsDirectory);
    
            // Get the base directory .
            generator.setTargetProjectBaseDirectory(baseDir.getAbsolutePath() + "/target");
    
            // Passes the Log to the implementation code.
            generator.setLog(getLog());
    
            // Add the project
            generator.add(mavenProject);
    
            // Builds the Diagram
            generator.generateDiagram();
        }
    }
    package example.impl;
    
    import java.io.File;
    import java.io.FileOutputStream;
    import java.io.IOException;
    import java.io.OutputStream;
    import java.util.ArrayList;
    import java.util.Comparator;
    import java.util.HashMap;
    import java.util.List;
    import java.util.Map;
    import java.util.StringJoiner;
    import java.util.stream.Collectors;
    
    import org.apache.maven.plugin.logging.Log;
    import org.apache.maven.project.MavenProject;
    import org.jboss.forge.roaster.Roaster;
    import org.jboss.forge.roaster.model.JavaType;
    import org.jboss.forge.roaster.model.source.Import;
    import org.jboss.forge.roaster.model.source.JavaAnnotationSource;
    import org.jboss.forge.roaster.model.source.JavaClassSource;
    import org.jboss.forge.roaster.model.source.JavaEnumSource;
    import org.jboss.forge.roaster.model.source.JavaInterfaceSource;
    import org.jboss.forge.roaster.model.source.JavaSource;
    import org.w3c.dom.DOMImplementation;
    import org.w3c.dom.Document;
    import org.w3c.dom.Element;
    
    public class DiagramImpl implements IDiagramGenerator {
    
        private String absolutePath = null;
        private Log log = null;
        private Boolean useTestFiles = false;
    
        private List<MavenProject> projects = new ArrayList<>();
    
        private List<String> sourceDirectories = new ArrayList<>();
    
        private List<String> sourceFiles = new ArrayList<>();
    
        private List<String> countWithNested = new ArrayList<>();
    
        private Map<String, List<String>> sourceFileImports = new HashMap<>();
    
        @Override
        public void add(MavenProject mavenProject) {
            if (mavenProject == null) {
                throw new IllegalArgumentException("no access to the maven project's plugin object");
            }
    
            log.info("Projects added...");
            projects = mavenProject.getCollectedProjects();
    
        }
    
        @Override
        public void use(Boolean useTestFiles) {
            this.useTestFiles = useTestFiles;
    
        }
    
        @Override
        public void setLog(Log log) {
            this.log = log;
        }
    
        @Override
        public void generateDiagram() {
            if (absolutePath == null) {
                throw new IllegalArgumentException("Bad Path " + absolutePath);
            }
    
            if (log == null) {
                throw new IllegalArgumentException("Unexpected no log passed in");
            }
    
            for (MavenProject project : projects) {
    
                List<String> locations = project.getCompileSourceRoots();
                for (String location : locations) {
                    log.info("Location of Directory -> " + location);
    
                }
                sourceDirectories.addAll(locations);
    
                if (useTestFiles) {
                    log.info("Adding the Test Files");
                    sourceDirectories.addAll(project.getTestCompileSourceRoots());
                }
    
            }
    
            // Find the Files in each directory.
            // Don't Follow links.
            for (String directory : sourceDirectories) {
                findSourceFiles(directory);
            }
    
            processCardinalityMap();
    
            printOutCardinality();
    
            log.info("Total Number of Java Files in the Project are: " + sourceFiles.size());
            log.info("Total Number of Classes/Interfaces/Enums in the Project are: " + countWithNested.size());
    
            generateImageFile();
    
        }
    
        private void generateImageFile() {
            Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);
    
            try(FileOutputStream fos = new FileOutputStream("diagram.csv");) {
                
                for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
                    
                    StringJoiner joiner = new StringJoiner(",");
                    for(String val : sourceFileImports.get(key)) {
                        joiner.add(val);
                    }
                    
                    String line = key + ",\"" + joiner.toString() + "\"";
                    fos.write(line.getBytes());
                    fos.write("\n".getBytes());
                }
    
            } catch (Exception e) {
                log.warn("Issue processing", e);
            }
    
        }
    
        public void printOutCardinality() {
    
            Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);
    
            //
            log.info("Cardinality count - imports into other classes");
            for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
                log.info(key + " -> " + sourceFileImports.get(key).size());
            }
    
        }
    
        public void processCardinalityMap() {
            // Import > List<Classes>
            // Stored in -> sourceFileImports
    
            for (String source : sourceFiles) {
                File srcFile = new File(source);
                try {
                    JavaType<?> jtFile = Roaster.parse(srcFile);
    
                    String parentJavaClass = jtFile.getQualifiedName();
    
                    if (jtFile instanceof JavaClassSource) {
                        countWithNested.add(parentJavaClass);
                        log.info("[C] -> " + parentJavaClass);
                        JavaClassSource jcs = (JavaClassSource) jtFile;
    
                        helperImports(parentJavaClass, jcs.getImports());
    
                        for (JavaSource<?> child : jcs.getNestedTypes()) {
    
                            String childLoc = child.getQualifiedName();
                            countWithNested.add(childLoc);
                            log.info("  [CC] -> " + childLoc);
                        }
                    }
    
                    else if (jtFile instanceof JavaEnumSource) {
                        log.info("[E] -> " + parentJavaClass);
                        countWithNested.add(parentJavaClass);
                        JavaEnumSource jes = (JavaEnumSource) jtFile;
    
                        helperImports(parentJavaClass, jes.getImports());
    
                        for (Object child : jes.getNestedTypes()) {
    
                            String childLoc = child.getClass().getName();
                            countWithNested.add(childLoc);
                            log.info("  [EC] -> " + childLoc);
                        }
    
                    } else if (jtFile instanceof JavaInterfaceSource) {
                        countWithNested.add(parentJavaClass);
    
                        log.info("[I] -> " + parentJavaClass);
                        JavaInterfaceSource jis = (JavaInterfaceSource) jtFile;
    
                        helperImports(parentJavaClass, jis.getImports());
    
                        for (Object child : jis.getNestedTypes()) {
    
                            String childLoc = child.getClass().getName();
                            countWithNested.add(childLoc);
                            log.info("  [IC] -> " + childLoc);
                        }
                    } else if (jtFile instanceof JavaAnnotationSource) {
                        countWithNested.add(parentJavaClass);
    
                        log.info("[A] -> " + parentJavaClass);
                        JavaAnnotationSource jis = (JavaAnnotationSource) jtFile;
    
                        helperImports(parentJavaClass, jis.getImports());
                    }
    
                    else {
                        log.info("[O] -> " + parentJavaClass);
                    }
    
                } catch (IOException e) {
                    log.info("unable to parse file " + srcFile);
                }
    
            }
            log.info("Parsed the Cardinality Map:");
    
        }
    
        private void helperImports(String parentJavaClass, List<Import> imports) {
            // sourceFileImports
            List<String> importOut = sourceFileImports.get(parentJavaClass);
            if (importOut == null) {
                sourceFileImports.put(parentJavaClass, new ArrayList<String>());
            }
    
            for (Import importX : imports) {
                String importXStr = importX.getQualifiedName();
                importOut = sourceFileImports.get(importXStr);
                if (importOut == null) {
                    importOut = new ArrayList<>();
                    sourceFileImports.put(importXStr, importOut);
                }
    
                importOut.add(parentJavaClass);
    
            }
    
        }
    
        public static void main(String... args) {
            SvgDiagramImpl impl = new SvgDiagramImpl();
            String proc = "Test.java";
            Log log = new Log() {
    
                @Override
                public boolean isDebugEnabled() {
                    return false;
                }
    
                @Override
                public void debug(CharSequence content) {
    
                }
    
                @Override
                public void debug(CharSequence content, Throwable error) {
    
                }
    
                @Override
                public void debug(Throwable error) {
    
                }
    
                @Override
                public boolean isInfoEnabled() {
                    return false;
                }
    
                @Override
                public void info(CharSequence content) {
    
                }
    
                @Override
                public void info(CharSequence content, Throwable error) {
    
                }
    
                @Override
                public void info(Throwable error) {
    
                }
    
                @Override
                public boolean isWarnEnabled() {
                    return false;
                }
    
                @Override
                public void warn(CharSequence content) {
    
                }
    
                @Override
                public void warn(CharSequence content, Throwable error) {
    
                }
    
                @Override
                public void warn(Throwable error) {
    
                }
    
                @Override
                public boolean isErrorEnabled() {
                    return false;
                }
    
                @Override
                public void error(CharSequence content) {
    
                }
    
                @Override
                public void error(CharSequence content, Throwable error) {
    
                }
    
                @Override
                public void error(Throwable error) {
    
                }
            };
            impl.setLog(log);
            impl.addSourceFile(proc);
            impl.processCardinalityMap();
        }
    
        private void addSourceFile(String proc) {
            sourceFiles.add(proc);
    
        }
    
        public void findSourceFiles(String directory) {
            File dir = new File(directory);
            if (dir.exists()) {
    
                File[] listFiles = dir.listFiles((file, name) -> {
                    return name.endsWith(".java");
                });
    
                // Add to source directory
                if (listFiles != null) {
                    for (File file : listFiles) {
                        sourceFiles.add(file.getAbsolutePath());
                        log.info(" File Added to Processing: " + file.getAbsolutePath());
                    }
                }
    
                File[] listFilesFolders = dir.listFiles((file, name) -> {
                    return file.isDirectory();
                });
    
                if (listFilesFolders != null) {
                    for (File tmpDir : listFilesFolders) {
                        findSourceFiles(tmpDir.getAbsolutePath());
                    }
                }
    
            } else {
                log.warn("Directory does not exist " + directory);
            }
        }
    
        @Override
        public void setTargetProjectBaseDirectory(String absolutePath) {
            this.absolutePath = absolutePath;
    
        }
    
    }
    
  • Raspberry Pi – Part IV – Simple Wiring Test

    I am plugging away working with my Pi. I went back to the Ada Fruit Site, and started working on one of the tutorials from the part I purchased – https://www.adafruit.com/products/2125   I want to be able to demonstrate a simple wiring with my Pi works.

    I selected Email Notifier. I read through it, and realized I needed to convert it to work for the B+ model. The pi4j website has a nice diagram for the pins http://pi4j.com/images/j8header-b-plus-large.png and I simplified the test a bit.

    I launched the simple python file

    pi@seconds ~ $ sudo python testPi.py

    Pi with a Green Light - Using Python
    Pi with a Green Light – Using Python

    I added the pi4j dependency to my maven pom

    <dependency>
        <groupId>com.pi4j</groupId>
        <artifactId>pi4j-core</artifactId>
       <version>1.0</version>
    </dependency>

    I create a servlet which replicated the same code from before (GPIO).

    I copied the war file to the local webapp directory, and restarted jetty

    pi@seconds /opt/jetty/web/bbq/webapps $ sudo cp ~/pi.webapp.war ./
    pi@seconds /opt/jetty/web/bbq/webapps $ sudo chown jetty:jetty pi.webapp.war
    pi@seconds /opt/jetty/web/bbq/webapps $ sudo /etc/init.d/jetty restart

    Refer to https://wiki.eclipse.org/Jetty/Howto/Deploy_Web_Applications  and I did have to add –module=jsp to the start.ini for my web configuration.

    I hit the servlet – http://192.168.1.200/pi.webapp/Control?status=true

    Exception
    Exception

    Which makes sense, since “Software using the Pi4J library must be run with ROOT level permissions.” (actually it doesn’t since I am running as root)…

    Issues remain.  I’ll tackle them on a different day 🙂   … Part 5 …