Categories
app dev

Gatsby & Carbon: Build with Github Action

As some of you know, I work on the IBM FHIR Server and with my colleagues, I have started automating some of the actions we take – Build, Test, Deploy, Deploy our website.

More specific to the “Deploy our website” automation, our website uses technologies, such as Gatsby, Carbon, Gatsby Carbon Theme. Fundamentally, a static site generation technology, like Jekyll, Gatsby uses Node, Yarn and some nice React code.

To build our site with GitHub actions, I built out a site workflow.  The key elements to this workflow are:

  • Triggers
  • Node.js and Ubuntu Images
  • Build
  • Add, Commit and Push to GH Pages
  • Debugging and Replicating Locally

Triggers

For the Triggers, I recommend limiting the site generation to master branches.  The master branch filter and on push, limits the re-deployment, also keep your site building on on docs/** changes.

on:
push:
paths:
– “docs/**”
branches:
– master
 
There is a subtlety the websites are cached for 10 minutes, confirmed on the site – Caching assets in website served from GitHub pages

Node.js and Ubuntu Images

I opted to use Ubuntu with Node.js
 
jobs:
build:
runs-on: ubuntu-latest

strategy:
matrix:
node-version: [12.x]
 
The important thing is ubuntu-latest which has some incompatibility with Gatsby Carbon’s build. 

Build

I build the system as follows:

Checkout the repo to a folder

steps:
-name: Grab the Master Branch
uses: actions/checkout@v1
with:
working-directory: fhir
ref: refs/heads/master
fetch-depth: 1
path: fhir
 
Activate Node
name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
 
Setup the build
 
echo “Check on Path”
pwd
cd docs/
npm install -g gatsby-cli
gatsby telemetry –disable
 
Install the packages, note, fsevents is not used on linux images, so use–no-optional (these plugins are suspect).
 
npm install –no-optional –save react react-copy-to-clipboard react-dom react-ga classnames carbon @carbon/addons-website carbon-components carbon-components-react carbon-addons-cloud carbon-icons gatsby gatsby-theme-carbon-starter markdown-it gatsby-plugin-manifest gatsby-plugin-slug gatsby-plugin-sitemap gatsby-plugin-sharp
 
With ubuntu, you can’t use gatsby build directly per https://github.com/gatsbyjs/gatsby/issues/17557, 
so I use the suggestion as a workaround due to path/issues in the gatsby component dependency of fsevents.
 
npm –prefix-paths run build
cp -R public/ ../../public/
 
Grab the GH-Pages branch
 
– name: Grab the GH Pages Branch
uses: actions/checkout@v1
with:
working-directory: gh-pages
ref: refs/heads/gh-pages
fetch-depth: 1
path: docs
token: ${{ secrets.GITHUB_TOKEN }}

Per Bypassing Jekyll on GitHub Pages, be sure to add the .nojekll to the root of the gh-pages. I added a guard in the shell script to check if the file is there, and create the file if it does not exist.

If you need Environment variables, you should add the environment variables to the step.

Add, Commit and Push to GH Pages

I add the gitignore and nojekll files while removing any cached files, before moving in the new files.

I also like to make sure when this runs there is a build.txt file to trace when the site is built. (This file contains the build time Thu Nov 21 19:39:49 UTC 2019

I then use the github environment variables passed in to push the contents to the repo the branch is from. 

-name: Commit and Add GH Pages

run: |

echo "cleaning up the prior files on the branch"

if [ ! -f .nojekyll ]

then

touch .nojekyll

rm -f _config.yml

fi

rm -f *.js webpack.stats.json styles-*.js styles-*.js.map webpack-runtime-*.js.map webpack-runtime-*.js manifest.webmanifest component---*.js* app-*.js*

rm -rf docs/node_modules docs/public docs/.cache

echo "Moving the files around for gh-pages"

cp -Rf ../public/* ./

find .

date > build.txt

git config --global user.email "${{ secrets.GITHUB_ACTOR }}@users.noreply.github.com"

git config --global user.name "Git Hub Site Automation"

git add .

git commit -m "Update to GH-Pages"

- name: Push changes to GH Pages

run: |

echo "Push Changes"

git branch

remote_repo="https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"

git push "${remote_repo}" HEAD:gh-pages

env:

GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

GITHUB_REPOSITORY: ${{ secrets.GITHUB_REPOSITORY }}

GITHUB_ACTOR: ${{ secrets.GITHUB_ACTOR }}

CI: true

Debugging and Replicating Locally

If you are troubleshooting, you can use a couple of approaches: 

1 – Create a Docker Image

Create the Image

docker run -itd –name gatsby-node -v docs:/data node:latest

Copy the Files

docker cp ~/FHIR/docs 6d810efb3b586739932166d424641003ee9b238de506543fcdd47eb7e7d41699:/data

Launch the shell and try the build

npm install –no-optional –save react react-copy-to-clipboard react-dom react-ga classnames carbon @carbon/addons-website carbon-components carbon-components-react carbon-addons-cloud carbon-icons gatsby gatsby-theme-carbon-starter markdown-it gatsby-plugin-manifest gatsby-plugin-slug gatsby-plugin-sitemap gatsby-plugin-sharp

Run the gatsby build

npm –prefix-paths run build

2. If you want complicated navigation, refer to https://github.com/gatsbyjs/gatsby/blob/master/www/src/data/sidebars/doc-links.yaml however… gatsby-carbon-theme’s sidebar links only uses the to value not the href value.

3.  If you have an issue with your deployment check a couple of things:

Check your Deployed Environment. You should see a deployment in the last few seconds.

Check your Settings You should see no issues, else investigate the site locally on the gh-pages branch, and check Troubleshooting Jekyll build errors for GitHub Pages sites

Best of luck with your build!

Categories
app dev

Migrating Source Code Git-to-Git

Migrating source code is a pain in the butt, I know.  There are about 9 million variations, and one of interest to me – git to github.com. 

There are a number of tools to clean up your git history and prepare to move.

  • Git and Scripting
  • BFG Repo Cleaner
  • Git-Python
  • JGit

I found Git-Python a bit cumbersome, BFG Repo Cleaner more than I needed/wanted, and Git / Scripting too much work. After some prototyping, I opted for JGit from Eclipse and some Git knowhow.

First, I switched to the source Git Repo branch I wanted to migrate and exported the commit list.

git rev-list HEAD > commits.txt

which results in

7452e8eb1f287e2ad2d8c2d005455197ba4183f2

baac5e4d0ce999d983c016d67175a898f50444b3

2a8e2ec7507e05555e277f214bf79119cda4f025

This commits.txt is useful down the line.

I am a Maven disciple so, I created a maven java project with Java 1.8 and the following dependencies:

        <dependency>

            <groupId>org.eclipse.jgit</groupId>

            <artifactId>org.eclipse.jgit</artifactId>

            <version>${jgit.version}</version>

        </dependency>

        <dependency>

            <groupId>com.google.guava</groupId>

            <artifactId>guava</artifactId>

            <version>20.0</version>

        </dependency>

        <dependency>

            <groupId>org.slf4j</groupId>

            <artifactId>slf4j-nop</artifactId>

            <version>1.7.25</version>

        </dependency>

I used the JGit to check the list of commits (note the REPO here must have .git at end).

try (Git git = Git.open(new File(SOURCE_GIT_REPO))) {

            printHeaderLine();

            System.out.println("Starting Branch is " + git.getRepository().getBranch());

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                String binSha = commit.name();

                commits.add(binSha);

            }

        }

I flip it around, so I can process OLDEST to NEWEST

        Collections.reverse(commits);

I used the git log (LogCommand in JGit) to find out all the times a FILE was modified, and do custom processing:

try (Git git = Git.open(new File(REPO))) {

            LogCommand logCommand = git.log().add(git.getRepository().resolve(Constants.HEAD)).addPath(fileName.replace(REPO, ""));

            Set<String> years = new HashSet<>();

            for (RevCommit revCommit : logCommand.call()) {

                Instant instant = Instant.ofEpochSecond(revCommit.getCommitTime());

// YOUR PROCESSING

            }

        }

}

To find out the files specifically in the HEAD of the repo, gets the files and paths, and puts it in a List

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            if (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (RevWalk walk = new RevWalk(git.getRepository());) {

                    RevTree tree = walk.parseTree(commit.getId());

                    try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                        treeWalk.addTree(tree);

                        treeWalk.setRecursive(true);

                        while (treeWalk.next()) {

                            headFiles.add(treeWalk.getPathString());

                        }

                    }

                }

            } 

        }

}

I built a history of changes.

try (Git git = Git.open(new File("test/.git"))) {

            Iterator<RevCommit> iter = git.log().call().iterator();

            while (iter.hasNext()) {

                RevCommit commit = iter.next();

                try (DiffFormatter df = new DiffFormatter(DisabledOutputStream.INSTANCE);) {

                    df.setRepository(git.getRepository());                    df.setDiffComparator(RawTextComparator.DEFAULT);

                    df.setDetectRenames(true);


                    CommitHistoryEntry.Builder builder =                            CommitHistoryEntry.builder().binsha(commit.name()).commitTime(commit.getCommitTime()).authorEmail(commit.getAuthorIdent().getEmailAddress()).shortMessage(commit.getShortMessage()).fullMessage(commit.getFullMessage());

                    RevCommit[] parents = commit.getParents();

                    if (parents != null && parents.length > 0) {

                        List<DiffEntry> diffs = df.scan(commit.getTree(), parents[0]);

                        builder.add(diffs);

                    } else {

                        builder.root(true);

                        try (RevWalk walk = new RevWalk(git.getRepository());) {

                            RevTree tree = walk.parseTree(commit.getId());

                            try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                                treeWalk.addTree(tree);

                                treeWalk.setRecursive(true);

                                while (treeWalk.next()) {                                   

builder.file(treeWalk.getPathString());

                                }

                            }

                        }

                    }

                    entries.add(entry);

                }

            }

I did implement the Visitor pattern to optimize the modifications to the commit details and cleanup any bad Identity mappings (folks had many emails and names which I unified) and cleanedup the email addresses.

Next, I created a destination git (all fresh and ready to go):

try (Git thisGit = Git.init().setDirectory(new File(REPO_DIR)).call()) {

   git = thisGit;

     }

One should make sure the path exists, and it doesn’t matter if you have files in it.

Commit the files in the git directory… you can commit without FILES!

CommitCommand commitCommand = git.commit();

        // Setup the Identity and date

        Date aWhen = new Date(entry.getCommitTime() * 1000);

        PersonIdent authorIdent =

                new PersonIdent(entry.getAuthorName(), entry.getAuthorEmail(), aWhen, TimeZone.getDefault());

        commitCommand.setCommitter(authorIdent);

        commitCommand.setAllowEmpty(true);

        commitCommand.setAuthor(authorIdent);

        commitCommand.setMessage(entry.getShortMessage());

        commitCommand.setNoVerify(true);

        commitCommand.setSign(false);

        commitCommand.call();

Note, you can set to almost any point in time.  As long as you don’t sign it, it’ll be OK.  I don’t recommend this as a general practice.

To grab the file, you can do a tree walk, and resolve to the object ID.

try (TreeWalk treeWalk = new TreeWalk(git.getRepository());) {

                            treeWalk.addTree(tree);

                            treeWalk.setRecursive(true);

                            int localCount = 0;

                            while (treeWalk.next()) {

                                String fileName = treeWalk.getPathString();

ObjectId objectId = treeWalk.getObjectId(0);

            ObjectLoader loader = git.getRepository().open(objectId);

            String fileOutput = GIT_OUTPUT + "/" + binSha + "/" + fileNameWithRelativePath;

            int last = fileOutput.lastIndexOf('/');

            String fileOutputDir = fileOutput.substring(0, last);

            File dir = new File(fileOutputDir);

            dir.mkdirs();

            // and then one can the loader to read the file

            try (FileOutputStream out =

                    new FileOutputStream(GIT_OUTPUT + "/" + binSha + "/"

                            + fileNameWithRelativePath);) {

                // System.out

                byte[] bytes = loader.getBytes();

                if (hasBeenModified(bytes, fileNameWithRelativePath)) {

                    loader.copyTo(out);

                    count++;

                    result = true;

                }

            }

Note, I did check if the file was duplicate, it saved a couple of steps.

If you want to add files, you can set:

commitCommand.setAll(addFiles);

git.add().addFilepattern(file).call();

Git in the background builds the DIFFs for any file that is not specially treated as binary in the .gitattributes file.

For each commit, I loaded the file – checking for stop-words, checked the copyright header, check the file type, and compared it against the head . 

Tip, if you need to reset from a bad test.

git reset --hard origin # reset the branch

rm -rf .git # reset the repo (also be sure to remove the files)

The moving of the branch, you can execute

cd <GIT_REPO>

git checkout <BRANCH_TO_MIGRATE>

git reset --hard origin

git pull

git gc --aggressive --prune=now

git push git@github.com:<MyOrg>/<DEST_REPO>.git <BRANCH_TO_MIGRATE>:master

Note, I did rename the master branch in the repo prior.  Voila.  550M+ Repo moved and cleaned up.

The repo is now migrated, and up-to-date.  I hope this helps you.

References

Rename Branch in Git

https://multiplestates.wordpress.com/2015/02/05/rename-a-local-and-remote-branch-in-git/

 

Rewrite History

https://help.github.com/en/articles/removing-sensitive-data-from-a-repository

https://stackoverflow.com/questions/tagged/git-rewrite-history

 

BFG Repo Cleaner

https://github.com/rtyley/bfg-repo-cleaner

https://rtyley.github.io/bfg-repo-cleaner/

 

JGit

https://www.vogella.com/tutorials/JGit/article.html

https://github.com/eclipse/jgit

http://wiki.eclipse.org/JGit/User_Guide#Repository

https://www.programcreek.com/java-api-examples/?class=org.eclipse.jgit.revwalk.RevWalk&method=parseCommit

https://www.eclipse.org/forums/index.php/t/213979/

https://stackoverflow.com/questions/46727610/how-to-get-the-list-of-files-as-a-part-of-commit-in-jgit

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/ListNotes.java

https://stackoverflow.com/questions/9683279/make-the-current-commit-the-only-initial-commit-in-a-git-repository

https://stackoverflow.com/questions/40590039/how-to-get-the-file-list-for-a-commit-with-jgit

https://doc.nuxeo.com/blog/jgit-example/

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/api/ReadFileFromCommit.java

https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit.test/tst/org/eclipse/jgit/api/AddCommandTest.java

https://stackoverflow.com/questions/12734760/jgit-how-to-add-all-files-to-staging-area

https://github.com/centic9/jgit-cookbook/blob/master/src/main/java/org/dstadler/jgit/porcelain/DiffFilesInCommit.java

 

Categories
app dev development

Code Graph showing the Layout of the Code base

I’ve been mixing data analysis and Java programming recently.  I wrote a tool to do the analysis (Maven/Python).

Here is the obfuscated output of the analysis, showing the hotspots.  I opted to show a thumbnail of the image here to protect the confidentiality of the project.  The generated image was also 78 Megabytes.  (a bit much, but you can zoom right in).

Complicated Graph

If you use a smaller set of classes and imports, the Maven plugin generates a reasonable diagram.csv file using

mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml

You then see the output diagram.csv.

To generate the layout dependencies of classes in your project.  Use the snippets, and the Jupyter Notebook – https://github.com/prb112/examples/blob/master/code-graph/code-graph.ipynb and view at https://nbviewer.jupyter.org/github/prb112/examples/blob/master/code-graph/code-graph.ipynb

Simple Graph

Reference

Code Graph on Git https://github.com/prb112/examples/tree/master/code-graph

CSV Sample Data diagram.csv

POM

<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>example</groupId>
	<artifactId>generate-diagram</artifactId>
	<version>99-SNAPSHOT</version>
	<packaging>maven-plugin</packaging>

	<name>generate-diagram</name>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
		<version.roaster>2.21.0.Final</version.roaster>
	</properties>



	<build>
		<pluginManagement>
			<plugins>
				<plugin>
					<artifactId>maven-jar-plugin</artifactId>
					<version>2.6</version>
					<executions>
						<execution>
							<goals>
								<goal>test-jar</goal>
							</goals>
						</execution>
					</executions>
					<configuration>
						<archive>
							<manifest>
								<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
								<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
							</manifest>
						</archive>
					</configuration>
				</plugin>
				<plugin>
					<groupId>org.apache.maven.plugins</groupId>
					<artifactId>maven-plugin-plugin</artifactId>
					<version>3.6.0</version>
					<configuration>
						<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
					</configuration>
					<executions>
						<execution>
							<id>mojo-descriptor</id>
							<goals>
								<goal>descriptor</goal>
							</goals>
							<phase>process-classes</phase>
							<configuration>
								<skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
							</configuration>
						</execution>
					</executions>
				</plugin>
			</plugins>
		</pluginManagement>

		<plugins>
			<plugin>
				<!-- Embeds the dependencies in fhir-tools into the jar. -->
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-shade-plugin</artifactId>
				<version>3.2.1</version>
				<executions>
					<execution>
						<phase>package</phase>
						<goals>
							<goal>shade</goal>
						</goals>
						<configuration>
							<artifactSet>
								<excludes>
									<exclude>org.testng:testng</exclude>
									<exclude>org.apache.maven:lib:tests</exclude>
									<exclude>org.apache.maven</exclude>
								</excludes>
							</artifactSet>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>

	<dependencies>
		<dependency>
			<groupId>org.jboss.forge.roaster</groupId>
			<artifactId>roaster-api</artifactId>
			<version>${version.roaster}</version>
		</dependency>
		<dependency>
			<groupId>org.jboss.forge.roaster</groupId>
			<artifactId>roaster-jdt</artifactId>
			<version>${version.roaster}</version>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-plugin-api</artifactId>
			<version>3.6.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugin-tools</groupId>
			<artifactId>maven-plugin-annotations</artifactId>
			<version>3.6.0</version>
			<optional>true</optional>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-core</artifactId>
			<version>3.6.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-artifact</artifactId>
			<version>3.6.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-model</artifactId>
			<version>3.6.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.maven</groupId>
			<artifactId>maven-compat</artifactId>
			<version>3.6.1</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.maven.plugin-testing</groupId>
			<artifactId>maven-plugin-testing-harness</artifactId>
			<version>3.3.0</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
</project>
package demo;

import java.io.File;
import java.util.Properties;

import org.apache.maven.execution.MavenSession;
import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.plugin.MojoFailureException;
import org.apache.maven.plugins.annotations.Execute;
import org.apache.maven.plugins.annotations.LifecyclePhase;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.plugins.annotations.Parameter;
import org.apache.maven.plugins.annotations.ResolutionScope;
import org.apache.maven.project.MavenProject;

import com.ibm.watsonhealth.fhir.tools.plugin.diagram.DiagramFactory;
import com.ibm.watsonhealth.fhir.tools.plugin.diagram.impl.IDiagramGenerator;

/**
 * This class coordinates the calls to the Diagram generation plugin
 * 
 * The phase is initialize. To find a list of phases -
 * https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference
 * 
 * Run the following to setup the plugin: <code>
 * mvn clean package install -f generate-diagram/pom.xml
 * </code>
 * 
 * Run the following to setup the classes in fhir-model: <code> 
 * mvn example:generate-diagram:99-SNAPSHOT:generate-diagram -f ./myproj/pom.xml
 * </code>
 * 
 * @author PBastide
 * 
 * @requiresDependencyResolution runtime
 *
 */
@Mojo(name = "generate-diagram", //$NON-NLS-1$
        requiresProject = true, requiresDependencyResolution = ResolutionScope.RUNTIME_PLUS_SYSTEM, requiresDependencyCollection = ResolutionScope.RUNTIME_PLUS_SYSTEM, defaultPhase = LifecyclePhase.GENERATE_SOURCES, requiresOnline = false, threadSafe = false, aggregator = true)
@Execute(phase = LifecyclePhase.GENERATE_SOURCES)
public class DiagramPlugin extends AbstractMojo {

    @Parameter(defaultValue = "${project}", required = true, readonly = true) //$NON-NLS-1$
    protected MavenProject mavenProject;

    @Parameter(defaultValue = "${session}")
    private MavenSession session;

    @Parameter(defaultValue = "${project.basedir}", required = true, readonly = true) //$NON-NLS-1$
    private File baseDir;

    @Override
    public void execute() throws MojoExecutionException, MojoFailureException {
        if (baseDir == null || !baseDir.exists()) {
            throw new MojoFailureException("The Base Directory is not found.  Throwing failure. ");
        }

        // Grab the Properties (the correct way)
        // https://maven.apache.org/plugin-developers/common-bugs.html#Using_System_Properties
        Properties userProps = session.getUserProperties();
        String useTestsDirectoryStr = userProps.getProperty("useTestsDirectory", "false");

        // Converts Limit value to boolean value.
        boolean useTestsDirectory = Boolean.parseBoolean(useTestsDirectoryStr);

        // Grab the right generator and set it up.
        IDiagramGenerator generator = DiagramFactory.getDiagramGenerator();

        // Set the use of tests directory
        generator.use(useTestsDirectory);

        // Get the base directory .
        generator.setTargetProjectBaseDirectory(baseDir.getAbsolutePath() + "/target");

        // Passes the Log to the implementation code.
        generator.setLog(getLog());

        // Add the project
        generator.add(mavenProject);

        // Builds the Diagram
        generator.generateDiagram();
    }
}
package example.impl;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.StringJoiner;
import java.util.stream.Collectors;

import org.apache.maven.plugin.logging.Log;
import org.apache.maven.project.MavenProject;
import org.jboss.forge.roaster.Roaster;
import org.jboss.forge.roaster.model.JavaType;
import org.jboss.forge.roaster.model.source.Import;
import org.jboss.forge.roaster.model.source.JavaAnnotationSource;
import org.jboss.forge.roaster.model.source.JavaClassSource;
import org.jboss.forge.roaster.model.source.JavaEnumSource;
import org.jboss.forge.roaster.model.source.JavaInterfaceSource;
import org.jboss.forge.roaster.model.source.JavaSource;
import org.w3c.dom.DOMImplementation;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

public class DiagramImpl implements IDiagramGenerator {

    private String absolutePath = null;
    private Log log = null;
    private Boolean useTestFiles = false;

    private List<MavenProject> projects = new ArrayList<>();

    private List<String> sourceDirectories = new ArrayList<>();

    private List<String> sourceFiles = new ArrayList<>();

    private List<String> countWithNested = new ArrayList<>();

    private Map<String, List<String>> sourceFileImports = new HashMap<>();

    @Override
    public void add(MavenProject mavenProject) {
        if (mavenProject == null) {
            throw new IllegalArgumentException("no access to the maven project's plugin object");
        }

        log.info("Projects added...");
        projects = mavenProject.getCollectedProjects();

    }

    @Override
    public void use(Boolean useTestFiles) {
        this.useTestFiles = useTestFiles;

    }

    @Override
    public void setLog(Log log) {
        this.log = log;
    }

    @Override
    public void generateDiagram() {
        if (absolutePath == null) {
            throw new IllegalArgumentException("Bad Path " + absolutePath);
        }

        if (log == null) {
            throw new IllegalArgumentException("Unexpected no log passed in");
        }

        for (MavenProject project : projects) {

            List<String> locations = project.getCompileSourceRoots();
            for (String location : locations) {
                log.info("Location of Directory -> " + location);

            }
            sourceDirectories.addAll(locations);

            if (useTestFiles) {
                log.info("Adding the Test Files");
                sourceDirectories.addAll(project.getTestCompileSourceRoots());
            }

        }

        // Find the Files in each directory.
        // Don't Follow links.
        for (String directory : sourceDirectories) {
            findSourceFiles(directory);
        }

        processCardinalityMap();

        printOutCardinality();

        log.info("Total Number of Java Files in the Project are: " + sourceFiles.size());
        log.info("Total Number of Classes/Interfaces/Enums in the Project are: " + countWithNested.size());

        generateImageFile();

    }

    private void generateImageFile() {
        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        try(FileOutputStream fos = new FileOutputStream("diagram.csv");) {
            
            for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
                
                StringJoiner joiner = new StringJoiner(",");
                for(String val : sourceFileImports.get(key)) {
                    joiner.add(val);
                }
                
                String line = key + ",\"" + joiner.toString() + "\"";
                fos.write(line.getBytes());
                fos.write("\n".getBytes());
            }

        } catch (Exception e) {
            log.warn("Issue processing", e);
        }

    }

    public void printOutCardinality() {

        Comparator<String> byName = (name1, name2) -> name1.compareTo(name2);

        //
        log.info("Cardinality count - imports into other classes");
        for (String key : sourceFileImports.keySet().stream().sorted(byName).collect(Collectors.toList())) {
            log.info(key + " -> " + sourceFileImports.get(key).size());
        }

    }

    public void processCardinalityMap() {
        // Import > List<Classes>
        // Stored in -> sourceFileImports

        for (String source : sourceFiles) {
            File srcFile = new File(source);
            try {
                JavaType<?> jtFile = Roaster.parse(srcFile);

                String parentJavaClass = jtFile.getQualifiedName();

                if (jtFile instanceof JavaClassSource) {
                    countWithNested.add(parentJavaClass);
                    log.info("[C] -> " + parentJavaClass);
                    JavaClassSource jcs = (JavaClassSource) jtFile;

                    helperImports(parentJavaClass, jcs.getImports());

                    for (JavaSource<?> child : jcs.getNestedTypes()) {

                        String childLoc = child.getQualifiedName();
                        countWithNested.add(childLoc);
                        log.info("  [CC] -> " + childLoc);
                    }
                }

                else if (jtFile instanceof JavaEnumSource) {
                    log.info("[E] -> " + parentJavaClass);
                    countWithNested.add(parentJavaClass);
                    JavaEnumSource jes = (JavaEnumSource) jtFile;

                    helperImports(parentJavaClass, jes.getImports());

                    for (Object child : jes.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
                        countWithNested.add(childLoc);
                        log.info("  [EC] -> " + childLoc);
                    }

                } else if (jtFile instanceof JavaInterfaceSource) {
                    countWithNested.add(parentJavaClass);

                    log.info("[I] -> " + parentJavaClass);
                    JavaInterfaceSource jis = (JavaInterfaceSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());

                    for (Object child : jis.getNestedTypes()) {

                        String childLoc = child.getClass().getName();
                        countWithNested.add(childLoc);
                        log.info("  [IC] -> " + childLoc);
                    }
                } else if (jtFile instanceof JavaAnnotationSource) {
                    countWithNested.add(parentJavaClass);

                    log.info("[A] -> " + parentJavaClass);
                    JavaAnnotationSource jis = (JavaAnnotationSource) jtFile;

                    helperImports(parentJavaClass, jis.getImports());
                }

                else {
                    log.info("[O] -> " + parentJavaClass);
                }

            } catch (IOException e) {
                log.info("unable to parse file " + srcFile);
            }

        }
        log.info("Parsed the Cardinality Map:");

    }

    private void helperImports(String parentJavaClass, List<Import> imports) {
        // sourceFileImports
        List<String> importOut = sourceFileImports.get(parentJavaClass);
        if (importOut == null) {
            sourceFileImports.put(parentJavaClass, new ArrayList<String>());
        }

        for (Import importX : imports) {
            String importXStr = importX.getQualifiedName();
            importOut = sourceFileImports.get(importXStr);
            if (importOut == null) {
                importOut = new ArrayList<>();
                sourceFileImports.put(importXStr, importOut);
            }

            importOut.add(parentJavaClass);

        }

    }

    public static void main(String... args) {
        SvgDiagramImpl impl = new SvgDiagramImpl();
        String proc = "Test.java";
        Log log = new Log() {

            @Override
            public boolean isDebugEnabled() {
                return false;
            }

            @Override
            public void debug(CharSequence content) {

            }

            @Override
            public void debug(CharSequence content, Throwable error) {

            }

            @Override
            public void debug(Throwable error) {

            }

            @Override
            public boolean isInfoEnabled() {
                return false;
            }

            @Override
            public void info(CharSequence content) {

            }

            @Override
            public void info(CharSequence content, Throwable error) {

            }

            @Override
            public void info(Throwable error) {

            }

            @Override
            public boolean isWarnEnabled() {
                return false;
            }

            @Override
            public void warn(CharSequence content) {

            }

            @Override
            public void warn(CharSequence content, Throwable error) {

            }

            @Override
            public void warn(Throwable error) {

            }

            @Override
            public boolean isErrorEnabled() {
                return false;
            }

            @Override
            public void error(CharSequence content) {

            }

            @Override
            public void error(CharSequence content, Throwable error) {

            }

            @Override
            public void error(Throwable error) {

            }
        };
        impl.setLog(log);
        impl.addSourceFile(proc);
        impl.processCardinalityMap();
    }

    private void addSourceFile(String proc) {
        sourceFiles.add(proc);

    }

    public void findSourceFiles(String directory) {
        File dir = new File(directory);
        if (dir.exists()) {

            File[] listFiles = dir.listFiles((file, name) -> {
                return name.endsWith(".java");
            });

            // Add to source directory
            if (listFiles != null) {
                for (File file : listFiles) {
                    sourceFiles.add(file.getAbsolutePath());
                    log.info(" File Added to Processing: " + file.getAbsolutePath());
                }
            }

            File[] listFilesFolders = dir.listFiles((file, name) -> {
                return file.isDirectory();
            });

            if (listFilesFolders != null) {
                for (File tmpDir : listFilesFolders) {
                    findSourceFiles(tmpDir.getAbsolutePath());
                }
            }

        } else {
            log.warn("Directory does not exist " + directory);
        }
    }

    @Override
    public void setTargetProjectBaseDirectory(String absolutePath) {
        this.absolutePath = absolutePath;

    }

}
Categories
app dev raspberry pi

Raspberry Pi: Setting up backup

I have a Raspberry Pi providing household automation and productivity services – WebDav, Backups and Calendar. I always worry about a jolt of power, a failed byte and something that is unrecoverable. Time for a Backup solution.

I plugged in a USB stick – 64GB, and immediately checked the file system is there and visible as SDA (unmounted).

pi@raspberrypi:~# sudo su - 

root@raspberrypi:~# lsblk 
 NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
 sda           8:0    1 58.2G  1 disk 
 └─sda1        8:1    1 58.2G  1 part 
 mmcblk0     179:0    0 14.9G  0 disk 
 ├─mmcblk0p1 179:1    0 43.9M  0 part /boot
 └─mmcblk0p2 179:2    0 14.8G  0 part /

I check to see which one is assigned to the SD card slot (mmc), I really don’t want to reformat Raspbian.  I see the USB stick is on /dev/sda. All of my subsequent commands use /dev/sda as part of the command.

root@raspberrypi:~# parted -ls
Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.
 Model: JetFlash Transcend 64GB (scsi)
 Disk /dev/sda: 62.5GB
 Sector size (logical/physical): 512B/512B
 Partition Table: unknown
 Disk Flags: 

 Model: SD SC16G (sd/mmc)
 Disk /dev/mmcblk0: 15.9GB
 Sector size (logical/physical): 512B/512B
 Partition Table: msdos
 Disk Flags: 
 Number  Start   End     Size    Type     File system  Flags
  1      4194kB  50.2MB  46.0MB  primary  fat32        lba
  2      50.3MB  15.9GB  15.9GB  primary  ext4

If you don’t see the relevant information on your HD or run into issues formatting the hard-drive, install hdparm.  and check with hdparm -r0 /dev/sda  .

TIP: I did run into an issue with an ISO written to a USB drive which locks the partition table and makes it unwriteable.

root@raspberrypi:~# apt-get install hdparm
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 The following package was automatically installed and is no longer required:
   realpath
 Use 'apt autoremove' to remove it.
 The following additional packages will be installed:
   powermgmt-base
 Suggested packages:
   apmd
 The following NEW packages will be installed:
   hdparm powermgmt-base
 0 upgraded, 2 newly installed, 0 to remove and 148 not upgraded.
 Need to get 114 kB of archives.
 After this operation, 278 kB of additional disk space will be used.
 Do you want to continue? [Y/n] y
 Get:1 http://raspbian.mirror.constant.com/raspbian stretch/main armhf hdparm armhf 9.51+ds-1+deb9u1 [105 kB]
 Get:2 http://raspbian-us.ngc292.space/raspbian stretch/main armhf powermgmt-base all 1.31+nmu1 [9,240 B]
 Fetched 114 kB in 0s (120 kB/s)           
 Selecting previously unselected package hdparm.
 (Reading database … 135688 files and directories currently installed.)
 Preparing to unpack …/hdparm_9.51+ds-1+deb9u1_armhf.deb …
 Unpacking hdparm (9.51+ds-1+deb9u1) …
 Selecting previously unselected package powermgmt-base.
 Preparing to unpack …/powermgmt-base_1.31+nmu1_all.deb …
 Unpacking powermgmt-base (1.31+nmu1) …
 Setting up powermgmt-base (1.31+nmu1) …
 Setting up hdparm (9.51+ds-1+deb9u1) …
 Processing triggers for man-db (2.7.6.1-2) …
Preparing to unpack …/hdparm_9.51+ds-1+deb9u1_armhf.deb …

Unpacking hdparm (9.51+ds-1+deb9u1) …

Selecting previously unselected package powermgmt-base.

Preparing to unpack …/powermgmt-base_1.31+nmu1_all.deb …

Unpacking powermgmt-base (1.31+nmu1) …

Setting up powermgmt-base (1.31+nmu1) …

Setting up hdparm (9.51+ds-1+deb9u1) …

Processing triggers for man-db (2.7.6.1-2) …
root@raspberrypi:~# hdparm -r0 /dev/sda
 /dev/sda:
  setting readonly to 0 (off)
  readonly      =  0 (off)

Now that I know the drive is writeable, I need to create the partition. I used

cfdisk

Navigate through the menu and select the maximum size
                                                                                        Disk: /dev/sda
Size: 58.2 GiB, 62495129600 bytes, 122060800 sectors
Label: dos, identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
>> /dev/sda1 2048 122060799 122058752 58.2G 83 Linux

Once you see “Syncing disks.”, you can format the disk. I formatted the partition sda1 with ext4 (I may want to encrypt in the future). Unmount and then format.

root@raspberrypi:~# umount /dev/sda1
root@raspberrypi:~# mkfs.ext4 /dev/sda1
mke2fs 1.43.4 (31-Jan-2017)
Found a dos partition table in /dev/sda1
Proceed anyway? (y,N) y
Creating filesystem with 2828032 4k blocks and 707136 inodes
Filesystem UUID: 363f1b4a-b0f5-4c7b-bf91-66f3823032d6
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

root@raspberrypi:~#

Make the backup directory, edit fstab and mount the directory.  Insert into fstab with your uuid “UUID=363f1b4a-b0f5-4c7b-bf91-66f3823032d6 /backups auto nosuid,nodev,nofail 0 0
”   The second to last avoids backup and the last one enables fsck on reboot.

root@raspberrypi:~# blkid 
/dev/mmcblk0p1: LABEL="boot" UUID="DDAB-3A15" TYPE="vfat" PARTUUID="b53687e8-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="5fa1ec37-3719-4b25-be14-1f7d29135a13" TYPE="ext4" PARTUUID="b53687e8-02"
/dev/mmcblk0: PTUUID="b53687e8" PTTYPE="dos"
/dev/sdb: UUID="363f1b4a-b0f5-4c7b-bf91-66f3823032d6" TYPE="ext4"
root@raspberrypi:~# mkdir /backups 
root@raspberrypi:~# vim /etc/fstab
root@raspberrypi:~# mount -a
root@raspberrypi:~# mount
UUID=363f1b4a-b0f5-4c7b-bf91-66f3823032d6 /backups auto nosuid,nodev,nofail 0 0


You should see backups listed. (Note: I bricked my Raspberry Pi with a bad FSTAB entry, and mounted it on my Mac using Paragon and removed the bad fstab entry. )

Update Crontab with daily backups.

crontab -e

Setup an editor for crontab.

root@raspberrypi:~# crontab -e
no crontab for root - using an empty one

Select an editor. To change later, run 'select-editor'.
1. /bin/ed
2. /bin/nano <---- easiest
3. /usr/bin/vim.basic
4. /usr/bin/vim.tiny

Choose 1-4 [2]: 3
crontab: installing new crontab

I added this line and copied it to /etc/cron.daily/

0 1 * * * /usr/bin/rsync -r /data/ /backups/`date +%w-%A`

crontab -l > pi-backup
mv /root/pi-backup /etc/cron.daily
run-parts /etc/cron.daily

Note, I had to add #!/bin/bash after I copied and remove the timing of the job.

Also, check to see if rsync is installed with which rsync and apt-get install rsync.

This enables backups on a daily basis rotating every 7 days.

Check back on the following day to see your backups

root@raspberrypi:~# /usr/bin/rsync -r /data/ /backups/`date +%w-%A`
root@raspberrypi:~# find /backups
/backups
/backups/lost+found
/backups/0-Sunday
/backups/0-Sunday/startup.sh

Good luck, I hope this helps you with your Raspberry Pi.

References

Categories
raspberry pi

Cups and Raspberry Pi – How to setup your print server

Print Server is one of those things that makes life easier.  I hooked up my Raspberry Pi and a Docker image in order to have a CUPS server and the Bonjour configuration.

Login to the Raspberry Pi and switch to root (sudo -s or sudo su –)

Check lsusb to see if you can see the USB device.

 root@raspberrypi:~# lsusb 
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@raspberryp:~#

If you don’t see it, check to see that the device is properly plugged in.

Check again, and you’ll see your manufacture for your printer listed. (highlighted in red)

 root@raspberrypi:~# lsusb 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 8564:1000 Transcend Information, Inc. JetFlash
Bus 001 Device 006: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@raspberrypi:~#

Further, I am able to see the exact model that is connected:

 root@raspberrypi:~# lsusb -v | grep -A 20 Hewlett-Packard 
Bus 001 Device 007: ID 03f0:7411 Hewlett-Packard
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x03f0 Hewlett-Packard
  idProduct          0x7411
  bcdDevice            1.00
  iManufacturer           1 HP
  iProduct                2 Photosmart C4600 series
  iSerial                 3 |||MASKED|||MASKED|||MASKED|||

I’ll use these details downstream in order to install and use the HP driver.

If you still don’t see the USB device, you should check dmesg and look at the pattern for USB.

 [3077006.701281] usb 1-1.2: new high-speed USB device number 7 using dwc_otg
[3077006.831891] usb 1-1.2: New USB device found, idVendor=03f0, idProduct=7411
[3077006.831908] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[3077006.831917] usb 1-1.2: Product: Photosmart C4600 series
[3077006.831925] usb 1-1.2: Manufacturer: HP

If you don’t see it, unplug and plug the device back in.

You can also use usb-devices

 T:  Bus=01 Lev=02 Prnt=02 Port=01 Cnt=02 Dev#=  7 Spd=480 MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=03f0 ProdID=7411 Rev=01.00
S:  Manufacturer=HP
S:  Product=Photosmart C4600 series
S:  SerialNumber=<<>><<>>
C:  #Ifs= 4 Cfg#= 1 Atr=c0 MxPwr=2mA
I:  If#= 0 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=cc Prot=00 Driver=(none)
I:  If#= 1 Alt= 0 #EPs= 2 Cls=07(print) Sub=01 Prot=02 Driver=usblp
I:  If#= 2 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)
I:  If#= 3 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=usb-storage

Copy the Dockerfile and the cupsd file locally (in the folder etc-cups/cupsd.conf). Cupsd is configured to listen on all interfaces: Listen 0.0.0.0:631

Build the Docker image (based off of this image https://github.com/a0js/rpi-cups and uses a different base image https://hub.docker.com/r/arm32v7/debian/ ) docker build -t cups-pi/cups-pi .

 root@raspberrypi:/data/cups-printer# docker build -t cups-pi/cups-pi . 
Sending build context to Docker daemon  8.192kB
Step 1/9 : FROM arm32v7/debian:latest
 ---> 64b4748d266b
Step 2/9 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 38ebdf2f5bb6
Step 3/9 : RUN apt-get update && apt-get install -y   sudo   locales   whois   cups   cups-client   cups-bsd   printer-driver-all   hpijs-ppds   hp-ppd   hplip
 ---> Using cache
 ---> 0b5b9eac6cef
Step 4/9 : RUN sed -i "s/^#\ \+\(en_US.UTF-8\)/\1/" /etc/locale.gen && locale-gen en_US en_US.UTF-8
 ---> Using cache
 ---> fd55737901b8
Step 5/9 : ENV LANG=en_US.UTF-8   LC_ALL=en_US.UTF-8   LANGUAGE=en_US:en
 ---> Using cache
 ---> d6cd104b40ec
Step 6/9 : RUN useradd   --groups=sudo,lp,lpadmin   --create-home   --home-dir=/home/print   --shell=/bin/bash   --password=$(mkpasswd print)   print   && sed -i '/%sudo[[:space:]]/ s/ALL[[:space:]]*$/NOPASSWD:ALL/' /etc/sudoers   && apt-get clean   && rm -rf /var/lib/apt/lists/*   && mkdir /var/lib/apt/lists/partial
 ---> Using cache
 ---> 156dcd02e397
Step 7/9 : COPY etc-cups/cupsd.conf /etc/cups/cupsd.conf
 ---> e1624a96970e
Step 8/9 : EXPOSE 631
 ---> Running in a16705e0f3d9
Removing intermediate container a16705e0f3d9
 ---> 60758af63011
Step 9/9 : ENTRYPOINT ["/usr/sbin/cupsd", "-f"]
 ---> Running in add02d421ea9
Removing intermediate container add02d421ea9
 ---> 8c6fe42423c8
Successfully built 8c6fe42423c8
Successfully tagged cups-pi/cups-pi:ltest

Run the Docker image to start the print server

root@raspberrypi:/data/cups-printer# docker run -d -p 631:631 --privileged -v /var/run/dbus:/var/run/dbus -v /dev/bus/usb:/dev/bus/usb -h myhostm.mydomain.org --name cupsm cups-pi/cups-pi:latest
fe6d9bc34c66911f05b011ef185fce95947efb965e90ef2b4ecdd0f1c3a32d68

It’s important to use the -h if you want to remotely access via hostname.

Login to the console http://myserver.mydomain.org:631/admin using print:print

Click Add Printer

Select a local printer “HP Photosmart C4600 series (HP Photosmart C4600 series)”

Click Continue

Click Share This Printer

Click Continue

Click Add Printer

Select Media Type – Letter

Click Set Default Options

Click Add

When you find the Printer, you can add the printer

You now have an image ready for use with a CUPS printer.  (If you rebuild, you may have to re-setup the default printer…. Just warning you ahead of time).  You can always look at http://myhost.mydomain.org:631/printers/HP_Photosmart_C4600_series

If you need to connect to the console, you can look at the docker ps

root@raspberrypi:/data/cups-printer# docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                                    NAMES
fe6d9bc34c66        cups-pi/cups-pi:latest   "/usr/sbin/cupsd -f"   5 minutes ago       Up 5 minutes        0.0.0.0:631->631/tcp                     cups

Extract the CONTAINER ID, and look at the /bin/sh

root@raspberrypi:/data/cups-printer# docker exec
-i -t fe6d9bc34c66 /bin/sh

Then look at the /var/log/cups files, specifically the error_log

Reference

List USB Devices Linux https://linuxhint.com/list-usb-devices-linux/

Dockerfile https://github.com/a0js/rpi-cups https://github.com/ryansch/docker-cups-rpi https://github.com/aadl/docker-cups/blob/master/2.2.1/Dockerfile

Arm 7 https://hub.docker.com/r/arm32v7/debian/

Cupsd.conf https://raw.githubusercontent.com/a0js/rpi-cups/master/etc-cups/cupsd.conf https://wiki.archlinux.org/index.php/CUPS/Printer_sharing

SSL / Advanced Configuration http://chschneider.eu/linux/server/cups.shtml http://192.168.86.176:631/admin (use IP if receiving a bad request indicator to the admin interface)

Advanced Ubuntu Configuration https://help.ubuntu.com/lts/serverguide/cups.html.en

Categories
app dev jupyter

Jupyter Notebook: Email Analysis to a Lotus Notes View

I wanted to do an analysis of my emails since I joined IBM, and see the flow of messages in-and-out of my inbox.

With my preferences for Jupyter Notebooks, I built a small notebook for analysis.

Steps
Open IBM Lotus Notes Rich Client

Open the Notes Database with the View you want to analyze.

Select the View you are interested in ‘All Documents’. For instance the All Documents view, like my inbox *obfuscated* with a purpose.

Click File > Export

Enter a file name – email.csv

Select Format “Comma Separate Value”

Click Export

Upload the Notebook to your Jupyter server

The notebook is describes the flow through my process. If you encounter ValueError: (‘Unknown string format:’, ’12/10/2018 08:34 AM’), you can refer to https://stackoverflow.com/a/8562577/1873438

iconv -c -f utf-8 -t ascii email.csv > email.csv.clean

You can break the data into month-year-day analysis with the following, and peek the results with df_emailA.head()

When you run the final cell, the code generates a Year-Month-Day count as a bar graph.

    # Title: Volume in Months when emails are sent.
    # Plots volume based on known year-mm-dd
    # to be included in the list, one must have data in those years.
    # Kind is a bar graph, so that the (Y - YYYY,MM can be read)
    y_m_df = df_emailA.groupby(['year','month','day']).year.count()
    y_m_df.plot(kind="bar")

    plt.title('Numbers submitted By YYYY-MM-DD')
    plt.xlabel('Email Flow')
    plt.ylabel('Year-Month-Day')
    plt.autoscale(enable=True, axis='both', tight=False)
    plt.rcParams['figure.figsize'] = [20, 200]

You’ll see the trend of emails I receive over the years.

Trends of Email
Categories
patent

Co-Inventing Model with Gephi

I was curious who my frequent co-inventors were, so I downloaded Gephi. I downloaded my patent data from Google Patents and built a fun little model from CSV data (similar to the attached).

SOURCE WEIGHT TARGET
Matthew E. Broomhall 121 PAUL Bastide
Robert E. Loredo 119 PAUL Bastide
Fang Lu 60 PAUL Bastide
Alaa Abou Mahmoud 43 PAUL Bastide
Lisa Seacat Deluca 23 PAUL Bastide
Lydia M. Do 17 PAUL Bastide
Dale M. Schultz 13 PAUL Bastide
Andrew E. Davis 7 PAUL Bastide
Ralph E. LeBlanc 7 PAUL Bastide
Sean Callanan 7 PAUL Bastide
Donna K. Byron 5 PAUL Bastide
Sandra L. Kogan 5 PAUL Bastide
Asima Silva 4 PAUL Bastide
Aaron J. Quirk 3 PAUL Bastide
Aaron M. Cohen 3 PAUL Bastide
Daniel B. Harris 3 PAUL Bastide
Eric S. Portner 3 PAUL Bastide
John M. Boyer 3 PAUL Bastide
Michael L. Taylor 3 PAUL Bastide
Alexander Pikovsky 2 PAUL Bastide
Corville O. Allen 2 PAUL Bastide
Dana L. Price 2 PAUL Bastide
Eric M. Wilcox 2 PAUL Bastide
Jeffrey R. Hoy 2 PAUL Bastide
John A. Jacobson 2 PAUL Bastide
Kulvir S. Bhogal 2 PAUL Bastide
Liam Harpur 2 PAUL Bastide
Marco A. Vicente 2 PAUL Bastide
Patrick J. O’Sullivan 2 PAUL Bastide
Scott J. Martin 2 PAUL Bastide
Shane M. Kilmon 2 PAUL Bastide
Stephen Crawford 2 PAUL Bastide
Thomas J. Evans IV 2 PAUL Bastide
Vijay Francis 2 PAUL Bastide
Weisong Wang 2 PAUL Bastide
Adam L. Cutler 1 PAUL Bastide
Amanda N. Savitzky 1 PAUL Bastide
Andrew L. Schirmer 1 PAUL Bastide
Arun Vishwanath 1 PAUL Bastide
Bernadette A. Carter 1 PAUL Bastide
Beth Anne M. Collopy 1 PAUL Bastide
Beth L. Hoffman 1 PAUL Bastide
Bradley W. Hurley 1 PAUL Bastide
Brenton P. Chasse 1 PAUL Bastide
Brian M. Walsh 1 PAUL Bastide
Carl J. Kraenzel 1 PAUL Bastide
Christopher W. Desforges 1 PAUL Bastide
Damian E.A. Garcia 1 PAUL Bastide
Dan DUMONT 1 PAUL Bastide
Dwarikanath Mahapatra 1 PAUL Bastide
Fred Raguillat 1 PAUL Bastide
Isabell Kiral-Kornek 1 PAUL Bastide
Jaime M. Stockton 1 PAUL Bastide
James A. Hart 1 PAUL Bastide
Jennifer L. Vargus 1 PAUL Bastide
Jodi RAJANIEMI 1 PAUL Bastide
Jose L. Lopez 1 PAUL Bastide
Juliana M. Leong 1 PAUL Bastide
Katherine M. Parsons 1 PAUL Bastide
Kelley L. ANDERS 1 PAUL Bastide
King Shing K. Lui 1 PAUL Bastide
Leah A. Lawrence 1 PAUL Bastide
Leho Nigul 1 PAUL Bastide
Lei Wang 1 PAUL Bastide
Lorelei M. McCollum 1 PAUL Bastide
Margo L. Ezekiel 1 PAUL Bastide
Mark Gargan 1 PAUL Bastide
Mary E. Miller 1 PAUL Bastide
Matthew Stephen Rosno 1 PAUL Bastide
Melissa A. Lord 1 PAUL Bastide
Michael G. Alexander 1 PAUL Bastide
Na Pei 1 PAUL Bastide
Neal Fishman 1 PAUL Bastide
Pei Sun 1 PAUL Bastide
Richard Gorzela 1 PAUL Bastide
Richard T. Bassemir 1 PAUL Bastide
Shelbee D. Smith-Eigenbrode 1 PAUL Bastide
Shu Qiang Li 1 PAUL Bastide
Shunguo Yan 1 PAUL Bastide
Stacy M. Cannon 1 PAUL Bastide
Stanley K. Jerrard-Dunne 1 PAUL Bastide
Stefan von Cavallar 1 PAUL Bastide
Susmita Saha 1 PAUL Bastide
Tamer E. Abuelsaad 1 PAUL Bastide
Thomas J. Evans 1 PAUL Bastide
Trudy L. Hewitt 1 PAUL Bastide
Xujin Liu 1 PAUL Bastide
Ying Mo 1 PAUL Bastide
PAUL Bastide 0 PAUL Bastide

Graph of Co-Inventors

 

 

Categories
troubleshooting WHC Development

CURL and LDAPS – How to Search and Debug

I hit an issue where I needed to Search LDAP from a machine I didn’t have access to install new RPMs on. I found this cool article on CURL and LDAP Search. I had to make some minor modifications to get it to work with a secure connection (–insecure ldaps:// and 636). I also added -v to diagnosis some connection problems.

curl "ldaps://127.0.0.1:636/DC=IBM.COM?cn,objectClass?sub?(objectClass=)" -u "cn=user1,ou=test_org3,o=dr,DC=IBM.COM" --insecure -v
Enter host password for user 'cn=user1,ou=test_org3,o=dr,DC=IBM.COM':
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 636 (#0)
* LDAP local: LDAP Vendor = OpenLDAP ; LDAP Version = 20428
* LDAP local: ldaps://127.0.0.1:636/DC=IBM.COM?cn,objectClass?sub?(objectClass=
)
* LDAP local: trying to establish encrypted connection
DN: dc=ibm.com
objectClass: domain
objectClass: top

DN: o=dr,dc=ibm.com
objectClass: organization
objectClass: top

DN: ou=test_org3,o=dr,dc=ibm.com
objectClass: organizationalunit
objectClass: top

You can then find the userids you need quickly. I left them off the output intentionally.

If you see connected, but no results, I suggest changing to the top level of the ldap, and using this string – ldaps://127.0.0.1:636/DC=IBM.COM?cn,objectClass?sub?(objectClass=*)

Categories
raspberry pi

Raspberry Pi – Part IV – Simple Wiring Test

I am plugging away working with my Pi. I went back to the Ada Fruit Site, and started working on one of the tutorials from the part I purchased – https://www.adafruit.com/products/2125   I want to be able to demonstrate a simple wiring with my Pi works.

I selected Email Notifier. I read through it, and realized I needed to convert it to work for the B+ model. The pi4j website has a nice diagram for the pins http://pi4j.com/images/j8header-b-plus-large.png and I simplified the test a bit.

I launched the simple python file

pi@seconds ~ $ sudo python testPi.py

Pi with a Green Light - Using Python
Pi with a Green Light – Using Python

I added the pi4j dependency to my maven pom

<dependency>
    <groupId>com.pi4j</groupId>
    <artifactId>pi4j-core</artifactId>
   <version>1.0</version>
</dependency>

I create a servlet which replicated the same code from before (GPIO).

I copied the war file to the local webapp directory, and restarted jetty

pi@seconds /opt/jetty/web/bbq/webapps $ sudo cp ~/pi.webapp.war ./
pi@seconds /opt/jetty/web/bbq/webapps $ sudo chown jetty:jetty pi.webapp.war
pi@seconds /opt/jetty/web/bbq/webapps $ sudo /etc/init.d/jetty restart

Refer to https://wiki.eclipse.org/Jetty/Howto/Deploy_Web_Applications  and I did have to add –module=jsp to the start.ini for my web configuration.

I hit the servlet – http://192.168.1.200/pi.webapp/Control?status=true

Exception
Exception

Which makes sense, since “Software using the Pi4J library must be run with ROOT level permissions.” (actually it doesn’t since I am running as root)…

Issues remain.  I’ll tackle them on a different day 🙂   … Part 5 …

Categories
raspberry pi

Raspberry Pi – Part III – Setting up Jetty

The Raspberry Pi is a device that has so many options open: You can extend the hardware. You can extend the software. I choose to tackle extending the software to start, so I can get the experience that I want setup, and running on the device.  The experience I am after is the web interface to control my Raspberry Pi.

For a Java developer, I conclude that a lightweight interface hosted on a Jetty server is probably easiest.  The Jetty server is a 12.9M download and 30M expanded. It’s also used frequently in devices, and why reinvent the wheel, use the approach that Industry is using.  Jetty is designed for a small memory footprint.   (I did consider using Tomcat Embedded, and came to the conclusion I’d probably want to many features.)

I downloaded the Jetty archive to my local user’s directory. Note, since I want to just use jetty.zip I’m renaming the output file with wget -O.

wget -O jetty.zip http://download.eclipse.org/jetty/stable-9/dist/jetty-distribution-9.2.10.v20150310.zip

I make a directory for the application server, and move over jetty.zip to that directory.

sudo mkdir -p /opt/jetty
sudo mv jetty.zip /opt/jetty

Next I launched into the sudoers root shell, and extract the Jetty archive.  I decided to move it into the runtime folder, so I can have an easy place to backup from.  (Remove the extra space from the .zip)

sudo -s 
cd /opt/jetty
unzip jetty.zip
mv jetty-distribution-9.2.10.v20150310/ runtime
rm jetty.zip

Next, I wanted to setup Jetty as a service, and check that Jetty starts

cp runtime/bin/jetty.sh /etc/init.d/jetty
echo JETTY_HOME=`pwd`/runtime > /etc/default/jetty
service jetty start
Starting Jetty: . . . OK Sun Apr 12 15:38:40 UTC 2015
service jetty stop
Stopping Jetty: OK

Next, I want to automatically start Jetty with the right runlevels

update-rc.d jetty defaults

I want to configure Jetty to run with a set user jetty.

mkdir -p /opt/jetty/web/bbq
mkdir -p /opt/jetty/temp
useradd --user-group --shell /bin/false --home-dir /opt/jetty/temp jetty

There is a base configuration that needs to be setup the base site.

cd /opt/jetty/web/bbq
java -jar /opt/jetty/runtime/start.jar --add-to-start=deploy,http,logging

Next, edit the default port value.

vi start.ini

## HTTP port to listen on
jetty.port=80

Finally, I wrap the configuration of the jetty service and app.

chown -R jetty:jetty /opt/jetty
echo "JETTY_HOME=/opt/jetty/runtime" > /etc/default/jetty
echo "JETTY_BASE=/opt/jetty/web/bbq" >> /etc/default/jetty
echo "TMPDIR=/opt/jetty/temp" >> /etc/default/jetty

A good check is to look at service jetty status, and confirm the settings, and then restart your Raspberry Pi. I did find that the startup time was significantly effected by the additional service.  ( +25 seconds from the original 15)

Navigate to http://192.168.1.200/ (or whatever IP you have used) and confirm the page loads Jetty.  If you see a 404, you’re off and ready for the next phase. (as am I) 🙂

Jetty Starting Point
Jetty Starting Point