Blog

  • Jupyter Notebook: Email Analysis to a Lotus Notes View

    I wanted to do an analysis of my emails since I joined IBM, and see the flow of messages in-and-out of my inbox.

    With my preferences for Jupyter Notebooks, I built a small notebook for analysis.

    Steps
    Open IBM Lotus Notes Rich Client

    Open the Notes Database with the View you want to analyze.

    Select the View you are interested in ‘All Documents’. For instance the All Documents view, like my inbox *obfuscated* with a purpose.

    Click File > Export

    Enter a file name – email.csv

    Select Format “Comma Separate Value”

    Click Export

    Upload the Notebook to your Jupyter server

    The notebook is describes the flow through my process. If you encounter ValueError: (‘Unknown string format:’, ’12/10/2018 08:34 AM’), you can refer to https://stackoverflow.com/a/8562577/1873438

    iconv -c -f utf-8 -t ascii email.csv > email.csv.clean

    You can break the data into month-year-day analysis with the following, and peek the results with df_emailA.head()

    When you run the final cell, the code generates a Year-Month-Day count as a bar graph.

        # Title: Volume in Months when emails are sent.
        # Plots volume based on known year-mm-dd
        # to be included in the list, one must have data in those years.
        # Kind is a bar graph, so that the (Y - YYYY,MM can be read)
        y_m_df = df_emailA.groupby(['year','month','day']).year.count()
        y_m_df.plot(kind="bar")
    
        plt.title('Numbers submitted By YYYY-MM-DD')
        plt.xlabel('Email Flow')
        plt.ylabel('Year-Month-Day')
        plt.autoscale(enable=True, axis='both', tight=False)
        plt.rcParams['figure.figsize'] = [20, 200]

    You’ll see the trend of emails I receive over the years.

    Trends of Email
  • Application Processes Branching

    Urban Code Deploy (UCD) is a tool we use to manage the deployment of our healthcare platform. I needed to branch between two different processes, and the setup and steps to get branching done between Application processes was not clearly documented.

    I built some custom bash logic to switch based on results:

    #!/bin/bash
    if [ -f /etc/SHORT_CIRCUIT ]
    then
    echo "result=YES"
    else
    echo "result=NO"
    fi
    

    I created a Post Processing Step

    var exit = properties.get('exitCode');
    
    scanner.register("result=", function(lineNumber, line) {
    var value=line.replace("result=","");
    properties.put("ShortCircuit",value);
    });
    
    if (exit == 0) {
    properties.put('Status', 'Success');
    }
    else {
    properties.put('Status', 'Failure');
    }
    
    scanner.scan();
    

    the scanner matches the text output for a value that has result= in it. I highly suggest having this in your scanner, otherwise you might see ucd propertyValue=script content: which indicates that it did not find any result to set.

    Once you get the value in the step, it’ll be available downstream once you set Process Properties. In this case, I pickup the property from the <STEP NAME><SLASH><PROPERTY NAME> and set it into ShortCircuitStatus.

    I then use it to make a decision downstream in a switch statement. I do suggest not using a default path in the switch statement, it ensures there is a deterministic and expected outcome of the prior process. I used YES/NO in the switch statement.

    References

  • Co-Inventing Model with Gephi

    I was curious who my frequent co-inventors were, so I downloaded Gephi. I downloaded my patent data from Google Patents and built a fun little model from CSV data (similar to the attached).

    SOURCE WEIGHT TARGET
    Matthew E. Broomhall 121 PAUL Bastide
    Robert E. Loredo 119 PAUL Bastide
    Fang Lu 60 PAUL Bastide
    Alaa Abou Mahmoud 43 PAUL Bastide
    Lisa Seacat Deluca 23 PAUL Bastide
    Lydia M. Do 17 PAUL Bastide
    Dale M. Schultz 13 PAUL Bastide
    Andrew E. Davis 7 PAUL Bastide
    Ralph E. LeBlanc 7 PAUL Bastide
    Sean Callanan 7 PAUL Bastide
    Donna K. Byron 5 PAUL Bastide
    Sandra L. Kogan 5 PAUL Bastide
    Asima Silva 4 PAUL Bastide
    Aaron J. Quirk 3 PAUL Bastide
    Aaron M. Cohen 3 PAUL Bastide
    Daniel B. Harris 3 PAUL Bastide
    Eric S. Portner 3 PAUL Bastide
    John M. Boyer 3 PAUL Bastide
    Michael L. Taylor 3 PAUL Bastide
    Alexander Pikovsky 2 PAUL Bastide
    Corville O. Allen 2 PAUL Bastide
    Dana L. Price 2 PAUL Bastide
    Eric M. Wilcox 2 PAUL Bastide
    Jeffrey R. Hoy 2 PAUL Bastide
    John A. Jacobson 2 PAUL Bastide
    Kulvir S. Bhogal 2 PAUL Bastide
    Liam Harpur 2 PAUL Bastide
    Marco A. Vicente 2 PAUL Bastide
    Patrick J. O’Sullivan 2 PAUL Bastide
    Scott J. Martin 2 PAUL Bastide
    Shane M. Kilmon 2 PAUL Bastide
    Stephen Crawford 2 PAUL Bastide
    Thomas J. Evans IV 2 PAUL Bastide
    Vijay Francis 2 PAUL Bastide
    Weisong Wang 2 PAUL Bastide
    Adam L. Cutler 1 PAUL Bastide
    Amanda N. Savitzky 1 PAUL Bastide
    Andrew L. Schirmer 1 PAUL Bastide
    Arun Vishwanath 1 PAUL Bastide
    Bernadette A. Carter 1 PAUL Bastide
    Beth Anne M. Collopy 1 PAUL Bastide
    Beth L. Hoffman 1 PAUL Bastide
    Bradley W. Hurley 1 PAUL Bastide
    Brenton P. Chasse 1 PAUL Bastide
    Brian M. Walsh 1 PAUL Bastide
    Carl J. Kraenzel 1 PAUL Bastide
    Christopher W. Desforges 1 PAUL Bastide
    Damian E.A. Garcia 1 PAUL Bastide
    Dan DUMONT 1 PAUL Bastide
    Dwarikanath Mahapatra 1 PAUL Bastide
    Fred Raguillat 1 PAUL Bastide
    Isabell Kiral-Kornek 1 PAUL Bastide
    Jaime M. Stockton 1 PAUL Bastide
    James A. Hart 1 PAUL Bastide
    Jennifer L. Vargus 1 PAUL Bastide
    Jodi RAJANIEMI 1 PAUL Bastide
    Jose L. Lopez 1 PAUL Bastide
    Juliana M. Leong 1 PAUL Bastide
    Katherine M. Parsons 1 PAUL Bastide
    Kelley L. ANDERS 1 PAUL Bastide
    King Shing K. Lui 1 PAUL Bastide
    Leah A. Lawrence 1 PAUL Bastide
    Leho Nigul 1 PAUL Bastide
    Lei Wang 1 PAUL Bastide
    Lorelei M. McCollum 1 PAUL Bastide
    Margo L. Ezekiel 1 PAUL Bastide
    Mark Gargan 1 PAUL Bastide
    Mary E. Miller 1 PAUL Bastide
    Matthew Stephen Rosno 1 PAUL Bastide
    Melissa A. Lord 1 PAUL Bastide
    Michael G. Alexander 1 PAUL Bastide
    Na Pei 1 PAUL Bastide
    Neal Fishman 1 PAUL Bastide
    Pei Sun 1 PAUL Bastide
    Richard Gorzela 1 PAUL Bastide
    Richard T. Bassemir 1 PAUL Bastide
    Shelbee D. Smith-Eigenbrode 1 PAUL Bastide
    Shu Qiang Li 1 PAUL Bastide
    Shunguo Yan 1 PAUL Bastide
    Stacy M. Cannon 1 PAUL Bastide
    Stanley K. Jerrard-Dunne 1 PAUL Bastide
    Stefan von Cavallar 1 PAUL Bastide
    Susmita Saha 1 PAUL Bastide
    Tamer E. Abuelsaad 1 PAUL Bastide
    Thomas J. Evans 1 PAUL Bastide
    Trudy L. Hewitt 1 PAUL Bastide
    Xujin Liu 1 PAUL Bastide
    Ying Mo 1 PAUL Bastide
    PAUL Bastide 0 PAUL Bastide

     

    Coinventing Graphic
  • When a value doesn’t exist?

    Recently, I ran into an issue with a resource referece in an Urban Code Deploy (UCD) resource that did not yet exist. When the value, that doesn’t yet exist I found this a great tip (UCD Documentation).

    Change from ${p:resource/value-not-yet-populated} to ${p?:resource/value-not-yet-populated}

    UCD replaces the missing value with an empty string. It saved my sanity and ease of use.

  • CURL and LDAPS – How to Search and Debug

    I hit an issue where I needed to Search LDAP from a machine I didn’t have access to install new RPMs on. I found this cool article on CURL and LDAP Search. I had to make some minor modifications to get it to work with a secure connection (–insecure ldaps:// and 636). I also added -v to diagnosis some connection problems.

    curl "ldaps://127.0.0.1:636/DC=IBM.COM?cn,objectClass?sub?(objectClass=)" -u "cn=user1,ou=test_org3,o=dr,DC=IBM.COM" --insecure -v
    Enter host password for user 'cn=user1,ou=test_org3,o=dr,DC=IBM.COM':
    * Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 636 (#0)
    * LDAP local: LDAP Vendor = OpenLDAP ; LDAP Version = 20428
    * LDAP local: ldaps://127.0.0.1:636/DC=IBM.COM?cn,objectClass?sub?(objectClass=
    )
    * LDAP local: trying to establish encrypted connection
    DN: dc=ibm.com
    objectClass: domain
    objectClass: top

    DN: o=dr,dc=ibm.com
    objectClass: organization
    objectClass: top

    DN: ou=test_org3,o=dr,dc=ibm.com
    objectClass: organizationalunit
    objectClass: top

    You can then find the userids you need quickly. I left them off the output intentionally.

    If you see connected, but no results, I suggest changing to the top level of the ldap, and using this string – ldaps://127.0.0.1:636/DC=IBM.COM?cn,objectClass?sub?(objectClass=*)

  • Maven and Ant

    I have been working on a project with IIB (IBM Integration Bus). If you are like me, you want to automate all those pesky build tasks. In IIB, I needed to package and build my project (BAR).

    I investigated scripting the build using the mqsipackagebar command . I noticed at the bottom a description of using it with ANT.

    I’ve used ANT in a few projects (in combination with Maven). I took the script as a starting point, added the jar files and property files (based on configuring environments without IIB). I configured the maven-antrun-plugin in order to integrate within my parent project’s maven build.

    Here is some sample code for you, I hope it helps you all get started:

  • ODBC and ESQL

    To setup ODBC bridge from IBM Integration Bus to Db2, download and install the ODBC bridge from the IBM Data Server Runtime Clients. It’s a 1.3 GB download, and you only need about a 40 MB file for your system.

    The installation is straight forward – Click Next, and Finish.

    The complexity and detailed steps are in the configuration of the ODBC Bridge and IIB.

    To setup the ODBC bridge and IIB, follow these directions:

    1. Start > Run

    2. Type ODBC

    3. Click on Data Sources

    4. Click on `Configure ODBC

    5. Enter the Connection Details

    Name Value
    Username db2admin
    Password fakepass!
    IP 9.32.18.166
    Port 50000
    Database name SAMPLE
    1. Name the alias as SAMPLE

    2. Launch a Command prompt

    3. Start > Run

    4. Enter cmd

    5. Change directory to server\bin

    6. Launch iib

    C:\Program Files\IBM\IIB\10.0.0.4\server\ 
    bin>mqsisetdbparms.exe TESTNODE_cheetah -n SAMPLE -u db2admin -p fakepass!
    
            BIP8071I: Successful command completion. 
            TESTNODE_cheetah should be your local node in Eclipse. 
    

    You want to restart the Node (you can look in the lower right of your toolkit)

    1. Right Click the Node

    2. Select Stop, Then Start

    Now, that it is setup, the steps to read the data, the ESQL is easy:

    You can then send a message using the IIB tests , and confirm using your database Db2 select * from DB2ADMIN.EMPLOYEE.

    To Write Data, create a New Input Message with

    {
    "EMPNO" : "2229",
    "FIRSTNME" : "Paul",
    "LASTNAME" : "B",
    "JOB" : "PRES",
    "EDLEVEL" : 18
    }
    
  • OAuth 2.0 Flow – A Metaphor

    The following video helps explain the OAuth 2.0 Flow and authorization. This video was originally shared as part of a prior project on Social Application Development.

  • Swagger Enumerations in YAML

    Recently, I have been spending time looking at the Swagger 2.0 Specification.  There XML representation and Enumerations are tricky.  Here is an example of a Swagger Enumeration in YAML.

    You can test the YAML at http://petstore.swagger.io/#!/default/get_demo

  • Raspberry Pi – Part IV – Simple Wiring Test

    I am plugging away working with my Pi. I went back to the Ada Fruit Site, and started working on one of the tutorials from the part I purchased – https://www.adafruit.com/products/2125   I want to be able to demonstrate a simple wiring with my Pi works.

    I selected Email Notifier. I read through it, and realized I needed to convert it to work for the B+ model. The pi4j website has a nice diagram for the pins http://pi4j.com/images/j8header-b-plus-large.png and I simplified the test a bit.

    I launched the simple python file

    pi@seconds ~ $ sudo python testPi.py

    Pi with a Green Light - Using Python
    Pi with a Green Light – Using Python

    I added the pi4j dependency to my maven pom

    <dependency>
        <groupId>com.pi4j</groupId>
        <artifactId>pi4j-core</artifactId>
       <version>1.0</version>
    </dependency>

    I create a servlet which replicated the same code from before (GPIO).

    I copied the war file to the local webapp directory, and restarted jetty

    pi@seconds /opt/jetty/web/bbq/webapps $ sudo cp ~/pi.webapp.war ./
    pi@seconds /opt/jetty/web/bbq/webapps $ sudo chown jetty:jetty pi.webapp.war
    pi@seconds /opt/jetty/web/bbq/webapps $ sudo /etc/init.d/jetty restart

    Refer to https://wiki.eclipse.org/Jetty/Howto/Deploy_Web_Applications  and I did have to add –module=jsp to the start.ini for my web configuration.

    I hit the servlet – http://192.168.1.200/pi.webapp/Control?status=true

    Exception
    Exception

    Which makes sense, since “Software using the Pi4J library must be run with ROOT level permissions.” (actually it doesn’t since I am running as root)…

    Issues remain.  I’ll tackle them on a different day 🙂   … Part 5 …