CodeMRI® Sys Admin Training Video

CodeMRI® Sys Admin Training

Training Video Transcript

Hi, I'm James Hamlett. I'm a senior software architect at Silverthread. And in this video we're gonna go over some of the more common usages with CodeMRI®. Have the cheat sheet up, which contains all the common usages I'll be going over. This cheat sheet can be found at Silver Thread's website on the Wiki site.

Before using CodeMRI®, a user will create a vault. Vault is the main workspace for CodeMRI® inside of a vault. A user can define one too many projects inside of a project. A user can define one too many systems. A system points to a code base and maintains a current snapshot of. Within a system, a user can also assign revision dates that keep track of specific snapshots of the codebase.

Vault Structure Explained

To better visualize a vault, we can look at this diagram of the cheat sheet and we see the vault here as a box. Inside of that box, we have multiple boxes. For projects inside of each project, we have multiple boxes for systems, and each system has multiple revision dates, so you can think of the vault projects and systems as boxes within boxes.

Create a CodeMRI® Vault

To create a vault, a user needs to open up a command line terminal. Create a directory. Name the vault, anything you'd like, and then change directory into your vault. Once you're in your vault, enter the command cmri vault create. This tells CodeMRI® to create everything it needs for a vault inside of that directory.

Issue Commands to CodeMRI®

Once a vault has been created, there's two ways to issue commands to code. The first way is to use the keyword cmri followed by the command. The second way is to enter into CodeMRI®'s, interactive shell. There's a few ways to enter into the shell. You can change directory into your vault and type in cmri shell.

You can from any directory, type in cmri - -vault with the path to your vault, and then shell, that will also put you in the. and if you don't want to CD into your vault and you don't want to type out your vault path every time, you can make a global variable for your operating system for cmri vault that points to the path of your vault.

You can do this on both Linux and Windows. For example, I have a vault in this directory and I've named it Vault. I can type in cmri shell. And it takes me right to it because I have my vault listed as a global variable. When you're in your vault, if you want to leave, you can just type exit and then you're out.

Use CodeMRI® Helper Commands

CodeMRI® also provides help commands. You can type in --help to get general help information. You can type in job list to get a list of all the jobs that are available and to see details about a particular job. You can type in, job describe, followed by the job name. So let's take a look at each one of.

First thing, I will enter my vault cmri shell, and then once inside my vault, type in --help and we'll display some general help information. Enter we will advance this by a line. The space bar will advance it by a page. You can type in Q to exit. We can look at all the jobs that are available by typing in job list. And this will provide a fairly exhaustive list of all the jobs available.

Now, there's a job we want to see specifically. Produce reports is a very common job, so let's look at the details, job describe, and then job name. So produce _ reports, and we can see here, It'll print out the name of the job. It'll print out the dependencies. The dependencies or other jobs this job depends on. This is quite nice, so there's no, prerequisites for typing in a job. There aren't other jobs the user has to run first CodeMRI® knows which jobs needs to run, and will run them on its own.

The job described function will also produce all the optional parameters as well as. Some explanation of each parameter, what they all mean. It'll describe what the job does. This particular job produces multiple reports and it lists the reports that it creates.

Set Up CodeMRI® Licensing

CodeMRI® requires licensing to run. There's two types of licensing, online mode and offline mode. For online mode, you contact Silverthread and have your license assigned, and then in CodeMRI® you will enter the command account, login with your email and password and you and your setup. For offline, there's a command machine-id generate and this will generate a file and you get that to Silverthread. And then Silvethread to return you an offline license. Offline license is typically for systems that don't have internet access.

Add & Remove Projects

Within a vault CodeMRI® uses projects and systems to make reference to codebases. The project is a high-level container that contains many systems, and the system references the codebase specifically. So, to create a project inside the cmri shell, you would enter project add --n for name, and then assign your project the name.

If you wanted to remove a project, you could select the project, and then type project remove that would remove the project from the vault. If you have multiple projects selected, you could enter project remove --force and that would remove all these selected projects. To remove a specific project, you could enter project remove --selection and enter the project name.

Let's take a look at that. So I'll go into my CodeMRI® shell. I'm in my shell in my vault, and you can always confirm your vault contents by typing in vault list. That will show you what's in the vault right now. The vault is the only thing there. To add a project, project add --n, and I have a publicly accessible codebase called Axis that I use for tests.

So we will create a project called Axis, and we can type vault list again to confirm what's in the vault and I now have a project called Axis, and that project is selected. It's selected by default when I create it. Since it's selected. If I wanted to remove it, project remove, and that project is now gone.

Vault list. Now if I re-add that project, the project is back, I can also remove it by saying project remove --selection Axis and it also removes it. I wouldn't normally do that when I just have one project in the vault, but if I had many projects in the vault, it's a little safer to specify the project. I would like to get rid of.

Add & Remove Systems

Creating a system in CodeMRI® is much like creating a project. It's a similar command. System add and we give it --selection with the project name. Cuz remember, all systems live under our project, so system add --selection <project name> --origin, and we give it the file path to where the codebase lives.

For this system that we're talking about, um, --name. We give the system a name and --version. We give the system a version. Deleting systems is very similar to deleting projects. If a single system is selected, we can type system remove, and that system will be removed from the vault. We have multiple systems selected. We can type in system removed –force, and that will remove all the systems that have been selected. To remove a specific system, we can type system remove --select. With the project name --system name, that's a little safer and let's take a look at all those. I will enter my vault. cmri shell.

I'm inside my vault. I have zero projects, zero system selected. I can type vault list at any time to view the contents of my vault. Right now, I have no content. Just the vault exists. So it will create a project add --n axis and then we will add a system, system add --selection with the project name that the system will be under --origin. For the path to my code--name, I will give my system a name, we'll call it Axis2, just to differentiate from the project name --version, and this is version 1.7.9, and then we hit enter. The system has successfully been added, and we will confirm that by looking at vault list and looking at the contents of the vault. Now we have a vault. We have a project Axis and a system Axis2 of 1.7.9.

Select & Deselect Projects and Systems

To run CodeMRI® against the system or systems. The system or systems need to be selected. To select systems enter select --add project name /system name to select all the systems under a project. Enter select --add project name /*. To remove projects, you enter select --remove project name /system name or select --remove project name /*.

Let's take a look at that. Oh, enter into my interactive shell. I'm going into my vault. What's inside my vault? There's no project selected, and no systems selected. Let's view the contents of the vault right now. There's one project called Axis and one system under Axis called Axis2. So, I want to select Axis2.

Select --add project name. Now I could enter Axis2, there's a number of ways I can do this. I could put in a wild card character the chart cuz there's only one system under access that was selected. Now that system is selected.

And we can see by viewing the vault through bot list that everything is selected. Let's deselect that. Select --remove Axis/*. That system has been removed. Another way to select that system, --add. And I always have to enter the project name Axis2, I could type in the full name, but the wild card works on partial names too, so I could type in Axis2*. Now that system is selected, we can verify that by looking in the vault list. And of course, I can type out the full name, but a shortcut is to use a wild card wherever you can. So that is selecting and deselecting systems within a vault.

Produce CodeMRI® Reports

Two of the most common jobs in CodeMRI® are to produce reports and to produce a technical health improvement plan report. To produce reports, you type in job run and then produce _ reports to run anything in CodeMRI®, any job you type in, job run, and then the name of the job. So, to produce reports, it's produce_reports.

To produce a technical health improvement plan or a THIP, it's job run produce_thip. So, let's take a look at that. Now we'll enter my vault here through the Shell cmri shell. I'm in my vault. There's no project selected, so I've just select a system in order to run a job against it. So, select --add Axis/*. Now I've selected a project and a system. We can look at vault list to verify what was selected in this case. I only have two things in there. I have the Axis project and the Axis to system under the Axis project, so those both selected, we can enter a job run.

Produce underscore reports, and the job's kicked off. This takes a little while to run. It depends on the size of the codebase is being run against. It could take anywhere from, uh, minutes to up to a few hours, depending upon the size of the codebase. So, let's skip ahead to where this job has finished.

I'm sure the produce reports is run successfully. And we can verify that by looking in the directory where the reports get created. So, let's look at that right now. If I open up a directory to my vault, which is in home James Vault, and I look under reports, under the Axis Project, under the Axis2 System, I'll find a directory for every language that CodeMRI® was able to scan.

In this case, it scanned the Java language. And inside of there, there are five reports that produce reports that it would create. So let's go back to the Shell, and inside the Shell we will create the next report, the technical health improvement plan, or the THIP, so we can run, job run, and then the name of the job produce_thip and job is running. And again, this could take a little while, depends a few minutes to a few hours, depending upon the size of the codebase. So, let's skip ahead to the end. The technical health report has completed successfully. And we can verify that by looking in the directory where the reports are created. Let's take a look there. And when we look in our directory of reports, we now have one more report, technical health Improvement plan or THIP. So that is how we create the reports.

Exploring the Most Common CodeMRI® Reports

Having run our jobs produce reports and produce THIP. We now have a directory in our vault under reports, under the Axis project, under the Axis2 system in the language of Java, we have the five reports from produce reports, and we have the technical health improvement plan from produce THIP. So now we can take a quick look at a couple of these and point out some of the more interesting things.

So if we look at the main report CodeMRI-, and then system name, we can see here on the summary page that it provides a summary about the system, the language that was scanned, the number of code files that it found, the number of lines of code, the average size of a file, and lines of code. And then, it provides some technical health summary information on design quality and code quality.

The design quality information is in the form of file cores. File cores are circular dependency relationships among a set of files. Code quality is expressed as McCabe complexity, McCabe complexity. The McCabe score is based on the number of unique executable paths through a file. So, if we look at design quality, in cores, we have two critical cores with over a hundred thousand lines of code, and then two emerging cores made up of over 40,000 lines of code.

A critical core is a set of files that form a circular relationship of dependencies. And to make it critical, it's over 150 files. An emerging core is between 30 and 150 files to make up that core for McCabe complexity. Um, files and low complexity are between 0 and 10. Medium complexity is from 11 to 20, and high complexity is anything over 20, and we can see here that we have five files that are in high complexity.

We have 273 in medium, and then the rest are in low complexity. This is all summary information and if you want detailed information here at the bottom, you can click on the file list tab and then we'll show you every file that was scanned. And it'll give you information on every single file. We can see the language, the extension, we can see. Let's scroll over here. So we can scroll through here, and we can see cyclic group size. So, this file isn't a cyclic group made up of 177. We can see fan in, fan out visibility, fan in visibility, fan out. Those numbers are how many other files are dependent on this file and how many other files this file depends upon.

And if we scroll over some more, we can see cyclomatic score. This is the McCabe score, as well as the category that it falls into. We can see lines of code and source lines of code. We can see comment ratio, which we haven't talked about yet. Comment ratio is the ratio of the number of lines of code that are comment versus the number of lines of code that are not comments.

So, for instance, this file here is a 70% ratio. That means for every 10 lines of code, there's seven lines of comments. Gives you an idea of how well a file is documented. If we scroll over some more, we have component information. CodeMRI® has the ability to define files as belonging to a component, and we'll discuss more on that in a few minutes.

We see percent of duplicate code. This is the percent of code that exists that is duplicated, uh, segments of code that are duplicated in more than one place, and this is the percent of the codebase that is considered to be duplicate. We'll have more on that later. We also have the ability to tag a file as being either a third-party file-generated code or a test file.

Uh, we'll talk about how to do that. That's also a common command that's useful if you're going through this and trying to develop metrics. A lot of the time you don't want to include a test file or a third-party files in any of those metrics. Another file we're gonna look at is the code duplication file.

We saw that in the made main code CodeMRI® file where it describes the percent of duplicate code in a particular file. The code duplication report is a roadmap to where all the duplication lives, and if we look in this file on the main tab, it shows. The percent of duplication in the system, and then the percent of removable duplication.

Now, of course, when you have duplication, say you have four segments of code that are identical across your code base, well, you're gonna have to keep one of those. So, percent duplication would be all four sets. Removable duplication would be three sets, since you have to keep one, we can look at the roadmap here, which is duplication instances.

Now, these are all grouped into sets as a piece of duplicate code is not necessarily contained within one file. Now, if we sort this, we can take a better look at this. Here you can see I've sorted on the duplication instance id. These are grouped by files that all share the same piece of duplicate code. And we can see here, uh, there's two files in duplication.

Instance ID two, meaning both of these files both have an identical segment of code. And if you look at the line here, it'll tell. If you go into that file, if you look starting online 103 and ending online 465, and then you go to the next file and start at line 101 and end on 463, the code in those two segments will match, and you're able to do some refactoring to uh, shrink your codebase.

And if we scroll down a little bit here, we'll see In duplication instance ID 51, there are a number of files that all share duplicate code. So doing some refactoring here would can shrink your code base considerably if it's possible. Another file that's full of good data is the detailed worksheet inside of the detailed worksheet. CodeMRI® discovers all the references that are made within the system. They're labeled from component down to file, down to entity. You know, as components are all empty. That's because no components were defined for this system. But if we look here, we can see that from this file and this entity within this file, starting at this line, is making a reference to something in this file.

Which is the entity. It also tells us whether this relationship is expected and whether this relationship is circular or not. Expected is something that would be filled in had components been defined and in relationships between the components defined, and this would indicate whether that relationship is breaking any of those defined rules.

And lastly, we can take a look at the Technical Health Improvement Plan. The Technical Health Improvement Plan lays out a roadmap for breaking up file cores. Here we have two plans, one for a core of 177 files, and then there's another plan for a core of 159 files. Both of these are critical cores. It lays out steps, and it shows that as follow the steps, how much of a reduction you can expect in your core. And the steps are laid out here, and it'll show you. These will all be under step one, and it'll say from this file and from this entity. And then, if you go to this file and this entity and find a way to resolve that dependency, and as you go down the line, you will start breaking up that core.

Okay, that's just a brief overview of the reports. There's a couple of others in here. These are chock-full of good information. Feel free to browse these as you can.

Define Code File Attributes

A common command in CodeMRI® is to attach file attributes to particular files that are being scanned. A few of the attributes that can be attached, we can flag a file as being test code. We can flag a file as being a third-party file, or we can just tag a file as being excluded, meaning it gets excluded from the metrics that are generated.

Uh, these commands are all relatively similar. To execute these commands, you would first select the system where the file resides, and then if you wanted to tag it as either test code or third party. You would enter our system, file-property add-n, and then either FP_test or fp_ThirdParty, depending upon what you're doing, and then a -v true, and then a -e.

The E is for expression, and the expression is the relative path to the directory where the files live that you want tagged. This does it on a directory basis, which is pretty common. Test codes usually all gonna live in a directory of its own as well as third-party plugins. And as I said, this is a relative path, so it doesn't start with a slash, and it starts after wherever the system directory is that was defined when the system was created in the vault. You tag a directory is excluded. It's similar. You select the project and the system, and then you type in system files, -exclusion, add, and then there's no need for a -N or -V, and we go straight to -E. And then again, we give it a relative path through the directory. In all of these, we can use wild cards, so when you enter the relative path, you can end it with a / and then a *, and that tells it to include everything in the directory.

Define Components & Component Subsets

CodeMRI® has the ability to let the user define groups of files and name them as components, and then generate blueprints based on those components. If we look at this, uh, in order to create a component, you would issue the command system component add,-- and then give your component the name, --expressions, and then provide a relative path to where the file group lives.

Each one of these files will be tagged with the component name that you choose to remove a component. It's a similar command system component remove --name, give it the component name, --expressions and give it the path to where the file group lives. Some usage for this, you could do the system add command, giving it the same component name on different subdirectories, and that would group all the files in these various directories into the same component.

You could also issue. A single command with your component name. And for expressions, you could provide a comma-separated list of the relative directory paths to where these files live. Another usage of the add and remove command is imagine you have a directory of files that you want to classify as a component.

Within that directory, there's a subdirectory of files. Which make up their own component and so you would like those to be named as their own component. You can do system component add, give it a component name, and then give it a relative path to the directory, and then system component. Remove from the same component name, but this time give it the subdirectory that's within the original directory.

That will, in effect, take all the files that are in that subdirectory and remove the tag of the component name. Then you could do system add and give it a new component name, and then give it that same subdirectory. It would then go and re-tag those files with a new name. Along with defining components, you can define relationships between the components.

You do that by entering system component dash relationship add and then -- from, and give it the name of the component where the relationship starts, and then a -- to, and then give it a comma-separated list of component names that will define a relationship between components for a code MRI purposes.

If you want to verify this, you can type in system architecture verify. That will give you a list of. What's been flagged as an error or a warning, or just information about the definitions. To view the components that you've defined, you can type in system component list, and that will give you a list of all the of all the components that have been defined.

If you want to see the relationships that have been defined, you can type in system component -relationship list. Um, if you want to verify that all the files have been tagged, you can rerun job run, produce reports, and that will produce all the reports. And then you can look in the file tab and you will see the component name next to each file that it's been assigned to.

Generate CodeMRI® Blureprint Diagrams

To see the blueprint diagram, you can run the job, job run produce_API_diagrams. This is an example of a diagram that gets created when you run produce API diagrams. As you can see, each box here represents a component that was defined and all the arrows between the box represent relationships. One thing to note, and it's very interesting, is that if you don't define relationships, or any relationship that isn't defined will show up when it's discovered and it will be colored differently from the relationships that were defined.

So for instance, if you're working with a legacy codebase and there's no architecture diagrams, there's no architecture documentation, A wonderful way to discover the architecture is to define the components and then run this diagram and it'll show you all the relationships. As you can see, this one looks fairly busy and there's a way to zoom in on that.

We can run another command. We can generate blueprints on a selected subset of components. We do that with the same command as we produce the API diagram. We do job run produce_API_diagrams, but we give it the parameter of --component, -subset, and we give it a list of components that we would like included in the subset, and then we give it –name, and we name the entire.

Another way to do this instead of providing a list of components, we can create a file inside the file. We can list the name of every component we would like included, and then we can execute the job by typing in job run produce API diagrams, --component -subset -file and provide a path to the file and again, --name.

And a name for the subset of the. Here we can look at the output on a test run from another system, and this is a subset of the data from the previous picture. This one is much more understandable when we zero in on just a couple of components, and so for systems that are very complicated that have a lot of relationships, this is perhaps an easy way. To piece by piece, try to understand the system

Define & Generate KPI Reports

CodeMRI® provides the ability for a user to define key performance indicators against existing metrics in order to generate tabular and graphical report data. The input for a KPI is a yaml file. The yaml file will contain definitions for the desired KPIs and for desired metrics to be seen. To get a list of all the metrics that are available for KPI, you can type in metrics list-kpi-metrics.

To generate KPI you first select systems of which KPI information will be gathered. and then the user will set a revision date against a particular snapshot that's done by entering system config, set the revision date, and then enter a date and time. That date and time will be attached to the current snapshot.

The user can then set the path to the yaml file by entering metrics config set -o and then the file path, and then to generate the metrics. The usual initiate a job with job run and the job's name is generate KPI results output for the KPIs presented in an Excel format with chart pivot tables and slicer buttons to modify the chart. Let's take a look at one.

Here we can see an example KPI output. It's a number of tabs divided up between KPI information and metric information. And the tabs along the bottom, there's an overview tab, which includes all the information of everything that was generated. And then it's split between, ratio information and then integer information.

So, if we look at KPI percentages, you can see we have a graph listing all the information that was collected with the accompanying pivot table, and all of this is customizable. As you modify fields, the charts will change. And so, you can look at this data in any combination that you like. The same is true for the metric data with the overview, percentage information and then straight integer information. And the reason those are divided up is because looking at percentages and whole integers in a chart doesn't really make sense. It's not apples to apples. It's apples to orange. So that's an example of our KPI output.

Thank you for watching!