Running CodeMRI and Generating Diagnostics
Table of Contents:
Health Diagnostics reports are the files produced by CMRI in Microsoft Excel format. They contain the diagnostic information regarding the health of the codebase including financial and timeframe metrics related to the impact of the codebase’s existing architectural design.
There are 4 steps:
First create a vault
Vault: a collection of one or more projects, containing a duplicate copy of scanned code, generated reports, and any other files produced by CMRI. The first step to starting CMRI in any directory is creating a vault. This only needs to be done the first time CMRI is run.
Then create/select a project
Project: a codebase or group of codebases that work together. Projects serve as a workspace to gather all the related systems/codebases you would like to scan and/or compare.
Then create/select a system
System: a snapshot of a project at a single point in time.
Finally generate the CodeMRI Diagnostics Reports
Creating a Vault
The first step to starting CMRI in any directory is creating a vault. Vaults are a collection of all created projects and generated reports at your site. They (optionally) contain a full duplicate copy of scanned code, generated reports, and any other files produced by CMRI. The first time you run CMRI it will prompt you to create a vault in the current directory or the directory passed into the cmri command; you must have a vault in the specified directory before having access to any of CMRI's functionality. You are not limited to using a single Vault; a new Vault will be created in any directory in which you run CMRI for the first time. Once a vault is created, restarting CMRI will boot directly into the CMRI command-line interface instead of going through the vault setup process again.
In Linux shell or Windows command prompt, navigate to a directory to place the Vault.
a. There must be enough space on the drive containing the Vault for the duplicate codebase CMRI produces for static analysis.
Run the command
cmri vault create to create the vault.
It should trigger the following output:
Would you like to create a new vault here? (y/n)
a. Enter n to cancel the CMRI startup.
b. Enter y to continue the CMRI startup.
After typing
y to start CMRI, answer the questions.
Contact information for Vault creation does not need to match the license email.
/PATHNAME does not seem to exist.
Would you like to create a new vault here? (y/n)
y
What organization owns this vault? (Silverthread Client)
The Daystrom Institute
What is the group that owns this vault? (Engineering)
Software Management
Who is the primary contact for this vault? (John Doe)
Rick Swoles
Where is the primary contact located? (Nowhere, OK)
Philadelphia, PA
What is the primary contact's e-mail address? (unknown@silverthreadinc.com)
What is the primary contact's phone number? (000-000-0000)
002-004-2018
4 CPUs, 8903 MB of memory available. Defaulting to 2 workers.
Welcome to CMRI Development Platform v1.19.3.300. Type help or ? to list commands.
==================================
0 projects and 0 systems selected.
The Vault is now created and CMRI is configured.
Creating a Project
Create & select a project to create the workspace in which the codebase(s) can be placed.
a. Use the project add command to add a project:
==================================
0 projects and 0 systems selected.project add --name ProjectName
==================================
1 projects and 0 systems selected.
Creating a System
Create & select a system which will point CMRI to a specific instance of a codebase.
a. Use the system add command to add a system. Use the --selection option to specify the project to add the system to.:
==================================
1 projects and 0 systems selected.
system add --selection ProjectName --origin ~/path/to/ProjectName_SOURCE.zip --name ProjectName --version 47.0
==================================
1 projects and 1 systems selected.
b. Alternatively, you can use the command system add --name <system name> to streamline the process into a single command.
Generating Diagnostics Reports
Diagnostics reports are the files produced by CMRI in Microsoft Excel format. They contain the diagnostic information regarding the health of the codebase including financial and timeframe metrics related to the impact of the codebase’s existing architectural design. The diagnostics information is relayed through various reports such as the Health Diagnostics, Schedule Estimator, and Refactoring ROI calculator. These diagnostics help your technical, project, and executive leadership teams connect the dots between waste & project overruns and the underlying technical root-causes hindering developer agility & maintainability. Health Diagnostics Reports benefit both technical and non-technical teams by enabling engineers to drill down on quality challenges, managers to understand delivery challenges, and both to communicate and address critical issues.
Log into your Silverthread account using the command account login. This step may not be required in certain instances.
account login
a. Enter the information associated with the CodeMRI.com account.
Scan the code by running
produce_reports:
job run produce_reports
Several jobs will execute in order, processing the code and producing a set of Microsoft Excel Spreadsheets. To view the final reports, navigate to your Vault root. The reports can be found in the Vault root under the
vault/reports/<project name>/<system name>/.
Understanding Diagnostics
Technical health directly impacts codebase economics:
Diagnostic information is relayed through various reports such a:
CodeMRI-<codebase>-<language> Health Diagnostics
Schedule-Estimator-<codebase>-<language> Schedule Estimator
CodeMRI-Refactoring-ROI <codebase>-<language> Refactoring ROI calculator.
These diagnostics help technical, project, and executive leadership teams connect the dots between waste & project overruns and the underlying technical root-causes hindering developer agility & maintainability. Health Diagnostics Reports benefit both technical and non-technical teams by enabling engineers to drill down on quality challenges, managers to understand delivery challenges, and both to communicate and address critical issues.
Technical Health: Design Quality
A healthy architecture is modular, layers, hierarchical and has tight APIs. An unhealthy architecture has linkages that form cyclicality between files:
Design quality metrics identify architectural problem areas within a codebase. A key architectural metric is the presence of a CORE:
The Software Engineering Institute (SEI) recommends that cores, if present, should be less than 150 files. Anything exceeding this Silverthread defines as a critical core. Silverthread’s research shows that there are economic impacts if the number of files in a core exceeds 30. Silverthread calls cores of more than 30 files emerging cores.
Technical Health: Code Quality
Code quality applies to individual entities, measuring the health of each individual brick in the wall. Fixing code quality problems is relatively simple as problems can be addressed locally within each file.
A standard code quality metric is McCabe Cyclomatic Complexity, which measures the complexity within a file. Silverthread includes this code quality metric because it has been proven to have a significant impact on codebase economics.
Economic Outcomes
Out of the box, the Silverthread Health Diagnostics report economics in three categories: Cost/Waste ($), Agility (time), and Risk (bugs). These metrics are based on proprietary models within the tool, but are not calibrated to a specific codebase and development organization. They should be considered directionally direct but not precise. More accurate numbers require calibration of the model using historic data from task tracking and version control systems. Model calibration to a specific codebase is a service provided by Silverthread, please contact Silverthread Sales.
Health Metrics
Technical Health: Design Quality
Design quality metrics apply to the architecture
The Software Engineering Institute (SEI) makes the following recommendations:
# Critical cores (exceeding 150 files) = 0
Silverthread recommends the following to minimize financial impact:
# Emerging cores (exceeding 30 files) = 0
Files affected by a single change = worse than 50th percentile of similar codebases are highlighted in red.
Technical Health: Code Quality
Code quality metrics apply to individual entities (files). Code quality problems can be fixed relatively simply because they can be addressed with the entity.
Silverthread reports the following:
High complexity (McCabe complexity of 21 or greater)
Medium complexity (McCabe complexity between 11 and 20)
Low complexity (McCabe complexity 10 or less)
NIST 500-235 recommends that McCabe complexity be kept below 10.
Economic Outcomes
Based on the technical health of the codebase, the Silverthread CodeMRI® Health Diagnostics develop a predictive economic model based on similar systems. Benchmarks show the Top 20% percentile prediction for similar codebases - 20% of codebases are better than this number while 80% are worse than this number.
The default preferences are listed below – these are user inputs in the Preferences tab of the report (Excel spreadsheet) and can be updated to more correctly describe the situation. However, for this report the preferences describe the default situation:
Cost per developer: $120,000
# developers: 5
Downstream cost per bug: $50
Programmer days per year: 261
Interest rate for capitalization: 6.5%
Cost of Ownership
Cost to produce 1000 lines of code (LOC) is predicted for the overall system and for working within each of the cores (if present). This is compared against the Top 20% of comparable systems.
Money wasted per additional $1M invested is the predicted amount spent (wasted) over what a Top 10% performing system would achieve. In summary, if the system under study were to be improved to be a Top 10% system from a design and code quality perspective, this would be zero.
Project Quality and Risk
All project quality and risk measures, as listed below, are predictive from the economic model, and based on similar systems.
Days to develop 1000 LOC
Bug to feature ratio (labor hours):
Number of bug LOC added/exposed per year
Number of bug LOC released last year
Generating Diagnostics for a Portfolio of Codebases
Portfolio is an Excel file that contains brief summary of the top-level health of multiple codebases. It is a concise, basic description of the codebases diagnostics that is intended for non-technical and or executive consumers. Using Portfolio, your team can build objective financial business cases for better software decisions and develop enterprise-wide policy and implementation recommendations.
Create & select any projects and systems in CMRI.
Run the
produce_portfolio job:
a. Note: it is not required to produce Health Diagnostics reports beforehand.
job run produce_portfolio
The Portfolio Excel file will be placed in the
vault/reports/Portfolio/ directory.
Customer Support
Phone: 800-674-9366 (9am - 5pm Eastern Time)
Email: support@silverthreadinc.com