Setting up CodeMRI® Platform Web
Table of Contents
- 1 Table of Contents
- 2 Overview
- 3 Requirements
- 4 Deployment Flavors
- 4.1 Selecting a Deployment Flavor
- 4.2 Simple
- 4.3 Shared
- 5 Capacity Planning
- 6 Simple Deployment Steps
- 7 Shared Deployment Steps
- 8 Running and Administration
- 9 Scanning Code
- 10 Viewing and Navigating Results
- 10.1 Portfolio Page
- 10.2 Diagnostic Page
- 11 Troubleshooting
Overview
CodeMRI® Platform Web provides a web-based application for viewing portfolio and diagnostic results. The web-based application has many benefits over the traditional Excel-based reports in terms of navigability and performance.
Requirements
A Linux distribution with Kernel 2.6 or greater.
Shared installations will require NFS version 4. See below for more details.
A volume with at least 20GB of disk space per codebase for large codebases.
At least 8GB of RAM.
A multi-core machine or a VM at least 2 VCPUs.
Deployment Flavors
Selecting a Deployment Flavor
Before you begin planning for a server setup, select a deployment flavor according to your team’s requirements:
Simple will provide the fastest way to get set up with CodeMRI® Platform Web, however it depends on everything running on a single machine.
Shared is ideal for integration with your build pipeline, and for teams that will need to share a single vault among multiple machines.
Feature | Simple | Shared |
---|---|---|
Portfolio View |
|
|
Diagnostic View |
|
|
Build Pipeline Integration |
|
|
Vault Sharing |
|
|
Simple
A self-contained setup in which the server will be run from the same machine, whether physical or virtual, as the machine executing the scans. This is the best setup for evaluation or proof-of-concept installations where you want things to come up and running as fast as possible, and where you do not need to run CodeMRI® on continuous integration systems.
Simple Deployment Layout
Shared
A shared vault setup means that the machine running the server and the machine(s) running the scans (also known as "processors") are different. This is the best setup for integration with your build pipeline. In this case, the vault will be shared over NFS for use by processors.
Shared Deployment Layout
Capacity Planning
Due to the complex nature of Silverthread’s analytics, you will need a partition with ample disk space for scans. For most installations we recommend a minimum disk size of 40GB. Here you will find a table of our recommendations based on codebase size:
Codebase Size | Space Required Per Scan |
---|---|
Large Codebase (50,000 or more files) | 20GB |
Medium Codebase (10,000 to 50,000 files) | 5GB |
Small Codebase (< 10,000 files) | 1GB |
We highly recommend monitoring your disk space closely, especially in a continuous integration environment. If you believe you have an exceptional case (e.g. a codebase with millions of files), please contact support@silverthreadinc.com for guidance.
Simple Deployment Steps
CodeMRI® Platform Installation
Step 1: Download and install CodeMRI®
Follow the instruction at https://silverthread.atlassian.net/wiki/spaces/SCKB/pages/2828861441
Server Setup
Step 1: Set up a vault
Create a vault (if none exists):
mkdir /desired/vault/location
cd /desired/vault/location
cmri vault create
Step 2: Run the server setup script:
# cd /srv/cmri/server
# ./setup.sh
Step 3: Select Deployment Type
Server deployments come in multiple flavors:
* Type 'simple' for a self-contained setup. A self-contained setup means that server will be run from the same machine,
whether physical or virtual, as the machine executing the scans. This is the best setup for evaluation or
proof-of-concept installations where you want things to come up and running as fast as possible.
* Type 'shared' for a shared-vault setup. A shared vault setup means that the machine running the server and the
machine(s) running the scans (also known as "processors") are different. This is the best setup for integration with
Continuous Integration (CI). In this case, the vault will be shared over NFS for use by processors. Before beginning
this process, you will need to ensure that your machine has NFS version 4 (NFSv4). This has been in the Linux kernel
since 2.6, so any reasonably modern Linux distribution should have access to NFSv4.
* Type 'skip' to cancel server setup altogether. This machine will not be set up as a server. Use this if you are not
interested in running the web-based application, or if this is a shared environment and you are setting up a worker
machine.
Please enter an installation type.
simple
Type simple
and press Enter.
Step 4: Provide Vault Information
Vault Location
In order to use the server, a data vault is required. Please choose a volume with enough space according to the Capacity Planning section above.
Server User
Next, provide the user who will be running the CodeMRI® Platform Web server. The user must be a name of an existing user on your OS.
If you have provided an existing vault, the wizard will set the vault up to communicate with the server. If you specify a directory without a pre-existing vault, the wizard will prompt you to create a new vault. If there is a pre-existing vault, a new folder called web_data will be created and populated within that vault.
Step 5: Confirm Network Settings
Bind and Port Settings
The CodeMRI® Web server requires 2 ports:
One for the web site itself.
A second for the administrative interface.
The setup wizard will prompt you with the defaults:
You will need to customize these values if you are running other services on the machine that already use one or more of the default port numbers.
Once the automated installation is complete, proceed with the following instructions:
Create an endpoint to associate the vault with a web portal. Using the default settings (IP Address = 0.0.0.0, Port = 8000) if default settings were selected, or otherwise using the custom values for IP Address and Port, enter the following in a terminal:
Run a script to ensure the database is up-to-date, enter the following in a terminal:
For the web-platform, it is HIGHLY recommended to perform the following (if not done in the previous platform setup):
That’s all. Once you have completed these steps, the server is set up and ready to be run. Please proceed to the Running and Administration section.
Shared Deployment Steps
Pre-Installation Checks
Before beginning this process, you will need to ensure that your machine has NFS version 4 (NFSv4). If you are unsure, you can check by running rpcinfo
as shown below:
If you see entries with 4
in the second column, you are running NFS version 4
. If you do not see any entries, you may not be running NFS at all; follow your distribution’s guide for installing and running NFS.
Once you have the NFS service running, be sure you can access the NFS service from other machines on your network. While not a complete test by any means, you can do a simple smoke test on another machine using netcat
:
If you cannot reach NFS from other machines, make sure the service is running locally. You may need to change your firewall settings. If you use firewalld
:
CodeMRI® Platform Installation
Step 1: Download and install CodeMRI®
Follow the instruction at https://silverthread.atlassian.net/wiki/spaces/SCKB/pages/2828861441
Server Setup
Step 1: Set up a vault
Create a vault (if none exists):
Step 2: Run the server setup script:
Step 3: Select Deployment Type
Type shared
and press Enter.
Step 4: Provide User Information
If you have already set up a designated CodeMRI® user on your server machine, you may elect to run the server as that user. Otherwise, you can accept the defaults and the wizard will create a codemri
user along with a corresponding group. The codemri
user has no home directory and is created as nologin
for security purposes.
Step 5: Confirm Network Information
Port and Binding Address
The CodeMRI® Web server requires 2 ports:
One for the web site itself.
A second for the administrative interface.
The setup wizard will prompt you with the defaults:
You may need to customize the default values if you are running any other services on the machine.
LAN IP Address
In order to reach the server, other machines connecting to the vault will read an IP address or hostname from the vault:
Other machines connecting to the centralized vault will need to know how to reach the machine the server is running on. Ensure the automatically detected value is correct. If not, type y
, press Enter, and provide the correct IP or hostname.
Step 6: Provide Vault Information
Vault Information
Enter the path to the vault (e.g. /usr/share/vault
). Be sure to choose a location with plenty of space, see the Capacity Planning section above for details.
If a vault is not present, the wizard will walk you through the process of creating the vault. Be sure to answer y
to the following prompt:
Enter your contact information as prompted, then the vault will be created and set up.
Step 7: Share Setup
Finally, the wizard will prompt you about network share information:
If the default settings are acceptable, type auto
.
If you need to set up another network sharing technology, or if your organization has stricter security requirements you will need to set up the share manually. Type manual
, and proceed to manually set up your network share.
If you have selected auto
, the wizard will give you a line to paste into the /etc/fstab
of machines you would like to have access to the vault:
Be sure to create the mount point as root via mkdir -p
before attempting to mount the NFS share.
For the web-platform, it is HIGHLY recommended to perform the following (if not done in the previous platform setup):
Please proceed to the Running and Administration section for further instructions.
Running and Administration
Starting and Stopping the Server
To start the server, run cmri-start-server.sh
. This command will setuid to the provided server user before running the server, so if you run this command as root, it will not run the server as root. The server is run in the background as a daemon; it will not exit after you exit the shell. navigate to http://server-ip-here:8000
to access the server. Default is 0.0.0.0:8000
To stop the server, run cmri-stop-server.sh
.
Administrative Interface
The web server comes packaged with supervisor which provides a remote administration interface allowing an administrator to start and stop services and view logs. Assuming you have default settings, navigate to http://server-ip-here:9201
to access supervisor. The default login details are:
To view logs as they are running, click Tail -f
on the component you wish to investigate:
Scanning Code
Exporting Results to CodeMRI® Web
Step 1: Ensure the vault is authenticated
This step does not apply to users running in an air-gapped environment. If you are running in an air-gapped environment, follow the licensing instructions you received directly from Silverthread and proceed to Step 2. If you have not received such instructions and are running in an air-gapped environment, contact support@silverthreadinc.com.
To authenticate your vault with Silverthread, run cmri account login
:
Enter your login credentials to authenticate your vault.
Step 2: Set up your codebase
(Optional) Environment Variable Setup
For the purpose of this guide, we will assume the data vault is located at /srv/vault
. Replace /srv/vault
with the path to the vault on your machine.
To make things easier, I highly recommend adding the following line to your .bashrc
:
After doing so, run source $HOME/.bashrc
to reload your shell. This will tell the cmri
program where the vault is, so you won’t have to worry about which directory you’re running the application from, and you won’t have to provide the vault path on the command line.
Create a project
Projects are specific codebases you wish to scan. Projects are containers for systems, which are snapshots of a particular codebase at a point in time. If you are familiar with version control, think of the project as the repository and the system as an individual commit within the repository.
To create a project, run cmri project add
:
Create a system
To create a system, you’ll need a name, version, location of source code, and optionally a revision date.
If you know the revision date of the system you are scanning, please set it. Otherwise, you can skip this step. However, if you skip this step, the web version of CodeMRI® will not be able to display trending graphs that show how a project evolves over time. revision_date must be set for every system in your project for trending to work.
If you are scanning code out of a directory, you can save time and disk space by turning on in-place scanning. This is especially useful if you do not wish to transfer a large codebase over NFS to a centralized vault:
Step 3a: Scan the codebase and create web data - if you are working from a fresh Data Vault that has been newly created
Once you have set up your project and system to your specifications, it is time to scan the codebase and export the results to the server. To accomplish this, run:
Scans can take anywhere from a few minutes for a small codebase to several days for an extremely large codebase. Silverthread will try to estimate the amount of time a scan will take early into the scan process. Actual time taken will vary from codebase to codebase, and may even be dependent on the programming language(s) used to construct the codebase. If you have concerns about very long running scans, contact support@silverthreadinc.com for assistance with tuning the scanning process.
Also note that scan results will not be immediately available once export_web_data
completes. It may take a few minutes for the server to process the data exported from the platform.
Step 3b: Create web data - if you are working with a Data Vault you have already been using for other things
If you are using a data vault with projects and systems already defined, and now want to look at them in a browser, you must generate web data in the Data Vault for use by the web server. If you want trending to work in the web view, you will optionally also have to retroactively set the ‘revision_date’ configuration option for each system in a project you want to see trending for.
For good examples of how to set revision_date, see examples laid out on the ‘CodeMRI Administration Example’ page.
To generate web data retroactively for systems already in a vault, run:
Note that scan results will not be immediately available once export_web_data
completes. It may take a few minutes for the server to process the data exported from the platform.
If you have already been using CodeMRI® to generate Excel reports or access data via command line, you have probably been running the following command to generate data in the Data Vault and generate Excel:
If you want to explore a system via both Excel reports and Web for future systems you scan, you will want to do the following so web data is also generated:
In addition, if you make changes to a system or its configuration, make sure to export_web_data
so that web data is up to date.
Viewing and Navigating Results
After you have scanned a system, you can view the results by navigating to (assuming default settings) http://<server-ip>:8000
.
Portfolio Page
The portfolio provides an overall view of all of your projects. Each numbered line shows a summary of the latest system for each project:
Results are separated by language. If your project contains multiple programming languages, you will see multiple results.
Diagnostic Page
To see detailed results, click on a row within the portfolio table. The health diagnostic page will appear showing detailed information about the system in question including technical health and economic health metrics:
Please contact support@silverthreadinc.com for guidance on interpreting results. We are always happy to help.
Troubleshooting
Like with any computer software, things on occasion don’t always do what you want them to do. Here are some common issues you may face and solutions to these problems:
Issue | Likely Cause | Resolution |
---|---|---|
Server running, can access website locally, cannot access from other machines. | Your system firewall is blocking the ports the server is exposed on. | This assumes default settings, if you have customized your ports, replace If you are running After completing the command sequence above, check that the service is accessible from other machines. If you are using another firewall technology, consult the user manual for your firewall. |
export_web_data completes, scan does not show up in portfolio. | The background worker process has stopped running. | Access the admin panel at Restarting the services and then refreshing the server page can work as well. If the worker is running, check the logs. It may be processing another import, or encountered an error processing the system you’ve exported. Contact support@silverthreadinc.com for assistance with errors. |