Set up running of new SWIP code through SWAMP

[von]

• uploaded pegasus-worker-4.8.0dev-x86_64_rhel_7.tar.gz (Python 2.x scripts)
• Assessed using all Python tools: Pylint, Flake8, Bandit
• Successfully completed (sorta), but haven’t analyzed.

[von]

The goal here is that new code developed for SWIP will be "assured" using the SWAMP, not the pre-existing Pegasus corpus. Can we create a snapshot of Pegasus' current weaknesses and then make a "diff" going forward (what are the semantically meaningful changes).

[von]

There will be other Pegasus changes concurrent with SWIP and we may not have the effort to run all of those through the SWAMP. Do we do the SWIP changes in a branch (that is merged often) and just run that branch through the SWAMP? (This is primarily a @rynge and @vahi question.)

[rheiland]

For starters, we've run an initial 'diff' between SWAMP (scarf, xml) results returned from 'beta3' and 4.8.0:
https://docs.google.com/document/d/1sQmKMvIjaTbL-LkLDdFijj8hnG0QkkmLnp76zW-akSM/edit#

We'll continue to improve this "diff" process (scripts, etc) to generate useful output.

[rheiland]

New approach: use github repos to feed SWAMP and use some continuous integration (CI) tool, e.g. TravisCI, to do whatever "diff" we end up writing. Unknown if we can extract just a portion (subdir) from a github repo. Will explore. Will also explore options for CI.

[rheiland]

I've been emailing SWAMP Support to try to understand their assessments (and reporting) better. The latest volley:
Can someone explain and walk me through a simple github assessment on SWAMP? I’ve created https://github.com/rheiland/test_swamp1 and created a release https://github.com/rheiland/test_swamp1/releases . I’d like to perform an assessment on this release. Note that it contains two subdirs, one containing a Python script; the other, two C files (neither one compilable into an executable).

I login to my SWAMP account using my username/password (not via github account). I then ‘Add New Package’ and select ‘Remote Git repo’. For ‘External URL’, I provide https://github.com/rheiland/test_swamp1/archive/v1.0.tar.gz and provide no ‘Checkout args’. On the next screen, I leave ‘Package path’ alone, i.e. “./“ and select C/C++ as Language. Next screen: I leave it as ‘No build’. Continue… Run Assessment, select a Tool, e.g. cppcheck, select a Platform. Run Assessment. Fails. ...

I've also asked them (again) about their contradictory reporting of failure/success of an assessment on web site vs. email.

[rheiland]

More email exchanges w/ SWAMP Support - trying to understand how I can automate, as much as possible, SWAMP and Pegasus releases in GitHub, e.g.:

Kyle,

Thanks for the reply. No, I didn’t expect SWAMP to do >1 language per run. But I’m still confused
with the workflow and my resulting error. I specify a URL to a release of my test repo to upload
it to SWAMP:
https://github.com/rheiland/test_swamp1/archive/v1.0.tar.gz

On the next screen, I don’t know what “Package path" I should supply. My gzipped tarball untars into:

/test_swamp1-1.0
     README.md
     /c_code
     /python3_code

So I tried entering “./python3_code” for my Package path and proceed with the assessment,
but it fails with the error “Unpacking archive ‘v1.0.tar’ failed".

Hoping you can provide some insight. I've attached some, hopefully, relevant screen captures.
-Randy

[rheiland]

getVuln.txt
While I wait for SWAMP Support to reply, I'm experimenting with Python to parse and provide relevant info from the SWAMP SCARF results, with two goals in mind: 1) provide a more helpful (concise, etc) listing of vulnerabilities from the assessment, 2) lead up to an automated "diff" of SCARF files.

$ python getVuln.py 

---- scarf-Bandit-4.8beta3-orig.xml:
AnalyzerReport
{'platform_name': 'ubuntu-16.04-64', 'build_root_dir': '/home/builder/build', 'package_version': '4.8.0beta3', 'parser_fw': 'resultparser', 'assessment_start_ts': '1503595449.7361057', 'parser_fw_version': '3.1.1', 'uuid': '6bd923a0-89e4-4e12-a8a3-922dbd9f0d42', 'tool_name': 'bandit', 'package_root_dir': 'pkg1', 'package_name': 'pegasus-python-source', 'tool_version': '1.3.0'}

---- HIGH priority ----
jinja2_autoescape_false HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH
subprocess_popen_with_shell_equals_true HIGH

---- MEDIUM priority ----
hardcoded_tmp_directory MEDIUM
hardcoded_tmp_directory MEDIUM
blacklist MEDIUM

[rheiland]

Got reply from SWAMP Support - it's not possible to specify a URL to a github Release. Rather, just a URL to the repo and, optionally, a branch, etc.. Also, the SWAMP expects files to have a suffix to indicate their type, e.g., ".py". Exchanged emails on swip-l about the reasoning for missing suffixes on Pegasus scripts in /bin.

Here's the reply from SWAMP Support (note large discrepancy in # of weaknesses between tools for "pegasus-integrity" Python script):

When creating a SWAMP package from a GitHub repo, SWAMP does a git clone with the "External URL" provided and then does a git checkout with the "Checkout Argument" provided. It then creates a compressed archive with the results and submits it as a package to the SWAMP, the same as if a user had uploaded an archive.
If no "Checkout Argument" is provided, the package will contain the latest code in the master branch.

To get the "External URL" from a GitHub repo, click the green "Clone or download" button on the repo's main page.
In this case that would be:
https://github.com/rheiland/test_swamp1.git

The "Check out argument", which is optional, would be the name of a branch, release, tag, or a commit number.
In this case that would be the release name:
v1.0

I tried this Creating a new package in the SWAMP with the following parameters:

"External URL" = https://github.com/rheiland/test_swamp1.git
"Check out argument" = v1.0
"Package path" = python3_code/
"Language" = Python
"Python Version" = Python 3
"Build System" = No build

I was able to successfully create this package and run assessments.

However, no files were actually assessed. I got back zero weaknesses from all three python tools.
This is because the python code is in a file in the Git Repo without any extension: "python3_code/pegasus-integrity"
without the file extension, SWAMP does not know that this file contains python code and does not assess it.
That is also why, when adding the package, on the source page, the SWAMP does not recognize that the package could be a python package.

I downloaded the pegasus-integrity file to my local system and renamed it "pegasus-integrity.py" and zipped it.
I then created a Python 3 no build package in the SWAMP uploading that file.

The SWAMP recognized that it was a python package.

I was able to assess it and get results: Flake8 found 117 potential weaknesses; Pylint found 1.
I suggest adding a file extension to the "pegasus-integrity" file in the GitHub repo and creating a new release.

[rheiland]

Significant updates to the getVuln.py mentioned above, but now called parseSCARF.py. Added it and a sample SCARF .xml file to a "/static_analysis" subdirectory in the github repo. And did a minor revision to the README.md

[rheiland]

Partial summary:
SWAMP:

• cannot perform more than one language assessment at once
• files need suffixes (e.g., ".py") to determine what type(s) of assessment tools are possible
• user cannot programmatically download SCARF (SWAMP Common Assessment Result Format) files
• "diff" of SCARF results will target differences between vulnerabilities (found by same assessment tool, but of different versions of files); rf. https://github.com/IU-CACR/SWIP/blob/master/static_analysis/sample_output.txt

Pegasus:

• unclear if/how we will assess SWIP-only edits to repo

[rheiland]

Recently looked into using SWAMP's REST API for programmatic functionality to perform assessments and retrieve results (SCARF output; SCARF=SWAMP Common Assessment Result Format). Unfortunately, the REST API does not currently allow retrieving SCARFs. SWAMP Support suggested I try their Java-CLI. Still a work in progress, but the Java-CLI does indeed let us retrieve SCARFs, e.g.:

$ alias jcli='java -jar ./java-cli-1.3.5-jar-with-dependencies.jar'
$ jcli   # get list of cmds
$ jcli projects --list
$ jlci assess --list -P b77ede58-bb9d-11e4-a4cd-001a4a814425
UUID                                  Package Name    Package Version Tool Name       Tool Version    Platform Name   Platform Version
04ee42ff-3a49-4e6e-8cac-46a63321029d  test_swamp1     latest          Pylint          latest          default         latest
89d15494-096f-4ea8-99c2-00fa501d8303  test_swamp1     latest          Flake8          latest          default         latest
ebb174cc-c5f0-4740-899f-6bb628fd4dac  test_swamp1     latest          Bandit          latest          default         latest
...

$ jcli status -P b77ede58-bb9d-11e4-a4cd-001a4a814425 -A 89d15494-096f-4ea8-99c2-00fa501d8303
SUCCESS, 117, ac65d12c-0a97-11e8-9925-001a4a814413
   ## means found 117 weaknesses
$ java -jar ./java-cli-1.3.5-jar-with-dependencies.jar results -P b77ede58-bb9d-11e4-a4cd-001a4a814425 -R ac65d12c-0a97-11e8-9925-001a4a814413 -F /Users/heiland/swamp.txt
$ file ~/swamp.txt
/Users/heiland/swamp.txt: XML 1.0 document text, ASCII text, with very long lines