Moved to - Fika

Coolname (We really need to find a cool name)

Coolname is a tool for augmenting test suites of projects with specific focus on increasing coverage of third party API calls.

How it works

Components: Theo consists of a preprocessor, an api-finder, a coverage-checker and a test-generator(WIP).

• Preprocessor is maven plugin that creates a mapping of dependencies and package names at project build time. This lets us identify all the package names of the dependencies used in the project. Currently, the format of the mapping is a simple text file with each line containing a package name and the corresponding dependency.

{
  "org.bouncycastle.oer.its.etsi103097": [
    "org.bouncycastle:bcutil-jdk18on:jar:1.81"
  ],
  "org.mockito.internal.framework" : [ 
    "org.mockito:mockito-core:jar:5.19.0" 
  ]
}

However, sometimes multiple dependencies could have a package with the same name. In these cases, the most accurate way to determine the correct dependency is through dynamic analysis. At this stage, we just record all the dependency names without trying to figure out which classes are actually loaded by the classloader.

• API-finder scans the source code of the project to identify all the third party API calls. It uses the mapping generated by the preprocessor to determine which package names belong to third party dependencies. The output is a json list of all the third party API calls found in the source code in the following format (The format can be changed depending on the requirements for later steps).

{
  "thirdPartyPaths" : [
    {
      "entryPoint": "org.apache.pdfbox.io.RandomAccessReadView.read",
      "thirdPartyMethod": "org.apache.logging.log4j.LogManager.getLogger",
      "thirdPartyPackage": "org.apache.logging.log4j",
      "path": [
        "org.apache.pdfbox.io.RandomAccessReadView.read",
        "org.apache.pdfbox.io.ScratchFileBuffer.read",
        "org.apache.pdfbox.io.ScratchFileBuffer.ensureAvailableBytesInPage",
        "org.apache.pdfbox.io.ScratchFileBuffer.addPage",
        "org.apache.pdfbox.io.ScratchFile.getNewPage",
        "org.apache.pdfbox.io.ScratchFile.enlarge",
        "org.apache.pdfbox.io.ScratchFile.<clinit>",
        "org.apache.logging.log4j.LogManager.getLogger"
      ]
    }, {
      "entryPoint" : "org.apache.pdfbox.pdmodel.encryption.SecurityProvider.getProvider",
      "thirdPartyMethod" : "org.bouncycastle.jce.provider.BouncyCastleProvider.<init>",
      "thirdPartyPackage" : "org.bouncycastle.jce.provider",
      "path" : [ "org.apache.pdfbox.pdmodel.encryption.SecurityProvider.getProvider", "org.bouncycastle.jce.provider.BouncyCastleProvider.<init>" ]
    } ]
}

• Coverage-checker takes the list of third party API calls and checks the test suite to see if there are any tests that cover these API calls. It uses code coverage reports generated by the JaCoCo plugin to determine which lines of code are covered by the tests. The output is a json list of all the third party API calls that are covered/not covered by any tests in the following format.

{
  "total" : 4,
  "covered" : 1,
  "notCovered" : 3,
  "coveragePercent" : 25.0,
  "coveredMethods" : [ {
    "method" : "org.apache.logging.log4j.LogManager.getLogger",
    "entryPoint" : "org.apache.pdfbox.io.RandomAccessRead.peek"
  } ],
  "notCoveredMethods" : [ {
    "method" : "org.apache.logging.log4j.Logger.debug",
    "entryPoint" : "org.apache.pdfbox.io.IOUtils.closeQuietly"
  }, {
    "method" : "org.apache.logging.log4j.Logger.warn",
    "entryPoint" : "org.apache.pdfbox.io.RandomAccessRead.peek"
  }, {
    "method" : "org.apache.logging.log4j.Logger.error",
    "entryPoint" : "org.apache.pdfbox.io.RandomAccessInputStream.read"
  } ]
}

• Test-generator (WIP) will take the list of uncovered third party API calls and generate tests that cover these API calls. This is a work in progress and will be implemented in future versions.

Usage

1. Clone and build the project (Java 17+ and Maven 3+ required)

git clone https://github.com/yogyagamage/coolname.git
mvn clean install

2. Run the preprocessor on your project

cd path/to/your/maven/project
mvn io.github.chains-project:preprocessor-maven-plugin:1.0-SNAPSHOT:preprocess -DoutputFile=path/to/output/file.json

3. Run the api-finder on your project

java -jar path/to/api-finder/target/api-finder-1.0-SNAPSHOT-jar-with-dependencies.jar process -m path/to/preprocessor/output/file.json -p package.name -j path/to/project/jar/with/dependencies -c path/to/jacoco/report/directory -s path/to/project/src/main/java 

If Jacoco reports do not exist please run the tests with JaCoCo enabled first.

If any package name should be ignored (if there are submodules which should not be considered as third party dependencies), add them to the file api-finder/resources/ignored-packages.txt, one package name per line.

Future work

• Add support to process mutation testing reports generated by PIT
  • Check how to identify thrid party API coverage from PIT reports
  • Modify the coverage-checker to process PIT reports
• Implement the test-generator component to generate tests for uncovered third party API calls.
  • Select an appropriate LLM by referring to related work
  • Deploy the LLM
  • Experiment with different prompt engineering techniques
  • Implement a feedback loop to improve the generated tests
  • Integrate the best approach into the toolchain
• Evaluate the tool on a set of open source projects
  • Select a set of open source projects that use third party dependencies
  • Run the tool on these projects and measure the increase in coverage of third party API calls
  • Analyze the results
• Write tests for this project :)