Cached And Distributed Execution

Tool for caching and distributed execution

Introduction

It works as a wrapper for executables to cache outputs based on its inputs.

fast lane

Create a .cade file in the working directory where your compiler is invoked. Prepend the call to cade to your compiler invocation.

For example, if your compiler command is:

gcc -I/path/to/include -o file.o -c file.c

Change it to:

cade gcc -I/path/to/include -o file.o -c file.c

An example .cade could look like:

{
  "cache": [
    {
      "filesystem": {
        "path": "C:/workspace/.cadec",
        "access": "ReadWrite"
      }
    }
  ]
}

How it works

The compile command is prepended with path to cade executable. cadewill:

1. check if a dependency file for the corresponding object file exists in cache
2. If found, it will create a hash based on the inputs (source file, compiler flags, included files) and check if an object file with the same hash exists in cache
3. If found, it will copy the object file, stderr and stdout from cache to the output location and replaces paths in stdout and stderr to match the current environment
4. If not found, it will execute the compile command, store the output file, dependency file, stdout and stderr in cache

Configuration

The configuration is done via a JSON file. See doc/schema.json for the schema.

The wrapper will look in the current working directory for a file named .cade.json.

The minumum configuration is:

{
  "cache": [
  ]
}

This will allow the tool to run, but it will be as you would compile without cade.

To enable caching, you need to add at least one cache backend. For example, to use a local directory as cache:

{
  "cache": [
    {
      "filesystem": {
        "path": "C:/workspace/.cadec",
        "access": "ReadWrite"
      }
    }
  ]
}

It is possible to use multiple cache backends. The order of the backends in the configuration file is the order in which they will be used.

For example:

{
  "cache": [
    {
      "filesystem": {
        "path": "C:/workspace/myproject/.cadec",
        "access": "ReadWrite"
      }
    },
    {
      "filesystem": {
        "path": "C:/workspace/.cadec",
        "access": "ReadWrite"
      }
    },
    {
      "redis": {
        "url": "server:6379",
        "expire": 604800,
        "access": "ReadWrite"
      }
    }
  ]
}

The first cache is project specific, the second is machine specific and the third is a shared cache for all users.

replace local paths

In case the cache is shared between multiple workspaces in different locations (either because the cache is shared with others our because the same code is checked out in different locations locally), you can replace local paths in stdout and stderr. This is done by configuring base_dir. It should be set to the path prefix of your local workspace which is not common between the workspaces. Typically this is the project root.

Logging

Cade supports logging stderr and stdout to a file. This can be used if you want to get access to compiler diagnostics (like warnings) for your build envoronment.

You can either log to a specific file per object file:

{
  "log": {
        "stderr": {
            "path": "{obj_path}.err",
            "append": false
        }
  },
  ...
}

or you can append all diagnostic outputs to a single file:

{
  "log": {
        "stderr": {
            "path": "compiler_diagnostics.log",
            "append": true
        }
  },
  ...
}

If you want to enable logging but not caching (e.g. for release builds), you can empty the cache array:

{
  "log": {
        "stderr": {
            "path": "compiler_diagnostics.log",
            "append": true
        }
  },
  "cache": [
  ]
}

See doc/schema.json for all available options.

supported compilers

Currently, the following compilers are supported:

• gcc
• tricore-gcc
• cctc (TASKING tricore)

New compilers can be added easily. Furthermore, it is planned to support other compilers via configuration file.

limitations

• Only single file compilation is supported. (Typilcal for build tools)
• Linking is not cached.
• Only compilers creating dependency files are supported. The dependency file must be passed to the compiler via command line argument.
• Compiler arguments are not checked if they are safe to be used for caching. The user is responsible to validate the arguments.

cmake integration

CMake is invoking the compiler in the CMAKE_BINARY_DIR. Therefore, you need to copy the .cade configuration file into the CMAKE_BINARY_DIR. You can do this by adding the following to your CMakeLists.txt:

configure_file(${CMAKE_SOURCE_DIR}/.cade ${CMAKE_BINARY_DIR}/.cade)

The side effect is that you can use CMake variables in your .cade file. For example, to set the base_dir to the source directory:

{
  "base_dir": "${CMAKE_SOURCE_DIR}",
  "cache": [
    {
      "filesystem": {
        "path": "${CMAKE_BINARY_DIR}/.cadec",
        "access": "ReadWrite"
      }
    }
  ]
}

Now, this configuration file can be added to your source control and will work for all developers working on the project.

You can include the provided example cmake file doc/cade.cmake in your CMakeLists.txt to setup cade automatically:

include(${PATH_TO_CADE}/doc/cade.cmake)