How to write to Partial sealed directory

[vladimir-cheverdyuk-altium]

Hi

I'm trying to run a tool from BuildXL that consumes files from some directory and writes new files to the same directory.
For what I read, it looks like I have to use partial sealed directory but I can't figure out how to specify that output of that tool should be that partial sealed directory.

For example:

const sealedDir = Transformer.sealPartialDirectory(dir1, globR(dir1));

I cannot specify sealedDir in outputs because it is not compatible. If I try to use sealedDir.root there, then BuildXL got some internal error.
If I try to use dir1 in outputs, I got error that dir1 coincides with the sealed directory.

What is correct way to use it?

Thank you,
Vlad

[smera]

Hi Vladimir,
For specifying inputs, you can either declare input files individually or use sealed directories. There are different flavors of sealed directories, and for inputs you can either use a full/partial one or source seal directories. At some level all these artifacts (including using individual files) are somewhat equivalent. Going finer-grained, the differences show up around how strict these declarations are (e.g. allowing outputs to be present in the same cone, letting any file be part of a tool sources as long as it is placed under a given cone, etc.). The other aspect of sealed directories vs individual files is that sealed directories allow you to treat all their members as a single entity (the seal directory) that you can reference and pass around.
On the other hand, for outputs, you have a similar choice. You either declare the outputs individually (and you can choose whether they are mandatory or optional) or use (sealed) output opaque directories. In your case, if sources and outputs are all intermixed, you can either declare the outputs explicitly or use shared opaque directories, which allows sources to be present in the same cone as well as multiple producers. The main difference between these approaches are

• Declaring individual output files will enable BuildXL to apply more static enforcements early in the build (e.g. detecting a double write before it actually happens, or enforcing that a tool is not producing outputs you were not expecting)
• Using an opaque directory does not require that you know all the outputs that will be produced upfront (you just need to a root, or set of roots, under which all the output files will be produced). In addition, most of the static validations that BuildXL performs will be turned into dynamic ones. So you'll still get them but they can't be statically predicted anymore and will start depending on execution behavior.

Thanks,
Serge

[vladimir-cheverdyuk-altium]

Hi Serge

Thank you for so detailed explanation.

My I ask slightly different question, I have solution in some directory and it contains multiple C++ projects that are located in sub-directories and during compilation each directory will have x64 directory created with intermediate output. I don't really care about that output because actual results will be produced in different directory.

Structure looks like this:

Root
  Prj1 source code
  Prj1\x64 - .lib, .obj etc
  Prj2 source code
  Prj2\x64 - .lib, .obj etc
....
  Prj100 source code
  Prj100\x64 - .lib, .obj etc
  root.sln
  root.vsprojx

What is the best to use in this case? I was trying to use sealed source code for Prj1 ... Prj100 and regular output directory for Prj1\x64 to Prj100\x64, but it is quite painful, specially considered that in reality each project has unique name.

I was planning to declare Root as sealed partial directory because x64 cleaned after checkout from VCS but I don't know how to specify intermediate output

Thank you,
Vlad

Vlad

[smera]

If you have multiple projects generating intermediate outputs under each project root, an easy way to deal with that which doesn't imply specifying each project root/individual outputs is to declare a catch-all shared opaque directory at the root of your source root. E.g. if you have projects src/prj1, src/prj2, etc. you can declare a shared opaque at 'src'. That would allow any output produced under that folder (and recursively below). This shared opaque can be declare for each pip (tool execution), and that means each pip will have an output directory that can be consumed by downstream tool. Each output directory (even though they will all share a common 'src' root) will contain only the outputs generated by the corresponding pip. So if you have a downstream tool that then creates a final deployment (presumably outside of the 'src' structure), this tool need to take a dependency on all the output directories representing intermediate outputs.
Hope this helps, please feel free to ping me if you have additional questions.

Thanks,
Serge.

[vladimir-cheverdyuk-altium]

How should I declare src directory? I tried as sealed source or seal partial and every time I got error

Vlad

[smera]

The problem with a source sealed directory is that it doesn't allow outputs to occur under it. A better candidate for your scenario is maybe a partial seal directory, where you can glob for all sources, and can be declares as an input for your pips.

Thanks,
Serge.

[vladimir-cheverdyuk-altium]

It does not work and I got error message:

C:\Projects\BuildXL\Test2\Hello.World.Project.dsc(49,33): error DX14109: Invalid graph since 'C:\Src', produced by 'Pip10B5B892974AE966, MSBuild.exe, HelloWorld, compileApiPip, {}', coincides with the sealed directory 'C:\Src', produced by 'Pip5A7E2A94930175B5, <SEALDIRECTORY>, HelloWorld, apiDir, {}, => Src(2765 entries) || 'Src' [2765 files - 0 output directories]'.

Code is here:

const apiDir = Transformer.sealPartialDirectory(apiRoot, globR(apiRoot));
const apiDirOut : Directory  = apiRoot;

export const compileApiPip = Transformer.execute({
    tool: msBuild16Tool,
    arguments: [
        Cmd.argument(Artifact.input(slnFile)),
        Cmd.argument("/target:Rebuild"),
        Cmd.argument("/p:Configuration=Production"),
        Cmd.argument("/p:Platform=x64")
    ],
    allowedSurvivingChildProcessNames: [
        "conhost.exe",
        "mspdbsrv.exe",
        "MSBuild.exe"
    ],
    dependencies: [
        prjFile,

        apiDir,
    ],
    implicitOutputs: [apiDirOut],
    outputs: [
        outDir
...

Perhaps I'm doing something wrong?

[smera]

Would you mind sharing the full code? I suspect 'outDir' in this case is an exclusive opaque directory, which wouldn't allow any inputs underneath. In order to specify that output as a shared opaque directory, you can follow the examples here.

[vladimir-cheverdyuk-altium]

outDir is mount to different directory.
I removed as much as possible from project and attached it.
Create empty C:\Code\Src\Api, and empty C:\Code\Installation and run project.

I got following error:

C:\Projects\BuildXL\Test3\Hello.World.Project.dsc(34,34): error DX14109: Invalid graph since 'C:\Code\Src\Api', produced by 'Pip58AFDC910DE2DC8E, MSBuild.exe, HelloWorld, compileApiPip, {}', coincides with the sealed directory 'C:\Code\Src\Api', produced by 'Pip4E08E5F54D725457, <SEALDIRECTORY>, HelloWorld, apiDir, {}, => Api(0 entries) || 'C:\Code\Src\Api' [0 files - 0 output directories]'.

Test3.zip

Thank you,
Vlad

[smera]

Test3-commented.zip
Tweaked some things and also added some general comments across the board. Please take a look and feel free to ask any questions you may have.

Thanks,
Serge.

[vladimir-cheverdyuk-altium]

Hi Serge

Thank you for your help. I found that I accidentally used wrong type of directory. Looks like I clicked on wrong link in Wiki.
I appreciate comments as well. I took config.dsc and module.config.dsc from HelloWorld example and I didn't change them.

I agree with comments for untrackedDirectories. I just need something working and then go to details. The same applies for qualifiers.

I have 2 more questions:

1. I see that content of outDir is wiped out. Usually Installation directory contains some static files that will be included in installation and PIP will just add dynamically generated files to it. Is it possible to prevent wiping this directory?
2. My understanding that whatever is produced by PIP will be put in cache with strong fingerprint of input. On next run, if strong fingerprint didn't change, BuildXL will just take data from cache. Is it possible to have some central location where this cache will be located so it can be used from different computers?

Vlad

[smera]

If 'outDir' will contain some files upfront, besides the ones produced by the tool, I suggest you turn that directory into a shared opaque one as well. Exclusive opaque directories are wiped out because they are not allowed to have sources or outputs from other tools (that the 'exclusive' part). So wiping them out guarantees build determinism. Making that a shared opaque will allow sources and other outputs to be there.

For having a shared cache across machines. Yes, that's an existent cache feature. Unfortunately, we don't have public documentation for it yet. In essence that's a service that has to be configured and has some tricky deployment steps. I opened up an internal work item to do some minimal documentation around this scenario, will let you know when we have that available.

Thanks,
Serge.

[vladimir-cheverdyuk-altium]

Thank you Serge. It works great with outDir. I will do more testings but looks like I have everything I need.

I will wait for documentation.

Thank you again for your help.

[vladimir-cheverdyuk-altium]

Hi Serge

I have another question about Test3-commented.zip:

        outputs: [
            // [Serge] You may want to turn this one into a share opaque as well if you plan to have other tools producing outputs under the same root
            outDir,
            // [Serge] changed this one to be a shared opaque. By default a directory being added as an output represents an exclusive opaque
            // An exclusive opaque directory does not allow sources or other opaque directories under its root. So the error you were getting
            // was about having this apiDirOut declared as an exclusive one, plus apiDir declaring sources under the same root
            {directory: apiDirOut, kind: "shared"}

Every build solution produces 63 Mb files into outDir. But it also produces 11 Gb intermediate content like lib, obj, pch and other files that nobody cares about. All of it goes to apiDirOut and looks like it placed into cache. As result, on second run BuildXL complains there is no space in cache. I can obviously increase cache size, but perhaps there is way to discard its content? It will save time registering that content in cache and obviously save a lot of space too.

Vlad

[smera]

Hi Vladimir,
As a general consideration, my recommendation is to increase the cache size. You can pass a cache configuration file with /cacheConfigFilePath:. Some extra points though:

• If there are produced files that are not consumed as part of the build (nor by post build consumers), you can untrack those directories. That will make bxl to not store them into the cache. This also mean they will not get replayed from the cache. So, if there are in-build consumers of those files, things will probably fail (depending on cache hit/miss behavior, and therefore not easy to spot).
• You can build with a folder-based filter (e.g. request bxl to filter out everything that doesn't produce stuff under 'outDir'. That won't change the cache size requirements but will instruct bxl to 1) not run pips that don't produce outputs under 'outDir' and 2) the default lazy materialization behavior will work such that files outside of 'outDir' won't get materialized from the cache unless there is a running pip that requires them.

Thanks,
Serge.

[vladimir-cheverdyuk-altium]

Hi Serge.
But if I will have so huge cache data and cache will be on different computer, then it means it will have to transfer 11 Gb to and from cache. It was my main concern with cache increase.

• What will be best way to untrack these directories? These are intermediate files and nobody consumes them. They are pretty much temp files required to create output and nothing more.
• How can I build such folder-based filter? If you can, just point me to documentation and I will try to figure it out.

Thank you,
Vlad