Handle missing and changed includes in direct mode

[webmaster128]

Closes #200.
Closes #209.

[codecov-io]

Current coverage is 78.30% (diff: 65.45%)

Merging #179 into master will decrease coverage by 0.27%

@@             master       #179   diff @@
==========================================
  Files             1          1          
  Lines           929        954    +25   
  Methods           0          0          
  Messages          0          0          
  Branches        155        159     +4   
==========================================
+ Hits            730        747    +17   
- Misses          166        171     +5   
- Partials         33         36     +3   

Powered by Codecov. Last update e25b61a...8e16bcd

[webmaster128]

I just realized that it is perfectly possible that one given manifest contains different includes lists.

manifestHash = sourceFile + compilerBinary + commandLine

Let sourceFile include A and A include B, making an includes list of ['A', 'B']. Now a change in A can lead to B being unused and the includes list becomes ['A']. This can go in both directions and can alternate between multiple includes lists.

I do not really have an idea how to solve this properly.

[webmaster128]

Adding the includes list to manifestHash cannot easily be done because when restoring, we do not have the includes list available before opening the manifest file.

[Jimilian]

@webmaster128 Can you describe why we have includes list in first place? Why do we need all this manifest magic?

[webmaster128]

Sure. When the cache sees a compile request it needs to know whether or not the resulting object is already in cache. The source filepath is not enough, because the source file content can change. Thus we hash the source file content. But even when the source file did not change, we might need to recompile because an included file (typically a header file) is changed.

There are two ways to determine if source and includes are unchanged:

1. The preprocessed output is the same. When we hash the result of the preprocessor and the content is equal, we know that neither the source file nor the includes did change.
2. The source file and all includes did not change. We iterate over all includes and if non of the includes changed and the source file did not change, we can restore.

Mode 2. is known as DIRECT MODE and 1. is known as NODIRECT MODE. Even if nodirect is a bit more solid, it is slower than direct mode. In our setup (strong could CPU and RAM, slow NAS storage) direct mode reduces restore time of the project about 1/3. In setups where reading the include file is cheep compared to compute power (SSD or RAM disk), I expect direct mode to be even more valuable.

Dealing with non-existing includes is not trivial. I took me a while to break it down to two essential tests but I don't know how to implement it. Some time ago it was solved in a way that was probably incorrect. At the moment you get crashes for every missing include file.

To work around that, disable direct mode for now by setting the environment variable CLCACHE_NODIRECT (see README).

[webmaster128]

But I think I got it now: include changed must have priority over include missing. This can be archived by storing hashes with the includes list and when iterating over it, store a includeNotFound flag, which is used when no include changed. As soon as one include is changed, this has priority and the includeNotFound is discarded.

I'll try to implement this within a couple of days.

[Jimilian]

I got new exception two times on different nodes:

Traceback (most recent call last):
           File "clcache.py", line 1568, in <module>
           File "clcache.py", line 1447, in main
           File "clcache.py", line 1477, in processCompileRequest
           File "clcache.py", line 1547, in processDirect
           File "clcache.py", line 1530, in <lambda>
           File "clcache.py", line 1358, in postprocessHeaderChangedMiss
           File "clcache.py", line 1295, in addObjectToCache
           File "clcache.py", line 282, in setEntry
           File "clcache.py", line 777, in copyOrLink
         FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\clcache\\objects\\4c\\4cdbbffca1bd7564d2756447347f0389\\object.tmp' -> 'C:\\clcache\\objects\\4c\\4cdbbffca1bd7564d2756447347f0389\\object'
         Failed to execute script clcache

Maybe such error should be skipped if content is same?

[webmaster128]

There is indeed some time between hasEntry() and addObjectToCache(), such that this situation can happen. But I think this is an entirely independent issue. Could you please open a ticket for this?

[webmaster128]

@Jimilian See also #155. Other people are having the issue as well (on release or master versions of clcache)

[frerich]

It's kinda hard to review this PR with so much action going on. I think I have a good idea of what problem this PR is trying to solve, and I'm very happy that there's a test for this - so I'm tempted to just merge this, if it's ready to go?

[webmaster128]

I think you can. Tested the patch with a real project and other contributors tested it as well.

[frerich]

Merged, thanks everyone involved for the work! Let's see how this goes.

[siu]

I think a regression was introduced here. Previously postprocessHeaderChangedMiss received a manifest object already initialized with the current contents of the manifest file. In this PR instead a new manifest is created from scratch, removing from the manifest references to any other object that had the same includes but different hash for the contents of the includes.

[webmaster128]

I think a regression was introduced here. Previously postprocessHeaderChangedMiss received a manifest object already initialized with the current contents of the manifest file. In this PR instead a new manifest is created from scratch, removing from the manifest references to any other object that had the same includes but different hash for the contents of the includes.

Yes it is. In order to archive correctness, a missing optimization was introduced. Alternating include contents are overriding each other, triggering a recompile.

[siu]

The "issue" I have with this is that that is a very common case when switching branches. It could be solved quickly by reading the contents of the manifest file from disk before updating + writing the updated one. The cache would still be correct.

[webmaster128]

I was aware of this but then moved it to the low-priority list, given that having crashlessness and correctness is more important than this (I guess) edge-case optimization.

Let me add a PR adding a test for this, where we can continue the discussion.

UPDATE: thanks for pointing out the use case. I did not think about switching branches.

[siu]

Please take also a look at my comment in #208 #208 (comment)

[webmaster128]

#212