install_clcache_msbuild.py : helper for msbuild integration

[pthom]

Since the integration of clcache with msbuild is rather cumbersome,
this script provide an helper in order to simplify the process.

Notes:

• Incremental builds: There still is an issue with msbuild incremental build : we always get a full rebuild. A merge of the PR https://github.com/frerich/clcache/pull/319/commits would make this script more relevant : as of now, it cannot reliably be used from inside Visual Studio, since any file modification always triggers a full rebuild.
• This PR is a cousin of the pull request Feature prepare environment script #172, which I discovered long after I started to work on this

It is compatible with MSVC 2010, 2015 and 2017 (tested with MSVC 2015 and 2017).

What this script does:

• Check that python3 and pip3 are installed and are in the PATH
• Check that the pip installed scripts are in the PATH (PYTHONHOME\Scripts)
• Call pip install . from the repo and check that clcache is then in the PATH.
  clcache will subsequently be used from the PYTHONHOME\Scripts directory.
• Modify the user msbuild preference files inside %AppData%\..\Local\Microsoft\MSBuild\v4.0
  so that clcache becomes the default compiler. (These prefs are shared between MSVC 2010 to 2017).
• Find all cl.exes version on your computer (for MSVC 2010 to MSVC 2017), and allows you
  to select the correct one, by showing a detailed list of their version and target architecture.
• Set the env variable CLCACHE_CL with the correct path to cl.exe

As additional options, this script can also

• change the cache location
• change the cache size
• change the timeout CLCACHE_OBJECT_CACHE_TIMEOUT_MS

Caveat

Since the msbuild preference files inside %AppData%\..\Local\Microsoft\MSBuild\v4.0 are shared
between different MSVC installations, clcache will be activated for all instances of MSVC.

Note

vswhere.exe is a tool provided by Microsoft in order to locate installations of MSVC >= 2017.

Usage

usage: install_clcache_msbuild.py [-h] [--cachedir CACHEDIR]
                                  [--cache_size CACHE_SIZE]
                                  [--clcache_timeout CLCACHE_TIMEOUT]
                                  {status,install,disable,enable_logs,disable_logs,show_cl_list,select_cl}

[codecov-io]

Codecov Report

Merging #330 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #330   +/-   ##
=======================================
  Coverage   88.23%   88.23%           
=======================================
  Files           4        4           
  Lines        1275     1275           
  Branches      190      190           
=======================================
  Hits         1125     1125           
  Misses        111      111           
  Partials       39       39

Flag	Coverage Δ
#integrationtests_memcached	66.85% <ø> (ø)	⬆️
#unittests	85.25% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d62cc1...0ba3f16. Read the comment docs.

[pthom]

Hum, this script only works for Visual Studio 2015.
The installation and settings have varied a lot for Visual studio 2017.
I will try to study this.

[pthom]

Ok, this script is now compatible with MSVC 2015 and 2017 (and should also work with MSVC 2012).
It enables to select the correct cl.exe version by presenting a list of the available cl.exe compilers, sorted by version, host Arch and target Arch.

Sample usage session:

> python install_clcache_msbuild.py install
Looking for python in PATH
C:\Python36-32\python.exe
Looking for pip in PATH
C:\Python36-32\Scripts\pip.exe

######################################################################
Installing clcache (installClcache)
######################################################################
====> pip install .(in folder F:\dvp\OpenSource\clcache)
Processing f:\dvp\opensource\clcache
Requirement already satisfied: pymemcache in c:\python36-32\lib\site-packages (from clcache==4.1.1.dev65+g105e486.d20181119) (2.0.0)
Requirement already satisfied: pyuv in c:\python36-32\lib\site-packages (from clcache==4.1.1.dev65+g105e486.d20181119) (1.4.0)
Requirement already satisfied: six in c:\python36-32\lib\site-packages (from pymemcache->clcache==4.1.1.dev65+g105e486.d20181119) (1.11.0)
Installing collected packages: clcache
  Found existing installation: clcache 4.1.1.dev65+g105e486.d20181119
    Uninstalling clcache-4.1.1.dev65+g105e486.d20181119:
      Successfully uninstalled clcache-4.1.1.dev65+g105e486.d20181119
  Running setup.py install for clcache ... done
Successfully installed clcache-4.1.1.dev65+g105e486.d20181119
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Looking for clcache in PATH
C:\Python36-32\Scripts\clcache.exe

######################################################################
Force clcache via Msbbuild user settings (copyMsvcPrefClcache)
######################################################################
Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.ARM.user.props
Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.Win32.user.props
Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.x64.user.props

######################################################################
Select cl compiler: (selectCl)
######################################################################
   #           version targetArch   hostArch                                                              folder (shortened)
   1              14.0      amd64      amd64                                               C:\ProgX86\MSVC 14.0\vc\bin\amd64
   2              14.0        arm      amd64                                           C:\ProgX86\MSVC 14.0\vc\bin\amd64_arm
   3              14.0        x86      amd64                                           C:\ProgX86\MSVC 14.0\vc\bin\amd64_x86
   4              14.0      amd64        x86                                           C:\ProgX86\MSVC 14.0\vc\bin\x86_amd64
   5              14.0        arm        x86                                             C:\ProgX86\MSVC 14.0\vc\bin\x86_arm
   6              14.0        x86        x86                                                     C:\ProgX86\MSVC 14.0\vc\bin
   7   15.7.27703.2047        x64        x64       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx64\x64
   8   15.7.27703.2047        x86        x64       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx64\x86
   9   15.7.27703.2047        x64        x86       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx86\x64
  10   15.7.27703.2047        x86        x86       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx86\x86
Enter the number corresponding to the desired compiler: 6
Selected : C:\Program Files (x86)\Microsoft Visual Studio 14.0\vc\bin\cl.exe
====> SETX CLCACHE_CL "C:\Program Files (x86)\Microsoft Visual Studio 14.0\vc\bin\cl.exe"

SUCCESS: Specified value was saved.

######################################################################
Note about clcache usage: (showClCacheUsage)
######################################################################
====> clcache --help
clcache.py v4.1.0-dev
  --help    : show this help
  -s        : print cache statistics
  -c        : clean cache
  -C        : clear cache
  -z        : reset cache statistics
  -M <size> : set maximum cache size (in bytes)

[frerich]

@pthom This looks really useful! Alas, I must admit that I never used msbuild myself in the past, so I don't feel like I would be the best person to maintain the script. As such, I'm hesitant to merge the PR.

Maybe it could be hosted on your GitHub account and then I add a link to it in the Wiki?

[pthom]

Hi, thanks for you answer.
I understand your concern and your proposition sounds ok.
I propose to setup a different repo, where ccache will be a submodule (and initialy it will contain the branch with the fix for incremental builds). I think I will name it "clcache-msbuild".

What do you think ?

[pthom]

Hello,

I created a repo for this utility at https://github.com/pthom/clcache-msbuild-install

When running, it will clone clcache and the run the installation script (clcache is not a submodule, but it is cloned during the installation).

Can you tell me your opinion?

Note: Since I wanted to use the version of clcache that includes the correction for incremental build, I used a a fork of clcache (https://github.com/pthom/clcache/tree/clcache-msbuild-install). This fork is essentialy the same as the PR #319.

Can you tell me if you intend to merge the PR in the future ? If so, I will be able to use the original clcache repo in this tool :-)