Features/windows agents with pathing issues fixed by jetersen · Pull Request #184 · jenkinsci/docker-workflow-plugin

jetersen · 2019-08-23T11:04:49Z

resolve #148
resolve #183

This is what we had issues with b0d3fae when testing out the branch made by @rbutcher

We have been running this in production since November/December of last year 😓

Allows us to use a declarative pipeline like

For a windows docker host

pipeline {
  agent {
    docker {
      label 'windows'
      image 'mcr.microsoft.com/powershell'
    }
  }
}

For a linux docker host

pipeline {
  agent {
    docker {
      label 'linux'
      image 'python'
    }
  }
}

- Fixing haphazard changes to Docker.groovy - Fixing FindBugs issue with annotations on WindowsDockerClient.launch()

jetersen · 2019-08-23T11:17:45Z

@dwnusbaum perhaps it would be best to setup import order and codestyle to avoid fighting over imports 😆

dwnusbaum · 2019-08-23T14:58:30Z

@Casz sure, if you want to open a PR to enforce some kind of import order, fine by me.

henryborchers · 2019-08-23T14:59:04Z

As someone who could REALLY USE Windows Docker agents, I'm extremely excited about this pull request. I've been wanting to integrate Windows docker agents for over a year now. However, I got so frustrated by the lack of interest from the maintainers of this plugin that I gave up waiting and started using bat commands to use it. @Casz, I really home you can get this PR accepted.

nvx · 2019-08-26T06:34:56Z

I've just run into the limitation of this plugin that it doesn't support windows and am glad to see a PR ready to fix it.

Is there anything else needed before this can be merged? This would make a lot of things easier for me.

res0nance · 2019-08-27T01:43:00Z

src/main/java/org/jenkinsci/plugins/docker/workflow/client/WindowsDockerClient.java

+            argb.add("-w", workdir);
+        }
+        for (Map.Entry<String, String> volume : volumes.entrySet()) {
+            argb.add("-v", volume.getKey() + ":" + volume.getValue());


Should we use --mount instead?

As an aside, I notice the regular docker client uses volumes instead of mounts as well. Should probably keep them consistent (which I suspect means switching DockerClient.java to mounts too mind you - and a separate PR)

jetersen · 2019-08-27T01:51:59Z

I'll properly end up spending the weekend fixing this up 😓

slide · 2019-09-13T19:12:13Z

src/main/resources/org/jenkinsci/plugins/docker/workflow/Docker.groovy


+    String shell() {
+        node {
+            script.isUnix() ? "sh" : "bat"


Would it be better to use powershell as the default for Windows based containers?

powershell or do you want pwsh? 😆

Actually, it looks like the powershell step currently only supports powershell.exe not pwsh.exe, so sticking with bat is the best option.

slide · 2019-09-13T19:14:02Z

Big +1 from me on this one.

jetersen · 2019-09-13T19:15:15Z

@slide I hope we can work on getting this tested in Jenkins CI :)

slide · 2019-09-13T19:16:46Z

@Casz what do you need for that? We have a label 'windock' for windows docker agents.

jetersen · 2019-09-13T19:20:43Z

Neat! All I would need, so good to know 😍

slide · 2019-09-13T20:28:44Z

Looks like several of the tests need to be updated to run only on Linux systems.

Nicabus · 2019-09-26T18:05:45Z

Click the button! We needses it! 😄

tiger5226 · 2019-09-29T17:42:53Z

@Casz I know this might be a big ask, but is there a way you could write up some docs on how to use this branch in an instance of jenkins? Maybe you should just fork this repo and publish this plugin? I would certainly use it. You can call it the docker4win-workflow-plugin

tiger5226 · 2019-09-29T19:41:16Z

I was actually able to get this repo forked, this PR merged, project build, and hpi file installed in jenkins in under a few hours. So not too bad at all. I can confirm...I have it working.

@Casz Not sure if it is related to this PR but I have to specify my registry instead of it assuming the global registry. So it inspects dockerhub first then tries my local repo. Not really a big deal though since it does try the globally referenced repo in the end. Again I doubt this is related, just wanted to mention it in case you it triggered something for you. Great work!

oleg-nenashev · 2019-10-04T16:28:11Z

I am about releasing it if there is a consensus.

gegles · 2019-10-10T01:17:34Z

This is so great, thanks guys, it works!

One feature request though, could you also make it work with agent { dockefile {} } ?
Even though it would probably take a while the first time to build the image (with Windows+Build tools containers) that would be awesome as we define our builder images Dockerfiles within the project codebase.

slide · 2019-10-10T01:22:02Z

@gegles Please file an issue in Jira at https://issues.jenkins.io. Adding additional comments to this PR is not really a good place to request features.

jetersen · 2019-10-10T04:55:37Z

@gegles it should already work.

Labeling agents becomes very important :)

I build this pipeline last week and been working great:

#! /usr/bin/env groovy

pipeline {
    agent {
        dockerfile {
            filename "Dockerfile.CLI"
            label "windows"
        }
    }
    stages {
        stage('Nuget restore') {
            steps {
                powershell "dotnet restore --source https://artifactory.company.com/api/nuget/v3/nuget"
            }
        }
        stage('Build solution') {
            steps {
                powershell "dotnet build -c:Release --no-restore -v:m"
            }
            post {
                always {
                    //  MSBuild warnings publisher
                    recordIssues tools: [msBuild()], enabledForFailure: true, qualityGates: [
                        [threshold: 1, type: 'TOTAL', unstable: false]
                    ]
                }
            }
        }
        stage('Pack') {
            steps {
                powershell 'remove-item ./publish/ -Force -Recurse -ErrorAction SilentlyContinue'
                powershell "dotnet publish projectCLI/projectCLI.csproj -c:Release -v n --no-restore --output publish"
                zip dir: 'publish', zipFile: 'publish/project.zip'
                archiveArtifacts 'publish/*.zip' 
            }//steps
        }//stage
    }//stages
}//pipeline

gegles · 2019-10-10T05:28:45Z

@Casz Thanks for your response. I tried again and here is what I see.

When I try using the following (with a pre-built image), it works:

stage('Windows') { 
   agent { 
      docker { 
         image 'my-win-builder' 
         label 'windows && docker && x86_64' 
     }
}

When I try with the following:

stage('Windows') { 
   agent { 
      dockerfile { 
         dir 'docker\\build\\windows' 
         label 'windows && docker && x86_64' 
     }
}

It fails with the classic Cannot run program "nohup" error.

Same issue if I move and rename my windows Dockerfile to the top-level dir:

stage('Windows') { 
   agent { 
      dockerfile { 
         filename 'Dockerfile.Windows' 
         label 'windows && docker && x86_64' 
     }
}

Any thoughts?

jetersen · 2019-10-10T05:38:16Z

oh it works for us because someone installed git with linux tools in our jenkins agent image 😓

cinst git --params \"/GitAndUnixToolsOnPath /SChannel\" -y --no-progress

I'll look into improving dockerfile then :)

gegles · 2019-10-10T05:41:47Z

I see. Glad you figured out the root cause. Yes, much appreciated if you can sort this out! I am not knowledgeable enough about Jenkins internals to do it myself... Let me know if there is a specific ticket to track this.

Cheers. G.

gegles · 2019-10-10T05:59:11Z

Ok, looking for a workaround, I've tried reinstalling git on our Windows agent the same way you did (and rebooted), but I still get the same error...

How do you guys connect Jenkins to the slave? we use ssh (even on windows).

albertosottile · 2019-10-10T05:59:33Z

@Casz Thank you very much for your efforts on this feature. Is it possible to use Windows containers also with the scripted pipeline syntax?

jetersen · 2019-10-10T06:03:23Z

@albertosottile should be the same.

@gegles we use JNLP

gegles · 2019-10-10T06:35:39Z

Ok, after some more trial and error, I got the workaround working by setting the DefaultShell for OpenSSH to the git bash shell:

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\Git\bin\bash.exe" -PropertyType String -Force

This will get us going for now. Thanks for all your work and for sharing.

oleg-nenashev · 2019-10-10T06:42:10Z

Thanks all. Would be great to get it documented for other adopters, better just in Github README for now

…

On Thu, Oct 10, 2019, 08:35 Guillaume Egles ***@***.***> wrote: Ok, after some more trial and error, I got the workaround working by setting the DefaultShell for OpenSSH to the git bash shell: New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\Git\bin\bash.exe" -PropertyType String -Force This will get us going for now. Thanks for all your work and for sharing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#184?email_source=notifications&email_token=AAW4RIEA4ERANBBLEXKNMOLQN3EMPA5CNFSM4IO6OVRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA27FOI#issuecomment-540406457>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAW4RIFXSTREXGABBOUYYNDQN3EMPANCNFSM4IO6OVRA> .

jetersen · 2019-10-10T07:08:44Z

Issue is located in https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/9b16261f04784ea7b538438f996c259340bb410c/pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy#L72

cc @bitwiseman

solvingj · 2019-10-10T11:59:36Z

Someone should tweet that this has been released, maybe Jenkins or cloudbees, we want to retweet.

jetersen · 2019-10-10T12:10:26Z

@solvingj Oleg already did: https://twitter.com/oleg_nenashev/status/1181991792352337926 😄

rbutcher · 2019-10-10T14:43:26Z

@Casz Thanks for taking this crusade over for me. After the chat over on my PR I thought this was dead in the water and have only been passively monitoring the progress of this. I was surprised when a buddy of mine at work told me this got merged yesterday. Crazy happy to see this merged. Thanks folks!

bverkron · 2019-10-10T16:49:45Z

It fails with the classic Cannot run program "nohup" error.

We ran into this as well. I had completely forgotten about it because I solved it while I was starting to refactor and rebuild the plugin myself before finding this PR.

I solved it via the following. Although this method is less ideal because it requires manual GUI steps. Powershell or other CLI methods are preferred for lots of reasons.

Right-click start menu > System > Advanced system settings > Environment variables > System variables (bottom section) > Select Path > Edit > New add this to a new entry C:\Program Files\Git\usr\bin

Log out / in or reboot

jetersen · 2019-10-10T16:52:59Z

I don't want workarounds 😄 I want actual fixes and the PR is already up: jenkinsci/pipeline-model-definition-plugin#354

bverkron · 2019-10-10T17:05:02Z

Excellent! That's the right way for sure!

bverkron · 2019-10-28T21:52:55Z

We have been running this PR for over a year now with no issues 🤔

Thanks, that's all the confirmation I needed. We likely won't troubleshoot this or submit a bug report because cmd itself is pretty antiquated and I'd much prefer to use something like powershell as our default anyway.

Ran into some situations where we might be forced to use bat. Submitted this issue.
https://issues.jenkins-ci.org/browse/JENKINS-59893

jetersen · 2019-11-05T18:55:50Z

@gegles should be solved in v1.4.0 of pipeline-model-definition

gegles · 2019-11-07T00:21:00Z

It seems tobe working!! Thank you!

penszo · 2020-03-18T08:23:56Z

Hi, so I'm trying to use it with latest version 1.23 of workflow plugin, and even with the simplest example of just running the container it fails to recognize that it is a windows not linux container:

Reason: Template provisioning failed.

com.github.dockerjava.api.exception.BadRequestException: {"message":"container 68e49ebb6925bc8c1f8ea92f2407843b6f94f35af8bc2c2ec5ad395c6efb0e9f encountered an error during CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {\"CommandLine\":\"/bin/sh\",\"WorkingDirectory\":\"C:\\\\\",\"EmulateConsole\":true,\"CreateStdInPipe\":true,\"CreateStdOutPipe\":true,\"ConsoleSize\":[0,0]}"}

jenkins script:

pipeline {
    agent {
        docker {
            label 'windows'
            image 'win-docker-tests'
        }
    }
    stages {
        stage('Docker Test') {
            steps {
                powershell "dir"
            }
        }
    }
}

any thoughts?

please help.

jetersen · 2020-03-18T08:26:12Z

@penszo please see the above comment above as there is a fix to that issue:
#184 (comment)
#184 (comment)

penszo · 2020-03-18T08:47:44Z

@jetersen thanks for super fast response.

Regarding your comments - I'm using the latest 1.6.0 version of this plugin. Still doesn't work for me :( and this is such a nice feature. I would love to have it up and running.

jetersen · 2020-03-18T09:16:30Z

@penszo I honestly only tested this using Windows Server 2019 since Microsoft recommends that for any docker workload

marianob85 · 2021-04-13T14:00:57Z

Hi, so I'm trying to use it with latest version 1.23 of workflow plugin, and even with the simplest example of just running the container it fails to recognize that it is a windows not linux container:

Reason: Template provisioning failed.

com.github.dockerjava.api.exception.BadRequestException: {"message":"container 68e49ebb6925bc8c1f8ea92f2407843b6f94f35af8bc2c2ec5ad395c6efb0e9f encountered an error during CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {\"CommandLine\":\"/bin/sh\",\"WorkingDirectory\":\"C:\\\\\",\"EmulateConsole\":true,\"CreateStdInPipe\":true,\"CreateStdOutPipe\":true,\"ConsoleSize\":[0,0]}"}

jenkins script:
pipeline {
    agent {
        docker {
            label 'windows'
            image 'win-docker-tests'
        }
    }
    stages {
        stage('Docker Test') {
            steps {
                powershell "dir"
            }
        }
    }
}
any thoughts?

please help.

@penszo Did you find a solution for this problem ? I'm struggling with the same problem on windows docker. Jenkins report:

com.github.dockerjava.api.exception.BadRequestException: {"message":"container f18410e6599a71a79288cea971da012df56ff85c64f363b8c5f20908d11201af encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2)\n[Event Detail: Provider: 00000000-0000-0000-0000-000000000000]\n[Event Detail: Provider: 00000000-0000-0000-0000-000000000000]\n[Event Detail: onecore\\vm\\compute\\management\\orchestration\\vmhostedcontainer\\processmanagement.cpp(173)\\vmcomputeagent.exe!00007FF753C89F4B: (caller: 00007FF753C3E13A) Exception(2) tid(394) 80070002 The system cannot find the file specified.\r\n CallContext:[\\Bridge_ProcessMessage\\VmHostedContainer_ExecuteProcess] \n Provider: 00000000-0000-0000-0000-000000000000] extra info: {\"CommandLine\":\"/bin/sh\",\"WorkingDirectory\":\"C:\\\\tmp\",\"Environment\":{\"COMPLUS_NGenProtectedProcess_FeatureEnabled\":\"0\",\"NUGET_VERSION\":\"5.8.1\",\"ROSLYN_COMPILER_LOCATION\":\"C:\\\\Program Files (x86)\\\\Microsoft Visual Studio\\\\2019\\\\BuildTools\\\\MSBuild\\\\Current\\\\Bin\\\\Roslyn\"},\"EmulateConsole\":true,\"CreateStdInPipe\":true,\"CreateStdOutPipe\":true,\"ConsoleSize\":[0,0]}"} at

rbutcher and others added 11 commits August 23, 2019 12:36

Added support to run Docker stages on Windows slaves

12f27b0

Updated changes to support docker.inside in reference to PR-98

3b3d798

Cleaning up imports

348b1d9

Fixing build issues

3bcadcd

- Fixing haphazard changes to Docker.groovy - Fixing FindBugs issue with annotations on WindowsDockerClient.launch()

Removing annotations

1722a14

Fixing DockerWindowsClient test_run()

2d19156

Updated whoAmI to properly get user on Windows

5c43bbe

Fixing findBugs issues with default charset

b9baff6

Fixed imports in WithContainerStep.java

20a2a26

fix windows path issue and skip whoami on Windows for now

b0d3fae

fix imports

6982db7

jetersen mentioned this pull request Aug 23, 2019

feat/windows slaves #148

Closed

res0nance reviewed Aug 27, 2019

View reviewed changes

jetersen mentioned this pull request Sep 13, 2019

Add Windows image packaging scripts jenkinsci/docker-ssh-agent#27

Merged

slide reviewed Sep 13, 2019

View reviewed changes

oleg-nenashev requested a review from slide October 4, 2019 16:28

jetersen deleted the features/windows_slaves branch October 9, 2019 18:56

jetersen mentioned this pull request Oct 10, 2019

[JENKINS-47703] - allow building dockerfile on Windows jenkinsci/pipeline-model-definition-plugin#354

Merged

jglick mentioned this pull request Nov 5, 2021

Simplify Docker.shell #243

Merged

jenkins-infra-bot mentioned this pull request Nov 8, 2019

[JENKINS-59893] bat calls hang in Windows Docker container in declarative pipeline script #629

Open

Comments

Conversation

jetersen commented Aug 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jetersen commented Aug 23, 2019

Uh oh!

dwnusbaum commented Aug 23, 2019

Uh oh!

henryborchers commented Aug 23, 2019

Uh oh!

nvx commented Aug 26, 2019

Uh oh!

res0nance Aug 27, 2019

Choose a reason for hiding this comment

Uh oh!

nvx Aug 27, 2019

Choose a reason for hiding this comment

Uh oh!

jetersen commented Aug 27, 2019

Uh oh!

slide Sep 13, 2019

Choose a reason for hiding this comment

Uh oh!

jetersen Sep 13, 2019

Choose a reason for hiding this comment

Uh oh!

slide Sep 13, 2019

Choose a reason for hiding this comment

Uh oh!

slide commented Sep 13, 2019

Uh oh!

jetersen commented Sep 13, 2019

Uh oh!

slide commented Sep 13, 2019

Uh oh!

jetersen commented Sep 13, 2019

Uh oh!

slide commented Sep 13, 2019

Uh oh!

Nicabus commented Sep 26, 2019

Uh oh!

tiger5226 commented Sep 29, 2019

Uh oh!

tiger5226 commented Sep 29, 2019

Uh oh!

oleg-nenashev commented Oct 4, 2019

Uh oh!

gegles commented Oct 10, 2019

Uh oh!

slide commented Oct 10, 2019

Uh oh!

jetersen commented Oct 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gegles commented Oct 10, 2019

Uh oh!

jetersen commented Oct 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gegles commented Oct 10, 2019

Uh oh!

gegles commented Oct 10, 2019

Uh oh!

albertosottile commented Oct 10, 2019

Uh oh!

jetersen commented Oct 10, 2019

Uh oh!

gegles commented Oct 10, 2019

Uh oh!

oleg-nenashev commented Oct 10, 2019 via email

Uh oh!

jetersen commented Oct 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

solvingj commented Oct 10, 2019

Uh oh!

jetersen commented Oct 10, 2019

Uh oh!

rbutcher commented Oct 10, 2019

jetersen commented Aug 23, 2019 •

edited

Loading

jetersen commented Oct 10, 2019 •

edited

Loading

jetersen commented Oct 10, 2019 •

edited

Loading

jetersen commented Oct 10, 2019 •

edited

Loading

jetersen commented Mar 18, 2020 •

edited

Loading