Set up CI with Azure Pipelines

.gitignore

@@ -38,6 +38,7 @@ src/WebUI/dotnet/WebPortal/userconfig.json
 src/WebUI/dotnet/WebPortal/configAuth.json
 src/WebUI/dotnet/WebPortal/Master-Templates.json
 src/WebUI/dotnet/WebPortal/hosting.json
+**package-lock.json
 **/wwwroot/*
 **/bin/Release/*
 **/bin/Debug/*
@@ -61,3 +62,8 @@ src/WebUI/dotnet/WebPortal/hosting.json
 /.vs/DLWorkspace/v15/.suo
 
 cluster-autoscaler
+src/ClusterBootstrap/services/monitor/grafana-config.yaml
+src/ClusterBootstrap/services/monitor/prometheus-alerting.yaml
+src/ClusterBootstrap/services/monitor/alert-templates.yaml
+src/ClusterBootstrap/services/jobmanager/dlws-scripts.yaml
+src/ClusterBootstrap/services/monitor/alerting/kill-idle.rules

azure-pipelines.yml

@@ -0,0 +1,40 @@
+# Starter pipeline
+# Start with a minimal pipeline that you can customize to build and deploy your code.
+# Add steps that build, run tests, deploy, and more:
+# https://aka.ms/yaml
+
+trigger:
+- dltsdev
+
+pool:
+  name: 'DLTS-Platform'
+
+# container: ubuntu:18.04
+
+variables: { SUBSCRIPTION_NAME: "'Bing DLTS'" }
+
+steps:
+- script: |
+    cd src/ClusterBootstrap/
+    sudo ./install_prerequisites.sh
+    az account set --subscription $(SUBSCRIPTION_NAME)
+    az account list | grep -A5 -B5 '"isDefault": true'
+  displayName: 'Install prerequisites'
+
+- script: |
+    cd src/ClusterBootstrap/
+    cp /mnt/_work/dlts_ci_config.yaml config.yaml
+    ./bash_step_by_step_deploy.sh
+  displayName: 'Deploy DLWorkspace'
+
+- script: |
+    echo TODO: verify the cluster is ready!
+  displayName: 'Verify deployment'
+
+- script: |
+    echo TODO: RUN functional tests!
+  displayName: 'Functional tests'
+
+- script: |
+    echo TODO: cleanup the deployment!
+  displayName: 'Cleanup'

docs/deployment/Azure/FAQ.md

@@ -1,32 +1,64 @@
-# Frequently Asked Questions (FAQ) for Azure Cluster Deployment. 
-
-Please refer to [this](../knownissues/Readme.md) for more general deployment issues. 
-
-## After setup, I cannot visit the deployed DL Workspace portal. 
-
-* Please wait a few minutes after the deployment script runs through to allow the portal container to be pulled and scheduled for execution. 
-
-## I can't execute Spark job on Azure. 
-
-The current default deployment procedure on Azure doesn't deploy HDFS/Spark. So Spark job execution is not available. 
-
-## For 'az login', when I type in the device code, the web page prompt me again for the code. 
-
-It seems that sometime the browser (Edge, Chrome) cache another identity not intended to be used with az login. To get around, please start the browser in (in-private) or (incognito) mode, you may then enter the proper device code. 
-
-## I have launched a job (e.g., TensorFlow-iPython-GPU). However, I am unable to access the endpoint with error 
-
-    ```This site can’t be reached
-    ....cloudapp.azure.com refused to connect.
-    ```
-
-    Please check the docker image of the job you are running. Sometime, the iPython (or SSH server) hasn't been properly started, which caused the endpoint to be not accessible.  
-
-## I notice that my azure command is failing. 
-
-Azure CLI may time out after inactivity. You may need to re-login via 'az login'. 
-
-## Common configuration errors. 
-
-* "merge_config( config["azure_cluster"], tmpconfig["azure_cluster"][config["azure_cluster"]["cluster_name"]], verbose )"
+# Frequently Asked Questions (FAQ) for Azure Cluster Deployment. 
+
+Please refer to [this](../knownissues/Readme.md) for more general deployment issues. 
+
+## After setup, I cannot visit the deployed DL Workspace portal. 
+
+* Please wait a few minutes after the deployment script runs through to allow the portal container to be pulled and scheduled for execution. 
+
+## sudo ./az_tools.py create failed.
+
+* Check whether your subscription is correct. Always execute ```az account list | grep -A5 -B5 '"isDefault": true'``` to double check.
+
+## Lost connection at the very first step of deploying infra node to Azure, or ```./deploy.py runscriptonall ./scripts/prepare_vm_disk.sh```
+
+* Check whether hostname and source address in config.yaml are correctly set. Also try to make sure that you can ssh to the node.
+
+## I cannot ssh to the node when my devbox is a physical server instead of a virtual one.
+
+* Source IP address in config.yaml should probably be public IP, which could be derived by ```curl ifconfig.me```, instead of private IP you use to ssh to the devbox deriving from ```hostname -I```. If you cannot even ssh to the node after creating it, try to first set a new rule in Azure portal, allowing any source and destination IP, and set destination portal ranges to 22. Then ssh to the node, and type ```who``` to get the actual IP that is used to login to the node. Delete the temporary rule and in Networking setting, add <broaden IP>/16 to valid source IP, where <broaden IP> is the ```who``` IP with last two numbers set to 0. (e.g., 167.220.2.105 to 167.220.0.0/16)
+
+## How do I know the node has been deployed?
+
+* You can log into the master node: ```./deploy.py connect master```
+
+## I could not build docker image/No such image/An image does not exist locally with the tag/The repository XXX does not have a Release file
+
+* Check whether your docker is able to correctly resolve dns. First try on your devbox to ping a certain website, then do it in docker, such as `docker run -it busybox`,
+	if the former setting can ping but not the later one, try to figure out whether your devbox need to visit public Internet via some private DNS. Then edit it in `/etc/docker/daemon.json` on your devbox. refer to [this article](https://medium.com/@faithfulanere/solved-docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-anything-9ea4dfdcdcf2)
+	use `systemd-resolve --status` to get more info about DNS if it is not managed by network-manager
+
+## I can connect master/infra node, but the UI is not working (cannot access from browser), how to debug?
+
+* Login to the master node, and use ```docker ps | grep web``` to get the ID corresponding to Web UI, then use ```docker logs --follow <WebUI ID>``` to figure out what happened.
+  a better way is to use ```sudo docker logs --tail 100 --follow $(sudo docker ps | grep webui | awk '{print $1}') ``` since the ID would change everytime the docker is restarted.
+  Everytime after modifying /etc/WebUI/userconfig.json etc., remember to restart that docker image: ```docker rm -f <WebUI ID>```
+
+## finished all deployment, but not able to connect to master node via ```./deploy.py connect master```, ssh denied even with ``` ssh -i deploy/sshkey/id_rsa core@<infra node url>```.  
+
+* Need to change owner ```sudo chown -R <usr_name>:<usr_name> DLWorkspace/```, can check ownership using ```ls -l```
+
+## I can't execute Spark job on Azure. 
+
+* The current default deployment procedure on Azure doesn't deploy HDFS/Spark. So Spark job execution is not available. 
+
+## For 'az login', when I type in the device code, the web page prompt me again for the code. 
+
+* It seems that sometime the browser (Edge, Chrome) cache another identity not intended to be used with az login. To get around, please start the browser in (in-private) or (incognito) mode, you may then enter the proper device code. 
+
+## I have launched a job (e.g., TensorFlow-iPython-GPU). However, I am unable to access the endpoint with error 
+
+    ```This site can’t be reached
+    ....cloudapp.azure.com refused to connect.
+    ```
+
+    Please check the docker image of the job you are running. Sometime, the iPython (or SSH server) hasn't been properly started, which caused the endpoint to be not accessible.  
+
+## I notice that my azure command is failing. 
+
+Azure CLI may time out after inactivity. You may need to re-login via 'az login'. 
+
+## Common configuration errors. 
+
+* "merge_config( config["azure_cluster"], tmpconfig["azure_cluster"][config["azure_cluster"]["cluster_name"]], verbose )"
   Please check if the cluster_name used in azure_cluster is the same as the DL workspace cluster name.

docs/deployment/Azure/Readme.md

@@ -4,20 +4,36 @@ This document describes the procedure to deploy a DL Workspace cluster on Azure.
 
 Please note that the procedure below doesn't deploy HDFS/Spark on DLWorkspace cluster on Azure (Spark job execution is not available on Azure Cluster).
 
-1. Follow [this document](../../DevEnvironment/Readme.md) to setup the dev environment of DLWorkspace. Login to your Azure subscription on your dev machine via:
+Prerequisite steps:
+First require the manager to add you into a subscription group., then either 
+1. go to that group from Azure Portal and add ubuntu server from resources, this virtual server is your devbox, or 
+2. if you have a physical machine, install ubuntu server system(18.04) on that and use it as your devbox
+then use the devbox to deploy node on cloud.
 
+Workflow:
+1. Please [configure](configure.md) your azure cluster. Put config.yaml under src/ClusterBootstrap
+
+2. Change directory to src/ClusterBootstrap on devbox, and install prerequisite packages:
+```
+cd src/ClusterBootstrap/ 
+sudo ./install_prerequisites.sh
 ```
+3. Login to Azure, setup proper subscription and confirm
+```
+SUBSCRIPTION_NAME="<subscription name>" 
 az login
+az account set --subscription "${SUBSCRIPTION_NAME}" 
+az account list | grep -A5 -B5 '"isDefault": true'
 ```
-
-2. Please [configure](configure.md) your azure cluster. 
-
-3. Set proper [authentication](../authentication/Readme.md).
-
-4. Initial cluster and generate certificates and keys:
+Configure your location, should be the same as you specified in config.yaml file:
+```AZ_LOCATION="<your location>"```
+Execute this command, log out(exit) and log in back
+```sudo usermod -aG docker zhe_ms```
+4. Initiate cluster and generate certificates and keys:
 ```
 ./deploy.py -y build
 ```
+
 5. Create Azure Cluster:
 ```
 ./az_tools.py create
@@ -40,9 +56,10 @@ Please note that if you are not Microsoft user, you should remove the
   ```
   where machine1 is your azure infrastructure node. (you may get the address by ./deploy.py display)
 
-  The script block execute the following command in sequences: (you do NOT need to run the following commands if you have run step 5)
-  1. Setup basic tools on the Ubuntu image. 
+  This command sequetially execute following steps:
+  1. Setup basic tools on VM and on the Ubuntu image. 
   ```
+  ./deploy.py runscriptonall ./scripts/prepare_vm_disk.sh
   ./deploy.py runscriptonall ./scripts/prepare_ubuntu.sh
   ```
 
@@ -57,16 +74,28 @@ Please note that if you are not Microsoft user, you should remove the
   ./deploy.py -y kubernetes labels
   ```
 
-  4. Build and deploy jobmanager, restfulapi, and webportal. Mount storage.
+  4. Start Nvidia device plugins:
   ```
+  ./deploy.py kubernetes start nvidia-device-plugin
+  ```
+
+  5. Build and deploy jobmanager, restfulapi, and webportal. Mount storage.
+  ```
+  ./deploy.py webui
   ./deploy.py docker push restfulapi
   ./deploy.py docker push webui
-  ./deploy.py webui
   ./deploy.py mount
   ./deploy.py kubernetes start jobmanager restfulapi webportal
   ```
 
-8. If you run into a deployment issue, please check [here](FAQ.md) first. 
-
-9. If you want to deploy a DLWorkspace cluster that can be autoscaled (i.e., automatically create/release VM when needed), please follow the following additional steps.
-
+8.  Manually connect to the infrastructure/master node:
+  ```./deploy.py connect master```
+  On master node(log in from devbox by ./deploy.py connect master), manually add ```"Grafana": "",``` to /etc/WebUI/userconfig.json, under "Restapi" entry.
+  Restart the WebUI docker:
+  Login to the master node, and use
+  ```docker ps | grep web``` 
+  to get the ID corresponding to Web UI, then restart that docker image: 
+  ```docker rm -f <WebUI ID>```
+  Wait for minutes for it to restart (can follow by using ```docker logs --follow <WebUI ID>```) and visit the infra node from web browser.
+
+9. If you run into a deployment issue, please check [here](FAQ.md) first.

docs/deployment/Azure/auto_scale.md

@@ -1,31 +1,31 @@
-# The following describe the procedures to autoscale a DLWorkspace cluster (i.e., automatically create/release VM when needed).
-
-1. Download auto_scaler binary
-   ```
-   wget https://github.com/DLWorkspace/autoscaler/releases/download/v1.9.0/cluster-autoscaler
-   ```
-
-2. Setup azure running environment and login (via az login)
-
-3. For the Azure machine types supported, please check the document at:
-   ```
-   src/ClusterBootstrap/templates/machine-types/azure/machineTypes.yaml
-   ```
-
-   A sample template is as follows. Please fill in additional worker VM SKUs if you need. 
-
-   ```
-    ---
-    Standard_NC6:
-    cpu: 6
-    memoryInMb: 56339
-    gpu: 1
-    Standard_D3_v2:
-    cpu: 4
-    memoryInMb: 14339
-  ```
-
-4. Start auto_scaler:
-   ```
-    ./cluster-autoscaler --v=5 --stderrthreshold=error --logtostderr=true --cloud-provider=aztools --skip-nodes-with-local-storage=false --nodes=0:10:Standard_NC6 --nodes=0:10:Standard_D3_v2 --leader-elect=false --scale-down-enabled=true --kubeconfig=./deploy/kubeconfig/kubeconfig.yaml --expander=least-waste
+# The following describe the procedures to autoscale a DLWorkspace cluster (i.e., automatically create/release VM when needed).
+
+1. Download auto_scaler binary
+   ```
+   wget https://github.com/DLWorkspace/autoscaler/releases/download/v1.9.0/cluster-autoscaler
+   ```
+
+2. Setup azure running environment and login (via az login)
+
+3. For the Azure machine types supported, please check the document at:
+   ```
+   src/ClusterBootstrap/templates/machine-types/azure/machineTypes.yaml
+   ```
+
+   A sample template is as follows. Please fill in additional worker VM SKUs if you need. 
+
+   ```
+    ---
+    Standard_NC6:
+    cpu: 6
+    memoryInMb: 56339
+    gpu: 1
+    Standard_D3_v2:
+    cpu: 4
+    memoryInMb: 14339
+  ```
+
+4. Start auto_scaler:
+   ```
+    ./cluster-autoscaler --v=5 --stderrthreshold=error --logtostderr=true --cloud-provider=aztools --skip-nodes-with-local-storage=false --nodes=0:10:Standard_NC6 --nodes=0:10:Standard_D3_v2 --leader-elect=false --scale-down-enabled=true --kubeconfig=./deploy/kubeconfig/kubeconfig.yaml --expander=least-waste
    ```