Merge pull request #687 from ananto-msft/master

Adding sample deployment script for Azure Arc Data Controller private preview
This commit is contained in:
Umachandar Jayachandran
2019-11-25 19:19:55 -08:00
committed by GitHub
9 changed files with 618 additions and 0 deletions
+4
View File
@@ -0,0 +1,4 @@
# Azure Arc Data Controller cluster
## __[Sample setup and deployment](deployment/)__
This folder contains necessary instruction to setup and deploy sample Azure Arc Data Controller cluster.
@@ -0,0 +1,8 @@
# Creating a Kubernetes cluster for Azure Arc Data Controller cluster
Azure Arc Data Controller cluster is deployed as docker containers on a Kubernetes cluster. These samples provide scripts that can be used to provision a Kubernetes clusters using different environments.
## __[Deploy a Kubernetes cluster using kubeadm](kubeadm/)__
Use the scripts in the **kubeadm** folder to deploy a Kubernetes cluster over one or more Linux machines (physical or virtualized) using `kubeadm` utility.
@@ -0,0 +1,10 @@
# Create a Kubernetes cluster using Kubeadm on Ubuntu 16.04 LTS or 18.04 LTS
## __[ubuntu](ubuntu/)__
This folder contains scripts that provide a template for deploying a Kubernetes cluster using kubeadm on one or more Linux machines.
## __[ubuntu-single-node-vm](ubuntu-single-node-vm/)__
This folder contains a sample script that can be used to create a single-node Kubernetes cluster on a Linux machine and deploy Azure Arc Data Controller cluster.
@@ -0,0 +1,47 @@
# Deploy a Azure Arc Data Controller on single node Kubernetes cluster (kubeadm)
Using this sample bash script, you will deploy a single node Kubernetes cluster using kubeadm and a Azure Arc Data Controller on top of it. The script must be run from the VM you are planning to use for your kubeadm deployment.
## Pre-requisites
1. A vanilla Ubuntu 16.04 or 18.04 virtual or physical machine. All dependencies will be setup by the script. Using Azure Linux VMs is not yet supported.
1. Machine should have at least 8 CPUs, 64GB RAM and 100GB disk space. After installing the images you will be left with 50GB for data/logs across all components.
1. Update existing packages using commands below to ensure that the OS image is up to date
``` bash
sudo apt update&&apt upgrade -y
sudo systemctl reboot
```
## Recommended Virtual Machine settings
1. Use static memory configuration for the virtual machine. For example, in hyper-v installations do not use dynamic memory allocation but instead allocate the recommended 64 GB or higher.
1. Use checkpoint or snapshot capability in your hyper visor so that you can rollback the virtual machine to a clean state.
## Instructions to deploy Azure Arc Data Controller
1. Download the script on the VM you are planning to use for the deployment
``` bash
curl --output setup-controller.sh https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/azure-arc/deployment/kubeadm/ubuntu-single-node-vm/setup-controller.sh
```
2. Make the script executable
``` bash
chmod +x setup-controller.sh
```
3. Run the script (make sure you are running with sudo)
``` bash
sudo ./setup-controller.sh
```
When prompted, provide your input for the password that will be used for all external endpoints: controller, SQL Server master and gateway. The password should be sufficiently complex based on existing rules for SQL Server password. The controller username is defaulted to *controlleradmin*.
## Cleanup
1. The [cleanup-controller.sh](cleanup-controller.sh/) script is provided as convenience to reset the environment in case of errors. However, we recommend that you use a virtual machine for testing purposes and use the snapshot capability in your hyper-visor to rollback the virtual machine to a clean state.
@@ -0,0 +1,98 @@
#!/bin/bash
if [ "$EUID" -ne 0 ]
then echo "Please run as root"
exit
fi
DIR_PREFIX=$1
kubeadm reset --force
# Clean up azdata-cli package.
#
unalias azdata
sudo dpkg --remove --force-all azdata-cli
systemctl stop kubelet
rm -rf /var/lib/cni/
rm -rf /var/lib/etcd/
rm -rf /run/flannel/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
rm -rf /etc/kubernetes/
ip link set cni0 down
#brctl delbr cni0
ip link set flannel.1 down
#brctl delbr flannel.1
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
rm -rf .azdata/
# Remove mounts.
#
SERVICE_STOP_FAILED=0
systemctl | grep "/var/lib/kubelet/pods" | while read -r line; do
# Retrieve the mount path
#
MOUNT_PATH=`echo "$line" | grep -v echo | egrep -oh -m 1 "(/var/lib/kubelet/pods).+"`
if [ -z "$MOUNT_PATH" ]; then
continue
fi
if [[ ! -d "$MOUNT_PATH" ]] && [[ ! -f "$MOUNT_PATH" ]]; then
SERVICE=$(echo $line | cut -f1 -d' ')
echo "Mount "$MOUNT_PATH" no longer exists."
echo "Stopping orphaned mount service: '$SERVICE'"
systemctl stop $SERVICE
if [ $? -ne 0 ]; then
SERVICE_STOP_FAILED=1
fi
echo ""
fi
done
if [ $SERVICE_STOP_FAILED -ne 0 ]; then
echo "Not all services were stopped successfully. Please check the above output for more inforamtion."
else
echo "All orphaned services successfully stopped."
fi
# Clean the mounted volumes.
#
for i in $(seq 1 40); do
vol="vol$i"
sudo umount /mnt/local-storage/$vol
sudo rm -rf /mnt/local-storage/$vol
done
# Reset kube
#
sudo apt-get purge -y kubeadm --allow-change-held-packages
sudo apt-get purge -y kubectl --allow-change-held-packages
sudo apt-get purge -y kubelet --allow-change-held-packages
sudo apt-get purge -y kubernetes-cni --allow-change-held-packages
sudo apt-get purge -y kube* --allow-change-held-packages
sudo apt -y autoremove
sudo rm -rf ~/.kube
# Clean up working folders.
#
export AZUREARCDATACONTROLLER_DIR=aadatacontroller
if [ -d "$AZUREARCDATACONTROLLER_DIR" ]; then
echo "Removing working directory $AZUREARCDATACONTROLLER_DIR."
rm -f -r $AZUREARCDATACONTROLLER_DIR
fi
@@ -0,0 +1,285 @@
#!/bin/bash
set -Eeuo pipefail
if [ "$EUID" -ne 0 ]
then echo "Please run as root"
exit
fi
# This is a script to create single-node Kubernetes cluster and deploy Azure Arc Data Controller on it.
#
export AZUREARCDATACONTROLLER_DIR=aadatacontroller
# Get password as input. It is used as default for controller, SQL Server Master instance (sa account).
#
while true; do
read -s -p "Create Password for Azure Arc Data Controller: " password
echo
read -s -p "Confirm your Password: " password2
echo
[ "$password" = "$password2" ] && break
echo "Password mismatch. Please try again."
done
# Name of virtualenv variable used.
#
export LOG_FILE="aadatacontroller.log"
export DEBIAN_FRONTEND=noninteractive
# Requirements file.
export AZDATA_PRIVATE_PREVIEW_DEB_PACKAGE="https://aka.ms/aadatacontrollerazdata"
# Kube version.
#
KUBE_DPKG_VERSION=1.16.3-00
KUBE_VERSION=1.16.3
# Wait for 5 minutes for the cluster to be ready.
#
TIMEOUT=600
RETRY_INTERVAL=5
# Variables used for azdata cluster creation.
#
export CONTROLLER_USERNAME=controlleradmin
export CONTROLLER_PASSWORD=$password
export ACCEPT_EULA=yes
export CLUSTER_NAME=azure-arc-system
export PV_COUNT="40"
# Make a directory for installing the scripts and logs.
#
rm -f -r $AZUREARCDATACONTROLLER_DIR
mkdir -p $AZUREARCDATACONTROLLER_DIR
cd $AZUREARCDATACONTROLLER_DIR/
touch $LOG_FILE
{
# Install all necessary packages: kuberenetes, docker, request, azdata.
#
echo ""
echo "######################################################################################"
echo "Starting installing packages..."
# Install docker.
#
apt-get update -q
apt --yes install \
software-properties-common \
apt-transport-https \
ca-certificates \
curl
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt update -q
apt-get install -q --yes docker-ce=18.06.2~ce~3-0~ubuntu --allow-downgrades
apt-mark hold docker-ce
usermod --append --groups docker $USER
# Prompt for private preview repository username and password provided by Microsoft
#
read -p 'Enter Azure Arc Data Controller repo username provided by Microsoft:' AADC_USERNAME
read -sp 'Enter Azure Arc Data Controller repo password provided by Microsoft:' AADC_PASSWORD
export DOCKER_USERNAME=$AADC_USERNAME
export DOCKER_PASSWORD=$AADC_PASSWORD
# Create working directory
#
rm -f -r setupscript
mkdir -p setupscript
cd setupscript/
# Download and install azdata package
#
curl --location $AZDATA_PRIVATE_PREVIEW_DEB_PACKAGE --output azdata_setup.deb
sudo dpkg -i azdata_setup.deb
cd -
azdata --version
echo "Azdata has been successfully installed."
# Load all pre-requisites for Kubernetes.
#
echo "###########################################################################"
echo "Starting to setup pre-requisites for kubernetes..."
# Setup the kubernetes preprequisites.
#
echo $(hostname -i) $(hostname) >> /etc/hosts
swapoff -a
sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
# Install docker and packages to allow apt to use a repository over HTTPS.
#
apt-get update -q
apt-get install -q -y ebtables ethtool
#apt-get install -y docker.ce
apt-get install -q -y apt-transport-https
# Setup daemon.
#
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart docker.
#
systemctl daemon-reload
systemctl restart docker
apt-get install -q -y kubelet=$KUBE_DPKG_VERSION kubeadm=$KUBE_DPKG_VERSION kubectl=$KUBE_DPKG_VERSION
# Holding the version of kube packages.
#
apt-mark hold kubelet kubeadm kubectl
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
. /etc/os-release
if [ "$UBUNTU_CODENAME" == "bionic" ]; then
modprobe br_netfilter
fi
# Disable Ipv6 for cluster endpoints.
#
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
echo net.ipv6.conf.all.disable_ipv6=1 >> /etc/sysctl.conf
echo net.ipv6.conf.default.disable_ipv6=1 >> /etc/sysctl.conf
echo net.ipv6.conf.lo.disable_ipv6=1 >> /etc/sysctl.conf
sysctl net.bridge.bridge-nf-call-iptables=1
# Setting up the persistent volumes for the kubernetes.
#
for i in $(seq 1 $PV_COUNT); do
vol="vol$i"
mkdir -p /mnt/local-storage/$vol
mount --bind /mnt/local-storage/$vol /mnt/local-storage/$vol
done
echo "Kubernetes pre-requisites have been completed."
# Setup kubernetes cluster including remove taint on master.
#
echo ""
echo "#############################################################################"
echo "Starting to setup Kubernetes master..."
# Initialize a kubernetes cluster on the current node.
#
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=$KUBE_VERSION
mkdir -p $HOME/.kube
mkdir -p /home/$SUDO_USER/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u $SUDO_USER):$(id -g $SUDO_USER) $HOME/.kube/config
# To enable a single node cluster remove the taint that limits the first node to master only service.
#
master_node=`kubectl get nodes --no-headers=true --output=custom-columns=NAME:.metadata.name`
kubectl taint nodes ${master_node} node-role.kubernetes.io/master:NoSchedule-
# Local storage provisioning.
#
kubectl apply -f https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/azure-arc/deployment/kubeadm/ubuntu/local-storage-provisioner.yaml
# Install the software defined network.
#
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# helm init
#
kubectl apply -f https://raw.githubusercontent.com/microsoft/sql-server-samples/master/samples/features/azure-arc/deployment/kubeadm/ubuntu/rbac.yaml
# Verify that the cluster is ready to be used.
#
echo "Verifying that the cluster is ready for use..."
while true ; do
if [[ "$TIMEOUT" -le 0 ]]; then
echo "Cluster node failed to reach the 'Ready' state. Kubeadm setup failed."
exit 1
fi
status=`kubectl get nodes --no-headers=true | awk '{print $2}'`
if [ "$status" == "Ready" ]; then
break
fi
sleep "$RETRY_INTERVAL"
TIMEOUT=$(($TIMEOUT-$RETRY_INTERVAL))
echo "Cluster not ready. Retrying..."
done
# Install the dashboard for Kubernetes.
#
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
echo "Kubernetes master setup done."
# Deploy azdata Azure Arc Data Cotnroller create cluster.
#
echo ""
echo "############################################################################"
echo "Starting to deploy azdata cluster..."
# Command to create cluster for single node cluster.
#
azdata control create -n $CLUSTER_NAME -c azure-arc-data-kubeadm-private-preview --accept-eula $ACCEPT_EULA
echo "Azure Arc Data Controller cluster created."
# Setting context to cluster.
#
kubectl config set-context --current --namespace $CLUSTER_NAME
# Login and get endpoint list for the cluster.
#
azdata login -n $CLUSTER_NAME
if [ -d "$HOME/.azdata/" ]; then
sudo chown -R $(id -u $SUDO_USER):$(id -g $SUDO_USER) $HOME/.azdata/
fi
echo "Cluster successfully setup. Run 'azdata --help' to see all available options."
}| tee $LOG_FILE
@@ -0,0 +1,25 @@
# Create a Kubernetes cluster using Kubeadm on Ubuntu 16.04 LTS or 18.04 LTS
In this example, we will deploy Kubernetes over multiple Linux machines (physical or virtualized) using kubeadm utility. These instructions have been tested primarily with Ubuntu 16.04 LTS & 18.04 LTS versions.
## Pre-requisites
1. Multiple Ubuntu Linux machines or virtual machines. Recommended configuration is 8 CPUs, 32 GB memory each and at least 100 GB storage for each machine. Minimum number of machines required is three machines
1. Designate one machine as the Kubernetes master
1. Rest of the machines will be used as the Kubernetes agents
**NOTE: Ensure there is sufficient local storage on your agents. Each volume will use up to 10GB by default. The script creates 25 volumes. Not all of the volumes will be used since it depends on the number of pods being deployed on each agent node. It is recommended to have at least 200 GB of storage on the agent nodes**
### Useful resources
[Creating a cluster using kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
[Troubleshooting kubeadm](https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/)
### Instructions
1. Start a sudo shell context and Execute [setup-k8s-prereqs.sh](setup-k8s-prereqs.sh/) script on each machine
1. Execute [setup-k8s-master.sh](setup-k8s-master.sh/) script on the machine designated as Kubernetes master (_not_ under sudo su as otherwise you'll setup K8S .kube/config permissions for root)
1. After successful initialization of the Kubernetes master, follow the kubeadm join commands output by the setup script on each agent machine
1. Execute [setup-volumes-agent.sh](setup-volumes-agent.sh/) script on each agent machine to create volumes for local storage
1. Execute ***kubectl apply -f local-storage-provisioner.yaml*** against the Kubernetes cluster to create the local storage provisioner. This will create a Storage Class named "local-storage".
@@ -0,0 +1,129 @@
apiVersion: v1
kind: Namespace
metadata:
name: local-storage
---
apiVersion: v1
kind: Service
metadata:
name: local-volume-provisioner
namespace: local-storage
labels:
app: local-volume-provisioner
spec:
type: ClusterIP
selector:
app: local-volume-provisioner
ports:
- name: metrics
port: 8080
protocol: TCP
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
apiVersion: v1
kind: ConfigMap
metadata:
name: local-provisioner-config
namespace: local-storage
data:
storageClassMap: |
local-storage:
hostDir: /mnt/local-storage
mountDir: /mnt/local-storage
blockCleanerCommand:
- "/scripts/shred.sh"
- "2"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: local-volume-provisioner
namespace: local-storage
labels:
app: local-volume-provisioner
spec:
selector:
matchLabels:
app: local-volume-provisioner
template:
metadata:
labels:
app: local-volume-provisioner
spec:
serviceAccountName: local-storage-admin
containers:
- image: "quay.io/external_storage/local-volume-provisioner:v2.1.0"
imagePullPolicy: "Always"
name: provisioner
securityContext:
privileged: true
env:
- name: MY_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- mountPath: /etc/provisioner/config
name: provisioner-config
readOnly: true
- mountPath: /mnt/local-storage
name: local-storage
mountPropagation: "HostToContainer"
volumes:
- name: provisioner-config
configMap:
name: local-provisioner-config
- name: local-storage
hostPath:
path: /mnt/local-storage
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: local-storage-admin
namespace: local-storage
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: local-storage-provisioner-pv-binding
namespace: local-storage
subjects:
- kind: ServiceAccount
name: local-storage-admin
namespace: local-storage
roleRef:
kind: ClusterRole
name: system:persistent-volume-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: local-storage-provisioner-node-clusterrole
namespace: local-storage
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: local-storage-provisioner-node-binding
namespace: local-storage
subjects:
- kind: ServiceAccount
name: local-storage-admin
namespace: local-storage
roleRef:
kind: ClusterRole
name: local-storage-provisioner-node-clusterrole
apiGroup: rbac.authorization.k8s.io
@@ -0,0 +1,12 @@
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: default-rbac
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io