Download the PhotonOS Kubernetes and HAProxy images from here http://storage.googleapis.com/capv-images (opens in a new tab)
HAProxy version used in the steps below: http://storage.googleapis.com/capv-images/extra/haproxy/release/v0.6.4/capv-haproxy-v0.6.4.ova (opens in a new tab)
PhotonOS Kubernetes version used in the steps below: http://storage.googleapis.com/capv-images/release/v1.18.2/photon-3-kube-v1.18.2.ova (opens in a new tab)
Deploy both OVA files in to vSphere, take a snapshot of them, and then covert to templates.
Create a new CentOS or Ubuntu VM to act as a temporary bootstrap server - steps below will be focused on CentOS. Alternatively, install the components below on your current device.
This machine will be used in later stages of your cluster lifecycle management, due to the presence of clusterctl
Install Docker
curl -fsSL https://get.docker.com -o get-docker.shsudo sh get-docker.shMAINUSER=$(logname)sudo usermod -aG docker $MAINUSERInstall KIND
KINDVERSION=$(curl -s https://github.com/kubernetes-sigs/kind/releases/latest/download 2>&1 | grep -Po [0-9]+\.[0-9]+\.[0-9]+)curl -L "https://github.com/kubernetes-sigs/kind/releases/download/v$KINDVERSION/kind-$(uname -s | tr '[:upper:]' '[:lower:]')-$(dpkg --print-architecture)" -o /usr/local/bin/kindchmod +x /usr/local/bin/kindInstall kubectl
curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"chmod +x ./kubectlsudo mv ./kubectl /usr/local/bin/kubectlInstall clusterctl
CLUSTERCTLVERSION=$(curl -s https://github.com/kubernetes-sigs/cluster-api/releases/latest/download 2>&1 | grep -Po [0-9]+\.[0-9]+\.[0-9]+)curl -L "https://github.com/kubernetes-sigs/cluster-api/releases/download/v$CLUSTERCTLVERSION/clusterctl-$(uname -s | tr '[:upper:]' '[:lower:]')-$(dpkg --print-architecture)" -o /usr/local/bin/clusterctlchmod +x /usr/local/bin/clusterctlCreate KIND cluster
kind create clusterCreate clusterctl.yaml to store configuration values. Replace the values where required. Take particular note of values between < and >.
mkdir -p ~/.cluster-apitee ~/.cluster-api/clusterctl.yaml >/dev/null <<EOF## -- Controller settings -- ##VSPHERE_USERNAME: "administrator@vsphere.local" # The username used to access the remote vSphere endpointVSPHERE_PASSWORD: "vmware1" # The password used to access the remote vSphere endpoint ## -- Required workload cluster default settings -- ##VSPHERE_SERVER: "vcenter1.domain.com" # The vCenter server IP or FQDNVSPHERE_DATACENTER: "Datacenter" # The vSphere datacenter to deploy the management cluster onVSPHERE_DATASTORE: "Datastore" # The vSphere datastore to deploy the management cluster onVSPHERE_NETWORK: "VM Network" # The VM network to deploy the management cluster onVSPHERE_RESOURCE_POOL: "<ClusterName>/Resources/<ResourcePoolName>" # The vSphere resource pool for your VMsVSPHERE_FOLDER: "vm/<FolderName>/<ChildFolderName>" # The VM folder for your VMs. Set to "" to use the root vSphere folderVSPHERE_TEMPLATE: "photon-3-kube-v1.18.2" # The VM template to use for your management cluster.VSPHERE_HAPROXY_TEMPLATE: "capv-haproxy-v0.6.4" # The VM template to use for the HAProxy load balancerVSPHERE_SSH_AUTHORIZED_KEY: "<ssh key>" # The public ssh authorized key on all machines in this cluster. Set to "" if you don't want to enable SSH, or are using another solution.EOFInitialise and Create the Bootstrap Management Cluster. Change the number of control-plane and worker nodes, kubernetes version etc as appropriate.
clusterctl init --infrastructure vsphere# if this fails use clusterctl init --infrastructure=vsphere:v0.6.6mkdir -p ~/local-control-planeclusterctl config cluster local-control-plane --infrastructure vsphere --kubernetes-version v1.18.2 --control-plane-machine-count 1 --worker-machine-count 1 > ~/local-control-plane/cluster.yamlReview the cluster config file and make any required changes. Some suggestions to review are CPU, Memory, Storage and Pods CIDR range.
You can use sed to do a find and replace within the file, the below examples are for changing the Storage and Memory on all virtual machines as part of this deployment and the Pods CIDR.
- Pod CIDR from
192.168.0.0/16to192.168.100.0/24 - Memory from
8192to4096 - Storage from
25to20
sed -i "s/192.168.0.0\/16/192.168.100.0\/24/g" ~/local-control-plane/cluster.yamlsed -i "s/memoryMiB: 8192/memoryMiB: 4096/g" ~/local-control-plane/cluster.yamlsed -i "s/diskGiB: 25/diskGiB: 20/g" ~/local-control-plane/cluster.yamlDeploy the cluster components. Monitor the deployment of virtual machines within vSphere.
kubectl apply -f ~/local-control-plane/cluster.yamlUse kubectl to see cluster progress. The control planes won’t be Ready until we install a CNI in a later step.
kubectl get cluster --all-namespaceskubectl get kubeadmcontrolplane --all-namespacesRetrieve the kubeconfig file for the local-control-plane cluster, and set it as KUBECONFIG environment variable.
kubectl get secret local-control-plane-kubeconfig -o=jsonpath='{.data.value}' | { base64 -d 2>/dev/null || base64 -D; } > ~/local-control-plane/kubeconfigexport KUBECONFIG="$HOME/local-control-plane/kubeconfig"Deploy Calico CNI to local-control-plane.
Because we have set KUBECONFIG in the above step our kubectl commands will run against the local-control-plane cluster. If you want to be certain, add --kubeconfig="$HOME/local-control-plane/kubeconfig" to the commands.
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml#Example with kubeconfig specified#kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml --kubeconfig="$HOME/local-control-plane/kubeconfig"Now we will initialise the local-control-plane cluster as our cluster-api management cluster.
clusterctl init --infrastructure vsphereRevert back to the kubeconfig for the bootstrap cluster, migrate management components from it to the local-control-plane cluster, and finally delete.
unset KUBECONFIGclusterctl move --to-kubeconfig="$HOME/local-control-plane/kubeconfig"kind delete cluster --name bootstrapSet the local-control-plane kubeconfig as our default
cp ~/local-control-plane/kubeconfig ~/.kube/configYou will now be able to perform cluster management operations. Some examples below.
#Retrieve related vSphere VMskubectl get vspheremachine#Retrieve related machineskubectl get machine#Retrieve additional machine detailskubectl describe machine <machine-name>#Retrieve additional cluster informationkubectl describe cluster local-control-plane#Get machinedeployment (replicaset equivalent for machines)kubectl get machinedeployment#Scale worker nodes machinedeploymentkubectl scale machinedeployment local-control-plane-md-0 --replicas=3#Get various machine resources at oncekubectl get cluster,machine,machinesets,machinedeployment,vspheremachine#Get most/all Cluster API resourceskubectl get clusters,machinedeployments,machinehealthchecks,machines,machinesets,providers,kubeadmcontrolplanes,machinepools,haproxyloadbalancers,vsphereclusters,vspheremachines,vspheremachinetemplates,vspherevmsThe next article will cover deploying a workload cluster that can be consumed by your development teams.
Next Article in this Series: Cluster API Workload Cluster (vSphere)
Reference: The Cluster API project (opens in a new tab)
