Skip to content
samperrin.com, home
Automation

vRA - Windows Cloudbase-init - Multi-part User Data Example

An example of Cloudbase-init that uses multi-part user data to run cloud-config and PowerShell code

Sam Perrin
8 min read

The following blog post runs through a vRA Cloud Template that deploys a Windows server using Cloudbase-init Multi-part user data, including cloud-config and PowerShell code to configure various things.

This post is broken into 4 main sections…

  • Cloud Template Inputs.
  • Cloud Template Resources.
  • Cloud-config for the Cloud.Machine resource.
  • A breakdown of the Cloud-config components.

Cloud Template Inputs

YAML
name: Windowsversion: 0.0.66formatVersion: 1inputs:  image:    type: string    default: windows-server-2019-standard    title: Operating System and Version    oneOf:      - title: Windows Server 2019 Standard        const: windows-server-2019-standard  flavor:    type: string    title: Server Size    oneOf:      - title: Standard Small (2CPU, 8GB RAM)        const: s1.small      - title: Standard Medium (2CPU, 16GB RAM)        const: s1.medium      - title: Standard Large (4CPU, 16GB RAM)        const: s1.large      - title: Standard Extra-Large (4CPU, 32GB RAM)        const: s1.xlarge  serverUser:    type: string  serverPassword:    type: string    encrypted: true    title: Server Password  serverCount:    type: integer    maximum: 5    minimum: 1    default: 1    title: Number of Servers  dataDisk1Size:    type: integer    title: Data Disk Size    minimum: 0    maximum: 1000    default: 0  domainPassword:    type: string    readOnly: false    encrypted: true

Cloud Template Resources

We have configured 3x resources on our Cloud Template, 1x Cloud.Volume (labelled DataDisk), 1x Cloud.Network (labelled Network) and 1x Cloud.Machine (labelled Server).

There are some specific properties worth calling out…

  1. On the Server resource customizeGuestOs: false - this stops vRA customising the VM, this would usually configure the VM Name and IP by creating a temporary customisation spec at deployment time.
  2. On the Server resource assignment: static - this pulls an IP from vRA’s IPAM.
YAML
resources:  DataDisk:    type: Cloud.Volume    allocatePerInstance: true    properties:      capacityGb: ${input.dataDisk1Size}      count: '${input.dataDisk1Size == 0 ? 0 : input.serverCount}'  Network:    type: Cloud.Network    properties:      networkType: outbound      constraints:        - tag: network.isolated:true  Server:    type: Cloud.Machine    allocatePerInstance: true    properties:      customizeGuestOs: false      image: ${input.image}      flavor: ${input.flavor}      count: ${input.serverCount}      attachedDisks: ${map_to_object(slice(resource.DataDisk[*].id, count.index, count.index + 1), "source")}      constraints:        - tag: cloud.zone.region:dc3      networks:        - network: ${resource.Network.id}          assignment: static      cloudConfig: #see below section
vRA Cloud Template

Cloudbase-Init (Cloud Config)

This cloudConfig utilises the multi-part user data (Multi-part content (opens in a new tab)), this allows us to run a variety of content types, including cloud-config and PowerShell scripts.

The example below uses a combination of Inputs and Property Groups to get its data. The cloudConfig does the following:

  1. Sets virtual machine hostname using cloud-config.
  2. Creates a local user based on two inputs, one for the username, one for the password and then adds that user to the Administrators group.
  3. Enables the “Remote Desktop” and “ICMPv4-In” firewall rules.
  4. Sets some Registry Key values.
  5. Formats the additional disk that was added in the Cloud Template.
  6. Configures various network adapter settings, including the IP and DNS.
  7. Sleeps between setting the network and Domain Join.
  8. Joins the server to the Domain.
  9. Removes any reference to entered passwords from the Cloudbase-init log files.
  10. Ejects any CD-ROMs.
  11. Reboots the server.

The virtual machine template is running Windows Server 2019, it has Cloudbase-init installed with the Cloudbase-init service set to “Automatic (Delayed)”. The Sysprep options were also unchecked on the Cloudbase-init installer. Refer to Michael Poore’s (opens in a new tab) blog post on this - Installing Cloudbase-Init on Windows for vRA Customisation (opens in a new tab).

Below is the Cloud-config in full, we break down each section at the bottom.

Cloud-config in full

YAML
cloudConfig: |Content-Type: multipart/mixed; boundary="==NewPart=="MIME-Version: 1.0 --==NewPart==Content-Type: text/cloud-config; charset="us-ascii"MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Disposition: attachment; filename="cloud-config" set_hostname: ${self.resourceName} --==NewPart==Content-Type: text/x-shellscript; charset="us-ascii"MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Disposition: attachment; filename="INIT.ps1" #ps1_sysnative $localUserPassword = ConvertTo-SecureString ${input.serverPassword} -AsPlainText -ForceNew-LocalUser -Name "${input.serverUser}" -Password $localUserPasswordAdd-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}" Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled TrueEnable-NetFirewallRule -DisplayGroup "Remote Desktop" Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0 Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA" Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $FalseGet-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $FalseGet-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com") $adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }$adapter | Remove-NetIpAddress -Confirm:$false$adapter | Remove-NetRoute -Confirm:$false$adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}$adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,', ')}")$adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6 Start-Sleep -Seconds 10 $domainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $domainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Restart:$false -Force (Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose $drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue shutdown /r /t 10

Identify Multi-part boundary

We establish our multi-part boundary, which tells cloudConfig where a new type starts. Our boundary parameter is ==NewPart==.

YAML
cloudConfig: |Content-Type: multipart/mixed; boundary="==NewPart=="MIME-Version: 1.0

Cloud-config Section

Our new section requires the use of our boundary parameter, this boundary also requires a prefix of --. In this section we set the virtual machine hostname using a reference the the resource itself - in otherwords, whatever the VM is called (based on Custom Names in vRA) we push it into the cloud-config section. This seems to automatically reboot the VM after the hostname is set.

In this section we specify the filename as cloud-config - Content-Disposition: attachment; filename="cloud-config". We can any cloud-config related commands in here. Refer to the Cloudbase-init docs for information - #cloud-config (opens in a new tab).

YAML
--==NewPart==Content-Type: text/cloud-config; charset="us-ascii"MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Disposition: attachment; filename="cloud-config" set_hostname: ${self.resourceName}

Start of the PowerShell Section

This section is the start of our PowerShell code, again we use our boundary and its double-dash prefix --==NewPart==. Again we set a filename but this time with the file extension .ps1 - Content-Disposition: attachment; filename="INIT.ps1". The addition of #ps1_sysnative tells cloudConfig what executable to use. Refer to the Cloudbase-init docs for information - #powershell (opens in a new tab).

YAML
--==NewPart==Content-Type: text/x-shellscript; charset="us-ascii"MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Disposition: attachment; filename="INIT.ps1" #ps1_sysnative

Local User Creation

We utilise the Inputs for serverPassword and serverUser to add and configure our local user.

PowerShell
$localUserPassword = ConvertTo-SecureString ${input.serverPassword} -AsPlainText -ForceNew-LocalUser -Name "${input.serverUser}" -Password $localUserPasswordAdd-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}"

Set Firewall Rules

PowerShell
Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled TrueEnable-NetFirewallRule -DisplayGroup "Remote Desktop"

Set Registry Values

We utilise values from a Property Group called defaultServerValues and specifically the value of property fqdnDomain. An example value could be domain.com.

PowerShell
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0

Format Additional Disk

This code looks for all disks that have a matching partition style and then initialises them before applying a driver letter and label. We only add a single disk in this Cloud Template, hence a single label - if you add more than one disk you will need to adapt the code to suit.

PowerShell
Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA"

Configure Network DNS Client Settings

Again we utilise values from a Property Group called defaultServerValues and specifically the value of property fqdnDomain. An example value could be domain.com.

PowerShell
Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $FalseGet-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $FalseGet-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com")

Configure Network DNS Client Settings

Our virtual machine template includes a single network adapter with the label Ethernet0, we first identify this network adapter object and we then configure the required values, including IP and DNS.

Within this Cloud Template we have configure the Machine resource to have assignment: static for it’s networking, so we are allocating an IP address from vRA’s IPAM.

On line 4 below we retrieve the IP address assigned to the virtual machine - ${self.networks[0].address}. For prefix and default gateway values we reference the attached Network resource - ${resource.Network.prefixLength} and ${resource.Network.gateway}.

Finally for line 5 we pull out the attached Networks DNS servers ${join(resource.Network.dns,', ')}. In this particular variable we also utilise one of vRA’s supported expression syntaxes join. Cloud Assembly expression syntax (opens in a new tab).

vRA Network Settings
PowerShell
$adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }$adapter | Remove-NetIpAddress -Confirm:$false$adapter | Remove-NetRoute -Confirm:$false$adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}$adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,', ')}")$adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6

Sleep

There is a slight delay between configuring the network and it being available, you could probably change this to a loop that better monitors the network status but during testing we found good success at the 10 second mark.

PowerShell
Start-Sleep -Seconds 10

Join Active Directory Domain

Within vRA we have configured Active Directory integration and we have made it available to all of the Projects that require it. This creates the Computer Object in our specified OU, which leaves us to only do the domain joining from within the deployed guest OS.

We utilise more Property Group values in this code as well as an additional Input for the domainPassword, our specific approach for this Input was to provide a Default Value for it on the Custom Form and set the Visibility to No, this way the user does not see it. The reason for this approach is because Secrets cannot be shared across the Organization, they are Project specific, which would have caused us to have multiple copies of this specific value (1x per Project).

In the first line we create a PowerShell Credentials object by using two Property Group values ${propgroup.defaultServerValues.windowsDomainJoinUsername} and ${propgroup.defaultServerValues.windowsDomain}, in combination with the @ symbol we construct a UPN that will be used for the domain join, such as user@example.com, the second part of the Credentials object is the domainPassword value.

This Credentials object is then used on the second line along with some additional Property Group values, ${propgroup.defaultServerValues.windowsDomain}, for example ad.domain.com and ${propgroup.defaultServerValues.windowsDomainServerDc3} which is a specific server that we want to talk to, such as server1.ad.domain.com - this input is not required normally, but due to some network constraints we need to speak to a specific server.

PowerShell
$domainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $domainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Restart:$false -Force

Clean the Cloudbase-init Log Files

Due to the configured logging level of Cloudbase-init we can see the inputted passwords in clear text. We run the below PowerShell code to look for the Input values for domainPassword and serverPassword and replace them with some placeholder values - <DOMAIN-JOIN-PASSWORD> and <LOCAL-USER-PASSWORD>. An alternative would be to change the Cloudbase-init logging level.

PowerShell
(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose

Eject CD-ROM

We remove the Cloud-config attached CD-ROM, im not sure if this is absolutely necessary but during testing we observed CD’s still being connected to the deployed VM.

PowerShell
$drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue

Shutdown VM

Finally, we reboot the VM, this is so the VM recognise it is domain joined.

PowerShell
shutdown /r /t 10
AutomationVMwarevRAvRealize