vRA - Windows Cloudbase-init - Multi-part User Data Example

The following blog post runs through a vRA Cloud Template that deploys a Windows server using Cloudbase-init Multi-part user data, including cloud-config and PowerShell code to configure various things.

This post is broken into 4 main sections…

  • Cloud Template Inputs.
  • Cloud Template Resources.
  • Cloud-config for the Cloud.Machine resource.
  • A breakdown of the Cloud-config components.

Cloud Template Inputs

name: Windows
version: 0.0.66
formatVersion: 1
inputs:
  image:
    type: string
    default: windows-server-2019-standard
    title: Operating System and Version
    oneOf:
      - title: Windows Server 2019 Standard
        const: windows-server-2019-standard
  flavor:
    type: string
    title: Server Size
    oneOf:
      - title: Standard Small (2CPU, 8GB RAM)
        const: s1.small
      - title: Standard Medium (2CPU, 16GB RAM)
        const: s1.medium
      - title: Standard Large (4CPU, 16GB RAM)
        const: s1.large
      - title: Standard Extra-Large (4CPU, 32GB RAM)
        const: s1.xlarge
  serverUser:
    type: string
  serverPassword:
    type: string
    encrypted: true
    title: Server Password
  serverCount:
    type: integer
    maximum: 5
    minimum: 1
    default: 1
    title: Number of Servers
  dataDisk1Size:
    type: integer
    title: Data Disk Size
    minimum: 0
    maximum: 1000
    default: 0
  domainPassword:
    type: string
    readOnly: false
    encrypted: true

Cloud Template Resources

We have configured 3x resources on our Cloud Template, 1x Cloud.Volume (labelled DataDisk), 1x Cloud.Network (labelled Network) and 1x Cloud.Machine (labelled Server).

There are some specific properties worth calling out…

  1. On the Server resource customizeGuestOs: false - this stops vRA customising the VM, this would usually configure the VM Name and IP by creating a temporary customisation spec at deployment time.
  2. On the Server resource assignment: static - this pulls an IP from vRA’s IPAM.
resources:
  DataDisk:
    type: Cloud.Volume
    allocatePerInstance: true
    properties:
      capacityGb: ${input.dataDisk1Size}
      count: '${input.dataDisk1Size == 0 ? 0 : input.serverCount}'
  Network:
    type: Cloud.Network
    properties:
      networkType: outbound
      constraints:
        - tag: network.isolated:true
  Server:
    type: Cloud.Machine
    allocatePerInstance: true
    properties:
      customizeGuestOs: false
      image: ${input.image}
      flavor: ${input.flavor}
      count: ${input.serverCount}
      attachedDisks: ${map_to_object(slice(resource.DataDisk[*].id, count.index, count.index + 1), "source")}
      constraints:
        - tag: cloud.zone.region:dc3
      networks:
        - network: ${resource.Network.id}
          assignment: static
      cloudConfig: #see below section

vRA Cloud Template

Cloudbase-Init (Cloud Config)

This cloudConfig utilises the multi-part user data (Multi-part content), this allows us to run a variety of content types, including cloud-config and PowerShell scripts.

The example below uses a combination of Inputs and Property Groups to get its data. The cloudConfig does the following:

  1. Sets virtual machine hostname using cloud-config.
  2. Creates a local user based on two inputs, one for the username, one for the password and then adds that user to the Administrators group.
  3. Enables the “Remote Desktop” and “ICMPv4-In” firewall rules.
  4. Sets some Registry Key values.
  5. Formats the additional disk that was added in the Cloud Template.
  6. Configures various network adapter settings, including the IP and DNS.
  7. Sleeps between setting the network and Domain Join.
  8. Joins the server to the Domain.
  9. Removes any reference to entered passwords from the Cloudbase-init log files.
  10. Ejects any CD-ROMs.
  11. Reboots the server.

The virtual machine template is running Windows Server 2019, it has Cloudbase-init installed with the Cloudbase-init service set to “Automatic (Delayed)”. The Sysprep options were also unchecked on the Cloudbase-init installer. Refer to Michael Poore’s blog post on this - Installing Cloudbase-Init on Windows for vRA Customisation.

Below is the Cloud-config in full, we break down each section at the bottom.

Cloud-config in full

cloudConfig: |
Content-Type: multipart/mixed; boundary="==NewPart=="
MIME-Version: 1.0

--==NewPart==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config"

set_hostname: ${self.resourceName}

--==NewPart==
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="INIT.ps1"

#ps1_sysnative

$localUserPassword = ConvertTo-SecureString ${input.serverPassword} -AsPlainText -Force
New-LocalUser -Name "${input.serverUser}" -Password $localUserPassword
Add-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}"

Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled True
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0

Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA"

Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $False
Get-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $False
Get-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}
Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com")

$adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }
$adapter | Remove-NetIpAddress -Confirm:$false
$adapter | Remove-NetRoute -Confirm:$false
$adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}
$adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,', ')}")
$adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6

Start-Sleep -Seconds 10

$domainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)
Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $domainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Restart:$false -Force

(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose

$drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"
$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue

shutdown /r /t 10

Identify Multi-part boundary

We establish our multi-part boundary, which tells cloudConfig where a new type starts. Our boundary parameter is ==NewPart==.

cloudConfig: |
Content-Type: multipart/mixed; boundary="==NewPart=="
MIME-Version: 1.0

Cloud-config Section

Our new section requires the use of our boundary parameter, this boundary also requires a prefix of --. In this section we set the virtual machine hostname using a reference the the resource itself - in otherwords, whatever the VM is called (based on Custom Names in vRA) we push it into the cloud-config section. This seems to automatically reboot the VM after the hostname is set.

In this section we specify the filename as cloud-config - Content-Disposition: attachment; filename="cloud-config". We can any cloud-config related commands in here. Refer to the Cloudbase-init docs for information - #cloud-config.

--==NewPart==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config"

set_hostname: ${self.resourceName}

Start of the PowerShell Section

This section is the start of our PowerShell code, again we use our boundary and its double-dash prefix --==NewPart==. Again we set a filename but this time with the file extension .ps1 - Content-Disposition: attachment; filename="INIT.ps1". The addition of #ps1_sysnative tells cloudConfig what executable to use. Refer to the Cloudbase-init docs for information - #powershell.

--==NewPart==
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="INIT.ps1"

#ps1_sysnative

Local User Creation

We utilise the Inputs for serverPassword and serverUser to add and configure our local user.

$localUserPassword = ConvertTo-SecureString ${input.serverPassword} -AsPlainText -Force
New-LocalUser -Name "${input.serverUser}" -Password $localUserPassword
Add-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}"

Set Firewall Rules

Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled True
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

Set Registry Values

We utilise values from a Property Group called defaultServerValues and specifically the value of property fqdnDomain. An example value could be domain.com.

Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0

Format Additional Disk

This code looks for all disks that have a matching partition style and then initialises them before applying a driver letter and label. We only add a single disk in this Cloud Template, hence a single label - if you add more than one disk you will need to adapt the code to suit.

Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA"

Configure Network DNS Client Settings

Again we utilise values from a Property Group called defaultServerValues and specifically the value of property fqdnDomain. An example value could be domain.com.

Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $False
Get-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $False
Get-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}
Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com")

Configure Network DNS Client Settings

Our virtual machine template includes a single network adapter with the label Ethernet0, we first identify this network adapter object and we then configure the required values, including IP and DNS.

Within this Cloud Template we have configure the Machine resource to have assignment: static for it’s networking, so we are allocating an IP address from vRA’s IPAM.

On line 4 below we retrieve the IP address assigned to the virtual machine - ${self.networks[0].address}. For prefix and default gateway values we reference the attached Network resource - ${resource.Network.prefixLength} and ${resource.Network.gateway}.

Finally for line 5 we pull out the attached Networks DNS servers ${join(resource.Network.dns,', ')}. In this particular variable we also utilise one of vRA’s supported expression syntaxes join. Cloud Assembly expression syntax.

vRA Network Settings

$adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }
$adapter | Remove-NetIpAddress -Confirm:$false
$adapter | Remove-NetRoute -Confirm:$false
$adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}
$adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,', ')}")
$adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6

Sleep

There is a slight delay between configuring the network and it being available, you could probably change this to a loop that better monitors the network status but during testing we found good success at the 10 second mark.

Start-Sleep -Seconds 10

Join Active Directory Domain

Within vRA we have configured Active Directory integration and we have made it available to all of the Projects that require it. This creates the Computer Object in our specified OU, which leaves us to only do the domain joining from within the deployed guest OS.

We utilise more Property Group values in this code as well as an additional Input for the domainPassword, our specific approach for this Input was to provide a Default Value for it on the Custom Form and set the Visibility to No, this way the user does not see it. The reason for this approach is because Secrets cannot be shared across the Organization, they are Project specific, which would have caused us to have multiple copies of this specific value (1x per Project).

In the first line we create a PowerShell Credentials object by using two Property Group values ${propgroup.defaultServerValues.windowsDomainJoinUsername} and ${propgroup.defaultServerValues.windowsDomain}, in combination with the @ symbol we construct a UPN that will be used for the domain join, such as user@example.com, the second part of the Credentials object is the domainPassword value.

This Credentials object is then used on the second line along with some additional Property Group values, ${propgroup.defaultServerValues.windowsDomain}, for example ad.domain.com and ${propgroup.defaultServerValues.windowsDomainServerDc3} which is a specific server that we want to talk to, such as server1.ad.domain.com - this input is not required normally, but due to some network constraints we need to speak to a specific server.

$domainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)
Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $domainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Restart:$false -Force

Clean the Cloudbase-init Log Files

Due to the configured logging level of Cloudbase-init we can see the inputted passwords in clear text. We run the below PowerShell code to look for the Input values for domainPassword and serverPassword and replace them with some placeholder values - <DOMAIN-JOIN-PASSWORD> and <LOCAL-USER-PASSWORD>. An alternative would be to change the Cloudbase-init logging level.

(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose

Eject CD-ROM

We remove the Cloud-config attached CD-ROM, im not sure if this is absolutely necessary but during testing we observed CD’s still being connected to the deployed VM.

$drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"
$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue

Shutdown VM

Finally, we reboot the VM, this is so the VM recognise it is domain joined.

shutdown /r /t 10
Published 17 Aug 2022

Tags


Automation Consultant, currently working at Xtravirt. Interested in all things automation/devops related.
Sam Perrin on Twitter