samperrin.com

Why Simplification could bring Business Value

Wed, 17 Jan 2024 00:00:00 GMT

Broadcom announced a simplification in their licensing model on December 11, since that time there has been mixed feelings within the industry, particularly around the shift to subscription and possible cost increases. This post aims to demonstrate some of the potential business value that can come from access to more products in the VMware Cloud Foundation offering.

The VMware Cloud Foundation (VCF) offering includes the following products;

SDDC Manager
vSphere Enterprise Plus
- vCenter Standard
- vSphere with Tanzu
- ESXi
vSAN Enterprise (1TiB/Core)
NSX Enterprise Plus (NSX Networking only)
HCX Enterprise
Aria Suite Enterprise
- Aria Suite Lifecycle (previously vRealize Suite Lifecycle Manager)
- Aria Automation (previously vRealize Automation)
- Aria Operations (previously vRealize Operations)
- Aria Operations for Logs (previously vRealize Log Insight)
Aria Operations for Network (previously vRealize Network Insight)

It also provides access to a several add-ons that can be purchased separately, such as;

VMware Cloud Disaster Recovery (VCDR)
VMware Load Balancer (NSX Advanced Load Balancer)
Tanzu Mission Control (TMC)
Tanzu Application Platform (TAP)

An excellent resource is William’s blog post that provides more information on the offerings and lists all of the add-ons: What’s in the new VMware vSphere Foundation (VVF) and VMware Cloud Foundation (VCF) offers?.

This post will not go into detail about each product or why they might be licensed the way they are, instead it will attempt to identify opportunities that these products could bring to the business.

Value is relative to the company, typically stakeholders, whether that is IT or other business functions, will decide what is of value and this usually aligns with the priorities of the business. For the sake of this post we will generalise and work on the basis that… Value is something that brings a positive change to the business, either through efficiency, cost saving or risk reduction.

Product	Efficiency	Cost Saving	Risk Reduction
SDDC Manager	Automates deployment, configuration, upgrading, and patching of the VCF stack (vCenter, ESXi, NSX), simplifying operations and reducing manual labour.	Reduces the need for manual deployments and configurations that require lengthy designing and planning.	Reduces the operational risk of workloads breaking during upgrades and patches. Security risk is reduced due to the ability to quickly patch an environment.
vSphere Enterprise Plus	Enhances operational efficiency and transforms existing infrastructure. Maximises infrastructure capacity utilisation and improves workload performance.	Achieves better price performance using the latest technology on industry-standard servers.	Decreases downtime and improves reliability with predictive analytics and built-in high availability capabilities.
vSAN Enterprise	Simplifies storage management and scales elastically in per-server increments.	Reduces storage cost and complexity, and achieves better price performance using the latest storage technology on industry-standard servers.	Ensures data protection and availability.
NSX Enterprise Plus	Automates network management, saving time and effort.	Reduces network provisioning time and improves operational efficiency through automation, allowing businesses can roll out new services faster.	Provides consistent management of networking and security policies, independent of physical network topology, reducing the risk of outages and strengthens network infrastructure security.
HCX Enterprise	Supports non-vSphere workload migration into vSphere, allowing businesses to migrate large quantities of virtual workloads from different environments, providing flexibility and reducing the time and effort required for migration.	Multiple appliances bundled as one service, with automated functions, eliminates the need for different tools, reducing TCO.	Minimizes risk by providing a mobility platform across environments by abstracting the underlying infrastructure, reducing the risk of migration failures, data loss and downtime, which help ensure the continuity of operations.
Aria Suite Lifecycle	Accelerates deployment and upgrades of the Aria Suite products.	Reduces ongoing management and operational costs.	Enforces alignment with VMware recommended reference architectures and validated designs, reducing the risk of non-compliance.
Aria Automation	Removes the requirement for manual tasks with advanced workflows and infrastructure as code.	Reduces the total cost of ownership by freeing up IT budget and resources that can be reallocated elsewhere.	Establishes consistent policies across multi-cloud environments, reduces human error through automated infrastructure deployment.
Aria Operations	Provides continuous performance and capacity optimisation, with intelligent remediation.	Reduces operating expenses and increases operational efficiency by reducing troubleshooting times and improving resolution time.	Reduces risk and enforces regulatory standards with integrated compliance. Can help identify under utilised and at risk infrastructure components.
Aria Operations for Logs	Extensible across physical, virtual, and cloud environments, enabling administrators to have a single location to collect, store, and analyse logs at scale.	Reduces operating expenses and increases operational efficiency by reducing troubleshooting times and improving resolution time.	Provides visibility into security-related events, mitigating costs from unplanned downtime, by using predictive analytics, machine learning, and root cause analysis tools.
Aria Operations for Network	Provides network visibility and analytics to accelerate application discovery and optimise network performance.	Reduces time spent monitoring the network, leading to cost savings.	Minimises the risk during application migration through discovery and identifies missing firewall policies.

This is a slightly different post to my usual, but hopefully with all of the recent changes within the VMware by Broadcom landscape, this provides some insight into where the VMware Cloud Foundation offering can benefit the business.

#Resources

On-demand Networking with Aria Automation and NSX

Wed, 20 Dec 2023 00:00:00 GMT

In Aria Automation you can create Network Profiles, these are the networks and network settings for deployments, one of those settings is for Isolation Policy, which has the options; None, On-demand Network and On-demand Security Group. In this post we will have a high-level look at what both of the on-demand options really mean and what is required to get them working with Aria Automation.

#On-demand Network

With this policy a network is created for each deployment using the settings specified within the Network Policies tab and all virtual machines within the deployment are attached to this network/segment.

#Network Profile Requirements Summary

NSX Cloud Account within Aria Automation.
Transport Zone configured within NSX.
An “External Network”, think of it as routable, available within NSX and Aria Automation, this is used to create an outbound SNAT rule from the on-demand networks.
The “External Network” needs to be configured within Aria Automation with Domain, CIDR, DNS and IP Range.
A Tier-0 router configured within NSX, this is used for outbound access for the on-demand networks.
An Edge Cluster configured within NSX, this is used for outbound access for the on-demand networks.
An IPAM configuration using either Aria Automation Internal IPAM or External IPAM, this is used to allocate the subnets to the on-demand networks.
A Cloud Template that has a NSX network resource using the networkType of outbound, private, or routed.

When a deployment associates itself with a Network Profile that is set to create an on-demand network, the following objects are created within NSX, which differ depending on the networkType.

Object/Resource	`outbound`	`private`	`routed`
Tier-1 Gateway	✓		✓
Network Segment	✓	✓	✓
NAT Rule (SNAT)	✓
DHCP Server*	✓	✓	✓

The created Tier-1 Gateway is attached to the Tier-0 Gateway and Edge Cluster specified in the Network Profile.
The created Segment is attached to the Transport Zone specified in the Network Profile and a subnet is allocated from the IPAM section within the Network Profile. The gateway address is set to first address in the allocated subnet.
If required, an SNAT rule is attached to the created Tier-1. This is configured so that the “Source” is the IP range allocated to the segment and the “Translated IP” comes from the External Network specified within the Network Profile and its pre-configured IP Range.
Virtual Machines within the Deployment are attached to the created Segment (if linked in the Cloud Template)
If the Network Profile > Network Policies > IP Address Management > IP Range Assignment is set to use DHCP or Static and DHCP then an NSX hosted DHCP Server is created. The size of this range depends on the option, for DHCP it is for ALL IP’s in the network, for Static and DHCP* it is half of the network.

#On-demand Security Group

With this policy a security group is created for each deployment and all of the virtual machines within the deployment are members of this security group.

#Network Profile Requirements Summary

NSX Cloud Account within Aria Automation.
Existing networks configured to attach VMs to.
If you are going to use static IP assignment within the Cloud Template then the networks need to be configured within Aria Automation with Domain, CIDR, DNS and IP Range.
A Cloud Template that has a NSX network resource using the networkType of outbound or private.

When a deployment associates itself with a Network Profile that is set to create an on-demand security group, the following DFW rules are created within NSX. The security group is usually named isolation-securitygroup-GUID. The VMs are added to the security group based on their allocated IP address (the IP’s are security group members).

#Network Type: Outbound

Rule Name	Sources	Destinations	Services	Applied To	Action
inbound-deny-all	Any	isolation-securitygroup	Any	isolation-securitygroup	Reject
outbound-allow-all	isolation-securitygroup	Any	Any	isolation-securitygroup	Allow
allow-intra-traffic	isolation-securitygroup	isolation-securitygroup	Any	isolation-securitygroup	Allow

#Network Type: Private

Rule Name	Sources	Destinations	Services	Applied To	Action
inbound-deny-all	Any	isolation-securitygroup	Any	isolation-securitygroup	Reject
allow-intra-traffic	isolation-securitygroup	isolation-securitygroup	Any	isolation-securitygroup	Allow
outbound-allow-all	isolation-securitygroup	Any	Any	isolation-securitygroup	Reject

Tip: Tag your resources in Aria Automation to easily identify objects created by AA - for example add a tag with key: ManagedBy and value: AriaAutomation to Network and Virtual Machine resources to enable the quick search and filtering of these tags in vSphere and NSX - be aware that the DHCP Server in NSX is not tagged!

Hopefully this has been helpful in understanding what is created within NSX when using Aria Automation’s on-demand isolation options.

#Resources

Learn more about network profiles in VMware Aria Automation

VMware Explore 2023 Session Agenda

Thu, 19 Oct 2023 00:00:00 GMT

I am not heading to VMware Explore this year but I thought I would spend a bit of time going through the Content Catalog. I have built out a rough agenda based on the products I am interested in, you can find that agenda below (I have not put much effort into avoiding duplicate time slots).

#Honorary Mentions

There are a couple of sessions from Xtravirt, these are;

Mastering Cross-Cloud Integration: Insights & Best Practices from Xtravirt [CEIB1454BCN]
Steve Wood, Head of Managed Services
Wednesday, Nov 8 2:00 PM - 2:45 PM CET
Explore real-world challenges solved by VMware Cross-Cloud Services through tangible use-cases and experiences, underlined by Xtravirt’s expertise. Dive into how a strategic partnership with a Cross-Cloud Managed Services Provider navigates complexities, ensures interoperability, and maintains compliance across varied cloud environments, enhancing efficiency and security in your cloud ecosystem, whilst optimising for scale.

Customer Panel: Cloud-Smart Insights from the Public Sector [CEIB1800BCN]
Robin Gardner, Director of Strategic Services
Tuesday, Nov 7 2:15 PM - 3:00 PM CET
As IT accelerates in this era of AI, Cross Cloud Services and Big Data, join Xtravirt and a panel of Higher Education and Government IT leaders for an open and honest discussion on how Public Sector organisations are evolving to embrace these new opportunities. Challenged by skills shortages, financial constraints, ransomware and the ever present burden of technical debt, the panel will share their experiences of navigating their own pathway to cloud smart. From universities to local government, how are establishments achieving the game-changing benefits new technologies offer?

#Selected Sessions

These are the sessions that have piqued my interest across the 4 days of VMware Explore Barcelona.

#Monday 6th Nov

Session	Session ID	Time (CET)
Deliver Cloud-Native and Traditional IT Resources on VMware Cloud	MAPT1605BCN	11:00 AM - 12:30 PM
Blueprint to Brilliance: Navigating the Marvels of VMware Cloud Foundation	CXS2060BCN	1:00 PM - 1:45 PM
VMware Edge Compute Stack Reference Architecture Deep Dive	CEIT1999BCN	1:00 PM - 2:30 PM
Livefire: Reveal Cluster Class Model for ClusterAPI Topologies Explore TAP	PART2247BCN	1:00 PM - 5:00 PM
Join the Revolution to Shape the Future of the Multi-Cloud Platform	DWS2147BCN	2:30 PM - 3:30 PM
Platform Engineering Done Right: Providing a Developer Platform with VMware Tanzu	MAPT1606BCN	3:00 PM - 4:30 PM
How to Migrate Apps Live to Any Cloud	VMTN2196BCN	4:00 PM - 4:15 PM
A Road to Modern Apps with VMware Tanzu	CODE1986BCN	4:00 PM - 4:20 PM

#Tuesday 7th Nov

Session	Session ID	Time (CET)
Network Observability for Private and Public Cloud with VMware Aria Operations for Networks	MAPB2297BCN	10:30 AM - 11:15 AM
Protecting VMware Workloads Running in Azure, AWS or Google Cloud with Veeam	CEIB1649BCN	10:30 AM - 11:15 AM
Extending the Capabilities of VMware Aria Automation	CXS1887BCN	11:30 AM - 12:15 PM
Unleash the Power of Cloud Native – Insights from a Leading Cloud Provider	MAPB1236BCN	1:00 PM - 1:45 PM
Apps and Infra Management with the New VMware Tanzu Integrated Solution Powered by Tanzu Hub	MAPB2300BCN	2:15 PM - 3:00 PM
Developing for VMware Aria Automation at Enterprise Scale	CEIB1235BCN	2:15 PM - 3:00 PM
What’s New with VMware Cloud Foundation and Operational Best Practices?	CEIB1446BCN	2:15 PM - 3:00 PM

#Wednesday 8th Nov

Session	Session ID	Time (CET)
Transforming the Retail Experience with VMware Edge Compute Stack	CEIB2001BCN	9:00 AM - 9:45 AM
Automate Kubernetes Platform Ops with VMware Tanzu for Kubernetes Operations	MAPB1392BCN	11:30 AM - 12:15 PM
Mastering VCF: Orange France Streamlines Cloud Without VMware Expertise	CEIB2131BCN	11:30 AM - 12:15 PM
Optimizing Distributed Cloud Architectures – Powered by Platform Equinix.	CEI2472BCNS	12:45 PM - 1:15 PM
AI Without GPUs: Run AI/ML Workloads on Intel AMX CPUs with vSphere 8 and Tanzu	MAPB2367BCN	12:45 PM - 1:30 PM
Automating with VMware Aria Automation Orchestrator	CXS1888BCN	1:00 PM - 1:45 PM
Mastering Cross-Cloud Integration: Insights & Best Practices from Xtravirt	CEIB1454BCN	2:00 PM - 2:45 PM
Achieve AI/ML-Based Insights with Full-Stack Contextual Observability	MAPB1747BCN	2:00 PM - 2:45 PM
Success Story – Accelerate DC Exit by Using Google Cloud VMware Engine	MAPB1604BCN	3:15 PM - 4:00 PM
Resilient IT Architecture in a Multi-Cloud World	VMTN2345BCN	4:00 PM - 4:25 PM
Need to Migrate Thousands of Workloads? No Problem.	[XS1373BCN	4:00 PM - 4:45 PM

#Thursday 9th Nov

Session	Session ID	Time (CET)
How to Migrate 25K Workloads to VMware Cloud Foundation with Security Posture	CSXB1301BCN	11:15 AM - 12:00 PM

#Products

The products I am interested in which I used to filter the Content Catalog by, I dont have a session for every product.

Tanzu / Cloud Native / Modern Apps	VMware Aria Suite	Everything Else
VMware Tanzu	VMware Aria Automation	Azure VMware Solution
VMware Tanzu Application Platform	VMware Aria Automation Assembler	VMware Cloud Director
VMware Tanzu CloudHealth	VMware Aria Automation Config	VMware Cloud Foundation
VMware Tanzu Hub	VMware Aria Automation Orchestrator	VMware Cloud Foundation with Tanzu
VMware Tanzu Mission Control	VMware Aria Operations for Networks	VMware Cloud on Equinix Metal
Cluster API	VMware Aria Universal Suite	VMware Edge
Project Harbor		VMware Edge Compute Stack
Project Antrea		VMware NSX Advanced Load Balancer
VMware Application Catalog		VMware Validated Design

#Resources

Content Catalog - VMware Explore 2023 Barcelona

Aria Automation Startup Troubleshooting

Thu, 20 Jul 2023 00:00:00 GMT

I recently had to bring up an old lab environment that was using an older version of vRA (8.8), but at appliance boot the Kubernetes API-Server containers continuously rebooted and I got an error with kubectl. This post covers some of the troubleshooting steps taken and how it was resolved, ultimately it was caused by an expired certificate.

kube-apiserver container existing and disappearing due to startup issues

I checked the service status for both kubelet and docker, both were active and running. Next I checked kubelet with journalctl - journalctl -u kubelet -r, the -r flag shows us the newest messages first.

There was a lot of repetitive logs stating the DNS name of our node was not found, but in amongst the noise there was also TLS and certificate related messages, include the path to the pem file that it was trying to use.

Kubelet service status with node not found logs

Kubelet service status with TLS/certificate errors in the logs

I then used an OpenSSL command to check the validity of the certificate: sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -text -noout | grep -A 2 Validity - this showed us that the certificate had expired.

openssl command to show the certificate validity

After some digging I came across the following VMware KB article, although the symptoms were not the same it did mention that the certificate had expired after a year, which did match what I was seeing - https://kb.vmware.com/s/article/82378

I am using a single vRA Appliance, so I followed the relevant steps within the KB, however I had some issues, which is the purpose of this post, the adjustments I made are below, along with the exact commands I ran…

Take a snapshot of the vRA VM.
Locate an etcd backup at /data/etcd-backup/ and copy the selected backup to /root.
- I ran ls -lh /data/etcd-backup/ - this showed me the available backups in date order (oldest to newest). I picked the most recent backup (not the .part file).
- I copied this to root with cp /data/etcd-backup/backup-1679068501.db /root.
- I checked the backup was in root with ls /root.
Reset Kubernetes by running vracli cluster leave.
- This takes a little while to run.
Restore the etcd backup in /root by using the /opt/scripts/recover_etcd.sh command. Example: /opt/scripts/recover_etcd.sh --confirm /root/backup-123456789.db.
- I ran /opt/scripts/recover_etcd.sh --confirm /root/backup-1679068501.db - the same as the example from VMware, updated with my backup file name.
Extract VA config from etcd with: kubectl get vaconfig -o yaml --export > /root/vaconfig.yaml.
- This is the first step where I had issues, initially I had the error Error: unknown flag: - -export, so I removed the flag from the command.
- I then got the same error I started with The connection to the server vra-k8s.local:6443 was refused - did you specify the right host or port?.
- I ran through the same initial troubleshooting steps, I checked docker (all good) and kubelet (not started).
- I started the kubelet service with systemctl start kubelet and kept an eye on the status to ensure it stayed running.
- I check with docker ps and could see the kube-apiserver container was running so then ran the kubectl get vaconfig -o yaml > /root/vaconfig.yaml command - this time it was successful.
- I validated that the /root/vaconfig.yaml file had contents with cat /root/vaconfig.yaml.
Reset Kubernetes once again using: vracli cluster leave.
- Again this takes a little while to run.
Run to Install the VA config: kubectl apply -f /root/vaconfig.yaml --force.
- The server seemed to be running after the previous leave step, but check its status at this point and wait for the kube-apiserver container to exist and be running.
Run vracli license to confirm that VA config is installed properly.
Run: /opt/scripts/deploy.sh.
- This takes a while time to run, but you should see that prelude has been successfully deployed.
- I had issues accessing the vRA UI, so I rebooted and it resolved itself.

#Resources

https://kb.vmware.com/s/article/82378

Monitor Cloud Extensibility Proxy embeded Aria Automation Orchestrator (vRealize Orchestrator) with Aria Operations Cloud (vRealize Operations Cloud)

Tue, 03 Jan 2023 00:00:00 GMT

Aria Operations Cloud (vRealize Operations Cloud) includes a management/integration pack for monitoring Aria Automation Orchestrator (vRealize Orchestrator), however when vRO is deployed within a Cloud Extensibility Proxy appliance the authentication process is to redirect the user to VMware Cloud Services where they sign in with their user ID and are then redirected to the vRO dashboard. If you use these same credentials for the vRO appliance when trying to connect via Aria Operations you will fail validation for the integration.

The way to monitor vRO with Aria Operations Cloud is to create an OAuth App within VMware Cloud Services and give it the appropriate Service Role to connect to the vRealize Orchestrator instance embedded in the CEXP.

In this post we will run through the short process of creating and entitling the OAuth App, setting up the vROps integration into CEXP based vRO and also adding additional workflows for monitoring.

#Create OAuth App

To create an OAuth App we need to use the VMware Cloud Services console and you must have access to the Organization section.

Go to VMware Cloud Services.
Go to Organization > OAuth Apps
Press CREATE APP
Select Server to server app and press CONTINUE
Provide the following information
1. App Name (e.g appliance name)
2. App Description (e.g. Cloud Extensibility Proxy vROps)
3. Access Token TTL (default value of 30 minutes is OK)
Select Service Roles: 'Cloud Assembly Administrator'
Click CREATE
Take note of both the App ID and App Secret values
Press CONTINUE
If prompted to Add to this organization, review the message and press Add

#Validate OAuth App

This is an optional step, but it is to ensure our OAuth App is correctly registered against our VMware Cloud Services Organization.

Go to VMware Cloud Services.
Go to Identity & Access Management > OAuth Apps.
Confirm the OAuth App you created in the Create OAuth App step is listed with the correct Organization Roles and Service Roles

#Add vRO Integration in Aria Operations Cloud

In this step we will be added the vRealize Orchestrator endpoint and we will be using our OAuth App ID and token for authentication.

Open Aria Operations Cloud (vRealize Operations Cloud)
Navigate to Data Sources > Integrations
Press Add Account and select vRealize Orchestrator Adapter
Provide the correct inputs, for the vRealize Orchestrator Host input use the appliance FQDN
For Credential press the + to Add New
1. Credential Kind = CSP OAuth App Credential
2. Credential name = A friendly identifier, e.g. CEXP/vRO CSP OAuth Credentials
3. Client Id = App ID value recorded from earlier
4. Client Secret = App Secrete value recorded from earlier
5. Press OK
Select the appropriate collector or collector group
Press VALIDATE CONNECTION
Press SAVE

#Verify Collection Status

This step is to ensure our newly added integration is collecting data.

Login to Aria Operations Cloud (vRealize Operations Cloud)
Navivate to Environment > Inventory
Expand Adapter Instances and select vRealize Orchestrator Adapter Instance
On the Objects tab you should see your vRealize Orchestrator instance
Under the Collection State column ensure State = Collecting
Under the Collection Status column ensure Status = Data recieving

#Add Workflow Packages

By default the vRealize Orchestrator management pack only monitors workflows and items stored in the packages com.vmware.library.vcenter (vCenter Workflows) and com.vmware.vrops.oob_content (Default Management Pack for vRealize Orchestrator workflows). If you have additional workflows that need to be monitored they will need to be compiled into a package and added into Aria Operations.

#Create vRO Package

Login to the vRealize Orchestrator Client.
Navigate to Assets > Packages.
Click New Package.
On the General tab, enter a name and description for the package.
On the Content tab, click ADD.
Select the objects that you want to add to the package and click ADD.
To finish creating the package, click CREATE.

#Add vRO Package to Aria Operations

Login to Aria Operations Cloud (vRealize Operations Cloud)
Navivate to Environment > Inventory
Expand Adapter Instances and select vRealize Orchestrator Adapter Instance
Select to highlight the target vRealize Orchestrator adapter instance.
Select the cog icon (actions) in the right pane and select Configure Package Discovery
Double click the entry in the Configure Package Discovery dialog box and add your package name to a new line.
Click BEGIN ACTION

Configure Package Discovery dialog box

#References

Exploring Aria Hub Search and GraphQL

Wed, 28 Dec 2022 00:00:00 GMT

In this post, we will learn how to search within Aria Hub and query our inventory using GraphQL and Altair. We will cover search examples and the use of operators, comparison functions and property names and values. We will also discuss using Aria Graph and Altair for advanced GraphQL queries.

#Aria Hub Search

As we briefly covered in the previous post Getting started with Aria Hub Free Tier we can search for objects using any of the objects data/properties - such as IP address, name, ID. Here we will cover some of our options for querying data within the UI.

#Search Examples

These queries can be used within the Aria Hub UI.

Find all EC2 Instances: entityType = AWS.EC2.Instance.
Find all EC2 Network ACL’s: entityType = AWS.EC2.NetworkAcl.
Find all EC2 Instances and EC2 Network ACL’s: entityType = AWS.EC2.NetworkAcl OR entityType = AWS.EC2.Instance. We use OR because if we use AND the object would need to be both entityTypes AWS.EC2.NetworkAcl and AWS.EC2.Instance at the same time, which it cant be.
Find everything within a particular AWS region: region = us-east-1.
Find the EC2 Network ACL’s in either of the two specified regions and provide a count of how many per region: AWS.EC2.NetworkAcl HAS (region = us-east-2 OR region = us-east-1) count(region).
Find EC2 Instances launched before a specific data (e.g. 31st December 2022): AWS.EC2.Instance HAS LaunchTime < 2022-12-31.
Find EC2 Instances launched within the last 7 days: AWS.EC2.Instance HAS LaunchTime > daysAgo(7).
Use wildcards, such as a PrivateIpAddress in the 172.31.82.x network: AWS.EC2.Instance HAS PrivateIpAddress = 172.31.82.*.
Find EC2 Instances that do not have InstanceType t2.large: AWS.EC2.Instance HAS InstanceType != t2.large. Can also be represented as: AWS.EC2.Instance HAS NOT InstanceType = t2.large.
Find EC2 Instances with PrivateIpAddress and PublicIpAddress properties: AWS.EC2.Instance HAS pn(PrivateIpAddress) AND propertyName(PublicIpAddress). Properties pn can be used interchangably with propertyName, the same for pv and propertyValue.

Inventory search within a specific region

Multiple entityType search

Search across multiple regions based on the entityType, displayed as a count per region

#Operators, Comparison Operators and Functions

#Operators

These Operators are case-insentive.

HAS - Apply one or more conditions to a resource type.
AND - Group conditions when all of them must be satisfied.
OR - Group conditions when at least one of them must be satisfied.
NOT - Reverse a condition.
() - Specify the order in which conditions are evaluated.

#Functions

hoursAgo(x) - The time x hours earlier than now.
daysAgo(x) - The time x days earlier than now.
monthsAgo(x) - The time x months earlier than now.
yearsAgo(x) - The time x years earlier than now.

#Comparison Operators

= - Equal to.
!= - Not equal to.
<= - Less than or equal to.
=> - Equal to or more than.
< - Less than.
> - More than.

#Identifying Object Properties

In the above queries we use various property names, we can find these names by browsing to an object of the type we are interested in, such as an AWS.EC2.Instance. Once selected we can expand the Properties pane, the left side of the table is the PropertyName (pn) and the right side of the table is the PropertyValue (pv).

Object Properties

#Altair

VMware Aria Graph provides built-in access to an open-source client for developing and testing your GraphQL queries and includes features like collections and pre- and post-scripts.

We have explored how to query data directly within the UI, but a key point to VMware Aria Hub is that it is powered by VMware Aria Graph. Aria Graph has been developed with an API-first approach, for us to start consuming these API’s, and specifically the data behind them, we need to use GraphQL.

There are a couple of browser based methods for this, the GraphiQL UI or with Altair, which is ”…a feature-rich GraphQL Client IDE for all platforms”. It is very similar to the GraphiQL UI but gives us a few more features.

GraphiQL UI

Altair UI

You can access Altair either through Aria Hub > Developer Resources > Altair GraphQL Client or directly using the following link: https://api.mgmt.cloud.vmware.com/aria/altair/.

#CSP Token

Before we can use Altair we need to generate a CSP Token, you can do this by going to the Cloud Services Portal > Select the dropdown by your username (top right) > My Account > API Tokens.

Press GENERATE TOKEN
Provide a Token Name
Select a Token TTL
Select Service Role > VMware Aria Hub
Press GENERATE and take note of the CSP token value - we will need this token for Altair

#Altair Setup

Once generated, head back to Altair. For us to be able to query data we need to authorize our requests, we can use the mutation that is provided by default to help generate a Bearer token.

At the bottom of your Altair window press the VARIABLES button and construct the input like below and then press (Run mutation). Take note of the returned Bearer token.

{
  "CSP_USER_TOKEN": "c-l7aUhW7OWzJaOAaq2Ho8MYEXAMPLECSPTOKENBrJF3i9M_3N6R_JFHglYX"
}

Once you have the Bearer token, within Altair select the Set Headers menu on the left (yellow box in the image below). For the Header key enter authorization and for the Header value enter the Bearer token, including the Bearer word. Press Save.

Altair IDE Setup

Altair IDE Headers

Now Altair is ready for us to start making some queries. Clear out the existing code in the Query panel or open a new tab by pressing + Add new at the top (make sure you set the header again).

#GraphQL Queries

Using Altair we can start to query our data that exists in Aria Hub. Lets repeat some of the searches that we did within the UI.

#Return all AWS EC2 Instances

This query returns all entities that match entityType: "AWS.EC2.Instance". For each of the returned entities we want the values for fields entityName and entityType.

Return all AWS EC2 Instances

{
  entityQuery {
    queryEntities(entityType: "AWS.EC2.Instance") {
      entities {
        entityName
        entityId
      }
    }
  }
}

#Return AWS EC2 Instances and all of their properties.

Similar to the above example, but this time we want to see all property names and values for each entity.

Return AWS EC2 Instances and all of their properties

{
  entityQuery {
    queryEntities(entityType: "AWS.EC2.Instance") {
      entities {
        entityName
        entityId
        properties {
          name
          value
        }
      }
    }
  }
}

#Return AWS EC2 Instances and only some properties.

In this search we only want the name and value for specific properties from each entity. In this instance PrivateDnsName, PrivateIpAddress and StateName.

Return AWS EC2 Instances and only some properties

{
  entityQuery {
    queryEntities(entityType: "AWS.EC2.Instance") {
      entities {
        entityName
        entityId
        properties(name: ["PrivateDnsName","PrivateIpAddress","StateName"]) {
          name
          value
        }
      }
    }
  }
}

#Introducing Variables

The previous examples were static queries, if we want to change what data gets returned we have to manually update the GraphQL query. To make it more dynamic we can start to use variables.

In the below query we take in four variables

entityId - the ID of the object we want to look at.
statKey the metric(s) we want values for
startTime - from what point we want to see the metrics
intervalMins - the interval of the timestamps for the metrics.

We can utilise the Altair variables window to input our values. The key of the variables payload must match the variable name in the GraphQL query, for example key entityId matches variable $entityId.

Return specific AWS EC2 Instance with metrics/stats

query awsStats($entityId: [EntityId!], $statKey: [String!], $startTime: DateTime, $intervalMins: Int) {
  entityQuery {
    queryEntities(entityId: $entityId) {
      entities {
        entityName
        entityId
        stats(
          input: {keys: $statKey, startTime: $startTime, intervalMins: $intervalMins}
        ) {
          key
          values
          timestampsMillis
        }
      }
    }
  }
}

{
  "entityId": "AWS.EC2.628637233703.us-east-1.Instance.i-069ecb9cb0e422211",
  "statKey": ["CPUUtilization", "NetworkIn"],
  "startTime": "2022-12-27T09:49:30.629Z",
  "intervalMins": 30
}

#Introducing Fragments

From the GraphQL docs, the core capability of Fragments is as follows…

Fragments let you construct sets of fields, and then include them in queries where you need to.

The query below, which is for a single AWS EC2 Instance, returns the following information

The name, type and ID or our queried entity.
The tags for our queried entity.
Relationships to children entities (entitiesIn), including their properties and their own children.
Relationships to parent entities (entitiesOut), including their properties
Stats keys of our queried entity.
Costing information of our queried entity.

If we look at the query below we are asking for the same information multiple times, for example on the queried entity we want entityId, entityType, entityName and properties, we then ask for the same with the parent and child entities that are connected. By using fragments we can reduce some of this repetition and utilise reuable information.

query entityDetails($entityId: EntityId!) {
  entityQuery {
    queryEntities(entityId: [$entityId]) {
      entities {
        entityId        entityType        entityName        properties {          name          value        }        tags {
          key
          value
        }
        entitiesIn: entitiesIn(first: 500, traversalScope: SAME_PARTITION) {
          count
          totalCount
          entities {
            entityId            entityType            entityName            properties {              name              value            }            service
            entitiesIn: entitiesIn(first: 500, traversalScope: SAME_PARTITION) {
              count
              totalCount
              entities {
                entityId
                entityType
                entityName
                provider
              }
            }
          }
        }
        entitiesOut: entitiesOut(first: 500, traversalScope: ANY_PARTITION) {
          count
          entities {
            entityId            entityType            entityName            properties {              name              value            }            service
          }
        }
        service
        stats(input: {namespace: "AWS"}) {
          key
        }
        costing {
          listPrices {
            publicPrice
            displayName
            description
          }
        }
      }
    }
  }
}

To construct a Fragment we need to know the schema type, we can find that out in two ways - through the Aria GraphQL docs or by adding the __typename field to our queries.

By using the __typeName method we have identified two schema types that we want to construct our Fragments on: Entity and EntityConnection.

__typeName field information

A fragment consists of two things, its name and the schema type we are fragmenting on - fragment <fragmentName> on <type>. By using them we can reduce the overall length of our query. We can use a Fragment by referencing its name ...<fragmentName>, as seen in the highlighted code below.

query entityDetails($entityId: EntityId!) {
  entityQuery {
    queryEntities(entityId: [$entityId]) {
      entities {
        ...entity        tags {
          key
          value
        }
        entitiesIn: entitiesIn(first: 500, traversalScope: SAME_PARTITION) {
          ...connection          entities {
            ...entity            entitiesIn: entitiesIn(first: 500, traversalScope: SAME_PARTITION) {
              ...connection              entities {
                ...childrenChildrenEntity              }
            }
          }
        }
        entitiesOut: entitiesOut(first: 500, traversalScope: ANY_PARTITION) {
          ...connection          entities {
            ...entity          }
        }
        stats(input: {namespace: "AWS"}) {
          key
        }
        costing {
          listPrices {
            publicPrice
            displayName
            description
          }
        }
      }
    }
  }
}

fragment entity on Entity {
  __typename
  entityId
  entityName
  entityType
  service
  properties {
    name 
    value
  }
}

fragment childrenChildrenEntity on Entity {
  entityId
  entityName
  entityType
  provider
}

fragment connection on EntityConnection {
  __typename
  count
  totalCount
}

#Using our Data

One example of returned data is the stats object. Depending on what you ask for this is typically returned as values and timestamps. They are returned as seperate keys, like the example below, however they are ordered the same - e.g. the top value relates to the top timestamp.

{
  "key": "CPUUtilization",
  "values": [
    "0.1832291531770028",
    "0.18350209991521224",
    "0.18358360369332824",
    "0.18885047821375775"
  ],
  "timestampsMillis": [
    1672226340000,
    1672224540000,
    1672222740000,
    1672220940000
  ]
}

With the metrics results we can take them and use an external tool, such as Excel to build out charts similar to the one within Aria Hub.

Aria Hub

Excel

Metrics in Aria Hub chart

Metrics in Excel chart

#Exploring Queries with Web Browser Developer Tools

Developer Tools within your web browser are a great way to find the queries being used to generate the graphics within the UI. Once you have Developer Tools open you need to find the relevant GraphQL query, in my case it was the one with operationName: "entityDetails".

This particular query shows the relationships as well as all of the information for the selected entity. With the query selected in Developer Tools, right click on the query field and press Copy Value for get the GraphQL query data. Do the same for the Variables entry.

The outputs from both of these can be used in Altair or other tools.

Web Browser Developer Tools to find GraphQL queries

Altair query with Developer Tools information

#REST API

When it comes to real world use it is unlikely you will spend all of your time within Altair, so lets explore how we can use a tool such as Postman to get the same information back via REST API.

First we need to setup Postman

Set the Authorization header to Bearer Token and paste in our token value.
Setup the payload Body
Use Postman Pre-request scripts to convert our raw GraphQL query to a JSON string
Request Method: POST
API Endpoint: https://api.mgmt.cloud.vmware.com/aria/graphql

Set Authorization to Bearer Token

Set Payload Body

Set Pre-Request Script

Returned Data

#Closing

This brings us to the end of this post, its been a long one. The Aria Hub Free Tier is limited to two Cloud Accounts, Azure and AWS. With the paid for version you can integrate additional clouds and data sources, such as Aria Automation and Aria Operations - with data from these sources your queries will become a lot more powerful.

#Resources

Getting started with Aria Hub Free Tier

Thu, 22 Dec 2022 00:00:00 GMT

The free tier of VMware Aria Hub was accounced VMware Explore back in November and I recently got access. This blog post covers the three steps required to get an AWS account connected and explores how we can start to query the inventory data.

The free tier allows you to connect up to two public cloud accounts from either Azure or AWS and there is no reliance on other VMware products or services, so this allows you to sign-up and experience how VMware Aria Hub powered by VMware Aria Graph can simplify the complexity of multi-cloud management.

#Data Source Setup

The very first step before we onboard our account is to access the service. Login to https://console.cloud.vmware.com and launch the VMware Aria Hub service.

VMware Cloud Services Portal - My Services

VMware Aria Hub Welcome screen

Select your data source, in this post we will be using AWS.

Create Data Source popup window

Provide a name and the AWS Account ID that you are connecting to.

Add AWS Data Source - Account Information step

Step 2 includes some steps to configure an IAM Role that will allow VMware Aria Hub to connect to our AWS account.

If you click the CREATE AWS IAM ROLE FOR VMWARE ARIA button it will open in a new window and auto populate the Account ID and External ID. The Account ID that is provided in this step is an external account.

Add AWS Data Source - Connect Account step

AWS IAM Role creation

We need to specifically attach the SecurityAudit policy to our new role.

AWS IAM Role creation - attach policies screen

No tags are required, but add them if your organisation uses them.

AWS IAM Role creation - add tags screen

Provide a name for your new role, I have used VMwareAriaHub. Make not of this role name as you will need it in a later step.

AWS IAM Role creation - review screen

Once the role has been created, select it and make note of the ARN.

AWS IAM Role summary screen

Paste the ARN value into the IAM Role ARN field in the Aria Hub portal. The External ID value should match the one you used when creating the IAM Role.

Add AWS Data Source - Connect Account step with ARN

This is the final step for onboarding our account and it is asking us for a couple of things

Enable CloudTrail in AWS for event monitoring purposes (we wont cover this step, but find CloudTrail and follow the steps to enable).
Run a script to configure an Event Stream stack that utilises CloudFormation.

Add AWS Data Source - Account Onboarding

Press the link Connect Event Stream to download the .sh script. I then uploaded this to AWS CloudShell. From CloudShell I ran the command specifed: bash cloud_account_onboarding.sh <ACCOUNT_ID>,<AWS_IAM_ROLE_NAME>

cloud_account_onboarding script output

Within AWS we can see under CloudFormation the creation of a new Stack.

cloudcoreo-events CloudFormation Stack - CREATE_IN_PROGRESS

cloudcoreo-events CloudFormation Stack - CREATE_COMPLETE

cloudcoreo-events CloudFormation Stack - Resources

#Explore the inventory

Now we have our data source connected we will start to see objects appear in our inventory. Aria Hub will start to pull in all types of inventory objects, including items that “out-of-the-box” from AWS themselevs, such as the IAM root user, EC2 network ACL’s etc. The objects are coming in from all AWS Regions.

Aria Hub Inventory

I have a single EC2 instance deployed, with the Name aria-demo and Instance ID i-069ecb9cb0e422211

AWS EC2 Instances

We can search for our instance in a couple of way, we can provide a generic search entityType = AWS.EC2.Instance and this will return all EC2 instances across all regions, or we can be specific and search using our Instance ID - or any of the properties attached to our EC2 instance, such as the Private IP Address.

Aria Hub Explore - Searching by entityType

Aria Hub Explore - Searching by Instance ID

The previous two screenshots show the same result in our case, but the next two start to show how powerful Aria Hub is.

When we search by the Private IP Address, Aria Hub finds two items that match the criteria, the AWS.EC2.Instance and also the AWS.EC2.NetworkInterface. Previously the NetworkInterface was shown as a connected object, when our Search Results only returned the Instance, but now we have two search results and we have the ability to expand each of their related objects.

Aria Hub Explore - Searching by PrivateIpAddress (EC2 Instance)

Aria Hub Explore - Searching by PrivateIpAddress (Network Interface)

That is it for this post, we have configured our AWS Data Source and started to explore our Inventory. In our next post Exploring Aria Hub Search and GraphQL we will continue the Inventory exploration and look at the Altair portal.

#Additional

After adding my data source I noticed a warning against the Data Source: Event stream is not connected, please configure event monitoring. This is not expected behaviour and I have raised this with VMware who are investigating. At present it does not seem to be impacting my discovered Inventory.

Data Source Event stream warning

UPDATE: (23rd December 2022)

It turns out this is because Secure State was not enabled within my Organization, after claiming the invite in the Aria Hub Free Tier welcome email for Secure State (which I initially missed!) and after deleting and onboarding the data source again, I am pleased to say the connection is much happier.

Data Source OK

vRA/AA - Windows Cloudbase-init - Function Update

Thu, 15 Dec 2022 00:00:00 GMT

Back in August I created the following post that showed an example Cloudbase-init multi-part user data script for use within an Aria Automation Template (vRealize Cloud Template) - vRA - Windows Cloudbase-init - Multi-part User Data Example

This post is a follow-on from that which expands on the Cloudbase-init code by utilising PowerShell functions. Cloudbase-init itself provides no guarantees that things will run, this can cause issues if you are relying on it to provide the intial customisation of virtual machines, so by using function we can provide some sort of control around when the next stages of the Cloudbase-init code are run - for example, we can wait for the network to be up before we attempt a domain join.

I will not be going into any detail around the Aria Automation Templates or the compute resources, instead we will focus just on the Cloudbase-init code. Refer back to the first post if you want to see the rest of the Template code.

#Establish the multi-part user data

You can find the full Cloudbase-init code at the bottom, but first we will break down what each main part is doing.

This is the start of our multi-part code. First we set our boundary ==NewPart==. When this appears we start a new “part” (section). Section 1 is using cloud-config and we set the hostname using the set_hostname Cloudbase-init plugin. Section 2 uses powershell, from this point down our code is run in PowerShell, we specifically use #ps1_sysnative.

  Content-Type: multipart/mixed; boundary="==NewPart=="
  MIME-Version: 1.0

  --==NewPart==
  Content-Type: text/cloud-config; charset="us-ascii"
  MIME-Version: 1.0
  Content-Transfer-Encoding: 7bit
  Content-Disposition: attachment; filename="cloud-config"

  set_hostname: ${self.resourceName}

  --==NewPart==
  Content-Type: text/x-shellscript; charset="us-ascii"
  MIME-Version: 1.0
  Content-Transfer-Encoding: 7bit
  Content-Disposition: attachment; filename="INIT.ps1"

  #ps1_sysnative

#The Functions

Next up is a logging function. Cloudbase-init itself creates a log output but it is easy to miss what is actually happening. This function is used throughout the Cloudbase-init code to write to a seperate log file. We have used this function quite heavily throughout our Cloudbase-init code, even within other functions, it will look similar to this: CustomLog("Create Local User - Started").

function CustomLog ($LogMessage) {
  $LogFilePath = "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\CustomLog.txt"
  Write-Output "$((Get-Date).ToString('T')) $LogMessage" | Out-File -FilePath $LogFilePath -Append
}

We establish our first of the “check” type functions. In this one we take in a property from a Property Group which is the FQDN of our Domain Server.

function DomainDnsTest {
  $DomainDnsTestResult = (Test-NetConnection -ComputerName "${propgroup.defaultServerValues.windowsDomainServer}" -Port 389).TcpTestSucceeded
  return $DomainDnsTestResult
}

This function check a property within Windows to see if it has domain joined.

function DomainJoinStatus {
  $DomainJoinStatusResult = (Get-WmiObject -Class Win32_ComputerSystem).PartOfDomain
  return $DomainJoinStatusResult
}

This function waits for the network interface status to return as “Up”. This status alone is not enough to know if a network is ready, hence why we create a DNS check too.

function WaitForNetwork {
  CustomLog("-- Starting WaitForNetwork. Starting status $((Get-NetAdapter -Name Ethernet0).Status)")
  While ((Get-NetAdapter -Name Ethernet0).Status -ne "Up") {
    CustomLog("-- Waiting for network Ethernet0 to be up. Current status $((Get-NetAdapter -Name Ethernet0).Status)")
    Start-Sleep -Seconds 2
  }
  CustomLog("-- Ending WaitForNetwork. Ending status $((Get-NetAdapter -Name Ethernet0).Status)")
}

This function utilises another nested function and waits for DNS to be ready.

function WaitForDns {
  CustomLog("-- Starting WaitForDns. Starting status $(DomainDnsTest)")
  While ((DomainDnsTest) -ne "True") {
    CustomLog("-- Waiting for DNS to succeed. Current status $(DomainDnsTest)")
    Start-Sleep -Seconds 2
  }
  CustomLog("-- Ending WaitForDns. Ending status $(DomainDnsTest)")
}

This functions code has some values replaced by the users Input or from Property Group values. It attempts to domain join the server. We use the Add-Computer cmdlet because the computer object already exists within Active Directory because Aria Automation has created it via its native integration.

function JoinDomain {
  CustomLog("-- Starting JoinDomain")
  WaitForNetwork
  WaitForDns
  $DomainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)
  Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $DomainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServer}" -Restart:$false -Force
  CustomLog("-- Ending JoinDomain")
}

Our final function, we use one of our earlier created functions (DomainJoinStatus) and we wait for that property to return true.

function WaitForDomainJoin {
  CustomLog("-- Starting WaitForDomainJoin. Starting status $(DomainJoinStatus)")
  While ((DomainJoinStatus) -ne "True") {
    CustomLog("---- Waiting for Domain to be True. Current status $(DomainJoinStatus)")
    CustomLog("---- Starting Sleep")
    Start-Sleep -Seconds 10
    CustomLog("---- Ending Sleep")
    JoinDomain
  }
  CustomLog("-- Ending WaitForDomainJoin. Ending status $(DomainJoinStatus)")
}

#The Remaining Parts

I have added comments inline with the rest of the code and each major section.

## We create a local user and add the user to the Administrators group.## Only the CustomLog function is used here.CustomLog("Create Local User - Started")
$localUserPassword = ConvertTo-SecureString "${input.serverPassword}" -AsPlainText -Force
New-LocalUser -Name "${input.serverUser}" -Password $localUserPassword
Add-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}"
CustomLog("Create Local User - Finished")

## We set some Windows Firewall settings.## Only the CustomLog function is used here.CustomLog("Set Local Firewall - Started")
Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled True
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0
CustomLog("Set Local Firewall - Finished")

## We format any additional disk drives that were added as part of the VM deployment request.## Only the CustomLog function is used here.CustomLog("Initialise Disk - Started")
Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA"
CustomLog("Initialise Disk - Finished")

## We configure some DNS settings. Only the CustomLog function is used here.## Only the CustomLog function is used here.CustomLog("Set DNS Settings - Started")
Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $False
Get-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $False
Get-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}
Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com", "sub.domain.com")
CustomLog("Set DNS Settings - Finished")

## We set the IP address, this uses the vRA IPAM and is injected at deployment time. ## Along with the CustomLog function we also use WaitForNetwork and WaitForDns.CustomLog("Set IP Settings - Started")
$adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }
$adapter | Remove-NetIpAddress -Confirm:$false
$adapter | Remove-NetRoute -Confirm:$false
$adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}
$adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,'", "')}")
$adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6
WaitForNetwork
WaitForDns
Set-NetConnectionProfile -InterfaceAlias Ethernet0 -NetworkCategory "Private"
CustomLog("Set IP Settings - Finished")

## At this point both Network and DNS should be ready so we attempt to domain join. ## We use the WaitForDomainJoin and CustomLog functions.CustomLog("Domain Join - Started")
WaitForDomainJoin
CustomLog("Domain Join - Finished")

## We tidy up some of the log files. ## Only the CustomLog function is used here.CustomLog("Tidying Logs - Started")
(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
CustomLog("Tidying Logs - Finished")

## We eject the CD drive that was attached automatically by vRA. ## Only the CustomLog function is used here.CustomLog("Ejecting CD Drives - Started")
$drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"
$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue
CustomLog("Ejecting CD Drives - Finished")

## We reboot the server. ## Only the CustomLog function is used here.CustomLog("Preparing to Reboot - Started")
shutdown /r /t 10
CustomLog("Preparing to Reboot - Finished")

#Cloud-config in full

cloudConfig: |
  Content-Type: multipart/mixed; boundary="==NewPart=="
  MIME-Version: 1.0

  --==NewPart==
  Content-Type: text/cloud-config; charset="us-ascii"
  MIME-Version: 1.0
  Content-Transfer-Encoding: 7bit
  Content-Disposition: attachment; filename="cloud-config"

  set_hostname: ${self.resourceName}

  --==NewPart==
  Content-Type: text/x-shellscript; charset="us-ascii"
  MIME-Version: 1.0
  Content-Transfer-Encoding: 7bit
  Content-Disposition: attachment; filename="INIT.ps1"

  #ps1_sysnative

  function CustomLog ($LogMessage) {
    $LogFilePath = "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\CustomLog.txt"
    Write-Output "$((Get-Date).ToString('T')) $LogMessage" | Out-File -FilePath $LogFilePath -Append
  }

  function DomainDnsTest {
    $DomainDnsTestResult = (Test-NetConnection -ComputerName "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Port 389).TcpTestSucceeded
    return $DomainDnsTestResult
  }

  function DomainJoinStatus {
    $DomainJoinStatusResult = (Get-WmiObject -Class Win32_ComputerSystem).PartOfDomain
    return $DomainJoinStatusResult
  }

  function WaitForNetwork {
    CustomLog("-- Starting WaitForNetwork. Starting status $((Get-NetAdapter -Name Ethernet0).Status)")
    While ((Get-NetAdapter -Name Ethernet0).Status -ne "Up") {
      CustomLog("-- Waiting for network Ethernet0 to be up. Current status $((Get-NetAdapter -Name Ethernet0).Status)")
      Start-Sleep -Seconds 2
    }
    CustomLog("-- Ending WaitForNetwork. Ending status $((Get-NetAdapter -Name Ethernet0).Status)")
  }

  function WaitForDns {
    CustomLog("-- Starting WaitForDns. Starting status $(DomainDnsTest)")
    While ((DomainDnsTest) -ne "True") {
      CustomLog("-- Waiting for DNS to succeed. Current status $(DomainDnsTest)")
      Start-Sleep -Seconds 2
    }
    CustomLog("-- Ending WaitForDns. Ending status $(DomainDnsTest)")
  }

  function JoinDomain {
    CustomLog("-- Starting JoinDomain")
    WaitForNetwork
    WaitForDns
    $DomainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)
    Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $DomainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Restart:$false -Force
    CustomLog("-- Ending JoinDomain")
  }

  function WaitForDomainJoin {
    CustomLog("-- Starting WaitForDomainJoin. Starting status $(DomainJoinStatus)")
    While ((DomainJoinStatus) -ne "True") {
      CustomLog("---- Waiting for Domain to be True. Current status $(DomainJoinStatus)")
      CustomLog("---- Starting Sleep")
      Start-Sleep -Seconds 10
      CustomLog("---- Ending Sleep")
      JoinDomain
    }
    CustomLog("-- Ending WaitForDomainJoin. Ending status $(DomainJoinStatus)")
  }

  CustomLog("Create Local User - Started")
  $localUserPassword = ConvertTo-SecureString "${input.serverPassword}" -AsPlainText -Force
  New-LocalUser -Name "${input.serverUser}" -Password $localUserPassword
  Add-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}"
  CustomLog("Create Local User - Finished")

  CustomLog("Set Local Firewall - Started")
  Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled True
  Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
  Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"
  Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"
  Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0
  CustomLog("Set Local Firewall - Finished")

  CustomLog("Initialise Disk - Started")
  Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA"
  CustomLog("Initialise Disk - Finished")

  CustomLog("Set DNS Settings - Started")
  Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $False
  Get-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $False
  Get-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}
  Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com", "sub.domain.com")
  CustomLog("Set DNS Settings - Finished")

  CustomLog("Set IP Settings - Started")
  $adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }
  $adapter | Remove-NetIpAddress -Confirm:$false
  $adapter | Remove-NetRoute -Confirm:$false
  $adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}
  $adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,'", "')}")
  $adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6
  WaitForNetwork
  WaitForDns
  Set-NetConnectionProfile -InterfaceAlias Ethernet0 -NetworkCategory "Private"
  CustomLog("Set IP Settings - Finished")

  CustomLog("Domain Join - Started")
  WaitForDomainJoin
  CustomLog("Domain Join - Finished")

  CustomLog("Tidying Logs - Started")
  (Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
  (Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
  CustomLog("Tidying Logs - Finished")

  CustomLog("Ejecting CD Drives - Started")
  $drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"
  $drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue
  CustomLog("Ejecting CD Drives - Finished")

  CustomLog("Preparing to Reboot - Started")
  shutdown /r /t 10
  CustomLog("Preparing to Reboot - Finished")

VMware Cloud Foundation - PowerShell Reporting Module

Mon, 05 Dec 2022 00:00:00 GMT

Day to day I am involved in architecting, designing, building, managing or fixing VMware Cloud Foundation environments. Given the potential size of these environments its easy to spend hours gathering information on them, so I was pleased when I was introduced to the VMware.CloudFoundation.Reporting module.

This is a short post highlighting this tool - the steps needed to get it installed and some information on the reports it provides.

#Available Reports

This module provides the following VMware Cloud Foundation reports:

Alert Report: Collects system alerts that are currently active across the VMware Cloud Foundation components.
Configuration Report: Collects information about the configuration settings in a VMware Cloud Foundation instance.
Health Report: he report contains detailed information about the health of the VMware Cloud Foundation system and its components by combining the SoS Utility health checks with additional health checks that were not previously available in earlier VMware Cloud Foundation releases.
Overview Report: Contains high-level information about the VMware Cloud Foundation system, can be used to provide a quick system overview.
Password Policy Report: Collects information about the password policy settings across VMware Cloud Foundation components.
Upgrade Pre-check Report: Initiates an upgrade precheck of a workload domain using the REST API and presents the results in an HTML report.

#Installing the Module

There are a few pre-requisite/supporting modules required for the VCF reporting module, you can install them all using the below:

Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
Install-Module -Name VMware.PowerCLI -MinimumVersion 12.4.1
Install-Module -Name VMware.vSphere.SsoAdmin -MinimumVersion 1.3.7
Install-Module -Name PowerVCF -MinimumVersion 2.2.0
Install-Module -Name PowerValidatedSolutions -MinimumVersion 1.7.0
Install-Module -Name VMware.CloudFoundation.Reporting

Validate the reporting module has been installed by running

Get-Command -Module VMware.CloudFoundation.Reporting

#Generate Reports

Replace the variable values with ones that suit your environment,

$sddcManagerFqdn = "sddc-manager.domain.com"
$sddcManagerUser = "admin@local"
$sddcManagerPass = "VMw@re1!VMw@re1!" 
$sddcManagerRootPass = "VMw@re1!"

$reportPath = "C:\VCF"

$mgmtWorkloadDomain = "lon-m01"
$viWorkloadDomain = "lon-w01"

#Overview report
Invoke-VcfOverviewReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath

#Health report
Invoke-VcfHealthReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -sddcManagerRootPass $sddcManagerRootPass -reportPath $reportPath -allDomains

#Alert report
Invoke-VcfAlertReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -allDomains

#Password Policy report
Invoke-VcfPasswordPolicy -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -allDomains

#Configuration report
Invoke-VcfConfigReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -allDomains

#Upgrade pre-check report
Invoke-VcfUpgradePrecheck -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -workloadDomain $mgmtWorkloadDomain
Invoke-VcfUpgradePrecheck -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -workloadDomain $viWorkloadDomain

#Bonus

The reports can also be generated in dark mode using the -darkMode flag when running the Invoke Report cmdlets

You can Get-Help for the available cmdlets: Get-Help -Name Invoke-VcfConfigReport

#Preview

#References

https://www.powershellgallery.com/packages/VMware.CloudFoundation.Reporting

vRA - Windows Cloudbase-init - Multi-part User Data Example

Wed, 17 Aug 2022 00:00:00 GMT

The following blog post runs through a vRA Cloud Template that deploys a Windows server using Cloudbase-init Multi-part user data, including cloud-config and PowerShell code to configure various things.

This post is broken into 4 main sections…

Cloud Template Inputs.
Cloud Template Resources.
Cloud-config for the Cloud.Machine resource.
A breakdown of the Cloud-config components.

#Cloud Template Inputs

name: Windows
version: 0.0.66
formatVersion: 1
inputs:
  image:
    type: string
    default: windows-server-2019-standard
    title: Operating System and Version
    oneOf:
      - title: Windows Server 2019 Standard
        const: windows-server-2019-standard
  flavor:
    type: string
    title: Server Size
    oneOf:
      - title: Standard Small (2CPU, 8GB RAM)
        const: s1.small
      - title: Standard Medium (2CPU, 16GB RAM)
        const: s1.medium
      - title: Standard Large (4CPU, 16GB RAM)
        const: s1.large
      - title: Standard Extra-Large (4CPU, 32GB RAM)
        const: s1.xlarge
  serverUser:
    type: string
  serverPassword:
    type: string
    encrypted: true
    title: Server Password
  serverCount:
    type: integer
    maximum: 5
    minimum: 1
    default: 1
    title: Number of Servers
  dataDisk1Size:
    type: integer
    title: Data Disk Size
    minimum: 0
    maximum: 1000
    default: 0
  domainPassword:
    type: string
    readOnly: false
    encrypted: true

#Cloud Template Resources

We have configured 3x resources on our Cloud Template, 1x Cloud.Volume (labelled DataDisk), 1x Cloud.Network (labelled Network) and 1x Cloud.Machine (labelled Server).

There are some specific properties worth calling out…

On the Server resource customizeGuestOs: false - this stops vRA customising the VM, this would usually configure the VM Name and IP by creating a temporary customisation spec at deployment time.
On the Server resource assignment: static - this pulls an IP from vRA’s IPAM.

resources:
  DataDisk:
    type: Cloud.Volume
    allocatePerInstance: true
    properties:
      capacityGb: ${input.dataDisk1Size}
      count: '${input.dataDisk1Size == 0 ? 0 : input.serverCount}'
  Network:
    type: Cloud.Network
    properties:
      networkType: outbound
      constraints:
        - tag: network.isolated:true
  Server:
    type: Cloud.Machine
    allocatePerInstance: true
    properties:
      customizeGuestOs: false
      image: ${input.image}
      flavor: ${input.flavor}
      count: ${input.serverCount}
      attachedDisks: ${map_to_object(slice(resource.DataDisk[*].id, count.index, count.index + 1), "source")}
      constraints:
        - tag: cloud.zone.region:dc3
      networks:
        - network: ${resource.Network.id}
          assignment: static
      cloudConfig: #see below section

vRA Cloud Template

#Cloudbase-Init (Cloud Config)

This cloudConfig utilises the multi-part user data (Multi-part content), this allows us to run a variety of content types, including cloud-config and PowerShell scripts.

The example below uses a combination of Inputs and Property Groups to get its data. The cloudConfig does the following:

Sets virtual machine hostname using cloud-config.
Creates a local user based on two inputs, one for the username, one for the password and then adds that user to the Administrators group.
Enables the “Remote Desktop” and “ICMPv4-In” firewall rules.
Sets some Registry Key values.
Formats the additional disk that was added in the Cloud Template.
Configures various network adapter settings, including the IP and DNS.
Sleeps between setting the network and Domain Join.
Joins the server to the Domain.
Removes any reference to entered passwords from the Cloudbase-init log files.
Ejects any CD-ROMs.
Reboots the server.

The virtual machine template is running Windows Server 2019, it has Cloudbase-init installed with the Cloudbase-init service set to “Automatic (Delayed)”. The Sysprep options were also unchecked on the Cloudbase-init installer. Refer to Michael Poore’s blog post on this - Installing Cloudbase-Init on Windows for vRA Customisation.

Below is the Cloud-config in full, we break down each section at the bottom.

#Cloud-config in full

cloudConfig: |
Content-Type: multipart/mixed; boundary="==NewPart=="
MIME-Version: 1.0

--==NewPart==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config"

set_hostname: ${self.resourceName}

--==NewPart==
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="INIT.ps1"

#ps1_sysnative

$localUserPassword = ConvertTo-SecureString ${input.serverPassword} -AsPlainText -Force
New-LocalUser -Name "${input.serverUser}" -Password $localUserPassword
Add-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}"

Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled True
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0

Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA"

Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $False
Get-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $False
Get-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}
Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com")

$adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }
$adapter | Remove-NetIpAddress -Confirm:$false
$adapter | Remove-NetRoute -Confirm:$false
$adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}
$adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,', ')}")
$adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6

Start-Sleep -Seconds 10

$domainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)
Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $domainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Restart:$false -Force

(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose

$drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"
$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue

shutdown /r /t 10

#Identify Multi-part boundary

We establish our multi-part boundary, which tells cloudConfig where a new type starts. Our boundary parameter is ==NewPart==.

cloudConfig: |
Content-Type: multipart/mixed; boundary="==NewPart=="
MIME-Version: 1.0

#Cloud-config Section

Our new section requires the use of our boundary parameter, this boundary also requires a prefix of --. In this section we set the virtual machine hostname using a reference the the resource itself - in otherwords, whatever the VM is called (based on Custom Names in vRA) we push it into the cloud-config section. This seems to automatically reboot the VM after the hostname is set.

In this section we specify the filename as cloud-config - Content-Disposition: attachment; filename="cloud-config". We can any cloud-config related commands in here. Refer to the Cloudbase-init docs for information - #cloud-config.

--==NewPart==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config"

set_hostname: ${self.resourceName}

#Start of the PowerShell Section

This section is the start of our PowerShell code, again we use our boundary and its double-dash prefix --==NewPart==. Again we set a filename but this time with the file extension .ps1 - Content-Disposition: attachment; filename="INIT.ps1". The addition of #ps1_sysnative tells cloudConfig what executable to use. Refer to the Cloudbase-init docs for information - #powershell.

--==NewPart==
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="INIT.ps1"

#ps1_sysnative

#Local User Creation

We utilise the Inputs for serverPassword and serverUser to add and configure our local user.

$localUserPassword = ConvertTo-SecureString ${input.serverPassword} -AsPlainText -Force
New-LocalUser -Name "${input.serverUser}" -Password $localUserPassword
Add-LocalGroupMember -Group "Administrators" -Member "${input.serverUser}"

#Set Firewall Rules

Set-NetFirewallRule -DisplayName "File and Printer Sharing (Echo Request - ICMPv4-In)" -enabled True
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

#Set Registry Values

We utilise values from a Property Group called defaultServerValues and specifically the value of property fqdnDomain. An example value could be domain.com.

Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name 'NV Domain' -Value "${propgroup.defaultServerValues.fqdnDomain}"
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' -Name SyncDomainWithMembership -Value "0"
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0

#Format Additional Disk

This code looks for all disks that have a matching partition style and then initialises them before applying a driver letter and label. We only add a single disk in this Cloud Template, hence a single label - if you add more than one disk you will need to adapt the code to suit.

Get-Disk | Where-Object PartitionStyle -Eq "RAW" | Initialize-Disk -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -NewFileSystemLabel "DATA"

#Configure Network DNS Client Settings

Again we utilise values from a Property Group called defaultServerValues and specifically the value of property fqdnDomain. An example value could be domain.com.

Get-NetAdapter | Set-DnsClient -RegisterThisConnectionsAddress $False
Get-NetAdapter | Set-DnsClient -UseSuffixWhenRegistering $False
Get-NetAdapter | Set-DnsClient -ConnectionSpecificSuffix ${propgroup.defaultServerValues.fqdnDomain}
Set-DnsClientGlobalSetting -SuffixSearchList @("domain.com")

#Configure Network DNS Client Settings

Our virtual machine template includes a single network adapter with the label Ethernet0, we first identify this network adapter object and we then configure the required values, including IP and DNS.

Within this Cloud Template we have configure the Machine resource to have assignment: static for it’s networking, so we are allocating an IP address from vRA’s IPAM.

On line 4 below we retrieve the IP address assigned to the virtual machine - ${self.networks[0].address}. For prefix and default gateway values we reference the attached Network resource - ${resource.Network.prefixLength} and ${resource.Network.gateway}.

Finally for line 5 we pull out the attached Networks DNS servers ${join(resource.Network.dns,', ')}. In this particular variable we also utilise one of vRA’s supported expression syntaxes join. Cloud Assembly expression syntax.

vRA Network Settings

$adapter = Get-NetAdapter | ? { $_.Name -eq "Ethernet0" }
$adapter | Remove-NetIpAddress -Confirm:$false
$adapter | Remove-NetRoute -Confirm:$false
$adapter | New-NetIpAddress -IPAddress ${self.networks[0].address} -PrefixLength ${resource.Network.prefixLength} -DefaultGateway ${resource.Network.gateway}
$adapter | Set-DnsClientServerAddress -ServerAddresses("${join(resource.Network.dns,', ')}")
$adapter | Disable-NetAdapterBinding -ComponentID ms_tcpip6

#Sleep

There is a slight delay between configuring the network and it being available, you could probably change this to a loop that better monitors the network status but during testing we found good success at the 10 second mark.

Start-Sleep -Seconds 10

#Join Active Directory Domain

Within vRA we have configured Active Directory integration and we have made it available to all of the Projects that require it. This creates the Computer Object in our specified OU, which leaves us to only do the domain joining from within the deployed guest OS.

We utilise more Property Group values in this code as well as an additional Input for the domainPassword, our specific approach for this Input was to provide a Default Value for it on the Custom Form and set the Visibility to No, this way the user does not see it. The reason for this approach is because Secrets cannot be shared across the Organization, they are Project specific, which would have caused us to have multiple copies of this specific value (1x per Project).

In the first line we create a PowerShell Credentials object by using two Property Group values ${propgroup.defaultServerValues.windowsDomainJoinUsername} and ${propgroup.defaultServerValues.windowsDomain}, in combination with the @ symbol we construct a UPN that will be used for the domain join, such as user@example.com, the second part of the Credentials object is the domainPassword value.

This Credentials object is then used on the second line along with some additional Property Group values, ${propgroup.defaultServerValues.windowsDomain}, for example ad.domain.com and ${propgroup.defaultServerValues.windowsDomainServerDc3} which is a specific server that we want to talk to, such as server1.ad.domain.com - this input is not required normally, but due to some network constraints we need to speak to a specific server.

$domainJoinCreds = New-Object System.Management.Automation.PSCredential "${propgroup.defaultServerValues.windowsDomainJoinUsername}@${propgroup.defaultServerValues.windowsDomain}" ,(ConvertTo-SecureString -String "${input.domainPassword}" -AsPlainText -Force)
Add-Computer -DomainName "${propgroup.defaultServerValues.windowsDomain}"-Credential $domainJoinCreds -Server "${propgroup.defaultServerValues.windowsDomainServerDc3}" -Restart:$false -Force

#Clean the Cloudbase-init Log Files

Due to the configured logging level of Cloudbase-init we can see the inputted passwords in clear text. We run the below PowerShell code to look for the Input values for domainPassword and serverPassword and replace them with some placeholder values - <DOMAIN-JOIN-PASSWORD> and <LOCAL-USER-PASSWORD>. An alternative would be to change the Cloudbase-init logging level.

(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.domainPassword}","<DOMAIN-JOIN-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose
(Get-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log") -replace "${input.serverPassword}","<LOCAL-USER-PASSWORD>" | Set-Content "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Verbose

#Eject CD-ROM

We remove the Cloud-config attached CD-ROM, im not sure if this is absolutely necessary but during testing we observed CD’s still being connected to the deployed VM.

$drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"
$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue

#Shutdown VM

Finally, we reboot the VM, this is so the VM recognise it is domain joined.

shutdown /r /t 10

Tanzu Community Edition - Azure Cluster

Thu, 10 Feb 2022 00:00:00 GMT

This post runs through the deployment of Tanzu Community Edition onto Azure - we will look at creating a Management Cluster, a Workload Cluster and the deployment of a demo application.

#Prepare Bootstrap Device/VM

Install the TCE Tanzu CLI - https://tanzucommunityedition.io/docs/latest/cli-installation/ \ You will need both kubectl and docker installed.

If you are using Windows and hit the x509 certificate error, follow the steps here to resolve: https://tanzucommunityedition.io/docs/latest/faq-cluster-bootstrapping/

When the CLI is installed;

tanzu management-cluster create --ui to open up the browser based installer on the bootstrap device.
Alternatively, use the following command to access the UI from another device tanzu management-cluster create --ui --bind <BOOTSTRAP_DEVICE_IP>:<PORT> --browser none

Tanzu Community Edition Installer UI

#Prepare Azure Environment

Follow the steps here to prepare for an Azure installation: https://tanzucommunityedition.io/docs/latest/azure-mgmt/

To quickly summarise what you will need to complete:

Get Tenant ID
Create App Registration
Get Application ID (Client ID)
Get Subscription ID
Configure Access control
Configure Client secret
Accept base image license
Configure SSH keys

#Deploy TCE Cluster

From the UI select the Azure deploy button, populate the Tenant ID, Client ID, Client Secret and Subscription ID fields and press Connect. \ Select the appropriate Region for your deployment, in my case it is UK South. \ Enter your public SSH key.

As this is my first TCE deployment I will be creating a new Resource Group - I have named it TCE

To save some costs on this initial installation we will deploy a Development cluster. \ Select an appropriate Instance Type from the dropdown list - you can use a service such as https://azureprice.net to find the out the cheapest option - for me it looks like Standard_F2s_v2 is the cheapest choice, giving me 2 vCPU and 4Gb memory, if I pay slightly more I can bump up to a Standard_D2s_v3 to give me 2 vCPU and 8Gb memory. Once all the fields are populated we will have our dev plan.

We do not need to provide any additional Metadata.
We can accept the default settings for the Antrea CNI.
We will leave Identity Management off for now.
Select an appropriate OS image, for this I have gone with Ubuntu 20.04.
We will skip the Tanzu Mission Control (TMC) registration.

Review and validate the configuration settings.

In the CLI Command Equivalent box you will see where the temporary YAML file is stored. Take note of the directory for this file, we can use it as a template for our Workload Cluster, it will look something like this: /home/tanzu/.config/tanzu/tkg/clusterconfigs/37elw85tmr.yaml

Press Deploy Management Cluster and follow along the installation

#Follow the Installation

The TCE deployment will run through multiple stages, you can monitor progress through the logs but you can also see whats happening via kubectl and the Azure portal.

Setup bootstrap cluster

After the image has been pulled you will see a container running within Docker, this is the bootstrap image which will do all the necessary things to then create a cluster within Azure.
When the Azure based management cluster is running, the bootstrap cluster will hand over to it.
Within the logs you will see reference to a kubeconfig file. Bootstrapper created. Kubeconfig: /home/tanzu/.kube-tkg/tmp/config_Jpd1gEfK - you can use this to run kubectl commands on the bootstrap cluster to see what is happening: kubectl get pod -A --kubeconfig /home/tanzu/.kube-tkg/tmp/config_Jpd1gEfK

Install providers on bootstrap cluster

This stage will bring up additional pods and deployments that are required for the provisioning of clusters into Azure.

Create management cluster

At this point activity will start within Azure, head to https://portal.azure.com and monitor the items that get created, such as a resource group, network, load balancers, security groups and VMs.
Within the logs you will see reference to another kubeconfig Saving management cluster kubeconfig into /home/tanzu/.kube/config.
You can set your context to the new management cluster to monitor progress

kubectl config get-contexts
CURRENT   NAME                                  CLUSTER          AUTHINFO               NAMESPACE
          tce-azure-mgmt-admin@tce-azure-mgmt   tce-azure-mgmt   tce-azure-mgmt-admin
          tce-mgmt-admin@tce-mgmt               tce-mgmt         tce-mgmt-admin
*         tce-wl01-admin@tce-wl01               tce-wl01         tce-wl01-admin
kubectl config use-context tce-azure-mgmt-admin@tce-azure-mgmt

Install providers on management cluster

This stage will bring up additional pods and deployments that are required for the management of clusters into Azure.

#Deploy Workload Cluster

We can use the yaml file that was created by the UI for our Management Cluster to create our Workload Cluster.

Copy the yaml used for the management cluster: cp /home/tanzu/.config/tanzu/tkg/clusterconfigs/37elw85tmr.yaml /home/tanzu/.config/tanzu/tkg/clusterconfigs/azure-wl01.yaml
Edit the yaml and tidy up to reduce any unnecessary info (see example below): vi /home/tanzu/.config/tanzu/tkg/clusterconfigs/azure-wl01.yaml
Deploy the cluster: tanzu cluster create tce-azure-wl01 --file /home/tanzu/.config/tanzu/tkg/clusterconfigs/azure-wl01.yaml -v 6

As we have not specified the names of VNET’s or SUBNETS’s, you will notice that they inherit the clusters name, such as tce-azure-wl01-vnet and tce-azure-wl01-node-subnet, other resources inherit this naming convention too. \ As we did specify the resource group (TCE), all the resources will be created inside this.

CLUSTER_PLAN: dev
NAMESPACE: default
CNI: antrea

AZURE_CONTROL_PLANE_MACHINE_TYPE: Standard_D2s_v3
AZURE_NODE_MACHINE_TYPE: Standard_D2s_v3
AZURE_TENANT_ID: <TENANT_ID>
AZURE_SUBSCRIPTION_ID: <SUBSCRIPTION_ID>
AZURE_CLIENT_ID: <CLIENT_ID>
AZURE_CLIENT_SECRET: <CLIENT_SECRET>
AZURE_LOCATION: uksouth
AZURE_SSH_PUBLIC_KEY_B64: <SSH_KEY> 
AZURE_RESOURCE_GROUP: TCE

CLUSTER_CIDR: 100.96.0.0/11
SERVICE_CIDR: 100.64.0.0/13
ENABLE_MHC: "true"
OS_ARCH: amd64
OS_NAME: ubuntu
OS_VERSION: "20.04"

Change to the context of your new workload cluster once it has been created:

tanzu cluster kubeconfig get tce-azure-wl01 --admin
kubectl config use-context tce-azure-wl01-admin@tce-azure-wl01

#Deploy Example Application

For this we will use the ACME fitness demo application - https://github.com/vmwarecloudadvocacy/acme_fitness_demo/tree/master/kubernetes-manifests

## Clone Repo
mkdir /home/tanzu/demo-apps
cd /home/tanzu/demo-apps
git clone https://github.com/vmwarecloudadvocacy/acme_fitness_demo.git
cd acme_fitness_demo/kubernetes-manifests

## Set ENV 
APP_PASSWORD=DemoApp123!
APP_NAMESPACE=acme-demo

## Deploy App
kubectl create ns ${APP_NAMESPACE}
kubectl create secret generic cart-redis-pass --from-literal=password=${APP_PASSWORD} -n ${APP_NAMESPACE} 
kubectl apply -f cart-redis-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f cart-total.yaml -n ${APP_NAMESPACE} 
kubectl create secret generic catalog-mongo-pass --from-literal=password=${APP_PASSWORD} -n ${APP_NAMESPACE} 
kubectl create -f catalog-db-initdb-configmap.yaml -n ${APP_NAMESPACE} 
kubectl apply -f catalog-db-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f catalog-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f payment-total.yaml -n ${APP_NAMESPACE} 
kubectl create secret generic order-postgres-pass --from-literal=password=${APP_PASSWORD} -n ${APP_NAMESPACE} 
kubectl apply -f order-db-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f order-total.yaml -n ${APP_NAMESPACE} 
kubectl create secret generic users-mongo-pass --from-literal=password=${APP_PASSWORD} -n ${APP_NAMESPACE} 
kubectl create secret generic users-redis-pass --from-literal=password=${APP_PASSWORD} -n ${APP_NAMESPACE} 
kubectl create -f users-db-initdb-configmap.yaml -n ${APP_NAMESPACE} 
kubectl apply -f users-db-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f users-redis-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f users-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f frontend-total.yaml -n ${APP_NAMESPACE} 
kubectl apply -f point-of-sales-total.yaml -n ${APP_NAMESPACE} 

## Monitor App
kubectl get pods -n ${APP_NAMESPACE} 
kubectl get service frontend -n ${APP_NAMESPACE}

Using the external IP listed against the service you should be able to browse to the website: http://<external-service-IP>

#Octant

Lets use Octant to view the cluster https://github.com/vmware-tanzu/octant/releases

cd /tmp
wget https://github.com/vmware-tanzu/octant/releases/download/v0.25.0/octant_0.25.0_Linux-64bit.tar.gz
tar -xzvf /tmp/octant_0.25.0_Linux-64bit.tar.gz
mv /tmp/octant_0.25.0_Linux-64bit /usr/bin/octant

I am starting octant with the following command so I can access the UI from another device, the value for the OCTANT_ACCEPTED_HOSTS variable is the IP of the octant host. \ OCTANT_ACCEPTED_HOSTS=$(/sbin/ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1) KUBECONFIG=/home/tanzu/.kube/config OCTANT_LISTENER_ADDR=0.0.0.0:8900 OCTANT_DISABLE_OPEN_BROWSER=true octant

#Tidy Up

#Delete workload cluster

Get the current Tanzu clusters and identify the name of the cluster to delete.

tanzu cluster list
NAME            NAMESPACE  STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES   PLAN
tce-azure-wl01  default    running  1/1           1/1      v1.21.2+vmware.1  <none>  dev

Start the cluster deletion. It will take a few minutes to clean up in Azure. Within the Azure portal you will start to see resources be deleted.

tanzu cluster delete tce-azure-wl01
Deleting workload cluster 'tce-azure-wl01'. Are you sure? [y/N]: y
Workload cluster 'tce-azure-wl01' is being deleted

Monitor the deletion of the cluster, you can use both kubectl and tanzu commands.

kubectl get cluster
NAME             PHASE
tce-azure-wl01   Deleting

tanzu cluster list
NAME            NAMESPACE  STATUS    CONTROLPLANE  WORKERS  KUBERNETES        ROLES   PLAN
tce-azure-wl01  default    deleting  1/1                    v1.21.2+vmware.1  <none>  dev

After the deletion has completed, the two commands will return no results.

kubectl get cluster
No resources found in default namespace.

tanzu cluster list
NAME  NAMESPACE  STATUS  CONTROLPLANE  WORKERS  KUBERNETES  ROLES  PLAN

You might need to delete the workload clusters context from your config file: kubectl config delete-context tce-azure-wl01-admin@tce-azure-wl01

#Delete management cluster

When the process to delete a management cluster is started a temporary kind cluster is stood up to manage the deletion. The kind cluster is then deleted after completion.

Retrieve the name of the management cluster.

tanzu management-cluster get
NAME            NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES
tce-azure-mgmt  tkg-system  running  1/1           1/1      v1.21.2+vmware.1  management

Start the deletion of the management cluster.

tanzu management-cluster delete tce-azure-mgmt
Deleting management cluster 'tce-azure-mgmt'. Are you sure? [y/N]: y

You can monitor the status of the temporary kind cluster through the use of docker and the temporary kubeconfig that is created.

Temporary kubeconfig files are stored in ~/.kube-tkg/tmp

docker ps -a
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED         STATUS         PORTS                       NAMES
e52dac6edc9b   projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1   "/usr/local/bin/entr…"   3 minutes ago   Up 2 minutes   127.0.0.1:37555->6443/tcp   tkg-kind-c82jls8gcv0cn7u54j30-control-plane

kubectl get cluster -A --kubeconfig /home/tanzu/.kube-tkg/tmp/config_jIv63FND
NAMESPACE    NAME             PHASE
tkg-system   tce-azure-mgmt   Deleting

Log output from deleting management cluster

tanzu management-cluster delete tce-azure-mgmt
Deleting management cluster 'tce-azure-mgmt'. Are you sure? [y/N]: y
Verifying management cluster...
Setting up cleanup cluster...
Installing providers to cleanup cluster...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-azure" Version="v0.4.15" TargetNamespace="capz-system"
Moving Cluster API objects from management cluster to cleanup cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Creating objects in the target cluster
Deleting objects from the source cluster
Waiting for the Cluster API objects to get ready after move...
Deleting management cluster...
Management cluster 'tce-azure-mgmt' deleted.
Deleting the management cluster context from the kubeconfig file '/home/tanzu/.kube/config'
warning: this removed your active context, use "kubectl config use-context" to select a different one

Management cluster deleted!

Your Azure portal should now be empty of any resources from this deployment.

Tanzu Community Edition - Tips and Concepts

Wed, 02 Feb 2022 00:00:00 GMT

VMware Tanzu Community Edition is a full-featured, easy to manage Kubernetes platform that is a freely available, community supported, open source distribution of VMware Tanzu.

The project enables the creation of application platforms through the use of Cluster API to provide declarative deployment and management of Kubernetes clusters.

This post covers a few tips and concepts for using and building with Tanzu Community Edition.

#1. Cluster Types

There are three primary types of clusters:

Management Clusters, a “permanent” cluster that is your interface for deploying managed Workload Clusters, use to perform lifecycle operations on managed clusters
Workload Clusters, a managed cluster, deployed via the Management Cluster, used to run workloads for services/apps, there can be multiple Workload Clusters under a single Management Cluster
Standalone Clusters, deployed via a temporary Management Cluster that is automatically deleted after provisioning, reduces overall system requirements and a “faster time to cluster”.

#2. Management Cluster Creation

There are two ways to deploy a Management Cluster:

Through the Tanzu Community Edition UI, which you can start by running: tanzu management-cluster create --ui
Directly through the Tanzu CLU, by running; tanzu management-cluster create --file FILE_PATH
Add the verbosity flags to get more output; -v / --verbose
You can create clusters in specific namespaces to keep them organised

#3. Delete Management Clusters

To delete a Management Cluster:

If it’s still up and accessible: tanzu management-cluster delete MGMT_CLUSTER_NAME
If it’s stuck and won’t delete, try adding the –force flag
If it’s been manually deleted or inaccessible: tanzu config server delete MGMT_CLUSTER_NAME
Add the --yes flag to avoid confirmation prompts

#4. Delete Workload Clusters

To delete Workload Clusters:

Delete them all; tanzu cluster delete
Delete a specific cluster: tanzu cluster delete WORKLOAD_CLUSTER_NAME
Add the --yes flag to avoid confirmation prompts

#5. Clusters on vSphere

If you are about to deploy your TCE clusters on vSphere, there are a few things to check:

The target network for the Nodes (Management and Workload) has DHCP enabled
A set of static IP addresses in the same subnet as DHCP but not in the DHCP range – the static IP’s are used by Kube-Vip for the API server endpoint
All Nodes can reach vCenter (FQDN/IP) or port 443
Traffic is allowed between the bootstrap machine and all Nodes on port 6433
Add DHCP reservations for deployed Nodes after they have been created

#6. Tanzu Packages

Use tanzu package to discover, configure and manage bundled software, known as packages, within clusters. Packages utilise a Package Repository that is installed into a cluster, this then allows the packages for installation. Some example packages include:

cert-manager https://cert-manager.io/docs/
Velero https://twitter.com/projectvelero
Harbor https://twitter.com/project_harbor
Contour https://twitter.com/projectcontour
ExternalDNS https://github.com/kubernetes-sigs/external-dns

#7. Scale Clusters

Management and Workload Clusters can both be scaled through the Tanzu CLI, with flags for scaling control plan and worker machine counts.

tanzu cluster scale WORKLOAD_CLUSTER_NAME --controlplane-machine-count 5 --worker-machine-count 10 --namespace NAMESPACE
tanzu cluster scale MGMT_CLUSTER_NAME --controlplane-machine-count 5 --worker-machine-count 10 --namespace NAMESPACE
Clusters can be vertically scaled by modifying MachineTemplates: https://cluster-api.sigs.k8s.io/tasks/updating-machine-templates.html

#8. Cluster API

Tanzu Community Edition uses Cluster API for the lifecycle management of Kubernetes clusters. A few definitions to be aware of:

Machine, an infrastructure component, such as a VM, hosting a Kubernetes Node.
MachineDeployment, behaves similarly to a Kubernetes Deployment, provides declarative updates to Machines and MachineSets.
MachineSet, like Kubernetes ReplicaSets, these form part of a MachineDeployment and ensure a stable set of Machines are running and in desired state.
MachineHealthCheck, monitors the status of Nodes, if deemed unhealthy, and if part of a MachineSet, the unhealthy Machine is deleted and a new one is created to replace it

#9. Cluster Bootstrapping on Windows

When deploying a Management Cluster on Windows you might get an x509 certificate error: x509: certificate signed by unknown authority. \ You can work around this by updating the Tanzu config file found at %USERPROFILE%.config\tanzu\tkg\config.yaml

Tanzu and vRealize - VMworld 2021

Wed, 13 Oct 2021 00:00:00 GMT

VMworld 2021 has been and gone, it was delivered as a virtual event again with the hope and aim that next years events will be in person - in both Barcelona and San Francisco, this year had a heavy focus around multi-cloud and how these clouds can provide advantages into application development and delivery.

With Multi-Cloud becoming a more viable concept, it raises plenty of new challenges around control, adoption and operability. VMware have captured what they believe are the core elements to a successful cloud transformation, they have labelled this the VMware Cloud Operating Model. This operating model was a key focus area for VMworld and the majority of other announcements feed directly into this.

Below I have summarised the announcements across the vRealize Suite and Tanzu Portfolios.

#vRealize Suite

With a continued focus on Multi-Cloud and Cloud Management, it is no surprise that the vRealize Suite has seen multiple enhancements and releases.

Project Ensemble: A tech preview was announced for Project Ensemble, that brings single UI with persona and app-centric views for a unified experience across all vRealize Cloud Management services.
vRealize Cloud Universal: Combine SaaS and on-premises capabilities for automation, operations, log analytics, and network visibility with vRealize Cloud Universal. You get the flexibility to deploy as SaaS or on-premises and the freedom to move between the two as needed.
vRealize Suite Lifecycle Manager 8.6: Topology view of vRealize product deployments, notification enhancements, and additional capability for managing vRealize products, such as snapshot management and cloud proxy deployment.
vRealize Automation Cloud and 8.6: New multi-cloud capabilities, particularly focused around Azure. New integration support for VMware Cloud Director environments and improvements to its VMware vSphere with Tanzu integration.
vRealize Operations Cloud and 8.6: Improved integration between vROPs and vRA and also public clouds, such as AWS, Azure and GCP. New management packs and bi-directional integration with CloudHealth
vRealize True Visibility Suite 2.0: Storage and Compute management packs now included with every licensed level of vROPs
vRealize AI Cloud: New beta features to support vRA Cloud and improved network optimisation within vRNI. The product also saw added capability around Smart Initial Sizing and Placement, adding KPI based statistics that better align to business value.
vRealize Log Insight Cloud and 8.6: Continued improvements around the logging capability for both cloud and on-premises products. Enhanced regional availability and AI powered root cause analysis for vRLI Cloud.
vRealize Cloud Universal Standard Add-on for Horizon (MP4H): A new monitoring add-on for all VMware Horizon customers that adds capabilities and enhancements into vROPs.
vRealize Network Insight Cloud and 6.4: New locations for vRNI Cloud, including Frankfurt and Central Canada. Added capabilities and integrations for NSX Federation, VMware Cloud and ServiceNow.
CloudHealth: Introduction of line items, allowing better visibility and control for chargeback and showback across multi-cloud environments.
CloudHealth Secure State: Two new capabilities focused on the security and risk visibility across cloud infrastructure; Kubernetes Security Posture Management (KSPM) and Explore - a powerful search feature for multi-cloud deployments, allowing the effective management
VMware Skyline: Skyline Advisor Pro is a new offering for Production, Premier, vRealize Cloud Universal, and Success 360 Customers that promises to provide faster, smarter and simpler insights into issues, remediation suggestions and findings across its managed environments.

#Tanzu

Tanzu Community Edition: A full featured and easy to manage Kubernetes platform based on the existing Tanzu Kubernetes Grid offering. TCE provides an opinionated approach to deploying Kubernetes clusters utilising Cluster API.
Tanzu Application Platform/Service: Continued improvements to the modular and application-aware platform for developers - some improvements include better developer tooling, such as IDE plugins and extensions, supply chain choreography based on the Cartographer open source project, and security enhancements.
Tanzu Mission Control: A new Starter Plan - a free tier to Tanzu Mission Control - allowing access to the multi-cloud management plane, for better visibility and control into Kubernetes clusters. TMC also has added capability around the lifecycle of Tanzu Kubernetes Grid clusters deployed into Azure. TMC Essentials will be included in VMware Cloud with Tanzu Services.
VMware Cloud with Tanzu Services: A portfolio of managed Kubernetes services built on top of VMware Cloud.
VMware Tanzu Kubernetes Grid GPU Support: VMware have announced GPU support with vSphere with Tanzu (TKGS) and also with Tanzu Kubernetes Grid (TKG) on AWS and Azure, helping organisations avoid AI silos for developers.
Tanzu Observability: AI/ML capabilities to aide in root cause analysis (Automated Probable Root Cause), enhanced Prometheus Query Language support, improved alerting processes and integrations into vROPs Cloud.

Sources \ https://blogs.vmware.com/management/2021/10/whats-new-in-cloud-management-at-vmworld-2021.html \ https://blogs.vmware.com/management/2021/10/introducing-vmware-cloud-director-support-in-vrealize-automation.html \ https://tanzu.vmware.com/content/blog/vmware-tanzu-community-edition-kubernetes-cloud-native-ecosystem \ https://tanzu.vmware.com/content/blog/

Custom Naming in vRA8

Wed, 14 Jul 2021 00:00:00 GMT

VMware vRealize Automation 8 provides several methods for applying custom naming to deployed virtual machines. This article will explore a few of these.

The testing/development within this article was based on vRA 8.2. VMware are providing frequent updates to the vRA product and some areas may be different if you are using an older/newer version.

We will only be focusing on the naming of the virtual machine object, not the hostname/FQDN within the VM OS.

The blog covers the following methods:

No Custom Naming / Out-of-the-Box Default
Name Property
Project Custom Naming Template
ABX Action
vRO Workflow

#No Custom Naming / Out-of-the-Box Default

If you do not apply any of the methods below, vRA automatically names the deployed virtual machine based on the following:

Resource name (e.g. SERVER or Cloud_Machine_1)
mcm### (provided by vRA)
############ (provided by vRA)

In this instance our Cloud Template looks like this…

name: Custom Naming Test
formatVersion: 1
inputs: {}
resources:
  SERVER: # This is the resource name used, by default it will be something like Cloud_Machine_#
    type: Cloud.Machine
    properties:
      image: 'TF-CentOS7'
      flavor: 'B'

Final result for our VM name: SERVER-mcm809-174067533632

#Name Property

Extending slightly on the method above, we could pass in a custom value to the name property under our Compute resource. I am using static values in the example Cloud Template, but this could be more dynamic through the use of Inputs.

name: Custom Naming Test
formatVersion: 1
inputs: {}
resources:
  SERVER:
    type: Cloud.Machine
    properties:
      name: 'myservername' # This overwrites the resource name above
      image: 'TF-CentOS7'
      flavor: 'B'

Final result for our VM name: myservername-mcm812-174067802073

#Project Custom Naming Template

A Project Custom Naming template can be applied to each Project within vRA. This naming template is then applied to machines, networks, security groups and disks provisioned under the Project.

The Custom Naming template used for this example looks like this: ${resource.serviceTier}-${resource.environment}-${resource.role}${####}${resource.datacentreLetter}

To make this Cloud Template work, we need to provide values for the properties role, serviceTier, environment and datacentreLetter. In the Cloud Template example below, under the machine resource we have the properties section, this is where we put our required values. Again, these could be driven by Inputs.

The #### value within the custom naming template is replaced at deployment time by numbers, this is managed by vRA. This is not required but helps to ensure resource names are unique if you don’t have any other method of checking for uniqueness (suggestions for this in the vRO section below)

name: Custom Naming Test
formatVersion: 1
inputs: {}
resources:
  SERVER:
    type: Cloud.Machine
    properties:
      role: 'web'
      serviceTier: 'silver'
      environment: 'dev'
      datacentreLetter: 's'
      image: 'TF-CentOS7'
      flavor: 'B'

Final result for our VM name: silver-dev-web0277s (serviceTier-environment-role####datacentreLetter)

In addition to using Properties from the Cloud Template you can utilise the Custom Properties section within the Project. These can then be referenced with ${resource.propertyName} in the Custom Naming template.

#ABX Action

vRA provides an ABX template that can be used for renaming a resource, by default this expects a property called newName. We will modify the action slightly to use the properties in our existing Cloud Template.

As the properties we added are “custom properties” we will need to pull them from the customProperties object as part of the inputs.

exports.handler = function handler(context, inputs) {

    //Get original resource name
    let oldName = inputs.resourceNames[0];
    //Get random number 
    let number = Math.floor(1000 + Math.random() * 9000);
    //Retrieve required properties
    let customProps = inputs.customProperties;
    let role = customProps.role;
    let serviceTier = customProps.serviceTier;
    let environment = customProps.environment;
    let datacentreLetter = customProps.datacentreLetter;

    //Construct new name
    let newName = serviceTier + "-" + environment + "-" + role + number + datacentreLetter;

    //Return new name as "outputs"
    let outputs = {};
    outputs.resourceNames = inputs.resourceNames;
    outputs.resourceNames[0] = newName;
    console.log("Setting machine name from '" + oldName + "' to '" + newName + "'");
    return outputs;
};

Once the action has been created we will add a Subscription and subscribe to the Compute allocation event.

Final result for our VM name: silver-prod-web6769s (serviceTier-environment-role####datacentreLetter)

#vRO Workflow

An alternative to the ABX action is to utilise vRealize Orchestrator. We can use almost the same code within our workflow to generate our name, with a few changes.

I won’t go in to detail on how to use vRO here, but there are some additional reference at the bottom of this post that provide a similar outcome but go into much more detail on how to construct the vRO Workflow.

Our workflow needs 1x Input inputProperties and 1x Output resourceNames. The input will be passed to vRO by vRA, and the output then returned to vRA.

For now our Workflow only needs one item, a Scriptable task that we will label Get Properties and Prepare Name.

The script will need the same inputs and outputs as our main workflow.

The code for our script item will be:

//Get original resource name
var oldName = inputProperties.resourceNames[0];
//Get random number 
var number = Math.floor(1000 + Math.random() * 9000);
//Retrieve required properties
var customProps = inputs.customProperties;
var role = customProps.role;
var serviceTier = customProps.serviceTier;
var environment = customProps.environment;
var datacentreLetter = customProps.datacentreLetter;

//Construct new name
var newName = serviceTier + "-" + environment + "-" + role + number + datacentreLetter;

//Return new name as "resourceNames"
resourceNames = [];
resourceNames[0] = newName;
System.log("Setting machine name from '" + oldName + "' to '" + newName + "'");

Once the workflow is ready, we just need to update the Subscription that we created earlier and swap out the ABX action for the vRO workflow.

Request your deployment and observe vRO for a Workflow run.

Final result for our VM name: silver-prod-web6477s (serviceTier-environment-role####datacentreLetter)

#Extending the Custom Naming

With the base of our Custom Naming in place, we need to think about duplicate detection and how we handle conflicting names.

One way we could do this is with vRO, we can add additional integrations and endpoints that connect with PowerShell Hosts, Active Directory, ITSM/CMDB systems, and most other things with vRO Plugins or REST API’s.

From here we could use these additional systems to confirm that the name we have generated is unique, for example;

Active Directory to ensure no other computer objects exist with the same name
PowerShell Hosts to connect to DNS servers to ensure there are no existing DNS entries for our new name
CMDB tools to check our name is unique across all other systems, and to also register our new name so we don’t generate it again in the future

Custom Naming is a key aspect of automated deployments, but be mindful that this will rely on other systems to be successful.

#Additional Reading

https://www.stevenbright.com/2020/02/custom-hostname-generation-in-vrealize-automation-8 \ https://blog.v12n.io/custom-naming-in-vrealize-automation-8x-2 \ https://virtuallypotato.com/vra8-custom-provisioning-part-two

Azure AKS - Failed to provision volume with StorageClass "default"

Mon, 07 Dec 2020 00:00:00 GMT

I wanted to deploy Concourse CI in to my Azure AKS cluster, which I hadn’t touched for a while, and encountered the error Failed to provision volume with StorageClass "default".

This post will cover my troubleshooting steps and ultimately the resolution - which was to update/rotate my Azure AKS credentials.

I followed the HELM install steps from Concourse CI https://github.com/concourse/concourse-chart.

#Troubleshooting Steps

To start I checked the status of the Pods deployed, they all showed Pending: kubectl get pods.

NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE
default       concourse-ci-postgresql-0                    0/1     Pending   0          5m11s
default       concourse-ci-web-d6bc9f97d-pr5zd             0/1     Running   1          5m11s
default       concourse-ci-worker-0                        0/1     Pending   0          5m11s
default       concourse-ci-worker-1                        0/1     Pending   0          5m11s

I investigated the logs from pod concourse-ci-web-d6bc9f97d-pr5zd: kubectl logs concourse-ci-web-d6bc9f97d-pr5zd.

These showed an error trying to reach the database. "error":"dial tcp: lookup concourse-ci-postgresql on 10.0.0.10:53: no such host"

{"timestamp":"2020-12-07T10:57:08.526432188Z","level":"info","source":"atc","message":"atc.cmd.start","data":{"session":"1"}}
{"timestamp":"2020-12-07T10:57:08.561956604Z","level":"error","source":"atc","message":"atc.db.failed-to-open-db-retrying","data":{"error":"dial tcp: lookup concourse-ci-postgresql on 10.0.0.10:53: no such host","session":"3"}}
{"timestamp":"2020-12-07T10:57:13.671634392Z","level":"error","source":"atc","message":"atc.db.failed-to-open-db-retrying","data":{"error":"dial tcp: lookup concourse-ci-postgresql on 10.0.0.10:53: no such host","session":"3"}}
{"timestamp":"2020-12-07T10:57:18.707090349Z","level":"error","source":"atc","message":"atc.db.failed-to-open-db-retrying","data":{"error":"dial tcp: lookup concourse-ci-postgresql on 10.0.0.10:53: no such host","session":"3"}}

Due to the database connection error above, I looked in to the concourse-ci-postgresql-0 pod. I did a describe this time to see what Events were being produced: kubectl describe pod concourse-ci-postgresql-0.

The top event shows us that it is having issues with scheduling and binding to PersistentVolumeClaims: Warning FailedScheduling 11m default-scheduler running "VolumeBinding" filter plugin for pod "concourse-ci-postgresql-0": pod has unbound immediate PersistentVolumeClaims.

Name:           concourse-ci-postgresql-0
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app.kubernetes.io/instance=concourse-ci
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=postgresql
                controller-revision-hash=concourse-ci-postgresql-5c7bfbd7bb
                helm.sh/chart=postgresql-9.2.0
                role=master
                statefulset.kubernetes.io/pod-name=concourse-ci-postgresql-0
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/concourse-ci-postgresql
Containers:
  concourse-ci-postgresql:
    Image:      docker.io/bitnami/postgresql:11.8.0-debian-10-r76
    Port:       5432/TCP
    Host Port:  0/TCP
    Requests:
      cpu:      250m
      memory:   256Mi
    Liveness:   exec [/bin/sh -c exec pg_isready -U "concourse" -d "dbname=concourse" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/sh -c -e exec pg_isready -U "concourse" -d "dbname=concourse" -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
] delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:           false
      POSTGRESQL_PORT_NUMBER:  5432
      POSTGRESQL_VOLUME_DIR:   /bitnami/postgresql
      PGDATA:                  /bitnami/postgresql/data
      POSTGRES_USER:           concourse
      POSTGRES_PASSWORD:       <set to the key 'postgresql-password' in secret 'concourse-ci-postgresql'>  Optional: false
      POSTGRESQL_ENABLE_LDAP:  no
      POSTGRESQL_ENABLE_TLS:   no
    Mounts:
      /bitnami/postgresql from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zpr9z (ro)
  Type           Status
  PodScheduled   False
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-concourse-ci-postgresql-0
    ReadOnly:   false
  dshm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  1Gi
  default-token-zpr9z:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zpr9z
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  11m   default-scheduler  running "VolumeBinding" filter plugin for pod "concourse-ci-postgresql-0": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  11m   default-scheduler  running "VolumeBinding" filter plugin for pod "concourse-ci-postgresql-0": pod has unbound immediate PersistentVolumeClaims

Next step was to have a look at the PersistentVolumeClaims: kubectl get pvc.

Again, like the pods, these were all Pending.

NAME                                       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
concourse-work-dir-concourse-ci-worker-0   Pending                                      default        14m
concourse-work-dir-concourse-ci-worker-1   Pending                                      default        14m
data-concourse-ci-postgresql-0             Pending                                      default        14m

I investigated one of the PVC’s: kubectl describe pvc concourse-work-dir-concourse-ci-worker-0.

Within the messages we can see quite clearly what the problem is: Failed to provision volume with StorageClass "default": Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/subscriptions/8af15f99-1234-43e2-8876-55bd7e6a727b/resourceGroups/mc_development_uksouth/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-f52878cf-5fb4-4de5-82f0-e9b284b29a9d?api-version=2019-07-01

Name:          concourse-work-dir-concourse-ci-worker-0
Namespace:     default
StorageClass:  default
Status:        Pending
Volume:
Labels:        app=concourse-ci-worker
               release=concourse-ci
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    concourse-ci-worker-0
Events:
  Type     Reason              Age                  From                         Message
  ----     ------              ----                 ----                         -------
  Warning  ProvisioningFailed  15m                  persistentvolume-controller  Failed to provision volume with StorageClass "default": Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/subscriptions/8af15f99-1234-43e2-8876-55bd7e6a727b/resourceGroups/mc_development_uksouth/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-f52878cf-5fb4-4de5-82f0-e9b284b29a9d?api-version=2019-07-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\nTrace ID: 8f13a52c-9f01-4012-9da2-4127e054cf00\r\nCorrelation ID: 054bec33-1b44-456f-b91b-4cb8bc4967e1\r\nTimestamp: 2020-12-07 10:55:24Z","error_codes":[7000222],"timestamp":"2020-12-07 10:55:24Z","trace_id":"8f13a52c-9f01-4012-9da2-4127e054cf00","correlation_id":"054bec33-1b44-456f-b91b-4cb8bc4967e1","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

Next step was to resolve the storage provisioning issues. Using information from the error above, and some Googling, I came across the following GitHub issue which showed the same error we were recieving https://github.com/Azure/AKS/issues/222.

This GitHub issue included a comment to the following Microsoft documentation page for updating credentials. https://docs.microsoft.com/en-us/azure/aks/update-credentials.

After completing the steps provided, I checked the same PVC as earlier to check on its status: kubectl describe pvc concourse-work-dir-concourse-ci-worker-0. We can now see that the volume has been provisioned: Successfully provisioned volume pvc-f52878cf-5fb4-4de5-82f0-e9b284b29a9d using kubernetes.io/azure-disk

Name:          concourse-work-dir-concourse-ci-worker-0
Namespace:     default
StorageClass:  default
Status:        Bound
Volume:        pvc-f52878cf-5fb4-4de5-82f0-e9b284b29a9d
Labels:        app=concourse-ci-worker
               release=concourse-ci
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    concourse-ci-worker-0
Events:
  Type     Reason                 Age                   From                         Message
  ----     ------                 ----                  ----                         -------
  Normal   ProvisioningSucceeded  66s                   persistentvolume-controller  Successfully provisioned volume pvc-f52878cf-5fb4-4de5-82f0-e9b284b29a9d using kubernetes.io/azure-disk

Checking the status of the pods again, we can see they are now Running/Creating: kubectl get pod.

NAME                               READY   STATUS              RESTARTS   AGE
concourse-ci-postgresql-0          0/1     ContainerCreating   0          46m
concourse-ci-web-d6bc9f97d-pr5zd   0/1     Running             14         46m
concourse-ci-worker-0              1/1     Running             0          46m
concourse-ci-worker-1              0/1     Pending             0          46m

#References

https://github.com/Azure/AKS/issues/222 \ https://docs.microsoft.com/en-us/azure/aks/update-credentials

Triggering vRO Workflow with SRM Recovery Plan

Tue, 01 Dec 2020 00:00:00 GMT

#Introduction

VMware Site Recovery Manager (SRM) provides you with the ability to create Custom Recovery Steps as part of a Recovery Plan.

There are two types of Custom Recovery Step, Command and Message Prompt. Each of these can be run as “top-level” steps in the recovery plan or on a “per-vm” basis during pre- and post- power on phases.

This post will explore triggering a vRO Workflow as a “top-level” step in a Recovery Plan, to summarise the steps we will go through:

Create vRO workflow.
Create shell script on SRM Appliance.
Create Recovery Plan with Custom Step.
Run a Recovery test.

If you are following this step for step, you will need:

Working deployment of SRM and the ability to create Recovery Plans.
Access to the SRM appliances to create local scripts.
Working deployment of vRO and access to create workflows.
An account with permissions to execute workflows.

#Create vRO Workflow

Create a new workflow with 3 inputs, all of type string:

recoveryPlanName
recoveryPlanMode
recoveryVmwareVcHost

Map these inputs to a Scripting object that has the following contents.

System.debug("Recovery Plan Name: " + recoveryPlanName);
System.debug("Recovery Plan Mode: " + recoveryPlanMode);
System.debug("Recovery Plan VC: " + recoveryVmwareVcHost);

Once the workflow has been created, take note of the workflow ID. If you are using the Java Client you will find this in the ID field on the “General” tab of the workflow. If you are using the HTML5 client you will find it in the URL bar when in edit mode.

You can also download the Workflow and import it: SRM Recovery Plan Details.zip

#Environment Variables

Site Recovery Manager sets the following environment variables for the duration of the command step. Тhe environment variables do not exist in Site Recovery Manager Server or the guest OS of the recovered VM when the command is completed. We will utilise these environment variables within our shell script, and you might have noticed they provide the values that our workflow is looking for as inputs.

VMware_RecoveryName: Name of the recovery plan that is running.
VMware_RecoveryMode Recovery mode.
VMware_VC_Host Host name of the vCenter Server at the recovery site.
VMware_VC_Port Network port used to contact vCenter Server.

#Create Shell Script

This script will trigger the vRO workflow we have just created. It will need to be stored on the SRM server. We are using the Appliance version of SRM, if you are using the Windows version you will need to modify the commands used and save it as .bat instead of .sh.

Connect to the SRM Appliance via SSH and login as user admin.

When the recovery plan is run, this script will be triggered by the Command Custom Step, and it is run on the recovery site appliance as user srm. I would suggest adding this script to both the protected site and recovery site Appliances - this will allow you to execute the script in either direction. This script executes a function that makes a POST request to the vRO server with a payload that maps to our Workflow inputs.

#Create the script file
vi /home/admin/trigger-vro-workflow.sh

Add the following contents. You will need to modify the values for the variables on the highlighted lines. (VRO_SERVER, VRO_WORKFLOW_ID, VRO_USERNAME, VRO_PASSWORD).

#!/bin/bash

#FQDN or IP of the vRO server.
VRO_SERVER="vroserver01.domain.com"#Workflow ID to execute.
VRO_WORKFLOW_ID="67748eec-e51c-8dd9-ac57-a7ca789a706c"#Username and Password with the required credentials to execute the specified workflow.
#Recommendation is to create a service account for just this purpose, with reduced permissions to workflows.
#Or modify the script to retirieve the credentials from something like Hashicorp Vault
VRO_USERNAME="administrator@vsphere.local"VRO_PASSWORD="password1!"
#Autogenerated
DATE=$(date "+%Y-%m-%d_%H-%M-%S")
VRO_URL=https://$VRO_SERVER/vco/api/workflows/$VRO_WORKFLOW_ID/executions

vro_function() {
  echo "$DATE: Recovery Plan $VMware_RecoveryName ran in $VMware_RecoveryMode mode"
  echo "vRO REST Call: $VRO_URL"

  curl -k $VRO_URL \
    -H "Accept: application/json" \
    -H "Content-Type:application/json" \
    --user $VRO_USERNAME:$VRO_PASSWORD \
    -d "$(
      cat <<EOF
    {
      "parameters": [
        {
          "value": {
            "string": {
              "value": "$VMware_RecoveryName"
            }
          },
          "type": "string",
          "name": "recoveryPlanName",
          "scope": "local"
        },
        {
          "value": {
            "string": {
              "value": "$VMware_RecoveryMode"
            }
          },
          "type": "string",
          "name": "recoveryPlanMode",
          "scope": "local"
        },
        {
          "value": {
            "string": {
              "value": "$VMware_VC_Host"
            }
          },
          "type": "string",
          "name": "recoveryVmwareVcHost",
          "scope": "local"
        }
      ]
    }
EOF
    )"
echo  #add new line in log file
echo "------------------"

} >> /tmp/SRM-VRO-Output.log 2>&1
vro_function

exit 0

When the script is triggered through the Custom step, it is run as user srm. To support this, we need to change the permissions on our script.

#Change the scripts access permissions, to allow the user "srm" to execute it
chmod 755 /home/admin/trigger-vro-workflow.sh

#Create/Modify SRM Recovery Plan

Create or Modify an SRM Recovery Plan, on the Recovery Steps tab, right click on Power on priority 1 VMs and select “Add Step Before”.

Type: Command on SRM Server
Name: vRO Workflow
Content: /bin/sh -c /home/admin/trigger-vro-workflow.sh
Timeout: 5 minutes

Execute a TEST of the Recovery Plan, you should hopefully see a success against the vRO Workflow step.

SRM Recovery Plan insert step context menu

Custom recovery step options and settings modal

SRM Recovery Plan steps after custom step has been added

SRM Recovery Plan test with successful mark against the custom step

#Check Appliance Log Files and vRO Workflow Log

Check the Workflow within vRO, you should see a new run. Check the logs to see details from our recovery plan. \ You should see an output similar to the below.

[2020-12-02 12:15:29.139] [D] Recovery Plan Name: recovery-plan-01
[2020-12-02 12:15:29.143] [D] Recovery Plan Mode: test
[2020-12-02 12:15:29.153] [D] Recovery Plan VC: drvcenter.domain.com

In the script we created earlier, we output all of the logs from our function to a file /tmp/SRM-VRO-Output.log. This will show details of the API request, and some additional logs.

2020-12-02_12-22-24: Recovery Plan recovery-plan-01 ran in test mode
vRO REST Call: https://vroserver01.domain.com/vco/api/workflows/67748eec-e51c-8dd9-ac57-a7ca789a706c/executions
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1360    0   627  100   733   8360   9773 --:--:-- --:--:-- --:--:-- 18133
{"id":"2a7edc1a-78b3-4710-b25d-593d1ffe1b29","state":"running","start-date":"2020-12-02T12:23:55Z","started-by":"administrator@vsphere.local","name":"SRM Recovery Plan Details","current-item-for-display":"__item-undefined__","input-parameters":[{"value":{"string":{"value":"recovery-plan-01"}},"type":"string","name":"recoveryPlanName","scope":"local"},{"value":{"string":{"value":"test"}},"type":"string","name":"recoveryPlanMode","scope":"local"},{"value":{"string":{"value":"drvcenter.domain.com"}},"type":"string","name":"recoveryVmwareVcHost","scope":"local"}],"output-parameters":[],"workflow-attributes":[]}
------------------

#Closing

The script and workflow within this post are very simple use cases. In a real world scenario you could modify the vRO Workflow to retrieve additional details about the Recovery Plan - as we have been provided with the Recovery Plan name - for instance, we could retrieve a list of all Virtual Machines protected by the plan and tag them on the recovery site with some data.

The script can also be used on a per-vm basis, this allows much more fine grained controls and it also provides us access to additional environment variables.

VMware_VM_Uuid: UUID used by vCenter Server to uniquely identify this virtual machine.
VMware_VM_Name: Name of this virtual machine, as set at the protected site.
VMware_VM_Ref: Managed object ID of the virtual machine.
VMware_VM_GuestName: Name of the guest OS as defined by the VIM API.

#Additional Resources

SRM Administration Guide PDF (docs.vmware.com) \ SRM Creating Custom Recovery Steps (docs.vmware.com) \ Adding SRM Protection to VMs via vRA (kskilling.com)

How to use the vCenter Managed Object Browser

Mon, 16 Nov 2020 00:00:00 GMT

The Managed Object Browser or MOB is a tool that is bundled with vCenter and ESXi, and it can be used to visually explore the structure of various vSphere related objects, such as Virtual Machines, Datastores and Clusters. This becomes invaluable when developing automation or reporting on infrastructure.

Each object within the MOB contains an array of data, from common information such as the objects name, configured CPU & Memory or its overall status, to less explored data such hardware device keys, VMX file paths and snapshot tree structure.

Not only can the MOB be used to explore this information, but it can also be used to invoke various methods against the various objects. For example, when viewing a Virtual Machine object you can invoke ShutdownGuest and CreateSnapshot tasks.

#MOB Key Components

Property Path: Used to identify where you are within an object. The path can be “walked” to reach specific information.
Properties/Attributes: The information on part of the path, referencing this property with something like PowerCLI or vRO will return the value of the property, for example vm.name would return the name of the virtual machine object you are working with.
Property Types: The property type shows what form the data is, such as string, number, or another object. When the value of a property is another object, the path can continue to be walked.
Parent Object Managed ID: This is often known as the moRef value, it is a unique ID generated by vCenter for a given object. This will be shown at the top of the page for the object you are in.
Browser Address Bar: The address bar gives you a direct link to wherever you are in the MOB, https://<vcenter-server-IP-or-FQDN>/mob/?moid=vm-725&doPath=guest - including the moRef of the vCenter object (e.g. vm-725)

#MOB Guided Tour

To connect to your vCenter MOB, navigate to https://<vcenter-server-IP-or-FQDN>/mob and login with valid credentials.

At the top of your page you should see a table, the first column is the property/attribute name, followed by its type, and then the value. This structure will be used throughout the MOB.

Click on the link for content.

The top level location within the MOB, which provides links to vSphere content

This second page shows us various Service Managers that vCenter provides, such as FileManager and OvfManager, these provide various actions and functions, but for now we are only interested in consuming existing information.

Click on the link against the property rootFolder, the value should be group-d1 (Datacenters)

Service contents and access to the rootFolder (Datacenters)

At the top of this next page you will see a childEntity property, you should see some familiar values here - your Datacenters, their names will be shown within brackets ( ).

Click on one of your Datacenters.

Top level Datacenter object

Within the Datacenter object screen you will see a lot of familiar information, this is the real start of what you would see within vCenter. On this page you will see links to your Datastores and Networks. At the bottom you will also see vmFolder.

You might have noticed that Datastores start with datastore- and Networks start with network-. As you browse the MOB you will see that various vSphere related objects have a naming prefix to them. (vm- for VMs, host- for Hosts etc).

Click the link next to vmFolder.

Top level VM folder object

Inside the top level VM folder, which has the name vm (find the name property), you will find all child folders as well as any virtual machines that have not been organised in to a folder. This is the root, or top level, vm folder - some applications such as govc rely on this when providing the path to an object, for example if you wanted to move a vm in to a folder: govc object.mv /dc1/vm/vm-foo-* /dc1/vm/folder-foo.

Find a virtual machine and click its link, if your VMs are organised into folders select one of those to find a VM object (remember to look for an object that starts vm-).

VM folder object with child entities, such as folders and ungrouped virtual machines

Take some time to look through this object, take note of the values against the datastore and network properties. Now compare these to what you see within the vCenter UI, they should match.

Next, click the link next to the guest property.

Virtual Machine object page with link to guest page highlighted

We are now looking at the guest information for our Virtual Machine. Take a look at the guestState and hostname properties. Once again, these should match what you see within vCenter - is the VM powered on or off? What is the hostname shown on the summary tab?

Virtual Machine guest information

Go back a page to the top level of the virtual machine object you selected, scroll down and select summary

Virtual Machine object page with link to summmary page highlighted

Take note of the Property Path at the top of the page, this currently shows summary.

Press the config link

Virtual Machine summary page with link to config page highlighted

Check the Property Path again, you should now see that we are in summary.config. The path shows us where we are within an object, so at the moment we are within vm.summary.confg.

Take a look at some of the properties listed here, such as the guestFullName, check it against vCenter. The full path to this property would be vm.summary.config.guestFullName

Virtual Machine summary.config page with highlighted properties "guestFullName" and "guestId"

#Example with PowerCLI

Install PowerCLI: Install-Module -Name VMware.PowerCLI.
Connect to vCenter: Connect-VIServer -Server "vcenter.domain.com" -User "username" -Password "password".
Get the VM object we looked at earlier in the MOB: $vm = Get-VM -Name "vm-name".

To access the data we would normally see on an object within the MOB, via PowerCLI, we need to use the ExtensionData property.

Output all ExtensionData, cross reference what you see as an output with what you see in the MOB: $vm.extensiondata.
Get the hostName of our VM object (step 6 above): $vm.extensiondata.guest.hostname.
Get the guestFullName of our VM object (step 8 above): $vm.extensiondata.summary.config.guestFullName.

Ignoring the ExtensionData property, we can see in the two simple examples above that the path we have gone down to get our data through PowerCLI matches the same path we went down to get the data from MOB.

#Example with vRO

Create a new workflow, name it VM Details.
Create a new Input call vm with the type VC:VirtualMachine.

Workflow Inputs tab, showing vm as an input of type VC:VirtualMachne

Drag a Scripting object on to the canvas, between the Start and End, rename it if necessary.

Workflow canvas with scripting object between Start and End items

Map the Input to the scripting object

Scripting object In tab showing an input mapping to the workflow Input "vm"

Update the code within the scripting object

Scripting object script"

System.log(vm.guest.hostName);
System.log(vm.summary.config.guestFullName);

Run the workflow, the output should show the same information as vCenter

VMware Automation/Orchestration Resources

Fri, 30 Oct 2020 00:00:00 GMT

A short collection of blogs, people and other resources that I have found helpful for VMware Automation/Orchestration related work.

If you feel something should be added to this list, please let me know! @sam_perrin.

VMware API References/Resources

Developer Documentation - Contains lots of additional links.
PowerCLI Reference
VMware {code}
VMware {code} Blog
vSphere Automation API (6.7)
vSphere Automation API (7.0)
vCenter MOB (Managed Object Browser) - web based app available to vCenter that allows you to examine objects that exist on the server, populated with runtime information and can help reveal the underlying structures of the object model.
- To access the MOB
- In a Web browser, navigate to the FQDN or IP address of your vCenter Server and add /mob to the end: https://vcenter-fqdn-ip/mob
- Enter the user account and password for the system when prompted.
- Using the MOB to Explore the Object Model

vRealize Automation (vRA) / vRealize Orchestrator (vRO)

People (most links to Twitter where possible)

vRA8 Code Stream - Lightweight Cluster API Container

Mon, 26 Oct 2020 00:00:00 GMT

The below code can be used to create a Dockerfile that will create a container with the following toolset;

curl
ovftool
govc
kubectl
clusterctl

This container can then be utilised by vRA8 Code Stream when using Docker endpoints.

https://docs.vmware.com/en/vRealize-Automation/8.1/Using-and-Managing-CodeStream/GUID-04481662-27C6-4F53-8EA0-D7B2C3DC04C1.html

Dockerfile

FROM alpine:3.12

ARG GOVC_VERSION=0.22.1
ARG OVFTOOL_FILENAME=VMware-ovftool-4.4.0-16360108-lin.x86_64.bundle

# Install additional tools
RUN apk add --no-cache curl ca-certificates bash jq python3 && \
    apk add --no-cache --virtual .build-apps wget unzip git coreutils libgcc

RUN wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.32-r0/glibc-2.32-r0.apk && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.32-r0/glibc-bin-2.32-r0.apk && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.32-r0/glibc-i18n-2.32-r0.apk
RUN apk add glibc-2.32-r0.apk glibc-bin-2.32-r0.apk glibc-i18n-2.32-r0.apk && \
    /usr/glibc-compat/bin/localedef -i en_GB -f UTF-8 en_GB.UTF-8 && \
    rm -f glibc-2.32-r0.apk glibc-bin-2.32-r0.apk glibc-i18n-2.32-r0.apk

# Install ovftool
RUN wget --quiet https://raw.githubusercontent.com/sam-perrin/codestream-cluster-api/master/files/$OVFTOOL_FILENAME && \
    chmod +x /$OVFTOOL_FILENAME && \
    /bin/sh /$OVFTOOL_FILENAME --console --eulas-agreed --required && \
    rm -f /$OVFTOOL_FILENAME

#Download and Extract GOVC
RUN wget --quiet https://github.com/vmware/govmomi/releases/download/v$GOVC_VERSION/govc_linux_amd64.gz && \
    gunzip -fq govc_linux_amd64.gz && \
    mv govc_linux_amd64 govc && \
    chown root govc && \
    chmod ug+r+x govc && \
    mv govc /usr/local/bin/.

# Install kubectl
RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" && \
    mv kubectl /usr/bin/kubectl && \
    chmod +x /usr/bin/kubectl

# Install clusterctl
RUN curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.3.8/clusterctl-linux-amd64 -o clusterctl && \
    mv clusterctl /usr/bin/clusterctl && \
    chmod +x /usr/bin/clusterctl

RUN apk del .build-apps

Cluster API vSphere - Using ClusterResourceSet to install Calico CNI

Wed, 23 Sep 2020 00:00:00 GMT

This post covers the steps required to get Cluster API vSphere (CAPV) to install the Calico CNI when a new cluster is deployed. This post assumes you are using haproxy with CAPV.

We will be utilising the ClusterResourceSet resource type, which was introduced in Cluster API 0.3.7 - in theory these steps can be followed regardless of the cloud provided you are using with Cluster API.

Even though the ClusterResourceSet was introduced in 0.3.7, I had issues getting it to correctly create the resource, so in the end I upgraded to 0.3.9. This might have been due to CAPV not knowing what the new resource type was until a later version of the provider.

ClusterResourceSet is an experimental feature. Follow the officially documented steps to enable. https://cluster-api.sigs.k8s.io/tasks/experimental-features/experimental-features.html#enabling-experimental-features-on-existing-management-clusters

In this post we use $HOME instead of ~ for consistency, --kubeconfig can sometimes be fussy when using ~.

If you haven’t yet setup clusterctl or if you do not have a working management control plane, follow this first: Cluster API Setup Steps (vSphere)

The versions used within this post:

clusterctl: 0.3.9
Cluster API: 0.3.9
Kubernetes: 1.18.2
kubectl: 1.18.2
capv provider: 0.7.1

Upgrade Cluster API

Install the latest binary of clusterctl with curl. Follow the steps below, or check the official documentation for additional info. https://cluster-api.sigs.k8s.io/user/quick-start.html

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.3.9/clusterctl-linux-amd64 -o clusterctl
chmod +x ./clusterctl
sudo mv ./clusterctl /usr/local/bin/clusterctl
clusterctl version
#sudo clusterctl version

Upgrade Cluster API components

Official documentation: https://cluster-api.sigs.k8s.io/clusterctl/commands/upgrade.html

clusterctl upgrade plan
#Run the upgrade with the command shown in response to the upgrade plan
clusterctl upgrade apply --management-group capi-system/cluster-api  --contract v1alpha3

Prepare Calico CNI ConfigMap and Custom Cluster Template file

Here we will download the Calico manifest and default cluster template file. We will then create a ConfigMap from the Calico manifest.

This section assumes your KUBECONFIG is set to your management control plane. If not, use --kubeconfig to specify your management control plane kubeconfig file. Check this for reference: Cluster API Setup Steps (vSphere)

Download the cluster-template-haproxy.yaml from the Cluster API vSphere (CAPV) releases page. https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/tag/v0.7.1

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/download/v0.7.1/cluster-template-haproxy.yaml -o "$HOME/custom-cluster-template-haproxy.yaml"

Download the Calico manifest file

curl -L https://docs.projectcalico.org/manifests/calico.yaml -o "$HOME/calico-manifest.yaml"

Create ConfigMap yaml from manifest file, it will be named calico-cni

kubectl create configmap calico-cni --from-file="$HOME/calico-manifest.yaml" -o yaml --dry-run=client > $HOME/calico-configmap.yaml

The contents of the calico-configmap.yaml is large, I have provided a full copy of my custom-cluster-template-haproxy.yaml file at the bottom of this post.

Update custom-cluster-template-haproxy.yaml and generate cluster resources

Here we will add the contents of the calico-configmap.yaml to our custom cluster template file and add the ClusterResourceSet, which then references the new ConfigMap.

echo "---" >> $HOME/custom-cluster-template-haproxy.yaml
cat $HOME/calico-configmap.yaml >> $HOME/custom-cluster-template-haproxy.yaml
cat >> $HOME/custom-cluster-template-haproxy.yaml <<'EOL'
---
apiVersion: addons.cluster.x-k8s.io/v1alpha3
kind: ClusterResourceSet
metadata:
  name: ${ CLUSTER_NAME }-crs-0
  namespace: '${ NAMESPACE }'
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: '${ CLUSTER_NAME }'
  resources:
  - kind: ConfigMap
    name: calico-cni
EOL

Using this new cluster template file we will now generate our cluster resources using clusterctl, we then deploy these resources using kubectl. Replace WORKLOAD_CLUSTER_NAME value with the name of your new workload cluster.

export WORKLOAD_CLUSTER_NAME="cluster01"
mkdir -p $HOME/$WORKLOAD_CLUSTER_NAME
clusterctl config cluster $WORKLOAD_CLUSTER_NAME --kubernetes-version v1.18.2 --control-plane-machine-count 1 --worker-machine-count 3 --from $HOME/custom-cluster-template-haproxy.yaml > $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml

# with kubeconfig specified
# clusterctl config cluster $WORKLOAD_CLUSTER_NAME --kubernetes-version v1.18.2 --control-plane-machine-count 1 --worker-machine-count 3 --from $HOME/custom-cluster-template-haproxy.yaml --kubeconfig=local-control-plane/kubeconfig > $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml

Review the cluster config file $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml and make any required changes. Some suggestions to review are CPU, Memory, Storage and Pods CIDR range.

You can use sed to do a find and replace within the file, the below examples are for changing the Storage and Memory on all virtual machines as part of this deployment and the Pods CIDR.

Pod CIDR from 192.168.0.0/16 to 192.168.200.0/24
Memory from 8192 to 4096
Storage from 25 to 20

sed -i "s/192.168.0.0\/16/192.168.200.0\/24/g" $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml
sed -i "s/memoryMiB: 8192/memoryMiB: 4096/g" $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml
sed -i "s/diskGiB: 25/diskGiB: 20/g" $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml

Deploy the cluster components. Monitor the deployment of virtual machines within vSphere.

kubectl apply -f $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml

Use kubectl to see the progress of your new cluster. As the CNI is now being deployed on cluster provision we should see it change to Ready.

kubectl get cluster --all-namespaces
kubectl get kubeadmcontrolplane --all-namespaces

Retrieve the kubeconfig file for the workload cluster, and then check the node status as well as the existence of calico pods.

kubectl get secret $WORKLOAD_CLUSTER_NAME-kubeconfig -o=jsonpath='{.data.value}' | { base64 -d 2>/dev/null || base64 -D; } > $HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig
kubectl get node -o wide --kubeconfig="$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig"
kubectl get pod -A -o wide --kubeconfig="$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig"

If all is well, we should see all nodes Ready and several pods running with names starting with calico-.

This completes the steps required to get calico deployed at cluster provisioning. ClusterResourceSet could be use to deploy a multitude of other resources at provisioning time, such as CSI’s.

Complete Custom Cluster Template Example

This can be used to generate your cluster resource files.

---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: '${ CLUSTER_NAME }'
  name: '${ CLUSTER_NAME }'
  namespace: '${ NAMESPACE }'
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: KubeadmControlPlane
    name: '${ CLUSTER_NAME }'
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: VSphereCluster
    name: '${ CLUSTER_NAME }'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: HAProxyLoadBalancer
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: '${ CLUSTER_NAME }'
  name: '${ CLUSTER_NAME }'
  namespace: '${ NAMESPACE }'
spec:
  user:
    authorizedKeys:
    - '${ VSPHERE_SSH_AUTHORIZED_KEY }'
    name: capv
  virtualMachineConfiguration:
    cloneMode: linkedClone
    datacenter: '${ VSPHERE_DATACENTER }'
    datastore: '${ VSPHERE_DATASTORE }'
    diskGiB: 25
    folder: '${ VSPHERE_FOLDER }'
    memoryMiB: 8192
    network:
      devices:
      - dhcp4: true
        networkName: '${ VSPHERE_NETWORK }'
    numCPUs: 2
    resourcePool: '${ VSPHERE_RESOURCE_POOL }'
    server: '${ VSPHERE_SERVER }'
    template: '${ VSPHERE_HAPROXY_TEMPLATE }'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: VSphereCluster
metadata:
  name: '${ CLUSTER_NAME }'
  namespace: '${ NAMESPACE }'
spec:
  cloudProviderConfiguration:
    global:
      insecure: true
      secretName: cloud-provider-vsphere-credentials
      secretNamespace: kube-system
    network:
      name: '${ VSPHERE_NETWORK }'
    providerConfig:
      cloud:
        controllerImage: gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.2.0
      storage:
        attacherImage: quay.io/k8scsi/csi-attacher:v2.0.0
        controllerImage: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.0.0
        livenessProbeImage: quay.io/k8scsi/livenessprobe:v1.1.0
        metadataSyncerImage: gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.0.0
        nodeDriverImage: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.0.0
        provisionerImage: quay.io/k8scsi/csi-provisioner:v1.4.0
        registrarImage: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
    virtualCenter:
      ${ VSPHERE_SERVER }:
        datacenters: '${ VSPHERE_DATACENTER }'
    workspace:
      datacenter: '${ VSPHERE_DATACENTER }'
      datastore: '${ VSPHERE_DATASTORE }'
      folder: '${ VSPHERE_FOLDER }'
      resourcePool: '${ VSPHERE_RESOURCE_POOL }'
      server: '${ VSPHERE_SERVER }'
  loadBalancerRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: HAProxyLoadBalancer
    name: '${ CLUSTER_NAME }'
  server: '${ VSPHERE_SERVER }'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: VSphereMachineTemplate
metadata:
  name: '${ CLUSTER_NAME }'
  namespace: '${ NAMESPACE }'
spec:
  template:
    spec:
      cloneMode: linkedClone
      datacenter: '${ VSPHERE_DATACENTER }'
      datastore: '${ VSPHERE_DATASTORE }'
      diskGiB: 25
      folder: '${ VSPHERE_FOLDER }'
      memoryMiB: 8192
      network:
        devices:
        - dhcp4: true
          networkName: '${ VSPHERE_NETWORK }'
      numCPUs: 2
      resourcePool: '${ VSPHERE_RESOURCE_POOL }'
      server: '${ VSPHERE_SERVER }'
      template: '${ VSPHERE_TEMPLATE }'
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: '${ CLUSTER_NAME }'
  namespace: '${ NAMESPACE }'
spec:
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: VSphereMachineTemplate
    name: '${ CLUSTER_NAME }'
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-provider: external
      controllerManager:
        extraArgs:
          cloud-provider: external
    initConfiguration:
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ ds.meta_data.hostname }}'
    joinConfiguration:
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ ds.meta_data.hostname }}'
    preKubeadmCommands:
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
    - echo "127.0.0.1   localhost" >>/etc/hosts
    - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
    useExperimentalRetryJoin: true
    users:
    - name: capv
      sshAuthorizedKeys:
      - '${ VSPHERE_SSH_AUTHORIZED_KEY }'
      sudo: ALL=(ALL) NOPASSWD:ALL
  replicas: ${ CONTROL_PLANE_MACHINE_COUNT }
  version: '${ KUBERNETES_VERSION }'
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfigTemplate
metadata:
  name: '${ CLUSTER_NAME }-md-0'
  namespace: '${ NAMESPACE }'
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          criSocket: /var/run/containerd/containerd.sock
          kubeletExtraArgs:
            cloud-provider: external
          name: '{{ ds.meta_data.hostname }}'
      preKubeadmCommands:
      - hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
      - echo "127.0.0.1   localhost" >>/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
      - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
      users:
      - name: capv
        sshAuthorizedKeys:
        - '${ VSPHERE_SSH_AUTHORIZED_KEY }'
        sudo: ALL=(ALL) NOPASSWD:ALL
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: '${ CLUSTER_NAME }'
  name: '${ CLUSTER_NAME }-md-0'
  namespace: '${ NAMESPACE }'
spec:
  clusterName: '${ CLUSTER_NAME }'
  replicas: ${ WORKER_MACHINE_COUNT }
  selector:
    matchLabels: {}
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: '${ CLUSTER_NAME }'
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: KubeadmConfigTemplate
          name: '${ CLUSTER_NAME }-md-0'
      clusterName: '${ CLUSTER_NAME }'
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: VSphereMachineTemplate
        name: '${ CLUSTER_NAME }'
      version: '${ KUBERNETES_VERSION }'
---
apiVersion: v1
data:
  calico-manifest.yaml: |+
    ---
    # Source: calico/templates/calico-config.yaml
    # This ConfigMap is used to configure a self-hosted Calico installation.
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: calico-config
      namespace: kube-system
    data:
      # Typha is disabled.
      typha_service_name: "none"
      # Configure the backend to use.
      calico_backend: "bird"
      # Configure the MTU to use for workload interfaces and tunnels.
      # - If Wireguard is enabled, set to your network MTU - 60
      # - Otherwise, if VXLAN or BPF mode is enabled, set to your network MTU - 50
      # - Otherwise, if IPIP is enabled, set to your network MTU - 20
      # - Otherwise, if not using any encapsulation, set to your network MTU.
      veth_mtu: "1440"

      # The CNI network configuration to install on each node. The special
      # values in this config will be automatically populated.
      cni_network_config: |-
        {
          "name": "k8s-pod-network",
          "cniVersion": "0.3.1",
          "plugins": [
            {
              "type": "calico",
              "log_level": "info",
              "log_file_path": "/var/log/calico/cni/cni.log",
              "datastore_type": "kubernetes",
              "nodename": "__KUBERNETES_NODE_NAME__",
              "mtu": __CNI_MTU__,
              "ipam": {
                  "type": "calico-ipam"
              },
              "policy": {
                  "type": "k8s"
              },
              "kubernetes": {
                  "kubeconfig": "__KUBECONFIG_FILEPATH__"
              }
            },
            {
              "type": "portmap",
              "snat": true,
              "capabilities": {"portMappings": true}
            },
            {
              "type": "bandwidth",
              "capabilities": {"bandwidth": true}
            }
          ]
        }

    ---
    # Source: calico/templates/kdd-crds.yaml


    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: bgpconfigurations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: BGPConfiguration
        listKind: BGPConfigurationList
        plural: bgpconfigurations
        singular: bgpconfiguration
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: BGPConfiguration contains the configuration for any BGP routing.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: BGPConfigurationSpec contains the values of the BGP configuration.
                properties:
                  asNumber:
                    description: 'ASNumber is the default AS number used by a node. [Default:
                      64512]'
                    format: int32
                    type: integer
                  communities:
                    description: Communities is a list of BGP community values and their
                      arbitrary names for tagging routes.
                    items:
                      description: Community contains standard or large community value
                        and its name.
                      properties:
                        name:
                          description: Name given to community value.
                          type: string
                        value:
                          description: Value must be of format `aa:nn` or `aa:nn:mm`.
                            For standard community use `aa:nn` format, where `aa` and
                            `nn` are 16 bit number. For large community use `aa:nn:mm`
                            format, where `aa`, `nn` and `mm` are 32 bit number. Where,
                            `aa` is an AS Number, `nn` and `mm` are per-AS identifier.
                          pattern: ^(\d+):(\d+)$|^(\d+):(\d+):(\d+)$
                          type: string
                      type: object
                    type: array
                  listenPort:
                    description: ListenPort is the port where BGP protocol should listen.
                      Defaults to 179
                    maximum: 65535
                    minimum: 1
                    type: integer
                  logSeverityScreen:
                    description: 'LogSeverityScreen is the log severity above which logs
                      are sent to the stdout. [Default: INFO]'
                    type: string
                  nodeToNodeMeshEnabled:
                    description: 'NodeToNodeMeshEnabled sets whether full node to node
                      BGP mesh is enabled. [Default: true]'
                    type: boolean
                  prefixAdvertisements:
                    description: PrefixAdvertisements contains per-prefix advertisement
                      configuration.
                    items:
                      description: PrefixAdvertisement configures advertisement properties
                        for the specified CIDR.
                      properties:
                        cidr:
                          description: CIDR for which properties should be advertised.
                          type: string
                        communities:
                          description: Communities can be list of either community names
                            already defined in `Specs.Communities` or community value
                            of format `aa:nn` or `aa:nn:mm`. For standard community use
                            `aa:nn` format, where `aa` and `nn` are 16 bit number. For
                            large community use `aa:nn:mm` format, where `aa`, `nn` and
                            `mm` are 32 bit number. Where,`aa` is an AS Number, `nn` and
                            `mm` are per-AS identifier.
                          items:
                            type: string
                          type: array
                      type: object
                    type: array
                  serviceClusterIPs:
                    description: ServiceClusterIPs are the CIDR blocks from which service
                      cluster IPs are allocated. If specified, Calico will advertise these
                      blocks, as well as any cluster IPs within them.
                    items:
                      description: ServiceClusterIPBlock represents a single allowed ClusterIP
                        CIDR block.
                      properties:
                        cidr:
                          type: string
                      type: object
                    type: array
                  serviceExternalIPs:
                    description: ServiceExternalIPs are the CIDR blocks for Kubernetes
                      Service External IPs. Kubernetes Service ExternalIPs will only be
                      advertised if they are within one of these blocks.
                    items:
                      description: ServiceExternalIPBlock represents a single allowed
                        External IP CIDR block.
                      properties:
                        cidr:
                          type: string
                      type: object
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: bgppeers.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: BGPPeer
        listKind: BGPPeerList
        plural: bgppeers
        singular: bgppeer
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: BGPPeerSpec contains the specification for a BGPPeer resource.
                properties:
                  asNumber:
                    description: The AS Number of the peer.
                    format: int32
                    type: integer
                  keepOriginalNextHop:
                    description: Option to keep the original nexthop field when routes
                      are sent to a BGP Peer. Setting "true" configures the selected BGP
                      Peers node to use the "next hop keep;" instead of "next hop self;"(default)
                      in the specific branch of the Node on "bird.cfg".
                    type: boolean
                  node:
                    description: The node name identifying the Calico node instance that
                      is peering with this peer. If this is not set, this represents a
                      global peer, i.e. a peer that peers with every node in the deployment.
                    type: string
                  nodeSelector:
                    description: Selector for the nodes that should have this peering.  When
                      this is set, the Node field must be empty.
                    type: string
                  peerIP:
                    description: The IP address of the peer followed by an optional port
                      number to peer with. If port number is given, format should be `[<IPv6>]:port`
                      or `<IPv4>:<port>` for IPv4. If optional port number is not set,
                      and this peer IP and ASNumber belongs to a calico/node with ListenPort
                      set in BGPConfiguration, then we use that port to peer.
                    type: string
                  peerSelector:
                    description: Selector for the remote nodes to peer with.  When this
                      is set, the PeerIP and ASNumber fields must be empty.  For each
                      peering between the local node and selected remote nodes, we configure
                      an IPv4 peering if both ends have NodeBGPSpec.IPv4Address specified,
                      and an IPv6 peering if both ends have NodeBGPSpec.IPv6Address specified.  The
                      remote AS number comes from the remote node’s NodeBGPSpec.ASNumber,
                      or the global default if that is not set.
                    type: string
                required:
                - asNumber
                - peerIP
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: blockaffinities.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: BlockAffinity
        listKind: BlockAffinityList
        plural: blockaffinities
        singular: blockaffinity
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: BlockAffinitySpec contains the specification for a BlockAffinity
                  resource.
                properties:
                  cidr:
                    type: string
                  deleted:
                    description: Deleted indicates that this block affinity is being deleted.
                      This field is a string for compatibility with older releases that
                      mistakenly treat this field as a string.
                    type: string
                  node:
                    type: string
                  state:
                    type: string
                required:
                - cidr
                - deleted
                - node
                - state
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: clusterinformations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: ClusterInformation
        listKind: ClusterInformationList
        plural: clusterinformations
        singular: clusterinformation
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: ClusterInformation contains the cluster specific information.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: ClusterInformationSpec contains the values of describing
                  the cluster.
                properties:
                  calicoVersion:
                    description: CalicoVersion is the version of Calico that the cluster
                      is running
                    type: string
                  clusterGUID:
                    description: ClusterGUID is the GUID of the cluster
                    type: string
                  clusterType:
                    description: ClusterType describes the type of the cluster
                    type: string
                  datastoreReady:
                    description: DatastoreReady is used during significant datastore migrations
                      to signal to components such as Felix that it should wait before
                      accessing the datastore.
                    type: boolean
                  variant:
                    description: Variant declares which variant of Calico should be active.
                    type: string
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: felixconfigurations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: FelixConfiguration
        listKind: FelixConfigurationList
        plural: felixconfigurations
        singular: felixconfiguration
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: Felix Configuration contains the configuration for Felix.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: FelixConfigurationSpec contains the values of the Felix configuration.
                properties:
                  allowIPIPPacketsFromWorkloads:
                    description: 'AllowIPIPPacketsFromWorkloads controls whether Felix
                      will add a rule to drop IPIP encapsulated traffic from workloads
                      [Default: false]'
                    type: boolean
                  allowVXLANPacketsFromWorkloads:
                    description: 'AllowVXLANPacketsFromWorkloads controls whether Felix
                      will add a rule to drop VXLAN encapsulated traffic from workloads
                      [Default: false]'
                    type: boolean
                  awsSrcDstCheck:
                    description: 'Set source-destination-check on AWS EC2 instances. Accepted
                      value must be one of "DoNothing", "Enabled" or "Disabled". [Default:
                      DoNothing]'
                    enum:
                    - DoNothing
                    - Enable
                    - Disable
                    type: string
                  bpfConnectTimeLoadBalancingEnabled:
                    description: 'BPFConnectTimeLoadBalancingEnabled when in BPF mode,
                      controls whether Felix installs the connection-time load balancer.  The
                      connect-time load balancer is required for the host to be able to
                      reach Kubernetes services and it improves the performance of pod-to-service
                      connections.  The only reason to disable it is for debugging purposes.  [Default:
                      true]'
                    type: boolean
                  bpfDataIfacePattern:
                    description: 'BPFDataIfacePattern is a regular expression that controls
                      which interfaces Felix should attach BPF programs to in order to
                      catch traffic to/from the network.  This needs to match the interfaces
                      that Calico workload traffic flows over as well as any interfaces
                      that handle incoming traffic to nodeports and services from outside
                      the cluster.  It should not match the workload interfaces (usually
                      named cali...). [Default: ^(en.*|eth.*|tunl0$)]'
                    type: string
                  bpfDisableUnprivileged:
                    description: 'BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled
                      sysctl to disable unprivileged use of BPF.  This ensures that unprivileged
                      users cannot access Calico''s BPF maps and cannot insert their own
                      BPF programs to interfere with Calico''s. [Default: true]'
                    type: boolean
                  bpfEnabled:
                    description: 'BPFEnabled, if enabled Felix will use the BPF dataplane.
                      [Default: false]'
                    type: boolean
                  bpfExternalServiceMode:
                    description: 'BPFExternalServiceMode in BPF mode, controls how connections
                      from outside the cluster to services (node ports and cluster IPs)
                      are forwarded to remote workloads.  If set to "Tunnel" then both
                      request and response traffic is tunneled to the remote node.  If
                      set to "DSR", the request traffic is tunneled but the response traffic
                      is sent directly from the remote node.  In "DSR" mode, the remote
                      node appears to use the IP of the ingress node; this requires a
                      permissive L2 network.  [Default: Tunnel]'
                    type: string
                  bpfKubeProxyEndpointSlicesEnabled:
                    description: BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls
                      whether Felix's embedded kube-proxy accepts EndpointSlices or not.
                    type: boolean
                  bpfKubeProxyIptablesCleanupEnabled:
                    description: 'BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF
                      mode, Felix will proactively clean up the upstream Kubernetes kube-proxy''s
                      iptables chains.  Should only be enabled if kube-proxy is not running.  [Default:
                      true]'
                    type: boolean
                  bpfKubeProxyMinSyncPeriod:
                    description: 'BPFKubeProxyMinSyncPeriod, in BPF mode, controls the
                      minimum time between updates to the dataplane for Felix''s embedded
                      kube-proxy.  Lower values give reduced set-up latency.  Higher values
                      reduce Felix CPU usage by batching up more work.  [Default: 1s]'
                    type: string
                  bpfLogLevel:
                    description: 'BPFLogLevel controls the log level of the BPF programs
                      when in BPF dataplane mode.  One of "Off", "Info", or "Debug".  The
                      logs are emitted to the BPF trace pipe, accessible with the command
                      `tc exec bpf debug`. [Default: Off].'
                    type: string
                  chainInsertMode:
                    description: 'ChainInsertMode controls whether Felix hooks the kernel’s
                      top-level iptables chains by inserting a rule at the top of the
                      chain or by appending a rule at the bottom. insert is the safe default
                      since it prevents Calico’s rules from being bypassed. If you switch
                      to append mode, be sure that the other rules in the chains signal
                      acceptance by falling through to the Calico rules, otherwise the
                      Calico policy will be bypassed. [Default: insert]'
                    type: string
                  dataplaneDriver:
                    type: string
                  debugDisableLogDropping:
                    type: boolean
                  debugMemoryProfilePath:
                    type: string
                  debugSimulateCalcGraphHangAfter:
                    type: string
                  debugSimulateDataplaneHangAfter:
                    type: string
                  defaultEndpointToHostAction:
                    description: 'DefaultEndpointToHostAction controls what happens to
                      traffic that goes from a workload endpoint to the host itself (after
                      the traffic hits the endpoint egress policy). By default Calico
                      blocks traffic from workload endpoints to the host itself with an
                      iptables “DROP” action. If you want to allow some or all traffic
                      from endpoint to host, set this parameter to RETURN or ACCEPT. Use
                      RETURN if you have your own rules in the iptables “INPUT” chain;
                      Calico will insert its rules at the top of that chain, then “RETURN”
                      packets to the “INPUT” chain once it has completed processing workload
                      endpoint egress policy. Use ACCEPT to unconditionally accept packets
                      from workloads after processing workload endpoint egress policy.
                      [Default: Drop]'
                    type: string
                  deviceRouteProtocol:
                    description: This defines the route protocol added to programmed device
                      routes, by default this will be RTPROT_BOOT when left blank.
                    type: integer
                  deviceRouteSourceAddress:
                    description: This is the source address to use on programmed device
                      routes. By default the source address is left blank, leaving the
                      kernel to choose the source address used.
                    type: string
                  disableConntrackInvalidCheck:
                    type: boolean
                  endpointReportingDelay:
                    type: string
                  endpointReportingEnabled:
                    type: boolean
                  externalNodesList:
                    description: ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes
                      which may source tunnel traffic and have the tunneled traffic be
                      accepted at calico nodes.
                    items:
                      type: string
                    type: array
                  failsafeInboundHostPorts:
                    description: 'FailsafeInboundHostPorts is a comma-delimited list of
                      UDP/TCP ports that Felix will allow incoming traffic to host endpoints
                      on irrespective of the security policy. This is useful to avoid
                      accidentally cutting off a host with incorrect configuration. Each
                      port should be specified as tcp:<port-number> or udp:<port-number>.
                      For back-compatibility, if the protocol is not specified, it defaults
                      to “tcp”. To disable all inbound host ports, use the value none.
                      The default value allows ssh access and DHCP. [Default: tcp:22,
                      udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]'
                    items:
                      description: ProtoPort is combination of protocol and port, both
                        must be specified.
                      properties:
                        port:
                          type: integer
                        protocol:
                          type: string
                      required:
                      - port
                      - protocol
                      type: object
                    type: array
                  failsafeOutboundHostPorts:
                    description: 'FailsafeOutboundHostPorts is a comma-delimited list
                      of UDP/TCP ports that Felix will allow outgoing traffic from host
                      endpoints to irrespective of the security policy. This is useful
                      to avoid accidentally cutting off a host with incorrect configuration.
                      Each port should be specified as tcp:<port-number> or udp:<port-number>.
                      For back-compatibility, if the protocol is not specified, it defaults
                      to “tcp”. To disable all outbound host ports, use the value none.
                      The default value opens etcd’s standard ports to ensure that Felix
                      does not get cut off from etcd as well as allowing DHCP and DNS.
                      [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667,
                      udp:53, udp:67]'
                    items:
                      description: ProtoPort is combination of protocol and port, both
                        must be specified.
                      properties:
                        port:
                          type: integer
                        protocol:
                          type: string
                      required:
                      - port
                      - protocol
                      type: object
                    type: array
                  featureDetectOverride:
                    description: FeatureDetectOverride is used to override the feature
                      detection. Values are specified in a comma separated list with no
                      spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=".
                      "true" or "false" will force the feature, empty or omitted values
                      are auto-detected.
                    type: string
                  genericXDPEnabled:
                    description: 'GenericXDPEnabled enables Generic XDP so network cards
                      that don''t support XDP offload or driver modes can use XDP. This
                      is not recommended since it doesn''t provide better performance
                      than iptables. [Default: false]'
                    type: boolean
                  healthEnabled:
                    type: boolean
                  healthHost:
                    type: string
                  healthPort:
                    type: integer
                  interfaceExclude:
                    description: 'InterfaceExclude is a comma-separated list of interfaces
                      that Felix should exclude when monitoring for host endpoints. The
                      default value ensures that Felix ignores Kubernetes'' IPVS dummy
                      interface, which is used internally by kube-proxy. If you want to
                      exclude multiple interface names using a single value, the list
                      supports regular expressions. For regular expressions you must wrap
                      the value with ''/''. For example having values ''/^kube/,veth1''
                      will exclude all interfaces that begin with ''kube'' and also the
                      interface ''veth1''. [Default: kube-ipvs0]'
                    type: string
                  interfacePrefix:
                    description: 'InterfacePrefix is the interface name prefix that identifies
                      workload endpoints and so distinguishes them from host endpoint
                      interfaces. Note: in environments other than bare metal, the orchestrators
                      configure this appropriately. For example our Kubernetes and Docker
                      integrations set the ‘cali’ value, and our OpenStack integration
                      sets the ‘tap’ value. [Default: cali]'
                    type: string
                  interfaceRefreshInterval:
                    description: InterfaceRefreshInterval is the period at which Felix
                      rescans local interfaces to verify their state. The rescan can be
                      disabled by setting the interval to 0.
                    type: string
                  ipipEnabled:
                    type: boolean
                  ipipMTU:
                    description: 'IPIPMTU is the MTU to set on the tunnel device. See
                      Configuring MTU [Default: 1440]'
                    type: integer
                  ipsetsRefreshInterval:
                    description: 'IpsetsRefreshInterval is the period at which Felix re-checks
                      all iptables state to ensure that no other process has accidentally
                      broken Calico’s rules. Set to 0 to disable iptables refresh. [Default:
                      90s]'
                    type: string
                  iptablesBackend:
                    description: IptablesBackend specifies which backend of iptables will
                      be used. The default is legacy.
                    type: string
                  iptablesFilterAllowAction:
                    type: string
                  iptablesLockFilePath:
                    description: 'IptablesLockFilePath is the location of the iptables
                      lock file. You may need to change this if the lock file is not in
                      its standard location (for example if you have mapped it into Felix’s
                      container at a different path). [Default: /run/xtables.lock]'
                    type: string
                  iptablesLockProbeInterval:
                    description: 'IptablesLockProbeInterval is the time that Felix will
                      wait between attempts to acquire the iptables lock if it is not
                      available. Lower values make Felix more responsive when the lock
                      is contended, but use more CPU. [Default: 50ms]'
                    type: string
                  iptablesLockTimeout:
                    description: 'IptablesLockTimeout is the time that Felix will wait
                      for the iptables lock, or 0, to disable. To use this feature, Felix
                      must share the iptables lock file with all other processes that
                      also take the lock. When running Felix inside a container, this
                      requires the /run directory of the host to be mounted into the calico/node
                      or calico/felix container. [Default: 0s disabled]'
                    type: string
                  iptablesMangleAllowAction:
                    type: string
                  iptablesMarkMask:
                    description: 'IptablesMarkMask is the mask that Felix selects its
                      IPTables Mark bits from. Should be a 32 bit hexadecimal number with
                      at least 8 bits set, none of which clash with any other mark bits
                      in use on the system. [Default: 0xff000000]'
                    format: int32
                    type: integer
                  iptablesNATOutgoingInterfaceFilter:
                    type: string
                  iptablesPostWriteCheckInterval:
                    description: 'IptablesPostWriteCheckInterval is the period after Felix
                      has done a write to the dataplane that it schedules an extra read
                      back in order to check the write was not clobbered by another process.
                      This should only occur if another application on the system doesn’t
                      respect the iptables lock. [Default: 1s]'
                    type: string
                  iptablesRefreshInterval:
                    description: 'IptablesRefreshInterval is the period at which Felix
                      re-checks the IP sets in the dataplane to ensure that no other process
                      has accidentally broken Calico’s rules. Set to 0 to disable IP sets
                      refresh. Note: the default for this value is lower than the other
                      refresh intervals as a workaround for a Linux kernel bug that was
                      fixed in kernel version 4.11. If you are using v4.11 or greater
                      you may want to set this to, a higher value to reduce Felix CPU
                      usage. [Default: 10s]'
                    type: string
                  ipv6Support:
                    type: boolean
                  kubeNodePortRanges:
                    description: 'KubeNodePortRanges holds list of port ranges used for
                      service node ports. Only used if felix detects kube-proxy running
                      in ipvs mode. Felix uses these ranges to separate host and workload
                      traffic. [Default: 30000:32767].'
                    items:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    type: array
                  logFilePath:
                    description: 'LogFilePath is the full path to the Felix log. Set to
                      none to disable file logging. [Default: /var/log/calico/felix.log]'
                    type: string
                  logPrefix:
                    description: 'LogPrefix is the log prefix that Felix uses when rendering
                      LOG rules. [Default: calico-packet]'
                    type: string
                  logSeverityFile:
                    description: 'LogSeverityFile is the log severity above which logs
                      are sent to the log file. [Default: Info]'
                    type: string
                  logSeverityScreen:
                    description: 'LogSeverityScreen is the log severity above which logs
                      are sent to the stdout. [Default: Info]'
                    type: string
                  logSeveritySys:
                    description: 'LogSeveritySys is the log severity above which logs
                      are sent to the syslog. Set to None for no logging to syslog. [Default:
                      Info]'
                    type: string
                  maxIpsetSize:
                    type: integer
                  metadataAddr:
                    description: 'MetadataAddr is the IP address or domain name of the
                      server that can answer VM queries for cloud-init metadata. In OpenStack,
                      this corresponds to the machine running nova-api (or in Ubuntu,
                      nova-api-metadata). A value of none (case insensitive) means that
                      Felix should not set up any NAT rule for the metadata path. [Default:
                      127.0.0.1]'
                    type: string
                  metadataPort:
                    description: 'MetadataPort is the port of the metadata server. This,
                      combined with global.MetadataAddr (if not ‘None’), is used to set
                      up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort.
                      In most cases this should not need to be changed [Default: 8775].'
                    type: integer
                  natOutgoingAddress:
                    description: NATOutgoingAddress specifies an address to use when performing
                      source NAT for traffic in a natOutgoing pool that is leaving the
                      network. By default the address used is an address on the interface
                      the traffic is leaving on (ie it uses the iptables MASQUERADE target)
                    type: string
                  natPortRange:
                    anyOf:
                    - type: integer
                    - type: string
                    description: NATPortRange specifies the range of ports that is used
                      for port mapping when doing outgoing NAT. When unset the default
                      behavior of the network stack is used.
                    pattern: ^.*
                    x-kubernetes-int-or-string: true
                  netlinkTimeout:
                    type: string
                  openstackRegion:
                    description: 'OpenstackRegion is the name of the region that a particular
                      Felix belongs to. In a multi-region Calico/OpenStack deployment,
                      this must be configured somehow for each Felix (here in the datamodel,
                      or in felix.cfg or the environment on each compute node), and must
                      match the [calico] openstack_region value configured in neutron.conf
                      on each node. [Default: Empty]'
                    type: string
                  policySyncPathPrefix:
                    description: 'PolicySyncPathPrefix is used to by Felix to communicate
                      policy changes to external services, like Application layer policy.
                      [Default: Empty]'
                    type: string
                  prometheusGoMetricsEnabled:
                    description: 'PrometheusGoMetricsEnabled disables Go runtime metrics
                      collection, which the Prometheus client does by default, when set
                      to false. This reduces the number of metrics reported, reducing
                      Prometheus load. [Default: true]'
                    type: boolean
                  prometheusMetricsEnabled:
                    description: 'PrometheusMetricsEnabled enables the Prometheus metrics
                      server in Felix if set to true. [Default: false]'
                    type: boolean
                  prometheusMetricsHost:
                    description: 'PrometheusMetricsHost is the host that the Prometheus
                      metrics server should bind to. [Default: empty]'
                    type: string
                  prometheusMetricsPort:
                    description: 'PrometheusMetricsPort is the TCP port that the Prometheus
                      metrics server should bind to. [Default: 9091]'
                    type: integer
                  prometheusProcessMetricsEnabled:
                    description: 'PrometheusProcessMetricsEnabled disables process metrics
                      collection, which the Prometheus client does by default, when set
                      to false. This reduces the number of metrics reported, reducing
                      Prometheus load. [Default: true]'
                    type: boolean
                  removeExternalRoutes:
                    description: Whether or not to remove device routes that have not
                      been programmed by Felix. Disabling this will allow external applications
                      to also add device routes. This is enabled by default which means
                      we will remove externally added routes.
                    type: boolean
                  reportingInterval:
                    description: 'ReportingInterval is the interval at which Felix reports
                      its status into the datastore or 0 to disable. Must be non-zero
                      in OpenStack deployments. [Default: 30s]'
                    type: string
                  reportingTTL:
                    description: 'ReportingTTL is the time-to-live setting for process-wide
                      status reports. [Default: 90s]'
                    type: string
                  routeRefreshInterval:
                    description: 'RouterefreshInterval is the period at which Felix re-checks
                      the routes in the dataplane to ensure that no other process has
                      accidentally broken Calico’s rules. Set to 0 to disable route refresh.
                      [Default: 90s]'
                    type: string
                  routeSource:
                    description: 'RouteSource configures where Felix gets its routing
                      information. - WorkloadIPs: use workload endpoints to construct
                      routes. - CalicoIPAM: the default - use IPAM data to construct routes.'
                    type: string
                  routeTableRange:
                    description: Calico programs additional Linux route tables for various
                      purposes.  RouteTableRange specifies the indices of the route tables
                      that Calico should use.
                    properties:
                      max:
                        type: integer
                      min:
                        type: integer
                    required:
                    - max
                    - min
                    type: object
                  sidecarAccelerationEnabled:
                    description: 'SidecarAccelerationEnabled enables experimental sidecar
                      acceleration [Default: false]'
                    type: boolean
                  usageReportingEnabled:
                    description: 'UsageReportingEnabled reports anonymous Calico version
                      number and cluster size to projectcalico.org. Logs warnings returned
                      by the usage server. For example, if a significant security vulnerability
                      has been discovered in the version of Calico being used. [Default:
                      true]'
                    type: boolean
                  usageReportingInitialDelay:
                    description: 'UsageReportingInitialDelay controls the minimum delay
                      before Felix makes a report. [Default: 300s]'
                    type: string
                  usageReportingInterval:
                    description: 'UsageReportingInterval controls the interval at which
                      Felix makes reports. [Default: 86400s]'
                    type: string
                  useInternalDataplaneDriver:
                    type: boolean
                  vxlanEnabled:
                    type: boolean
                  vxlanMTU:
                    description: 'VXLANMTU is the MTU to set on the tunnel device. See
                      Configuring MTU [Default: 1440]'
                    type: integer
                  vxlanPort:
                    type: integer
                  vxlanVNI:
                    type: integer
                  wireguardEnabled:
                    description: 'WireguardEnabled controls whether Wireguard is enabled.
                      [Default: false]'
                    type: boolean
                  wireguardInterfaceName:
                    description: 'WireguardInterfaceName specifies the name to use for
                      the Wireguard interface. [Default: wg.calico]'
                    type: string
                  wireguardListeningPort:
                    description: 'WireguardListeningPort controls the listening port used
                      by Wireguard. [Default: 51820]'
                    type: integer
                  wireguardMTU:
                    description: 'WireguardMTU controls the MTU on the Wireguard interface.
                      See Configuring MTU [Default: 1420]'
                    type: integer
                  wireguardRoutingRulePriority:
                    description: 'WireguardRoutingRulePriority controls the priority value
                      to use for the Wireguard routing rule. [Default: 99]'
                    type: integer
                  xdpEnabled:
                    description: 'XDPEnabled enables XDP acceleration for suitable untracked
                      incoming deny rules. [Default: true]'
                    type: boolean
                  xdpRefreshInterval:
                    description: 'XDPRefreshInterval is the period at which Felix re-checks
                      all XDP state to ensure that no other process has accidentally broken
                      Calico''s BPF maps or attached programs. Set to 0 to disable XDP
                      refresh. [Default: 90s]'
                    type: string
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: globalnetworkpolicies.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: GlobalNetworkPolicy
        listKind: GlobalNetworkPolicyList
        plural: globalnetworkpolicies
        singular: globalnetworkpolicy
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                properties:
                  applyOnForward:
                    description: ApplyOnForward indicates to apply the rules in this policy
                      on forward traffic.
                    type: boolean
                  doNotTrack:
                    description: DoNotTrack indicates whether packets matched by the rules
                      in this policy should go through the data plane's connection tracking,
                      such as Linux conntrack.  If True, the rules in this policy are
                      applied before any data plane connection tracking, and packets allowed
                      by this policy are marked as not to be tracked.
                    type: boolean
                  egress:
                    description: The ordered set of egress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  ingress:
                    description: The ordered set of ingress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  namespaceSelector:
                    description: NamespaceSelector is an optional field for an expression
                      used to select a pod based on namespaces.
                    type: string
                  order:
                    description: Order is an optional field that specifies the order in
                      which the policy is applied. Policies with higher "order" are applied
                      after those with lower order.  If the order is omitted, it may be
                      considered to be "infinite" - i.e. the policy will be applied last.  Policies
                      with identical order will be applied in alphanumerical order based
                      on the Policy "Name".
                    type: number
                  preDNAT:
                    description: PreDNAT indicates to apply the rules in this policy before
                      any DNAT.
                    type: boolean
                  selector:
                    description: "The selector is an expression used to pick pick out
                      the endpoints that the policy should be applied to. \n Selector
                      expressions follow this syntax: \n \tlabel == \"string_literal\"
                      \ ->  comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
                      \  ->  not equal; also matches if label is not present \tlabel in
                      { \"a\", \"b\", \"c\", ... }  ->  true if the value of label X is
                      one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
                      ... }  ->  true if the value of label X is not one of \"a\", \"b\",
                      \"c\" \thas(label_name)  -> True if that label is present \t! expr
                      -> negation of expr \texpr && expr  -> Short-circuit and \texpr
                      || expr  -> Short-circuit or \t( expr ) -> parens for grouping \tall()
                      or the empty selector -> matches all endpoints. \n Label names are
                      allowed to contain alphanumerics, -, _ and /. String literals are
                      more permissive but they do not support escape characters. \n Examples
                      (with made-up labels): \n \ttype == \"webserver\" && deployment
                      == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
                      \"dev\" \t! has(label_name)"
                    type: string
                  serviceAccountSelector:
                    description: ServiceAccountSelector is an optional field for an expression
                      used to select a pod based on service accounts.
                    type: string
                  types:
                    description: "Types indicates whether this policy applies to ingress,
                      or to egress, or to both.  When not explicitly specified (and so
                      the value on creation is empty or nil), Calico defaults Types according
                      to what Ingress and Egress rules are present in the policy.  The
                      default is: \n - [ PolicyTypeIngress ], if there are no Egress rules
                      (including the case where there are   also no Ingress rules) \n
                      - [ PolicyTypeEgress ], if there are Egress rules but no Ingress
                      rules \n - [ PolicyTypeIngress, PolicyTypeEgress ], if there are
                      both Ingress and Egress rules. \n When the policy is read back again,
                      Types will always be one of these values, never empty or nil."
                    items:
                      description: PolicyType enumerates the possible values of the PolicySpec
                        Types field.
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: globalnetworksets.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: GlobalNetworkSet
        listKind: GlobalNetworkSetList
        plural: globalnetworksets
        singular: globalnetworkset
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: GlobalNetworkSet contains a set of arbitrary IP sub-networks/CIDRs
              that share labels to allow rules to refer to them via selectors.  The labels
              of GlobalNetworkSet are not namespaced.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: GlobalNetworkSetSpec contains the specification for a NetworkSet
                  resource.
                properties:
                  nets:
                    description: The list of IP networks that belong to this set.
                    items:
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: hostendpoints.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: HostEndpoint
        listKind: HostEndpointList
        plural: hostendpoints
        singular: hostendpoint
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: HostEndpointSpec contains the specification for a HostEndpoint
                  resource.
                properties:
                  expectedIPs:
                    description: "The expected IP addresses (IPv4 and IPv6) of the endpoint.
                      If \"InterfaceName\" is not present, Calico will look for an interface
                      matching any of the IPs in the list and apply policy to that. Note:
                      \tWhen using the selector match criteria in an ingress or egress
                      security Policy \tor Profile, Calico converts the selector into
                      a set of IP addresses. For host \tendpoints, the ExpectedIPs field
                      is used for that purpose. (If only the interface \tname is specified,
                      Calico does not learn the IPs of the interface for use in match
                      \tcriteria.)"
                    items:
                      type: string
                    type: array
                  interfaceName:
                    description: "Either \"*\", or the name of a specific Linux interface
                      to apply policy to; or empty.  \"*\" indicates that this HostEndpoint
                      governs all traffic to, from or through the default network namespace
                      of the host named by the \"Node\" field; entering and leaving that
                      namespace via any interface, including those from/to non-host-networked
                      local workloads. \n If InterfaceName is not \"*\", this HostEndpoint
                      only governs traffic that enters or leaves the host through the
                      specific interface named by InterfaceName, or - when InterfaceName
                      is empty - through the specific interface that has one of the IPs
                      in ExpectedIPs. Therefore, when InterfaceName is empty, at least
                      one expected IP must be specified.  Only external interfaces (such
                      as “eth0”) are supported here; it isn't possible for a HostEndpoint
                      to protect traffic through a specific local workload interface.
                      \n Note: Only some kinds of policy are implemented for \"*\" HostEndpoints;
                      initially just pre-DNAT policy.  Please check Calico documentation
                      for the latest position."
                    type: string
                  node:
                    description: The node name identifying the Calico node instance.
                    type: string
                  ports:
                    description: Ports contains the endpoint's named ports, which may
                      be referenced in security policy rules.
                    items:
                      properties:
                        name:
                          type: string
                        port:
                          type: integer
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                      required:
                      - name
                      - port
                      - protocol
                      type: object
                    type: array
                  profiles:
                    description: A list of identifiers of security Profile objects that
                      apply to this endpoint. Each profile is applied in the order that
                      they appear in this list.  Profile rules are applied after the selector-based
                      security policy.
                    items:
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ipamblocks.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPAMBlock
        listKind: IPAMBlockList
        plural: ipamblocks
        singular: ipamblock
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPAMBlockSpec contains the specification for an IPAMBlock
                  resource.
                properties:
                  affinity:
                    type: string
                  allocations:
                    items:
                      type: integer
                      # TODO: This nullable is manually added in. We should update controller-gen
                      # to handle []*int properly itself.
                      nullable: true
                    type: array
                  attributes:
                    items:
                      properties:
                        handle_id:
                          type: string
                        secondary:
                          additionalProperties:
                            type: string
                          type: object
                      type: object
                    type: array
                  cidr:
                    type: string
                  deleted:
                    type: boolean
                  strictAffinity:
                    type: boolean
                  unallocated:
                    items:
                      type: integer
                    type: array
                required:
                - allocations
                - attributes
                - cidr
                - deleted
                - strictAffinity
                - unallocated
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ipamconfigs.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPAMConfig
        listKind: IPAMConfigList
        plural: ipamconfigs
        singular: ipamconfig
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPAMConfigSpec contains the specification for an IPAMConfig
                  resource.
                properties:
                  autoAllocateBlocks:
                    type: boolean
                  strictAffinity:
                    type: boolean
                required:
                - autoAllocateBlocks
                - strictAffinity
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ipamhandles.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPAMHandle
        listKind: IPAMHandleList
        plural: ipamhandles
        singular: ipamhandle
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPAMHandleSpec contains the specification for an IPAMHandle
                  resource.
                properties:
                  block:
                    additionalProperties:
                      type: integer
                    type: object
                  handleID:
                    type: string
                required:
                - block
                - handleID
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ippools.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPPool
        listKind: IPPoolList
        plural: ippools
        singular: ippool
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPPoolSpec contains the specification for an IPPool resource.
                properties:
                  blockSize:
                    description: The block size to use for IP address assignments from
                      this pool. Defaults to 26 for IPv4 and 112 for IPv6.
                    type: integer
                  cidr:
                    description: The pool CIDR.
                    type: string
                  disabled:
                    description: When disabled is true, Calico IPAM will not assign addresses
                      from this pool.
                    type: boolean
                  ipip:
                    description: 'Deprecated: this field is only used for APIv1 backwards
                      compatibility. Setting this field is not allowed, this field is
                      for internal use only.'
                    properties:
                      enabled:
                        description: When enabled is true, ipip tunneling will be used
                          to deliver packets to destinations within this pool.
                        type: boolean
                      mode:
                        description: The IPIP mode.  This can be one of "always" or "cross-subnet".  A
                          mode of "always" will also use IPIP tunneling for routing to
                          destination IP addresses within this pool.  A mode of "cross-subnet"
                          will only use IPIP tunneling when the destination node is on
                          a different subnet to the originating node.  The default value
                          (if not specified) is "always".
                        type: string
                    type: object
                  ipipMode:
                    description: Contains configuration for IPIP tunneling for this pool.
                      If not specified, then this is defaulted to "Never" (i.e. IPIP tunneling
                      is disabled).
                    type: string
                  nat-outgoing:
                    description: 'Deprecated: this field is only used for APIv1 backwards
                      compatibility. Setting this field is not allowed, this field is
                      for internal use only.'
                    type: boolean
                  natOutgoing:
                    description: When nat-outgoing is true, packets sent from Calico networked
                      containers in this pool to destinations outside of this pool will
                      be masqueraded.
                    type: boolean
                  nodeSelector:
                    description: Allows IPPool to allocate for a specific node by label
                      selector.
                    type: string
                  vxlanMode:
                    description: Contains configuration for VXLAN tunneling for this pool.
                      If not specified, then this is defaulted to "Never" (i.e. VXLAN
                      tunneling is disabled).
                    type: string
                required:
                - cidr
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: kubecontrollersconfigurations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: KubeControllersConfiguration
        listKind: KubeControllersConfigurationList
        plural: kubecontrollersconfigurations
        singular: kubecontrollersconfiguration
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: KubeControllersConfigurationSpec contains the values of the
                  Kubernetes controllers configuration.
                properties:
                  controllers:
                    description: Controllers enables and configures individual Kubernetes
                      controllers
                    properties:
                      namespace:
                        description: Namespace enables and configures the namespace controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                      node:
                        description: Node enables and configures the node controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          hostEndpoint:
                            description: HostEndpoint controls syncing nodes to host endpoints.
                              Disabled by default, set to nil to disable.
                            properties:
                              autoCreate:
                                description: 'AutoCreate enables automatic creation of
                                  host endpoints for every node. [Default: Disabled]'
                                type: string
                            type: object
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                          syncLabels:
                            description: 'SyncLabels controls whether to copy Kubernetes
                              node labels to Calico nodes. [Default: Enabled]'
                            type: string
                        type: object
                      policy:
                        description: Policy enables and configures the policy controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                      serviceAccount:
                        description: ServiceAccount enables and configures the service
                          account controller. Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                      workloadEndpoint:
                        description: WorkloadEndpoint enables and configures the workload
                          endpoint controller. Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                    type: object
                  etcdV3CompactionPeriod:
                    description: 'EtcdV3CompactionPeriod is the period between etcdv3
                      compaction requests. Set to 0 to disable. [Default: 10m]'
                    type: string
                  healthChecks:
                    description: 'HealthChecks enables or disables support for health
                      checks [Default: Enabled]'
                    type: string
                  logSeverityScreen:
                    description: 'LogSeverityScreen is the log severity above which logs
                      are sent to the stdout. [Default: Info]'
                    type: string
                required:
                - controllers
                type: object
              status:
                description: KubeControllersConfigurationStatus represents the status
                  of the configuration. It's useful for admins to be able to see the actual
                  config that was applied, which can be modified by environment variables
                  on the kube-controllers process.
                properties:
                  environmentVars:
                    additionalProperties:
                      type: string
                    description: EnvironmentVars contains the environment variables on
                      the kube-controllers that influenced the RunningConfig.
                    type: object
                  runningConfig:
                    description: RunningConfig contains the effective config that is running
                      in the kube-controllers pod, after merging the API resource with
                      any environment variables.
                    properties:
                      controllers:
                        description: Controllers enables and configures individual Kubernetes
                          controllers
                        properties:
                          namespace:
                            description: Namespace enables and configures the namespace
                              controller. Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                          node:
                            description: Node enables and configures the node controller.
                              Enabled by default, set to nil to disable.
                            properties:
                              hostEndpoint:
                                description: HostEndpoint controls syncing nodes to host
                                  endpoints. Disabled by default, set to nil to disable.
                                properties:
                                  autoCreate:
                                    description: 'AutoCreate enables automatic creation
                                      of host endpoints for every node. [Default: Disabled]'
                                    type: string
                                type: object
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                              syncLabels:
                                description: 'SyncLabels controls whether to copy Kubernetes
                                  node labels to Calico nodes. [Default: Enabled]'
                                type: string
                            type: object
                          policy:
                            description: Policy enables and configures the policy controller.
                              Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                          serviceAccount:
                            description: ServiceAccount enables and configures the service
                              account controller. Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                          workloadEndpoint:
                            description: WorkloadEndpoint enables and configures the workload
                              endpoint controller. Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                        type: object
                      etcdV3CompactionPeriod:
                        description: 'EtcdV3CompactionPeriod is the period between etcdv3
                          compaction requests. Set to 0 to disable. [Default: 10m]'
                        type: string
                      healthChecks:
                        description: 'HealthChecks enables or disables support for health
                          checks [Default: Enabled]'
                        type: string
                      logSeverityScreen:
                        description: 'LogSeverityScreen is the log severity above which
                          logs are sent to the stdout. [Default: Info]'
                        type: string
                    required:
                    - controllers
                    type: object
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: networkpolicies.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: NetworkPolicy
        listKind: NetworkPolicyList
        plural: networkpolicies
        singular: networkpolicy
      scope: Namespaced
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                properties:
                  egress:
                    description: The ordered set of egress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  ingress:
                    description: The ordered set of ingress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  order:
                    description: Order is an optional field that specifies the order in
                      which the policy is applied. Policies with higher "order" are applied
                      after those with lower order.  If the order is omitted, it may be
                      considered to be "infinite" - i.e. the policy will be applied last.  Policies
                      with identical order will be applied in alphanumerical order based
                      on the Policy "Name".
                    type: number
                  selector:
                    description: "The selector is an expression used to pick pick out
                      the endpoints that the policy should be applied to. \n Selector
                      expressions follow this syntax: \n \tlabel == \"string_literal\"
                      \ ->  comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
                      \  ->  not equal; also matches if label is not present \tlabel in
                      { \"a\", \"b\", \"c\", ... }  ->  true if the value of label X is
                      one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
                      ... }  ->  true if the value of label X is not one of \"a\", \"b\",
                      \"c\" \thas(label_name)  -> True if that label is present \t! expr
                      -> negation of expr \texpr && expr  -> Short-circuit and \texpr
                      || expr  -> Short-circuit or \t( expr ) -> parens for grouping \tall()
                      or the empty selector -> matches all endpoints. \n Label names are
                      allowed to contain alphanumerics, -, _ and /. String literals are
                      more permissive but they do not support escape characters. \n Examples
                      (with made-up labels): \n \ttype == \"webserver\" && deployment
                      == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
                      \"dev\" \t! has(label_name)"
                    type: string
                  serviceAccountSelector:
                    description: ServiceAccountSelector is an optional field for an expression
                      used to select a pod based on service accounts.
                    type: string
                  types:
                    description: "Types indicates whether this policy applies to ingress,
                      or to egress, or to both.  When not explicitly specified (and so
                      the value on creation is empty or nil), Calico defaults Types according
                      to what Ingress and Egress are present in the policy.  The default
                      is: \n - [ PolicyTypeIngress ], if there are no Egress rules (including
                      the case where there are   also no Ingress rules) \n - [ PolicyTypeEgress
                      ], if there are Egress rules but no Ingress rules \n - [ PolicyTypeIngress,
                      PolicyTypeEgress ], if there are both Ingress and Egress rules.
                      \n When the policy is read back again, Types will always be one
                      of these values, never empty or nil."
                    items:
                      description: PolicyType enumerates the possible values of the PolicySpec
                        Types field.
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: networksets.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: NetworkSet
        listKind: NetworkSetList
        plural: networksets
        singular: networkset
      scope: Namespaced
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: NetworkSet is the Namespaced-equivalent of the GlobalNetworkSet.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: NetworkSetSpec contains the specification for a NetworkSet
                  resource.
                properties:
                  nets:
                    description: The list of IP networks that belong to this set.
                    items:
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---
    ---
    # Source: calico/templates/calico-kube-controllers-rbac.yaml

    # Include a clusterrole for the kube-controllers component,
    # and bind it to the calico-kube-controllers serviceaccount.
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: calico-kube-controllers
    rules:
      # Nodes are watched to monitor for deletions.
      - apiGroups: [""]
        resources:
          - nodes
        verbs:
          - watch
          - list
          - get
      # Pods are queried to check for existence.
      - apiGroups: [""]
        resources:
          - pods
        verbs:
          - get
      # IPAM resources are manipulated when nodes are deleted.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - ippools
        verbs:
          - list
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - blockaffinities
          - ipamblocks
          - ipamhandles
        verbs:
          - get
          - list
          - create
          - update
          - delete
      # kube-controllers manages hostendpoints.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - hostendpoints
        verbs:
          - get
          - list
          - create
          - update
          - delete
      # Needs access to update clusterinformations.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - clusterinformations
        verbs:
          - get
          - create
          - update
      # KubeControllersConfiguration is where it gets its config
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - kubecontrollersconfigurations
        verbs:
          # read its own config
          - get
          # create a default if none exists
          - create
          # update status
          - update
          # watch for changes
          - watch
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: calico-kube-controllers
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: calico-kube-controllers
    subjects:
    - kind: ServiceAccount
      name: calico-kube-controllers
      namespace: kube-system
    ---

    ---
    # Source: calico/templates/calico-node-rbac.yaml
    # Include a clusterrole for the calico-node DaemonSet,
    # and bind it to the calico-node serviceaccount.
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: calico-node
    rules:
      # The CNI plugin needs to get pods, nodes, and namespaces.
      - apiGroups: [""]
        resources:
          - pods
          - nodes
          - namespaces
        verbs:
          - get
      - apiGroups: [""]
        resources:
          - endpoints
          - services
        verbs:
          # Used to discover service IPs for advertisement.
          - watch
          - list
          # Used to discover Typhas.
          - get
      # Pod CIDR auto-detection on kubeadm needs access to config maps.
      - apiGroups: [""]
        resources:
          - configmaps
        verbs:
          - get
      - apiGroups: [""]
        resources:
          - nodes/status
        verbs:
          # Needed for clearing NodeNetworkUnavailable flag.
          - patch
          # Calico stores some configuration information in node annotations.
          - update
      # Watch for changes to Kubernetes NetworkPolicies.
      - apiGroups: ["networking.k8s.io"]
        resources:
          - networkpolicies
        verbs:
          - watch
          - list
      # Used by Calico for policy information.
      - apiGroups: [""]
        resources:
          - pods
          - namespaces
          - serviceaccounts
        verbs:
          - list
          - watch
      # The CNI plugin patches pods/status.
      - apiGroups: [""]
        resources:
          - pods/status
        verbs:
          - patch
      # Calico monitors various CRDs for config.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - globalfelixconfigs
          - felixconfigurations
          - bgppeers
          - globalbgpconfigs
          - bgpconfigurations
          - ippools
          - ipamblocks
          - globalnetworkpolicies
          - globalnetworksets
          - networkpolicies
          - networksets
          - clusterinformations
          - hostendpoints
          - blockaffinities
        verbs:
          - get
          - list
          - watch
      # Calico must create and update some CRDs on startup.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - ippools
          - felixconfigurations
          - clusterinformations
        verbs:
          - create
          - update
      # Calico stores some configuration information on the node.
      - apiGroups: [""]
        resources:
          - nodes
        verbs:
          - get
          - list
          - watch
      # These permissions are only required for upgrade from v2.6, and can
      # be removed after upgrade or on fresh installations.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - bgpconfigurations
          - bgppeers
        verbs:
          - create
          - update
      # These permissions are required for Calico CNI to perform IPAM allocations.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - blockaffinities
          - ipamblocks
          - ipamhandles
        verbs:
          - get
          - list
          - create
          - update
          - delete
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - ipamconfigs
        verbs:
          - get
      # Block affinities must also be watchable by confd for route aggregation.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - blockaffinities
        verbs:
          - watch
      # The Calico IPAM migration needs to get daemonsets. These permissions can be
      # removed if not upgrading from an installation using host-local IPAM.
      - apiGroups: ["apps"]
        resources:
          - daemonsets
        verbs:
          - get

    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: calico-node
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: calico-node
    subjects:
    - kind: ServiceAccount
      name: calico-node
      namespace: kube-system

    ---
    # Source: calico/templates/calico-node.yaml
    # This manifest installs the calico-node container, as well
    # as the CNI plugins and network config on
    # each master and worker node in a Kubernetes cluster.
    kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: calico-node
      namespace: kube-system
      labels:
        k8s-app: calico-node
    spec:
      selector:
        matchLabels:
          k8s-app: calico-node
      updateStrategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
      template:
        metadata:
          labels:
            k8s-app: calico-node
        spec:
          nodeSelector:
            kubernetes.io/os: linux
          hostNetwork: true
          tolerations:
            # Make sure calico-node gets scheduled on all nodes.
            - effect: NoSchedule
              operator: Exists
            # Mark the pod as a critical add-on for rescheduling.
            - key: CriticalAddonsOnly
              operator: Exists
            - effect: NoExecute
              operator: Exists
          serviceAccountName: calico-node
          # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
          # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
          terminationGracePeriodSeconds: 0
          priorityClassName: system-node-critical
          initContainers:
            # This container performs upgrade from host-local IPAM to calico-ipam.
            # It can be deleted if this is a fresh installation, or if you have already
            # upgraded to use calico-ipam.
            - name: upgrade-ipam
              image: calico/cni:v3.16.1
              command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
              envFrom:
              - configMapRef:
                  # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                  name: kubernetes-services-endpoint
                  optional: true
              env:
                - name: KUBERNETES_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: CALICO_NETWORKING_BACKEND
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: calico_backend
              volumeMounts:
                - mountPath: /var/lib/cni/networks
                  name: host-local-net-dir
                - mountPath: /host/opt/cni/bin
                  name: cni-bin-dir
              securityContext:
                privileged: true
            # This container installs the CNI binaries
            # and CNI network config file on each node.
            - name: install-cni
              image: calico/cni:v3.16.1
              command: ["/opt/cni/bin/install"]
              envFrom:
              - configMapRef:
                  # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                  name: kubernetes-services-endpoint
                  optional: true
              env:
                # Name of the CNI config file to create.
                - name: CNI_CONF_NAME
                  value: "10-calico.conflist"
                # The CNI network config to install on each node.
                - name: CNI_NETWORK_CONFIG
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: cni_network_config
                # Set the hostname based on the k8s node name.
                - name: KUBERNETES_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                # CNI MTU Config variable
                - name: CNI_MTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # Prevents the container from sleeping forever.
                - name: SLEEP
                  value: "false"
              volumeMounts:
                - mountPath: /host/opt/cni/bin
                  name: cni-bin-dir
                - mountPath: /host/etc/cni/net.d
                  name: cni-net-dir
              securityContext:
                privileged: true
            # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
            # to communicate with Felix over the Policy Sync API.
            - name: flexvol-driver
              image: calico/pod2daemon-flexvol:v3.16.1
              volumeMounts:
              - name: flexvol-driver-host
                mountPath: /host/driver
              securityContext:
                privileged: true
          containers:
            # Runs calico-node container on each Kubernetes node. This
            # container programs network policy and routes on each
            # host.
            - name: calico-node
              image: calico/node:v3.16.1
              envFrom:
              - configMapRef:
                  # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                  name: kubernetes-services-endpoint
                  optional: true
              env:
                # Use Kubernetes API as the backing datastore.
                - name: DATASTORE_TYPE
                  value: "kubernetes"
                # Wait for the datastore.
                - name: WAIT_FOR_DATASTORE
                  value: "true"
                # Set based on the k8s node name.
                - name: NODENAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                # Choose the backend to use.
                - name: CALICO_NETWORKING_BACKEND
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: calico_backend
                # Cluster type to identify the deployment type
                - name: CLUSTER_TYPE
                  value: "k8s,bgp"
                # Auto-detect the BGP IP address.
                - name: IP
                  value: "autodetect"
                # Enable IPIP
                - name: CALICO_IPV4POOL_IPIP
                  value: "Always"
                # Enable or Disable VXLAN on the default IP pool.
                - name: CALICO_IPV4POOL_VXLAN
                  value: "Never"
                # Set MTU for tunnel device used if ipip is enabled
                - name: FELIX_IPINIPMTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # Set MTU for the VXLAN tunnel device.
                - name: FELIX_VXLANMTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # Set MTU for the Wireguard tunnel device.
                - name: FELIX_WIREGUARDMTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # The default IPv4 pool to create on startup if none exists. Pod IPs will be
                # chosen from this range. Changing this value after installation will have
                # no effect. This should fall within `--cluster-cidr`.
                # - name: CALICO_IPV4POOL_CIDR
                #   value: "192.168.0.0/16"
                # Disable file logging so `kubectl logs` works.
                - name: CALICO_DISABLE_FILE_LOGGING
                  value: "true"
                # Set Felix endpoint to host default action to ACCEPT.
                - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
                  value: "ACCEPT"
                # Disable IPv6 on Kubernetes.
                - name: FELIX_IPV6SUPPORT
                  value: "false"
                # Set Felix logging to "info"
                - name: FELIX_LOGSEVERITYSCREEN
                  value: "info"
                - name: FELIX_HEALTHENABLED
                  value: "true"
              securityContext:
                privileged: true
              resources:
                requests:
                  cpu: 250m
              livenessProbe:
                exec:
                  command:
                  - /bin/calico-node
                  - -felix-live
                  - -bird-live
                periodSeconds: 10
                initialDelaySeconds: 10
                failureThreshold: 6
              readinessProbe:
                exec:
                  command:
                  - /bin/calico-node
                  - -felix-ready
                  - -bird-ready
                periodSeconds: 10
              volumeMounts:
                - mountPath: /lib/modules
                  name: lib-modules
                  readOnly: true
                - mountPath: /run/xtables.lock
                  name: xtables-lock
                  readOnly: false
                - mountPath: /var/run/calico
                  name: var-run-calico
                  readOnly: false
                - mountPath: /var/lib/calico
                  name: var-lib-calico
                  readOnly: false
                - name: policysync
                  mountPath: /var/run/nodeagent
                # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
                # parent directory.
                - name: sysfs
                  mountPath: /sys/fs/
                  # Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
                  # If the host is known to mount that filesystem already then Bidirectional can be omitted.
                  mountPropagation: Bidirectional
          volumes:
            # Used by calico-node.
            - name: lib-modules
              hostPath:
                path: /lib/modules
            - name: var-run-calico
              hostPath:
                path: /var/run/calico
            - name: var-lib-calico
              hostPath:
                path: /var/lib/calico
            - name: xtables-lock
              hostPath:
                path: /run/xtables.lock
                type: FileOrCreate
            - name: sysfs
              hostPath:
                path: /sys/fs/
                type: DirectoryOrCreate
            # Used to install CNI.
            - name: cni-bin-dir
              hostPath:
                path: /opt/cni/bin
            - name: cni-net-dir
              hostPath:
                path: /etc/cni/net.d
            # Mount in the directory for host-local IPAM allocations. This is
            # used when upgrading from host-local to calico-ipam, and can be removed
            # if not using the upgrade-ipam init container.
            - name: host-local-net-dir
              hostPath:
                path: /var/lib/cni/networks
            # Used to create per-pod Unix Domain Sockets
            - name: policysync
              hostPath:
                type: DirectoryOrCreate
                path: /var/run/nodeagent
            # Used to install Flex Volume Driver
            - name: flexvol-driver-host
              hostPath:
                type: DirectoryOrCreate
                path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    ---

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: calico-node
      namespace: kube-system

    ---
    # Source: calico/templates/calico-kube-controllers.yaml
    # See https://github.com/projectcalico/kube-controllers
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      # The controllers can only have a single active instance.
      replicas: 1
      selector:
        matchLabels:
          k8s-app: calico-kube-controllers
      strategy:
        type: Recreate
      template:
        metadata:
          name: calico-kube-controllers
          namespace: kube-system
          labels:
            k8s-app: calico-kube-controllers
        spec:
          nodeSelector:
            kubernetes.io/os: linux
          tolerations:
            # Mark the pod as a critical add-on for rescheduling.
            - key: CriticalAddonsOnly
              operator: Exists
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
          serviceAccountName: calico-kube-controllers
          priorityClassName: system-cluster-critical
          containers:
            - name: calico-kube-controllers
              image: calico/kube-controllers:v3.16.1
              env:
                # Choose which controllers to run.
                - name: ENABLED_CONTROLLERS
                  value: node
                - name: DATASTORE_TYPE
                  value: kubernetes
              readinessProbe:
                exec:
                  command:
                  - /usr/bin/check-status
                  - -r

    ---

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: calico-kube-controllers
      namespace: kube-system

    ---
    # Source: calico/templates/calico-etcd-secrets.yaml

    ---
    # Source: calico/templates/calico-typha.yaml

    ---
    # Source: calico/templates/configure-canal.yaml


kind: ConfigMap
metadata:
  creationTimestamp: null
  name: calico-cni
apiVersion: v1
data:
  calico-manifest.yaml: |+
    ---
    # Source: calico/templates/calico-config.yaml
    # This ConfigMap is used to configure a self-hosted Calico installation.
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: calico-config
      namespace: kube-system
    data:
      # Typha is disabled.
      typha_service_name: "none"
      # Configure the backend to use.
      calico_backend: "bird"
      # Configure the MTU to use for workload interfaces and tunnels.
      # - If Wireguard is enabled, set to your network MTU - 60
      # - Otherwise, if VXLAN or BPF mode is enabled, set to your network MTU - 50
      # - Otherwise, if IPIP is enabled, set to your network MTU - 20
      # - Otherwise, if not using any encapsulation, set to your network MTU.
      veth_mtu: "1440"

      # The CNI network configuration to install on each node. The special
      # values in this config will be automatically populated.
      cni_network_config: |-
        {
          "name": "k8s-pod-network",
          "cniVersion": "0.3.1",
          "plugins": [
            {
              "type": "calico",
              "log_level": "info",
              "log_file_path": "/var/log/calico/cni/cni.log",
              "datastore_type": "kubernetes",
              "nodename": "__KUBERNETES_NODE_NAME__",
              "mtu": __CNI_MTU__,
              "ipam": {
                  "type": "calico-ipam"
              },
              "policy": {
                  "type": "k8s"
              },
              "kubernetes": {
                  "kubeconfig": "__KUBECONFIG_FILEPATH__"
              }
            },
            {
              "type": "portmap",
              "snat": true,
              "capabilities": {"portMappings": true}
            },
            {
              "type": "bandwidth",
              "capabilities": {"bandwidth": true}
            }
          ]
        }

    ---
    # Source: calico/templates/kdd-crds.yaml


    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: bgpconfigurations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: BGPConfiguration
        listKind: BGPConfigurationList
        plural: bgpconfigurations
        singular: bgpconfiguration
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: BGPConfiguration contains the configuration for any BGP routing.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: BGPConfigurationSpec contains the values of the BGP configuration.
                properties:
                  asNumber:
                    description: 'ASNumber is the default AS number used by a node. [Default:
                      64512]'
                    format: int32
                    type: integer
                  communities:
                    description: Communities is a list of BGP community values and their
                      arbitrary names for tagging routes.
                    items:
                      description: Community contains standard or large community value
                        and its name.
                      properties:
                        name:
                          description: Name given to community value.
                          type: string
                        value:
                          description: Value must be of format `aa:nn` or `aa:nn:mm`.
                            For standard community use `aa:nn` format, where `aa` and
                            `nn` are 16 bit number. For large community use `aa:nn:mm`
                            format, where `aa`, `nn` and `mm` are 32 bit number. Where,
                            `aa` is an AS Number, `nn` and `mm` are per-AS identifier.
                          pattern: ^(\d+):(\d+)$|^(\d+):(\d+):(\d+)$
                          type: string
                      type: object
                    type: array
                  listenPort:
                    description: ListenPort is the port where BGP protocol should listen.
                      Defaults to 179
                    maximum: 65535
                    minimum: 1
                    type: integer
                  logSeverityScreen:
                    description: 'LogSeverityScreen is the log severity above which logs
                      are sent to the stdout. [Default: INFO]'
                    type: string
                  nodeToNodeMeshEnabled:
                    description: 'NodeToNodeMeshEnabled sets whether full node to node
                      BGP mesh is enabled. [Default: true]'
                    type: boolean
                  prefixAdvertisements:
                    description: PrefixAdvertisements contains per-prefix advertisement
                      configuration.
                    items:
                      description: PrefixAdvertisement configures advertisement properties
                        for the specified CIDR.
                      properties:
                        cidr:
                          description: CIDR for which properties should be advertised.
                          type: string
                        communities:
                          description: Communities can be list of either community names
                            already defined in `Specs.Communities` or community value
                            of format `aa:nn` or `aa:nn:mm`. For standard community use
                            `aa:nn` format, where `aa` and `nn` are 16 bit number. For
                            large community use `aa:nn:mm` format, where `aa`, `nn` and
                            `mm` are 32 bit number. Where,`aa` is an AS Number, `nn` and
                            `mm` are per-AS identifier.
                          items:
                            type: string
                          type: array
                      type: object
                    type: array
                  serviceClusterIPs:
                    description: ServiceClusterIPs are the CIDR blocks from which service
                      cluster IPs are allocated. If specified, Calico will advertise these
                      blocks, as well as any cluster IPs within them.
                    items:
                      description: ServiceClusterIPBlock represents a single allowed ClusterIP
                        CIDR block.
                      properties:
                        cidr:
                          type: string
                      type: object
                    type: array
                  serviceExternalIPs:
                    description: ServiceExternalIPs are the CIDR blocks for Kubernetes
                      Service External IPs. Kubernetes Service ExternalIPs will only be
                      advertised if they are within one of these blocks.
                    items:
                      description: ServiceExternalIPBlock represents a single allowed
                        External IP CIDR block.
                      properties:
                        cidr:
                          type: string
                      type: object
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: bgppeers.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: BGPPeer
        listKind: BGPPeerList
        plural: bgppeers
        singular: bgppeer
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: BGPPeerSpec contains the specification for a BGPPeer resource.
                properties:
                  asNumber:
                    description: The AS Number of the peer.
                    format: int32
                    type: integer
                  keepOriginalNextHop:
                    description: Option to keep the original nexthop field when routes
                      are sent to a BGP Peer. Setting "true" configures the selected BGP
                      Peers node to use the "next hop keep;" instead of "next hop self;"(default)
                      in the specific branch of the Node on "bird.cfg".
                    type: boolean
                  node:
                    description: The node name identifying the Calico node instance that
                      is peering with this peer. If this is not set, this represents a
                      global peer, i.e. a peer that peers with every node in the deployment.
                    type: string
                  nodeSelector:
                    description: Selector for the nodes that should have this peering.  When
                      this is set, the Node field must be empty.
                    type: string
                  peerIP:
                    description: The IP address of the peer followed by an optional port
                      number to peer with. If port number is given, format should be `[<IPv6>]:port`
                      or `<IPv4>:<port>` for IPv4. If optional port number is not set,
                      and this peer IP and ASNumber belongs to a calico/node with ListenPort
                      set in BGPConfiguration, then we use that port to peer.
                    type: string
                  peerSelector:
                    description: Selector for the remote nodes to peer with.  When this
                      is set, the PeerIP and ASNumber fields must be empty.  For each
                      peering between the local node and selected remote nodes, we configure
                      an IPv4 peering if both ends have NodeBGPSpec.IPv4Address specified,
                      and an IPv6 peering if both ends have NodeBGPSpec.IPv6Address specified.  The
                      remote AS number comes from the remote node’s NodeBGPSpec.ASNumber,
                      or the global default if that is not set.
                    type: string
                required:
                - asNumber
                - peerIP
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: blockaffinities.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: BlockAffinity
        listKind: BlockAffinityList
        plural: blockaffinities
        singular: blockaffinity
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: BlockAffinitySpec contains the specification for a BlockAffinity
                  resource.
                properties:
                  cidr:
                    type: string
                  deleted:
                    description: Deleted indicates that this block affinity is being deleted.
                      This field is a string for compatibility with older releases that
                      mistakenly treat this field as a string.
                    type: string
                  node:
                    type: string
                  state:
                    type: string
                required:
                - cidr
                - deleted
                - node
                - state
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: clusterinformations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: ClusterInformation
        listKind: ClusterInformationList
        plural: clusterinformations
        singular: clusterinformation
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: ClusterInformation contains the cluster specific information.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: ClusterInformationSpec contains the values of describing
                  the cluster.
                properties:
                  calicoVersion:
                    description: CalicoVersion is the version of Calico that the cluster
                      is running
                    type: string
                  clusterGUID:
                    description: ClusterGUID is the GUID of the cluster
                    type: string
                  clusterType:
                    description: ClusterType describes the type of the cluster
                    type: string
                  datastoreReady:
                    description: DatastoreReady is used during significant datastore migrations
                      to signal to components such as Felix that it should wait before
                      accessing the datastore.
                    type: boolean
                  variant:
                    description: Variant declares which variant of Calico should be active.
                    type: string
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: felixconfigurations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: FelixConfiguration
        listKind: FelixConfigurationList
        plural: felixconfigurations
        singular: felixconfiguration
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: Felix Configuration contains the configuration for Felix.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: FelixConfigurationSpec contains the values of the Felix configuration.
                properties:
                  allowIPIPPacketsFromWorkloads:
                    description: 'AllowIPIPPacketsFromWorkloads controls whether Felix
                      will add a rule to drop IPIP encapsulated traffic from workloads
                      [Default: false]'
                    type: boolean
                  allowVXLANPacketsFromWorkloads:
                    description: 'AllowVXLANPacketsFromWorkloads controls whether Felix
                      will add a rule to drop VXLAN encapsulated traffic from workloads
                      [Default: false]'
                    type: boolean
                  awsSrcDstCheck:
                    description: 'Set source-destination-check on AWS EC2 instances. Accepted
                      value must be one of "DoNothing", "Enabled" or "Disabled". [Default:
                      DoNothing]'
                    enum:
                    - DoNothing
                    - Enable
                    - Disable
                    type: string
                  bpfConnectTimeLoadBalancingEnabled:
                    description: 'BPFConnectTimeLoadBalancingEnabled when in BPF mode,
                      controls whether Felix installs the connection-time load balancer.  The
                      connect-time load balancer is required for the host to be able to
                      reach Kubernetes services and it improves the performance of pod-to-service
                      connections.  The only reason to disable it is for debugging purposes.  [Default:
                      true]'
                    type: boolean
                  bpfDataIfacePattern:
                    description: 'BPFDataIfacePattern is a regular expression that controls
                      which interfaces Felix should attach BPF programs to in order to
                      catch traffic to/from the network.  This needs to match the interfaces
                      that Calico workload traffic flows over as well as any interfaces
                      that handle incoming traffic to nodeports and services from outside
                      the cluster.  It should not match the workload interfaces (usually
                      named cali...). [Default: ^(en.*|eth.*|tunl0$)]'
                    type: string
                  bpfDisableUnprivileged:
                    description: 'BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled
                      sysctl to disable unprivileged use of BPF.  This ensures that unprivileged
                      users cannot access Calico''s BPF maps and cannot insert their own
                      BPF programs to interfere with Calico''s. [Default: true]'
                    type: boolean
                  bpfEnabled:
                    description: 'BPFEnabled, if enabled Felix will use the BPF dataplane.
                      [Default: false]'
                    type: boolean
                  bpfExternalServiceMode:
                    description: 'BPFExternalServiceMode in BPF mode, controls how connections
                      from outside the cluster to services (node ports and cluster IPs)
                      are forwarded to remote workloads.  If set to "Tunnel" then both
                      request and response traffic is tunneled to the remote node.  If
                      set to "DSR", the request traffic is tunneled but the response traffic
                      is sent directly from the remote node.  In "DSR" mode, the remote
                      node appears to use the IP of the ingress node; this requires a
                      permissive L2 network.  [Default: Tunnel]'
                    type: string
                  bpfKubeProxyEndpointSlicesEnabled:
                    description: BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls
                      whether Felix's embedded kube-proxy accepts EndpointSlices or not.
                    type: boolean
                  bpfKubeProxyIptablesCleanupEnabled:
                    description: 'BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF
                      mode, Felix will proactively clean up the upstream Kubernetes kube-proxy''s
                      iptables chains.  Should only be enabled if kube-proxy is not running.  [Default:
                      true]'
                    type: boolean
                  bpfKubeProxyMinSyncPeriod:
                    description: 'BPFKubeProxyMinSyncPeriod, in BPF mode, controls the
                      minimum time between updates to the dataplane for Felix''s embedded
                      kube-proxy.  Lower values give reduced set-up latency.  Higher values
                      reduce Felix CPU usage by batching up more work.  [Default: 1s]'
                    type: string
                  bpfLogLevel:
                    description: 'BPFLogLevel controls the log level of the BPF programs
                      when in BPF dataplane mode.  One of "Off", "Info", or "Debug".  The
                      logs are emitted to the BPF trace pipe, accessible with the command
                      `tc exec bpf debug`. [Default: Off].'
                    type: string
                  chainInsertMode:
                    description: 'ChainInsertMode controls whether Felix hooks the kernel’s
                      top-level iptables chains by inserting a rule at the top of the
                      chain or by appending a rule at the bottom. insert is the safe default
                      since it prevents Calico’s rules from being bypassed. If you switch
                      to append mode, be sure that the other rules in the chains signal
                      acceptance by falling through to the Calico rules, otherwise the
                      Calico policy will be bypassed. [Default: insert]'
                    type: string
                  dataplaneDriver:
                    type: string
                  debugDisableLogDropping:
                    type: boolean
                  debugMemoryProfilePath:
                    type: string
                  debugSimulateCalcGraphHangAfter:
                    type: string
                  debugSimulateDataplaneHangAfter:
                    type: string
                  defaultEndpointToHostAction:
                    description: 'DefaultEndpointToHostAction controls what happens to
                      traffic that goes from a workload endpoint to the host itself (after
                      the traffic hits the endpoint egress policy). By default Calico
                      blocks traffic from workload endpoints to the host itself with an
                      iptables “DROP” action. If you want to allow some or all traffic
                      from endpoint to host, set this parameter to RETURN or ACCEPT. Use
                      RETURN if you have your own rules in the iptables “INPUT” chain;
                      Calico will insert its rules at the top of that chain, then “RETURN”
                      packets to the “INPUT” chain once it has completed processing workload
                      endpoint egress policy. Use ACCEPT to unconditionally accept packets
                      from workloads after processing workload endpoint egress policy.
                      [Default: Drop]'
                    type: string
                  deviceRouteProtocol:
                    description: This defines the route protocol added to programmed device
                      routes, by default this will be RTPROT_BOOT when left blank.
                    type: integer
                  deviceRouteSourceAddress:
                    description: This is the source address to use on programmed device
                      routes. By default the source address is left blank, leaving the
                      kernel to choose the source address used.
                    type: string
                  disableConntrackInvalidCheck:
                    type: boolean
                  endpointReportingDelay:
                    type: string
                  endpointReportingEnabled:
                    type: boolean
                  externalNodesList:
                    description: ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes
                      which may source tunnel traffic and have the tunneled traffic be
                      accepted at calico nodes.
                    items:
                      type: string
                    type: array
                  failsafeInboundHostPorts:
                    description: 'FailsafeInboundHostPorts is a comma-delimited list of
                      UDP/TCP ports that Felix will allow incoming traffic to host endpoints
                      on irrespective of the security policy. This is useful to avoid
                      accidentally cutting off a host with incorrect configuration. Each
                      port should be specified as tcp:<port-number> or udp:<port-number>.
                      For back-compatibility, if the protocol is not specified, it defaults
                      to “tcp”. To disable all inbound host ports, use the value none.
                      The default value allows ssh access and DHCP. [Default: tcp:22,
                      udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]'
                    items:
                      description: ProtoPort is combination of protocol and port, both
                        must be specified.
                      properties:
                        port:
                          type: integer
                        protocol:
                          type: string
                      required:
                      - port
                      - protocol
                      type: object
                    type: array
                  failsafeOutboundHostPorts:
                    description: 'FailsafeOutboundHostPorts is a comma-delimited list
                      of UDP/TCP ports that Felix will allow outgoing traffic from host
                      endpoints to irrespective of the security policy. This is useful
                      to avoid accidentally cutting off a host with incorrect configuration.
                      Each port should be specified as tcp:<port-number> or udp:<port-number>.
                      For back-compatibility, if the protocol is not specified, it defaults
                      to “tcp”. To disable all outbound host ports, use the value none.
                      The default value opens etcd’s standard ports to ensure that Felix
                      does not get cut off from etcd as well as allowing DHCP and DNS.
                      [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667,
                      udp:53, udp:67]'
                    items:
                      description: ProtoPort is combination of protocol and port, both
                        must be specified.
                      properties:
                        port:
                          type: integer
                        protocol:
                          type: string
                      required:
                      - port
                      - protocol
                      type: object
                    type: array
                  featureDetectOverride:
                    description: FeatureDetectOverride is used to override the feature
                      detection. Values are specified in a comma separated list with no
                      spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=".
                      "true" or "false" will force the feature, empty or omitted values
                      are auto-detected.
                    type: string
                  genericXDPEnabled:
                    description: 'GenericXDPEnabled enables Generic XDP so network cards
                      that don''t support XDP offload or driver modes can use XDP. This
                      is not recommended since it doesn''t provide better performance
                      than iptables. [Default: false]'
                    type: boolean
                  healthEnabled:
                    type: boolean
                  healthHost:
                    type: string
                  healthPort:
                    type: integer
                  interfaceExclude:
                    description: 'InterfaceExclude is a comma-separated list of interfaces
                      that Felix should exclude when monitoring for host endpoints. The
                      default value ensures that Felix ignores Kubernetes'' IPVS dummy
                      interface, which is used internally by kube-proxy. If you want to
                      exclude multiple interface names using a single value, the list
                      supports regular expressions. For regular expressions you must wrap
                      the value with ''/''. For example having values ''/^kube/,veth1''
                      will exclude all interfaces that begin with ''kube'' and also the
                      interface ''veth1''. [Default: kube-ipvs0]'
                    type: string
                  interfacePrefix:
                    description: 'InterfacePrefix is the interface name prefix that identifies
                      workload endpoints and so distinguishes them from host endpoint
                      interfaces. Note: in environments other than bare metal, the orchestrators
                      configure this appropriately. For example our Kubernetes and Docker
                      integrations set the ‘cali’ value, and our OpenStack integration
                      sets the ‘tap’ value. [Default: cali]'
                    type: string
                  interfaceRefreshInterval:
                    description: InterfaceRefreshInterval is the period at which Felix
                      rescans local interfaces to verify their state. The rescan can be
                      disabled by setting the interval to 0.
                    type: string
                  ipipEnabled:
                    type: boolean
                  ipipMTU:
                    description: 'IPIPMTU is the MTU to set on the tunnel device. See
                      Configuring MTU [Default: 1440]'
                    type: integer
                  ipsetsRefreshInterval:
                    description: 'IpsetsRefreshInterval is the period at which Felix re-checks
                      all iptables state to ensure that no other process has accidentally
                      broken Calico’s rules. Set to 0 to disable iptables refresh. [Default:
                      90s]'
                    type: string
                  iptablesBackend:
                    description: IptablesBackend specifies which backend of iptables will
                      be used. The default is legacy.
                    type: string
                  iptablesFilterAllowAction:
                    type: string
                  iptablesLockFilePath:
                    description: 'IptablesLockFilePath is the location of the iptables
                      lock file. You may need to change this if the lock file is not in
                      its standard location (for example if you have mapped it into Felix’s
                      container at a different path). [Default: /run/xtables.lock]'
                    type: string
                  iptablesLockProbeInterval:
                    description: 'IptablesLockProbeInterval is the time that Felix will
                      wait between attempts to acquire the iptables lock if it is not
                      available. Lower values make Felix more responsive when the lock
                      is contended, but use more CPU. [Default: 50ms]'
                    type: string
                  iptablesLockTimeout:
                    description: 'IptablesLockTimeout is the time that Felix will wait
                      for the iptables lock, or 0, to disable. To use this feature, Felix
                      must share the iptables lock file with all other processes that
                      also take the lock. When running Felix inside a container, this
                      requires the /run directory of the host to be mounted into the calico/node
                      or calico/felix container. [Default: 0s disabled]'
                    type: string
                  iptablesMangleAllowAction:
                    type: string
                  iptablesMarkMask:
                    description: 'IptablesMarkMask is the mask that Felix selects its
                      IPTables Mark bits from. Should be a 32 bit hexadecimal number with
                      at least 8 bits set, none of which clash with any other mark bits
                      in use on the system. [Default: 0xff000000]'
                    format: int32
                    type: integer
                  iptablesNATOutgoingInterfaceFilter:
                    type: string
                  iptablesPostWriteCheckInterval:
                    description: 'IptablesPostWriteCheckInterval is the period after Felix
                      has done a write to the dataplane that it schedules an extra read
                      back in order to check the write was not clobbered by another process.
                      This should only occur if another application on the system doesn’t
                      respect the iptables lock. [Default: 1s]'
                    type: string
                  iptablesRefreshInterval:
                    description: 'IptablesRefreshInterval is the period at which Felix
                      re-checks the IP sets in the dataplane to ensure that no other process
                      has accidentally broken Calico’s rules. Set to 0 to disable IP sets
                      refresh. Note: the default for this value is lower than the other
                      refresh intervals as a workaround for a Linux kernel bug that was
                      fixed in kernel version 4.11. If you are using v4.11 or greater
                      you may want to set this to, a higher value to reduce Felix CPU
                      usage. [Default: 10s]'
                    type: string
                  ipv6Support:
                    type: boolean
                  kubeNodePortRanges:
                    description: 'KubeNodePortRanges holds list of port ranges used for
                      service node ports. Only used if felix detects kube-proxy running
                      in ipvs mode. Felix uses these ranges to separate host and workload
                      traffic. [Default: 30000:32767].'
                    items:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    type: array
                  logFilePath:
                    description: 'LogFilePath is the full path to the Felix log. Set to
                      none to disable file logging. [Default: /var/log/calico/felix.log]'
                    type: string
                  logPrefix:
                    description: 'LogPrefix is the log prefix that Felix uses when rendering
                      LOG rules. [Default: calico-packet]'
                    type: string
                  logSeverityFile:
                    description: 'LogSeverityFile is the log severity above which logs
                      are sent to the log file. [Default: Info]'
                    type: string
                  logSeverityScreen:
                    description: 'LogSeverityScreen is the log severity above which logs
                      are sent to the stdout. [Default: Info]'
                    type: string
                  logSeveritySys:
                    description: 'LogSeveritySys is the log severity above which logs
                      are sent to the syslog. Set to None for no logging to syslog. [Default:
                      Info]'
                    type: string
                  maxIpsetSize:
                    type: integer
                  metadataAddr:
                    description: 'MetadataAddr is the IP address or domain name of the
                      server that can answer VM queries for cloud-init metadata. In OpenStack,
                      this corresponds to the machine running nova-api (or in Ubuntu,
                      nova-api-metadata). A value of none (case insensitive) means that
                      Felix should not set up any NAT rule for the metadata path. [Default:
                      127.0.0.1]'
                    type: string
                  metadataPort:
                    description: 'MetadataPort is the port of the metadata server. This,
                      combined with global.MetadataAddr (if not ‘None’), is used to set
                      up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort.
                      In most cases this should not need to be changed [Default: 8775].'
                    type: integer
                  natOutgoingAddress:
                    description: NATOutgoingAddress specifies an address to use when performing
                      source NAT for traffic in a natOutgoing pool that is leaving the
                      network. By default the address used is an address on the interface
                      the traffic is leaving on (ie it uses the iptables MASQUERADE target)
                    type: string
                  natPortRange:
                    anyOf:
                    - type: integer
                    - type: string
                    description: NATPortRange specifies the range of ports that is used
                      for port mapping when doing outgoing NAT. When unset the default
                      behavior of the network stack is used.
                    pattern: ^.*
                    x-kubernetes-int-or-string: true
                  netlinkTimeout:
                    type: string
                  openstackRegion:
                    description: 'OpenstackRegion is the name of the region that a particular
                      Felix belongs to. In a multi-region Calico/OpenStack deployment,
                      this must be configured somehow for each Felix (here in the datamodel,
                      or in felix.cfg or the environment on each compute node), and must
                      match the [calico] openstack_region value configured in neutron.conf
                      on each node. [Default: Empty]'
                    type: string
                  policySyncPathPrefix:
                    description: 'PolicySyncPathPrefix is used to by Felix to communicate
                      policy changes to external services, like Application layer policy.
                      [Default: Empty]'
                    type: string
                  prometheusGoMetricsEnabled:
                    description: 'PrometheusGoMetricsEnabled disables Go runtime metrics
                      collection, which the Prometheus client does by default, when set
                      to false. This reduces the number of metrics reported, reducing
                      Prometheus load. [Default: true]'
                    type: boolean
                  prometheusMetricsEnabled:
                    description: 'PrometheusMetricsEnabled enables the Prometheus metrics
                      server in Felix if set to true. [Default: false]'
                    type: boolean
                  prometheusMetricsHost:
                    description: 'PrometheusMetricsHost is the host that the Prometheus
                      metrics server should bind to. [Default: empty]'
                    type: string
                  prometheusMetricsPort:
                    description: 'PrometheusMetricsPort is the TCP port that the Prometheus
                      metrics server should bind to. [Default: 9091]'
                    type: integer
                  prometheusProcessMetricsEnabled:
                    description: 'PrometheusProcessMetricsEnabled disables process metrics
                      collection, which the Prometheus client does by default, when set
                      to false. This reduces the number of metrics reported, reducing
                      Prometheus load. [Default: true]'
                    type: boolean
                  removeExternalRoutes:
                    description: Whether or not to remove device routes that have not
                      been programmed by Felix. Disabling this will allow external applications
                      to also add device routes. This is enabled by default which means
                      we will remove externally added routes.
                    type: boolean
                  reportingInterval:
                    description: 'ReportingInterval is the interval at which Felix reports
                      its status into the datastore or 0 to disable. Must be non-zero
                      in OpenStack deployments. [Default: 30s]'
                    type: string
                  reportingTTL:
                    description: 'ReportingTTL is the time-to-live setting for process-wide
                      status reports. [Default: 90s]'
                    type: string
                  routeRefreshInterval:
                    description: 'RouterefreshInterval is the period at which Felix re-checks
                      the routes in the dataplane to ensure that no other process has
                      accidentally broken Calico’s rules. Set to 0 to disable route refresh.
                      [Default: 90s]'
                    type: string
                  routeSource:
                    description: 'RouteSource configures where Felix gets its routing
                      information. - WorkloadIPs: use workload endpoints to construct
                      routes. - CalicoIPAM: the default - use IPAM data to construct routes.'
                    type: string
                  routeTableRange:
                    description: Calico programs additional Linux route tables for various
                      purposes.  RouteTableRange specifies the indices of the route tables
                      that Calico should use.
                    properties:
                      max:
                        type: integer
                      min:
                        type: integer
                    required:
                    - max
                    - min
                    type: object
                  sidecarAccelerationEnabled:
                    description: 'SidecarAccelerationEnabled enables experimental sidecar
                      acceleration [Default: false]'
                    type: boolean
                  usageReportingEnabled:
                    description: 'UsageReportingEnabled reports anonymous Calico version
                      number and cluster size to projectcalico.org. Logs warnings returned
                      by the usage server. For example, if a significant security vulnerability
                      has been discovered in the version of Calico being used. [Default:
                      true]'
                    type: boolean
                  usageReportingInitialDelay:
                    description: 'UsageReportingInitialDelay controls the minimum delay
                      before Felix makes a report. [Default: 300s]'
                    type: string
                  usageReportingInterval:
                    description: 'UsageReportingInterval controls the interval at which
                      Felix makes reports. [Default: 86400s]'
                    type: string
                  useInternalDataplaneDriver:
                    type: boolean
                  vxlanEnabled:
                    type: boolean
                  vxlanMTU:
                    description: 'VXLANMTU is the MTU to set on the tunnel device. See
                      Configuring MTU [Default: 1440]'
                    type: integer
                  vxlanPort:
                    type: integer
                  vxlanVNI:
                    type: integer
                  wireguardEnabled:
                    description: 'WireguardEnabled controls whether Wireguard is enabled.
                      [Default: false]'
                    type: boolean
                  wireguardInterfaceName:
                    description: 'WireguardInterfaceName specifies the name to use for
                      the Wireguard interface. [Default: wg.calico]'
                    type: string
                  wireguardListeningPort:
                    description: 'WireguardListeningPort controls the listening port used
                      by Wireguard. [Default: 51820]'
                    type: integer
                  wireguardMTU:
                    description: 'WireguardMTU controls the MTU on the Wireguard interface.
                      See Configuring MTU [Default: 1420]'
                    type: integer
                  wireguardRoutingRulePriority:
                    description: 'WireguardRoutingRulePriority controls the priority value
                      to use for the Wireguard routing rule. [Default: 99]'
                    type: integer
                  xdpEnabled:
                    description: 'XDPEnabled enables XDP acceleration for suitable untracked
                      incoming deny rules. [Default: true]'
                    type: boolean
                  xdpRefreshInterval:
                    description: 'XDPRefreshInterval is the period at which Felix re-checks
                      all XDP state to ensure that no other process has accidentally broken
                      Calico''s BPF maps or attached programs. Set to 0 to disable XDP
                      refresh. [Default: 90s]'
                    type: string
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: globalnetworkpolicies.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: GlobalNetworkPolicy
        listKind: GlobalNetworkPolicyList
        plural: globalnetworkpolicies
        singular: globalnetworkpolicy
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                properties:
                  applyOnForward:
                    description: ApplyOnForward indicates to apply the rules in this policy
                      on forward traffic.
                    type: boolean
                  doNotTrack:
                    description: DoNotTrack indicates whether packets matched by the rules
                      in this policy should go through the data plane's connection tracking,
                      such as Linux conntrack.  If True, the rules in this policy are
                      applied before any data plane connection tracking, and packets allowed
                      by this policy are marked as not to be tracked.
                    type: boolean
                  egress:
                    description: The ordered set of egress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  ingress:
                    description: The ordered set of ingress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  namespaceSelector:
                    description: NamespaceSelector is an optional field for an expression
                      used to select a pod based on namespaces.
                    type: string
                  order:
                    description: Order is an optional field that specifies the order in
                      which the policy is applied. Policies with higher "order" are applied
                      after those with lower order.  If the order is omitted, it may be
                      considered to be "infinite" - i.e. the policy will be applied last.  Policies
                      with identical order will be applied in alphanumerical order based
                      on the Policy "Name".
                    type: number
                  preDNAT:
                    description: PreDNAT indicates to apply the rules in this policy before
                      any DNAT.
                    type: boolean
                  selector:
                    description: "The selector is an expression used to pick pick out
                      the endpoints that the policy should be applied to. \n Selector
                      expressions follow this syntax: \n \tlabel == \"string_literal\"
                      \ ->  comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
                      \  ->  not equal; also matches if label is not present \tlabel in
                      { \"a\", \"b\", \"c\", ... }  ->  true if the value of label X is
                      one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
                      ... }  ->  true if the value of label X is not one of \"a\", \"b\",
                      \"c\" \thas(label_name)  -> True if that label is present \t! expr
                      -> negation of expr \texpr && expr  -> Short-circuit and \texpr
                      || expr  -> Short-circuit or \t( expr ) -> parens for grouping \tall()
                      or the empty selector -> matches all endpoints. \n Label names are
                      allowed to contain alphanumerics, -, _ and /. String literals are
                      more permissive but they do not support escape characters. \n Examples
                      (with made-up labels): \n \ttype == \"webserver\" && deployment
                      == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
                      \"dev\" \t! has(label_name)"
                    type: string
                  serviceAccountSelector:
                    description: ServiceAccountSelector is an optional field for an expression
                      used to select a pod based on service accounts.
                    type: string
                  types:
                    description: "Types indicates whether this policy applies to ingress,
                      or to egress, or to both.  When not explicitly specified (and so
                      the value on creation is empty or nil), Calico defaults Types according
                      to what Ingress and Egress rules are present in the policy.  The
                      default is: \n - [ PolicyTypeIngress ], if there are no Egress rules
                      (including the case where there are   also no Ingress rules) \n
                      - [ PolicyTypeEgress ], if there are Egress rules but no Ingress
                      rules \n - [ PolicyTypeIngress, PolicyTypeEgress ], if there are
                      both Ingress and Egress rules. \n When the policy is read back again,
                      Types will always be one of these values, never empty or nil."
                    items:
                      description: PolicyType enumerates the possible values of the PolicySpec
                        Types field.
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: globalnetworksets.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: GlobalNetworkSet
        listKind: GlobalNetworkSetList
        plural: globalnetworksets
        singular: globalnetworkset
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: GlobalNetworkSet contains a set of arbitrary IP sub-networks/CIDRs
              that share labels to allow rules to refer to them via selectors.  The labels
              of GlobalNetworkSet are not namespaced.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: GlobalNetworkSetSpec contains the specification for a NetworkSet
                  resource.
                properties:
                  nets:
                    description: The list of IP networks that belong to this set.
                    items:
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: hostendpoints.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: HostEndpoint
        listKind: HostEndpointList
        plural: hostendpoints
        singular: hostendpoint
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: HostEndpointSpec contains the specification for a HostEndpoint
                  resource.
                properties:
                  expectedIPs:
                    description: "The expected IP addresses (IPv4 and IPv6) of the endpoint.
                      If \"InterfaceName\" is not present, Calico will look for an interface
                      matching any of the IPs in the list and apply policy to that. Note:
                      \tWhen using the selector match criteria in an ingress or egress
                      security Policy \tor Profile, Calico converts the selector into
                      a set of IP addresses. For host \tendpoints, the ExpectedIPs field
                      is used for that purpose. (If only the interface \tname is specified,
                      Calico does not learn the IPs of the interface for use in match
                      \tcriteria.)"
                    items:
                      type: string
                    type: array
                  interfaceName:
                    description: "Either \"*\", or the name of a specific Linux interface
                      to apply policy to; or empty.  \"*\" indicates that this HostEndpoint
                      governs all traffic to, from or through the default network namespace
                      of the host named by the \"Node\" field; entering and leaving that
                      namespace via any interface, including those from/to non-host-networked
                      local workloads. \n If InterfaceName is not \"*\", this HostEndpoint
                      only governs traffic that enters or leaves the host through the
                      specific interface named by InterfaceName, or - when InterfaceName
                      is empty - through the specific interface that has one of the IPs
                      in ExpectedIPs. Therefore, when InterfaceName is empty, at least
                      one expected IP must be specified.  Only external interfaces (such
                      as “eth0”) are supported here; it isn't possible for a HostEndpoint
                      to protect traffic through a specific local workload interface.
                      \n Note: Only some kinds of policy are implemented for \"*\" HostEndpoints;
                      initially just pre-DNAT policy.  Please check Calico documentation
                      for the latest position."
                    type: string
                  node:
                    description: The node name identifying the Calico node instance.
                    type: string
                  ports:
                    description: Ports contains the endpoint's named ports, which may
                      be referenced in security policy rules.
                    items:
                      properties:
                        name:
                          type: string
                        port:
                          type: integer
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                      required:
                      - name
                      - port
                      - protocol
                      type: object
                    type: array
                  profiles:
                    description: A list of identifiers of security Profile objects that
                      apply to this endpoint. Each profile is applied in the order that
                      they appear in this list.  Profile rules are applied after the selector-based
                      security policy.
                    items:
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ipamblocks.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPAMBlock
        listKind: IPAMBlockList
        plural: ipamblocks
        singular: ipamblock
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPAMBlockSpec contains the specification for an IPAMBlock
                  resource.
                properties:
                  affinity:
                    type: string
                  allocations:
                    items:
                      type: integer
                      # TODO: This nullable is manually added in. We should update controller-gen
                      # to handle []*int properly itself.
                      nullable: true
                    type: array
                  attributes:
                    items:
                      properties:
                        handle_id:
                          type: string
                        secondary:
                          additionalProperties:
                            type: string
                          type: object
                      type: object
                    type: array
                  cidr:
                    type: string
                  deleted:
                    type: boolean
                  strictAffinity:
                    type: boolean
                  unallocated:
                    items:
                      type: integer
                    type: array
                required:
                - allocations
                - attributes
                - cidr
                - deleted
                - strictAffinity
                - unallocated
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ipamconfigs.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPAMConfig
        listKind: IPAMConfigList
        plural: ipamconfigs
        singular: ipamconfig
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPAMConfigSpec contains the specification for an IPAMConfig
                  resource.
                properties:
                  autoAllocateBlocks:
                    type: boolean
                  strictAffinity:
                    type: boolean
                required:
                - autoAllocateBlocks
                - strictAffinity
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ipamhandles.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPAMHandle
        listKind: IPAMHandleList
        plural: ipamhandles
        singular: ipamhandle
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPAMHandleSpec contains the specification for an IPAMHandle
                  resource.
                properties:
                  block:
                    additionalProperties:
                      type: integer
                    type: object
                  handleID:
                    type: string
                required:
                - block
                - handleID
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: ippools.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: IPPool
        listKind: IPPoolList
        plural: ippools
        singular: ippool
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: IPPoolSpec contains the specification for an IPPool resource.
                properties:
                  blockSize:
                    description: The block size to use for IP address assignments from
                      this pool. Defaults to 26 for IPv4 and 112 for IPv6.
                    type: integer
                  cidr:
                    description: The pool CIDR.
                    type: string
                  disabled:
                    description: When disabled is true, Calico IPAM will not assign addresses
                      from this pool.
                    type: boolean
                  ipip:
                    description: 'Deprecated: this field is only used for APIv1 backwards
                      compatibility. Setting this field is not allowed, this field is
                      for internal use only.'
                    properties:
                      enabled:
                        description: When enabled is true, ipip tunneling will be used
                          to deliver packets to destinations within this pool.
                        type: boolean
                      mode:
                        description: The IPIP mode.  This can be one of "always" or "cross-subnet".  A
                          mode of "always" will also use IPIP tunneling for routing to
                          destination IP addresses within this pool.  A mode of "cross-subnet"
                          will only use IPIP tunneling when the destination node is on
                          a different subnet to the originating node.  The default value
                          (if not specified) is "always".
                        type: string
                    type: object
                  ipipMode:
                    description: Contains configuration for IPIP tunneling for this pool.
                      If not specified, then this is defaulted to "Never" (i.e. IPIP tunneling
                      is disabled).
                    type: string
                  nat-outgoing:
                    description: 'Deprecated: this field is only used for APIv1 backwards
                      compatibility. Setting this field is not allowed, this field is
                      for internal use only.'
                    type: boolean
                  natOutgoing:
                    description: When nat-outgoing is true, packets sent from Calico networked
                      containers in this pool to destinations outside of this pool will
                      be masqueraded.
                    type: boolean
                  nodeSelector:
                    description: Allows IPPool to allocate for a specific node by label
                      selector.
                    type: string
                  vxlanMode:
                    description: Contains configuration for VXLAN tunneling for this pool.
                      If not specified, then this is defaulted to "Never" (i.e. VXLAN
                      tunneling is disabled).
                    type: string
                required:
                - cidr
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: kubecontrollersconfigurations.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: KubeControllersConfiguration
        listKind: KubeControllersConfigurationList
        plural: kubecontrollersconfigurations
        singular: kubecontrollersconfiguration
      scope: Cluster
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: KubeControllersConfigurationSpec contains the values of the
                  Kubernetes controllers configuration.
                properties:
                  controllers:
                    description: Controllers enables and configures individual Kubernetes
                      controllers
                    properties:
                      namespace:
                        description: Namespace enables and configures the namespace controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                      node:
                        description: Node enables and configures the node controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          hostEndpoint:
                            description: HostEndpoint controls syncing nodes to host endpoints.
                              Disabled by default, set to nil to disable.
                            properties:
                              autoCreate:
                                description: 'AutoCreate enables automatic creation of
                                  host endpoints for every node. [Default: Disabled]'
                                type: string
                            type: object
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                          syncLabels:
                            description: 'SyncLabels controls whether to copy Kubernetes
                              node labels to Calico nodes. [Default: Enabled]'
                            type: string
                        type: object
                      policy:
                        description: Policy enables and configures the policy controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                      serviceAccount:
                        description: ServiceAccount enables and configures the service
                          account controller. Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                      workloadEndpoint:
                        description: WorkloadEndpoint enables and configures the workload
                          endpoint controller. Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform reconciliation
                              with the Calico datastore. [Default: 5m]'
                            type: string
                        type: object
                    type: object
                  etcdV3CompactionPeriod:
                    description: 'EtcdV3CompactionPeriod is the period between etcdv3
                      compaction requests. Set to 0 to disable. [Default: 10m]'
                    type: string
                  healthChecks:
                    description: 'HealthChecks enables or disables support for health
                      checks [Default: Enabled]'
                    type: string
                  logSeverityScreen:
                    description: 'LogSeverityScreen is the log severity above which logs
                      are sent to the stdout. [Default: Info]'
                    type: string
                required:
                - controllers
                type: object
              status:
                description: KubeControllersConfigurationStatus represents the status
                  of the configuration. It's useful for admins to be able to see the actual
                  config that was applied, which can be modified by environment variables
                  on the kube-controllers process.
                properties:
                  environmentVars:
                    additionalProperties:
                      type: string
                    description: EnvironmentVars contains the environment variables on
                      the kube-controllers that influenced the RunningConfig.
                    type: object
                  runningConfig:
                    description: RunningConfig contains the effective config that is running
                      in the kube-controllers pod, after merging the API resource with
                      any environment variables.
                    properties:
                      controllers:
                        description: Controllers enables and configures individual Kubernetes
                          controllers
                        properties:
                          namespace:
                            description: Namespace enables and configures the namespace
                              controller. Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                          node:
                            description: Node enables and configures the node controller.
                              Enabled by default, set to nil to disable.
                            properties:
                              hostEndpoint:
                                description: HostEndpoint controls syncing nodes to host
                                  endpoints. Disabled by default, set to nil to disable.
                                properties:
                                  autoCreate:
                                    description: 'AutoCreate enables automatic creation
                                      of host endpoints for every node. [Default: Disabled]'
                                    type: string
                                type: object
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                              syncLabels:
                                description: 'SyncLabels controls whether to copy Kubernetes
                                  node labels to Calico nodes. [Default: Enabled]'
                                type: string
                            type: object
                          policy:
                            description: Policy enables and configures the policy controller.
                              Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                          serviceAccount:
                            description: ServiceAccount enables and configures the service
                              account controller. Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                          workloadEndpoint:
                            description: WorkloadEndpoint enables and configures the workload
                              endpoint controller. Enabled by default, set to nil to disable.
                            properties:
                              reconcilerPeriod:
                                description: 'ReconcilerPeriod is the period to perform
                                  reconciliation with the Calico datastore. [Default:
                                  5m]'
                                type: string
                            type: object
                        type: object
                      etcdV3CompactionPeriod:
                        description: 'EtcdV3CompactionPeriod is the period between etcdv3
                          compaction requests. Set to 0 to disable. [Default: 10m]'
                        type: string
                      healthChecks:
                        description: 'HealthChecks enables or disables support for health
                          checks [Default: Enabled]'
                        type: string
                      logSeverityScreen:
                        description: 'LogSeverityScreen is the log severity above which
                          logs are sent to the stdout. [Default: Info]'
                        type: string
                    required:
                    - controllers
                    type: object
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: networkpolicies.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: NetworkPolicy
        listKind: NetworkPolicyList
        plural: networkpolicies
        singular: networkpolicy
      scope: Namespaced
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                properties:
                  egress:
                    description: The ordered set of egress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  ingress:
                    description: The ordered set of ingress rules.  Each rule contains
                      a set of packet match criteria and a corresponding action to apply.
                    items:
                      description: "A Rule encapsulates a set of match criteria and an
                        action.  Both selector-based security Policy and security Profiles
                        reference rules - separated out as a list of rules for both ingress
                        and egress packet matching. \n Each positive match criteria has
                        a negated version, prefixed with ”Not”. All the match criteria
                        within a rule must be satisfied for a packet to match. A single
                        rule can contain the positive and negative version of a match
                        and both must be satisfied for the rule to match."
                      properties:
                        action:
                          type: string
                        destination:
                          description: Destination contains the match criteria that apply
                            to destination entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                        http:
                          description: HTTP contains match criteria that apply to HTTP
                            requests.
                          properties:
                            methods:
                              description: Methods is an optional field that restricts
                                the rule to apply only to HTTP requests that use one of
                                the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                                methods are OR'd together.
                              items:
                                type: string
                              type: array
                            paths:
                              description: 'Paths is an optional field that restricts
                                the rule to apply to HTTP requests that use one of the
                                listed HTTP Paths. Multiple paths are OR''d together.
                                e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                                ONLY specify either a `exact` or a `prefix` match. The
                                validator will check for it.'
                              items:
                                description: 'HTTPPath specifies an HTTP path to match.
                                  It may be either of the form: exact: <path>: which matches
                                  the path exactly or prefix: <path-prefix>: which matches
                                  the path prefix'
                                properties:
                                  exact:
                                    type: string
                                  prefix:
                                    type: string
                                type: object
                              type: array
                          type: object
                        icmp:
                          description: ICMP is an optional field that restricts the rule
                            to apply to a specific type and code of ICMP traffic.  This
                            should only be specified if the Protocol field is set to "ICMP"
                            or "ICMPv6".
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        ipVersion:
                          description: IPVersion is an optional field that restricts the
                            rule to only match a specific IP version.
                          type: integer
                        metadata:
                          description: Metadata contains additional information for this
                            rule
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              description: Annotations is a set of key value pairs that
                                give extra information about the rule
                              type: object
                          type: object
                        notICMP:
                          description: NotICMP is the negated version of the ICMP field.
                          properties:
                            code:
                              description: Match on a specific ICMP code.  If specified,
                                the Type value must also be specified. This is a technical
                                limitation imposed by the kernel’s iptables firewall,
                                which Calico uses to enforce the rule.
                              type: integer
                            type:
                              description: Match on a specific ICMP type.  For example
                                a value of 8 refers to ICMP Echo Request (i.e. pings).
                              type: integer
                          type: object
                        notProtocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: NotProtocol is the negated version of the Protocol
                            field.
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        protocol:
                          anyOf:
                          - type: integer
                          - type: string
                          description: "Protocol is an optional field that restricts the
                            rule to only apply to traffic of a specific IP protocol. Required
                            if any of the EntityRules contain Ports (because ports only
                            apply to certain protocols). \n Must be one of these string
                            values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                            \"UDPLite\" or an integer in the range 1-255."
                          pattern: ^.*
                          x-kubernetes-int-or-string: true
                        source:
                          description: Source contains the match criteria that apply to
                            source entity.
                          properties:
                            namespaceSelector:
                              description: "NamespaceSelector is an optional field that
                                contains a selector expression. Only traffic that originates
                                from (or terminates at) endpoints within the selected
                                namespaces will be matched. When both NamespaceSelector
                                and Selector are defined on the same rule, then only workload
                                endpoints that are matched by both selectors will be selected
                                by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                                implies that the Selector is limited to selecting only
                                workload endpoints in the same namespace as the NetworkPolicy.
                                \n For NetworkPolicy, `global()` NamespaceSelector implies
                                that the Selector is limited to selecting only GlobalNetworkSet
                                or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                                NamespaceSelector implies the Selector applies to workload
                                endpoints across all namespaces."
                              type: string
                            nets:
                              description: Nets is an optional field that restricts the
                                rule to only apply to traffic that originates from (or
                                terminates at) IP addresses in any of the given subnets.
                              items:
                                type: string
                              type: array
                            notNets:
                              description: NotNets is the negated version of the Nets
                                field.
                              items:
                                type: string
                              type: array
                            notPorts:
                              description: NotPorts is the negated version of the Ports
                                field. Since only some protocols have ports, if any ports
                                are specified it requires the Protocol match in the Rule
                                to be set to "TCP" or "UDP".
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            notSelector:
                              description: NotSelector is the negated version of the Selector
                                field.  See Selector field for subtleties with negated
                                selectors.
                              type: string
                            ports:
                              description: "Ports is an optional field that restricts
                                the rule to only apply to traffic that has a source (destination)
                                port that matches one of these ranges/values. This value
                                is a list of integers or strings that represent ranges
                                of ports. \n Since only some protocols have ports, if
                                any ports are specified it requires the Protocol match
                                in the Rule to be set to \"TCP\" or \"UDP\"."
                              items:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^.*
                                x-kubernetes-int-or-string: true
                              type: array
                            selector:
                              description: "Selector is an optional field that contains
                                a selector expression (see Policy for sample syntax).
                                \ Only traffic that originates from (terminates at) endpoints
                                matching the selector will be matched. \n Note that: in
                                addition to the negated version of the Selector (see NotSelector
                                below), the selector expression syntax itself supports
                                negation.  The two types of negation are subtly different.
                                One negates the set of matched endpoints, the other negates
                                the whole match: \n \tSelector = \"!has(my_label)\" matches
                                packets that are from other Calico-controlled \tendpoints
                                that do not have the label “my_label”. \n \tNotSelector
                                = \"has(my_label)\" matches packets that are not from
                                Calico-controlled \tendpoints that do have the label “my_label”.
                                \n The effect is that the latter will accept packets from
                                non-Calico sources whereas the former is limited to packets
                                from Calico-controlled endpoints."
                              type: string
                            serviceAccounts:
                              description: ServiceAccounts is an optional field that restricts
                                the rule to only apply to traffic that originates from
                                (or terminates at) a pod running as a matching service
                                account.
                              properties:
                                names:
                                  description: Names is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account whose name is in the list.
                                  items:
                                    type: string
                                  type: array
                                selector:
                                  description: Selector is an optional field that restricts
                                    the rule to only apply to traffic that originates
                                    from (or terminates at) a pod running as a service
                                    account that matches the given label selector. If
                                    both Names and Selector are specified then they are
                                    AND'ed.
                                  type: string
                              type: object
                          type: object
                      required:
                      - action
                      type: object
                    type: array
                  order:
                    description: Order is an optional field that specifies the order in
                      which the policy is applied. Policies with higher "order" are applied
                      after those with lower order.  If the order is omitted, it may be
                      considered to be "infinite" - i.e. the policy will be applied last.  Policies
                      with identical order will be applied in alphanumerical order based
                      on the Policy "Name".
                    type: number
                  selector:
                    description: "The selector is an expression used to pick pick out
                      the endpoints that the policy should be applied to. \n Selector
                      expressions follow this syntax: \n \tlabel == \"string_literal\"
                      \ ->  comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
                      \  ->  not equal; also matches if label is not present \tlabel in
                      { \"a\", \"b\", \"c\", ... }  ->  true if the value of label X is
                      one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
                      ... }  ->  true if the value of label X is not one of \"a\", \"b\",
                      \"c\" \thas(label_name)  -> True if that label is present \t! expr
                      -> negation of expr \texpr && expr  -> Short-circuit and \texpr
                      || expr  -> Short-circuit or \t( expr ) -> parens for grouping \tall()
                      or the empty selector -> matches all endpoints. \n Label names are
                      allowed to contain alphanumerics, -, _ and /. String literals are
                      more permissive but they do not support escape characters. \n Examples
                      (with made-up labels): \n \ttype == \"webserver\" && deployment
                      == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
                      \"dev\" \t! has(label_name)"
                    type: string
                  serviceAccountSelector:
                    description: ServiceAccountSelector is an optional field for an expression
                      used to select a pod based on service accounts.
                    type: string
                  types:
                    description: "Types indicates whether this policy applies to ingress,
                      or to egress, or to both.  When not explicitly specified (and so
                      the value on creation is empty or nil), Calico defaults Types according
                      to what Ingress and Egress are present in the policy.  The default
                      is: \n - [ PolicyTypeIngress ], if there are no Egress rules (including
                      the case where there are   also no Ingress rules) \n - [ PolicyTypeEgress
                      ], if there are Egress rules but no Ingress rules \n - [ PolicyTypeIngress,
                      PolicyTypeEgress ], if there are both Ingress and Egress rules.
                      \n When the policy is read back again, Types will always be one
                      of these values, never empty or nil."
                    items:
                      description: PolicyType enumerates the possible values of the PolicySpec
                        Types field.
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---

    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: networksets.crd.projectcalico.org
    spec:
      group: crd.projectcalico.org
      names:
        kind: NetworkSet
        listKind: NetworkSetList
        plural: networksets
        singular: networkset
      scope: Namespaced
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            description: NetworkSet is the Namespaced-equivalent of the GlobalNetworkSet.
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: NetworkSetSpec contains the specification for a NetworkSet
                  resource.
                properties:
                  nets:
                    description: The list of IP networks that belong to this set.
                    items:
                      type: string
                    type: array
                type: object
            type: object
        served: true
        storage: true
    status:
      acceptedNames:
        kind: ""
        plural: ""
      conditions: []
      storedVersions: []

    ---
    ---
    # Source: calico/templates/calico-kube-controllers-rbac.yaml

    # Include a clusterrole for the kube-controllers component,
    # and bind it to the calico-kube-controllers serviceaccount.
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: calico-kube-controllers
    rules:
      # Nodes are watched to monitor for deletions.
      - apiGroups: [""]
        resources:
          - nodes
        verbs:
          - watch
          - list
          - get
      # Pods are queried to check for existence.
      - apiGroups: [""]
        resources:
          - pods
        verbs:
          - get
      # IPAM resources are manipulated when nodes are deleted.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - ippools
        verbs:
          - list
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - blockaffinities
          - ipamblocks
          - ipamhandles
        verbs:
          - get
          - list
          - create
          - update
          - delete
      # kube-controllers manages hostendpoints.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - hostendpoints
        verbs:
          - get
          - list
          - create
          - update
          - delete
      # Needs access to update clusterinformations.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - clusterinformations
        verbs:
          - get
          - create
          - update
      # KubeControllersConfiguration is where it gets its config
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - kubecontrollersconfigurations
        verbs:
          # read its own config
          - get
          # create a default if none exists
          - create
          # update status
          - update
          # watch for changes
          - watch
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: calico-kube-controllers
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: calico-kube-controllers
    subjects:
    - kind: ServiceAccount
      name: calico-kube-controllers
      namespace: kube-system
    ---

    ---
    # Source: calico/templates/calico-node-rbac.yaml
    # Include a clusterrole for the calico-node DaemonSet,
    # and bind it to the calico-node serviceaccount.
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: calico-node
    rules:
      # The CNI plugin needs to get pods, nodes, and namespaces.
      - apiGroups: [""]
        resources:
          - pods
          - nodes
          - namespaces
        verbs:
          - get
      - apiGroups: [""]
        resources:
          - endpoints
          - services
        verbs:
          # Used to discover service IPs for advertisement.
          - watch
          - list
          # Used to discover Typhas.
          - get
      # Pod CIDR auto-detection on kubeadm needs access to config maps.
      - apiGroups: [""]
        resources:
          - configmaps
        verbs:
          - get
      - apiGroups: [""]
        resources:
          - nodes/status
        verbs:
          # Needed for clearing NodeNetworkUnavailable flag.
          - patch
          # Calico stores some configuration information in node annotations.
          - update
      # Watch for changes to Kubernetes NetworkPolicies.
      - apiGroups: ["networking.k8s.io"]
        resources:
          - networkpolicies
        verbs:
          - watch
          - list
      # Used by Calico for policy information.
      - apiGroups: [""]
        resources:
          - pods
          - namespaces
          - serviceaccounts
        verbs:
          - list
          - watch
      # The CNI plugin patches pods/status.
      - apiGroups: [""]
        resources:
          - pods/status
        verbs:
          - patch
      # Calico monitors various CRDs for config.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - globalfelixconfigs
          - felixconfigurations
          - bgppeers
          - globalbgpconfigs
          - bgpconfigurations
          - ippools
          - ipamblocks
          - globalnetworkpolicies
          - globalnetworksets
          - networkpolicies
          - networksets
          - clusterinformations
          - hostendpoints
          - blockaffinities
        verbs:
          - get
          - list
          - watch
      # Calico must create and update some CRDs on startup.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - ippools
          - felixconfigurations
          - clusterinformations
        verbs:
          - create
          - update
      # Calico stores some configuration information on the node.
      - apiGroups: [""]
        resources:
          - nodes
        verbs:
          - get
          - list
          - watch
      # These permissions are only required for upgrade from v2.6, and can
      # be removed after upgrade or on fresh installations.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - bgpconfigurations
          - bgppeers
        verbs:
          - create
          - update
      # These permissions are required for Calico CNI to perform IPAM allocations.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - blockaffinities
          - ipamblocks
          - ipamhandles
        verbs:
          - get
          - list
          - create
          - update
          - delete
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - ipamconfigs
        verbs:
          - get
      # Block affinities must also be watchable by confd for route aggregation.
      - apiGroups: ["crd.projectcalico.org"]
        resources:
          - blockaffinities
        verbs:
          - watch
      # The Calico IPAM migration needs to get daemonsets. These permissions can be
      # removed if not upgrading from an installation using host-local IPAM.
      - apiGroups: ["apps"]
        resources:
          - daemonsets
        verbs:
          - get

    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: calico-node
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: calico-node
    subjects:
    - kind: ServiceAccount
      name: calico-node
      namespace: kube-system

    ---
    # Source: calico/templates/calico-node.yaml
    # This manifest installs the calico-node container, as well
    # as the CNI plugins and network config on
    # each master and worker node in a Kubernetes cluster.
    kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: calico-node
      namespace: kube-system
      labels:
        k8s-app: calico-node
    spec:
      selector:
        matchLabels:
          k8s-app: calico-node
      updateStrategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
      template:
        metadata:
          labels:
            k8s-app: calico-node
        spec:
          nodeSelector:
            kubernetes.io/os: linux
          hostNetwork: true
          tolerations:
            # Make sure calico-node gets scheduled on all nodes.
            - effect: NoSchedule
              operator: Exists
            # Mark the pod as a critical add-on for rescheduling.
            - key: CriticalAddonsOnly
              operator: Exists
            - effect: NoExecute
              operator: Exists
          serviceAccountName: calico-node
          # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
          # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
          terminationGracePeriodSeconds: 0
          priorityClassName: system-node-critical
          initContainers:
            # This container performs upgrade from host-local IPAM to calico-ipam.
            # It can be deleted if this is a fresh installation, or if you have already
            # upgraded to use calico-ipam.
            - name: upgrade-ipam
              image: calico/cni:v3.16.1
              command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
              envFrom:
              - configMapRef:
                  # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                  name: kubernetes-services-endpoint
                  optional: true
              env:
                - name: KUBERNETES_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: CALICO_NETWORKING_BACKEND
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: calico_backend
              volumeMounts:
                - mountPath: /var/lib/cni/networks
                  name: host-local-net-dir
                - mountPath: /host/opt/cni/bin
                  name: cni-bin-dir
              securityContext:
                privileged: true
            # This container installs the CNI binaries
            # and CNI network config file on each node.
            - name: install-cni
              image: calico/cni:v3.16.1
              command: ["/opt/cni/bin/install"]
              envFrom:
              - configMapRef:
                  # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                  name: kubernetes-services-endpoint
                  optional: true
              env:
                # Name of the CNI config file to create.
                - name: CNI_CONF_NAME
                  value: "10-calico.conflist"
                # The CNI network config to install on each node.
                - name: CNI_NETWORK_CONFIG
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: cni_network_config
                # Set the hostname based on the k8s node name.
                - name: KUBERNETES_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                # CNI MTU Config variable
                - name: CNI_MTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # Prevents the container from sleeping forever.
                - name: SLEEP
                  value: "false"
              volumeMounts:
                - mountPath: /host/opt/cni/bin
                  name: cni-bin-dir
                - mountPath: /host/etc/cni/net.d
                  name: cni-net-dir
              securityContext:
                privileged: true
            # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
            # to communicate with Felix over the Policy Sync API.
            - name: flexvol-driver
              image: calico/pod2daemon-flexvol:v3.16.1
              volumeMounts:
              - name: flexvol-driver-host
                mountPath: /host/driver
              securityContext:
                privileged: true
          containers:
            # Runs calico-node container on each Kubernetes node. This
            # container programs network policy and routes on each
            # host.
            - name: calico-node
              image: calico/node:v3.16.1
              envFrom:
              - configMapRef:
                  # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                  name: kubernetes-services-endpoint
                  optional: true
              env:
                # Use Kubernetes API as the backing datastore.
                - name: DATASTORE_TYPE
                  value: "kubernetes"
                # Wait for the datastore.
                - name: WAIT_FOR_DATASTORE
                  value: "true"
                # Set based on the k8s node name.
                - name: NODENAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                # Choose the backend to use.
                - name: CALICO_NETWORKING_BACKEND
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: calico_backend
                # Cluster type to identify the deployment type
                - name: CLUSTER_TYPE
                  value: "k8s,bgp"
                # Auto-detect the BGP IP address.
                - name: IP
                  value: "autodetect"
                # Enable IPIP
                - name: CALICO_IPV4POOL_IPIP
                  value: "Always"
                # Enable or Disable VXLAN on the default IP pool.
                - name: CALICO_IPV4POOL_VXLAN
                  value: "Never"
                # Set MTU for tunnel device used if ipip is enabled
                - name: FELIX_IPINIPMTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # Set MTU for the VXLAN tunnel device.
                - name: FELIX_VXLANMTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # Set MTU for the Wireguard tunnel device.
                - name: FELIX_WIREGUARDMTU
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: veth_mtu
                # The default IPv4 pool to create on startup if none exists. Pod IPs will be
                # chosen from this range. Changing this value after installation will have
                # no effect. This should fall within `--cluster-cidr`.
                # - name: CALICO_IPV4POOL_CIDR
                #   value: "192.168.0.0/16"
                # Disable file logging so `kubectl logs` works.
                - name: CALICO_DISABLE_FILE_LOGGING
                  value: "true"
                # Set Felix endpoint to host default action to ACCEPT.
                - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
                  value: "ACCEPT"
                # Disable IPv6 on Kubernetes.
                - name: FELIX_IPV6SUPPORT
                  value: "false"
                # Set Felix logging to "info"
                - name: FELIX_LOGSEVERITYSCREEN
                  value: "info"
                - name: FELIX_HEALTHENABLED
                  value: "true"
              securityContext:
                privileged: true
              resources:
                requests:
                  cpu: 250m
              livenessProbe:
                exec:
                  command:
                  - /bin/calico-node
                  - -felix-live
                  - -bird-live
                periodSeconds: 10
                initialDelaySeconds: 10
                failureThreshold: 6
              readinessProbe:
                exec:
                  command:
                  - /bin/calico-node
                  - -felix-ready
                  - -bird-ready
                periodSeconds: 10
              volumeMounts:
                - mountPath: /lib/modules
                  name: lib-modules
                  readOnly: true
                - mountPath: /run/xtables.lock
                  name: xtables-lock
                  readOnly: false
                - mountPath: /var/run/calico
                  name: var-run-calico
                  readOnly: false
                - mountPath: /var/lib/calico
                  name: var-lib-calico
                  readOnly: false
                - name: policysync
                  mountPath: /var/run/nodeagent
                # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
                # parent directory.
                - name: sysfs
                  mountPath: /sys/fs/
                  # Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
                  # If the host is known to mount that filesystem already then Bidirectional can be omitted.
                  mountPropagation: Bidirectional
          volumes:
            # Used by calico-node.
            - name: lib-modules
              hostPath:
                path: /lib/modules
            - name: var-run-calico
              hostPath:
                path: /var/run/calico
            - name: var-lib-calico
              hostPath:
                path: /var/lib/calico
            - name: xtables-lock
              hostPath:
                path: /run/xtables.lock
                type: FileOrCreate
            - name: sysfs
              hostPath:
                path: /sys/fs/
                type: DirectoryOrCreate
            # Used to install CNI.
            - name: cni-bin-dir
              hostPath:
                path: /opt/cni/bin
            - name: cni-net-dir
              hostPath:
                path: /etc/cni/net.d
            # Mount in the directory for host-local IPAM allocations. This is
            # used when upgrading from host-local to calico-ipam, and can be removed
            # if not using the upgrade-ipam init container.
            - name: host-local-net-dir
              hostPath:
                path: /var/lib/cni/networks
            # Used to create per-pod Unix Domain Sockets
            - name: policysync
              hostPath:
                type: DirectoryOrCreate
                path: /var/run/nodeagent
            # Used to install Flex Volume Driver
            - name: flexvol-driver-host
              hostPath:
                type: DirectoryOrCreate
                path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    ---

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: calico-node
      namespace: kube-system

    ---
    # Source: calico/templates/calico-kube-controllers.yaml
    # See https://github.com/projectcalico/kube-controllers
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      # The controllers can only have a single active instance.
      replicas: 1
      selector:
        matchLabels:
          k8s-app: calico-kube-controllers
      strategy:
        type: Recreate
      template:
        metadata:
          name: calico-kube-controllers
          namespace: kube-system
          labels:
            k8s-app: calico-kube-controllers
        spec:
          nodeSelector:
            kubernetes.io/os: linux
          tolerations:
            # Mark the pod as a critical add-on for rescheduling.
            - key: CriticalAddonsOnly
              operator: Exists
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
          serviceAccountName: calico-kube-controllers
          priorityClassName: system-cluster-critical
          containers:
            - name: calico-kube-controllers
              image: calico/kube-controllers:v3.16.1
              env:
                # Choose which controllers to run.
                - name: ENABLED_CONTROLLERS
                  value: node
                - name: DATASTORE_TYPE
                  value: kubernetes
              readinessProbe:
                exec:
                  command:
                  - /usr/bin/check-status
                  - -r

    ---

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: calico-kube-controllers
      namespace: kube-system

    ---
    # Source: calico/templates/calico-etcd-secrets.yaml

    ---
    # Source: calico/templates/calico-typha.yaml

    ---
    # Source: calico/templates/configure-canal.yaml


kind: ConfigMap
metadata:
  creationTimestamp: null
  name: calico-cni
---
apiVersion: addons.cluster.x-k8s.io/v1alpha3
kind: ClusterResourceSet
metadata:
  name: ${ CLUSTER_NAME }-crs-0
  namespace: '${ NAMESPACE }'
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: '${ CLUSTER_NAME }'
  resources:
  - kind: ConfigMap
    name: calico-cni

Cluster API - Using a Private Container Registry

Mon, 17 Aug 2020 00:00:00 GMT

The pre-configured Cluster API Provider vSphere (CAPV) images use containerd as their Container Runtime, and to get this to pull images from a non-secure, or self-signed certificate container registry you need to make a few changes to the /etc/containerd/config.toml file.

These are the steps I followed to get a private Harbor repository working over https with a self-signed certificate, that my Kubernetes clusters did not trust.

If you have already deployed your workload cluster that needs to use this Private Registry, the easiest way to get this working is to manually adjust the file on each of the Kubernetes nodes, and restart the relevant service.

Existing Workload Cluster

SSH (default username is capv if created with CAPV) to each node in the workload cluster, and run the following (this will replace any existing content of the file).

sudo su
cat > /etc/containerd/config.toml << EOF
version = 2
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "k8s.gcr.io/pause:3.2"
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."REGISTRY-FQDN"]
        endpoint = ["https://REGISTRY-FQDN"]
    [plugins."io.containerd.grpc.v1.cri".registry.configs]
      [plugins."io.containerd.grpc.v1.cri".registry.configs."REGISTRY-FQDN".tls]
        insecure_skip_verify = true
EOF
sudo systemctl restart containerd
sudo systemctl status containerd

A nicer way is to add the configuration above in to your cluster.yaml and have it populate the /etc/containerd/config.toml file whenever you create another cluster with CAPV.

New Workload Cluster

Update your cluster.yaml to include a files: section for the KubeadmConfigTemplate and KubeadmControlPlane resource types. The files: section needs to be added in the below locations:

KubeadmConfigTemplate (worker nodes): Add files: section to spec.template.spec
KubeadmControlPlane (control plane nodes): Add files: section to spec.kubeadmConfigSpec

Replace REGISTRY-FQDN with the FQDN or IP address of your registry. Ensure that your endpoint has the correct protocol (http or https).

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: cluster01
  namespace: default
spec:
  kubeadmConfigSpec:
    files:
      - path: /etc/containerd/config.toml
        content: |
          version = 2
          [plugins]
            [plugins."io.containerd.grpc.v1.cri"]
              sandbox_image = "k8s.gcr.io/pause:3.2"
              [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
                [plugins."io.containerd.grpc.v1.cri".registry.mirrors."REGISTRY-FQDN"]
                  endpoint = ["https://REGISTRY-FQDN"]
              [plugins."io.containerd.grpc.v1.cri".registry.configs]
                [plugins."io.containerd.grpc.v1.cri".registry.configs."REGISTRY-FQDN".tls]
                  insecure_skip_verify = true

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfigTemplate
metadata:
  name: cluster01-md-0
  namespace: default
spec:
  template:
    spec:
      files:
        - path: /etc/containerd/config.toml
          content: |
            version = 2
            [plugins]
              [plugins."io.containerd.grpc.v1.cri"]
                sandbox_image = "k8s.gcr.io/pause:3.2"
                [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
                  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."REGISTRY-FQDN"]
                    endpoint = ["https://REGISTRY-FQDN"]
                [plugins."io.containerd.grpc.v1.cri".registry.configs]
                  [plugins."io.containerd.grpc.v1.cri".registry.configs."REGISTRY-FQDN".tls]
                    insecure_skip_verify = true

Deploy the new cluster from your Cluster API management cluster: kubectl apply -f cluster.yaml

Once the cluster is up, you can validate that the file has the correct content by SSH’ing in to each of your clusters nodes and running:

sudo cat /etc/containerd/config.toml

Create Kubernetes Secret

Within your new workload cluster, you can now docker login REGISTRY-FQDN, and utilise the stored credentials to create a Kubernetes Secret that can be used within deployments to pull from your private registry.

#Retrieve new workload cluster kubeconfig
export WORKLOAD_CLUSTER_NAME="<CLUSTER-NAME>"
mkdir -p $HOME/$WORKLOAD_CLUSTER_NAME
kubectl get secret $WORKLOAD_CLUSTER_NAME-kubeconfig -o=jsonpath='{.data.value}' | { base64 -d 2>/dev/null || base64 -D; } > $HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig
#Enter a valid username/password for the registry
docker login REGISTRY-FQDN
#Afer successful login, create a kubernetes secret based on the docker config file
kubectl create secret generic regcred --from-file=.dockerconfigjson=$HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson --kubeconfig=$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig

Deploying from the Private Repo/Registry

Create a deployment using the created secret. This step assumes you have a container image available to pull from your Privary Registry.

The image path could look something like this harbor01.domain.com/myrepo/my-node-app:latest.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: node-app
  name: node-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: node-app
  template:
    metadata:
      labels:
        app: node-app
    spec:
      containers:
      - image: REGISTRY-FQDN/REPO/CONTAINER:TAG
        name: node-app
      imagePullSecrets:
      - name: regcred

Validate your app has been deployed

kubectl get deployment node-app --kubeconfig=$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig
kubectl get pods -l app=node-app --kubeconfig=$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig

Snippet from cluster.yaml including files:

This is a snippet from the full cluster.yaml that Cluster API generates, but it shows a document that includes files:.

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: wlc01
  namespace: default
spec:
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: VSphereMachineTemplate
    name: wlc01
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-provider: external
      controllerManager:
        extraArgs:
          cloud-provider: external
    initConfiguration:
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ ds.meta_data.hostname }}'
    joinConfiguration:
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          cloud-provider: external
        name: '{{ ds.meta_data.hostname }}'
    preKubeadmCommands:
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
    - echo "127.0.0.1   localhost" >>/etc/hosts
    - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
    files:
      - path: /etc/containerd/config.toml
        content: |
          version = 2
          [plugins]
            [plugins."io.containerd.grpc.v1.cri"]
              sandbox_image = "k8s.gcr.io/pause:3.2"
              [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
                [plugins."io.containerd.grpc.v1.cri".registry.mirrors."REGISTRY-FQDN"]
                  endpoint = ["https://REGISTRY-FQDN"]
              [plugins."io.containerd.grpc.v1.cri".registry.configs]
                [plugins."io.containerd.grpc.v1.cri".registry.configs."REGISTRY-FQDN".tls]
                  insecure_skip_verify = true
    useExperimentalRetryJoin: true
    users:
    - name: capv
      sshAuthorizedKeys:
      - ssh-rsa OMITTED
      sudo: ALL=(ALL) NOPASSWD:ALL
  replicas: 1
  version: v1.18.2

Other Articles in this Series\ Cluster API Setup Steps (vSphere)\ Cluster API Workload Cluster (vSphere)

Deploying your first App into a Workload Cluster

Tue, 11 Aug 2020 00:00:00 GMT

Prerequisites

Access to a working Workload Cluster (see Cluster API Workload Cluster (vSphere)) via kubectl

Set your kubeconfig so you can access the workload cluster.

If you followed the steps in the linked article above, you should hopefully have the environment variable $WORKLOAD_CLUSTER_NAME already set, and its kubeconfig file is located in $HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig.

This step assumes you followed Cluster API Workload Cluster (vSphere).

#Replace the value to match the name of your workload cluster
export WORKLOAD_CLUSTER_NAME="wlc01"
export KUBECONFIG="$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig"

Create a development namespace

kubectl create namespace development

Create a simple deployment. The below will create a YAML file in the $HOME/Development directory.

mkdir -p $HOME/Development
tee $HOME/Development/my-nginx-deployment.yaml >/dev/null <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx 
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
EOF

Apply the Deployment, this will deploy nginx with the deployment name my-nginx-deployment.

kubectl apply -f "$HOME/Development/my-nginx-deployment.yaml" --namespace development

Check to see if our Deployment has been created. You should hopefully see all Pods READY and Running

kubectl get deployment -n development
kubectl get pod -n development

Alternative imperative commands to create the same deployment, with a slightly different name.

kubectl create deployment my-nginx-deployment2 --image=nginx:1.14.2 --namespace=development
kubectl

Previous Article in this Series: Cluster API Setup Steps (vSphere)

Cluster API Workload Cluster (vSphere)

Mon, 10 Aug 2020 00:00:00 GMT

Prerequisites

PhotonOS Kubernetes template within vSphere
HAProxy template within vSphere
Working Management Cluster (follow these steps Cluster API Setup Steps (vSphere))
Access to the Management Cluster with kubectl
Access to a machine with clusterctl

Create clusterctl.yaml to store configuration values. Replace the values where required. Take particular note of values between < and >.

mkdir -p $HOME/.cluster-api
tee $HOME/.cluster-api/clusterctl.yaml >/dev/null <<EOF
## -- Controller settings -- ##
VSPHERE_USERNAME: "administrator@vsphere.local"            # The username used to access the remote vSphere endpoint
VSPHERE_PASSWORD: "vmware1"                                # The password used to access the remote vSphere endpoint

## -- Required workload cluster default settings -- ##
VSPHERE_SERVER: "vcenter1.domain.com"                                 # The vCenter server IP or FQDN
VSPHERE_DATACENTER: "Datacenter"                                      # The vSphere datacenter to deploy the management cluster on
VSPHERE_DATASTORE: "Datastore"                                        # The vSphere datastore to deploy the management cluster on
VSPHERE_NETWORK: "VM Network"                                         # The VM network to deploy the management cluster on
VSPHERE_RESOURCE_POOL: "<ClusterName>/Resources/<ResourcePoolName>"   # The vSphere resource pool for your VMs
VSPHERE_FOLDER: "vm/<FolderName>/<ChildFolderName>"                   # The VM folder for your VMs. Set to "" to use the root vSphere folder
VSPHERE_TEMPLATE: "photon-3-kube-v1.18.2"                             # The VM template to use for your management cluster.
VSPHERE_HAPROXY_TEMPLATE: "capv-haproxy-v0.6.4"                       # The VM template to use for the HAProxy load balancer
VSPHERE_SSH_AUTHORIZED_KEY: "<ssh key>"                               # The public ssh authorized key on all machines in this cluster. Set to "" if you don't want to enable SSH, or are using another solution.
EOF

Change the value of WORKLOAD_CLUSTER_NAME to be the name of your new workload cluster.

export WORKLOAD_CLUSTER_NAME="wlc01"
mkdir -p $HOME/$WORKLOAD_CLUSTER_NAME
clusterctl config cluster $WORKLOAD_CLUSTER_NAME --infrastructure vsphere --kubernetes-version v1.18.2 --control-plane-machine-count 1 --worker-machine-count 3 > $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml

Review the cluster config file and make any required changes. Some suggestions to review are CPU, Memory, Storage and Pods CIDR range.

You can use sed to do a find and replace within the file, the below examples are for changing the Storage and Memory on all virtual machines as part of this deployment and the Pods CIDR.

Pod CIDR from 192.168.0.0/16 to 192.168.200.0/24
Memory from 8192 to 4096
Storage from 25 to 20

sed -i "s/192.168.0.0\/16/192.168.200.0\/24/g" $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml
sed -i "s/memoryMiB: 8192/memoryMiB: 4096/g" $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml
sed -i "s/diskGiB: 25/diskGiB: 20/g" $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml

Deploy the cluster components. Monitor the deployment of virtual machines within vSphere.

kubectl apply -f $HOME/$WORKLOAD_CLUSTER_NAME/cluster.yaml

Use kubectl to see cluster progress. The control planes won’t be Ready until we install a CNI in a later step.

kubectl get cluster --all-namespaces
kubectl get kubeadmcontrolplane --all-namespaces

Retrieve the kubeconfig file for the workload cluster, and set it as KUBECONFIG environment variable.

kubectl get secret $WORKLOAD_CLUSTER_NAME-kubeconfig -o=jsonpath='{.data.value}' | { base64 -d 2>/dev/null || base64 -D; } > $HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig
export KUBECONFIG="$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig"

Deploy Calico CNI to $WORKLOAD_CLUSTER_NAME. (or any other CNI)

Because we have set KUBECONFIG in the above step our kubectl commands will run against the local-control-plane cluster. If you want to be certain, add --kubeconfig="$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig" to the commands.

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
#Example with kubeconfig specified
#kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml --kubeconfig="$HOME/$WORKLOAD_CLUSTER_NAME/kubeconfig"

Reset your KUBECONFIG back to the management cluster

unset KUBECONFIG
export KUBECONFIG=$HOME/local-control-plane/kubeconfig

You will now be able to continue performing cluster management operations. Some examples below.

#Retrieve related vSphere VMs
kubectl get vspheremachine
#Retrieve related machines
kubectl get machine
#Retrieve additional machine details
kubectl describe machine <machine-name>
#Retrieve additional cluster information
kubectl describe cluster local-control-plane
#Get machinedeployment (replicaset equivalent for machines)
kubectl get machinedeployment
#Scale worker nodes machinedeployment
kubectl scale machinedeployment $WORKLOAD_CLUSTER_NAME-md-0 --replicas=4
#Get various machine resources at once
kubectl get cluster,machine,machinesets,machinedeployment,vspheremachine
#Get most/all Cluster API resources
kubectl get clusters,machinedeployments,machinehealthchecks,machines,machinesets,providers,kubeadmcontrolplanes,machinepools,haproxyloadbalancers,vsphereclusters,vspheremachines,vspheremachinetemplates,vspherevms

The next article will cover deploying an application to your new workload cluster.

Previous Article in this Series: Cluster API Setup Steps (vSphere)

Cluster API Setup Steps (vSphere)

Sun, 02 Aug 2020 00:00:00 GMT

Download the PhotonOS Kubernetes and HAProxy images from here http://storage.googleapis.com/capv-images

HAProxy version used in the steps below: http://storage.googleapis.com/capv-images/extra/haproxy/release/v0.6.4/capv-haproxy-v0.6.4.ova

PhotonOS Kubernetes version used in the steps below: http://storage.googleapis.com/capv-images/release/v1.18.2/photon-3-kube-v1.18.2.ova

Deploy both OVA files in to vSphere, take a snapshot of them, and then covert to templates.

Create a new CentOS or Ubuntu VM to act as a temporary bootstrap server - steps below will be focused on CentOS. Alternatively, install the components below on your current device.

This machine will be used in later stages of your cluster lifecycle management, due to the presence of clusterctl

Install Docker

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
MAINUSER=$(logname)
sudo usermod -aG docker $MAINUSER

Install KIND

KINDVERSION=$(curl -s https://github.com/kubernetes-sigs/kind/releases/latest/download 2>&1 | grep -Po [0-9]+\.[0-9]+\.[0-9]+)
curl -L "https://github.com/kubernetes-sigs/kind/releases/download/v$KINDVERSION/kind-$(uname -s | tr '[:upper:]' '[:lower:]')-$(dpkg --print-architecture)" -o /usr/local/bin/kind
chmod +x /usr/local/bin/kind

Install kubectl

curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Install clusterctl

CLUSTERCTLVERSION=$(curl -s https://github.com/kubernetes-sigs/cluster-api/releases/latest/download 2>&1 | grep -Po [0-9]+\.[0-9]+\.[0-9]+)
curl -L "https://github.com/kubernetes-sigs/cluster-api/releases/download/v$CLUSTERCTLVERSION/clusterctl-$(uname -s | tr '[:upper:]' '[:lower:]')-$(dpkg --print-architecture)" -o /usr/local/bin/clusterctl
chmod +x /usr/local/bin/clusterctl

Create KIND cluster

kind create cluster

Create clusterctl.yaml to store configuration values. Replace the values where required. Take particular note of values between < and >.

mkdir -p ~/.cluster-api
tee ~/.cluster-api/clusterctl.yaml >/dev/null <<EOF
## -- Controller settings -- ##
VSPHERE_USERNAME: "administrator@vsphere.local"            # The username used to access the remote vSphere endpoint
VSPHERE_PASSWORD: "vmware1"                                # The password used to access the remote vSphere endpoint

## -- Required workload cluster default settings -- ##
VSPHERE_SERVER: "vcenter1.domain.com"                                 # The vCenter server IP or FQDN
VSPHERE_DATACENTER: "Datacenter"                                      # The vSphere datacenter to deploy the management cluster on
VSPHERE_DATASTORE: "Datastore"                                        # The vSphere datastore to deploy the management cluster on
VSPHERE_NETWORK: "VM Network"                                         # The VM network to deploy the management cluster on
VSPHERE_RESOURCE_POOL: "<ClusterName>/Resources/<ResourcePoolName>"   # The vSphere resource pool for your VMs
VSPHERE_FOLDER: "vm/<FolderName>/<ChildFolderName>"                   # The VM folder for your VMs. Set to "" to use the root vSphere folder
VSPHERE_TEMPLATE: "photon-3-kube-v1.18.2"                             # The VM template to use for your management cluster.
VSPHERE_HAPROXY_TEMPLATE: "capv-haproxy-v0.6.4"                       # The VM template to use for the HAProxy load balancer
VSPHERE_SSH_AUTHORIZED_KEY: "<ssh key>"                               # The public ssh authorized key on all machines in this cluster. Set to "" if you don't want to enable SSH, or are using another solution.
EOF

Initialise and Create the Bootstrap Management Cluster. Change the number of control-plane and worker nodes, kubernetes version etc as appropriate.

clusterctl init --infrastructure vsphere
# if this fails use clusterctl init --infrastructure=vsphere:v0.6.6
mkdir -p ~/local-control-plane
clusterctl config cluster local-control-plane --infrastructure vsphere --kubernetes-version v1.18.2 --control-plane-machine-count 1 --worker-machine-count 1 > ~/local-control-plane/cluster.yaml

Review the cluster config file and make any required changes. Some suggestions to review are CPU, Memory, Storage and Pods CIDR range.

You can use sed to do a find and replace within the file, the below examples are for changing the Storage and Memory on all virtual machines as part of this deployment and the Pods CIDR.

Pod CIDR from 192.168.0.0/16 to 192.168.100.0/24
Memory from 8192 to 4096
Storage from 25 to 20

sed -i "s/192.168.0.0\/16/192.168.100.0\/24/g" ~/local-control-plane/cluster.yaml
sed -i "s/memoryMiB: 8192/memoryMiB: 4096/g" ~/local-control-plane/cluster.yaml
sed -i "s/diskGiB: 25/diskGiB: 20/g" ~/local-control-plane/cluster.yaml

Deploy the cluster components. Monitor the deployment of virtual machines within vSphere.

kubectl apply -f ~/local-control-plane/cluster.yaml

Use kubectl to see cluster progress. The control planes won’t be Ready until we install a CNI in a later step.

kubectl get cluster --all-namespaces
kubectl get kubeadmcontrolplane --all-namespaces

Retrieve the kubeconfig file for the local-control-plane cluster, and set it as KUBECONFIG environment variable.

kubectl get secret local-control-plane-kubeconfig -o=jsonpath='{.data.value}' | { base64 -d 2>/dev/null || base64 -D; } > ~/local-control-plane/kubeconfig
export KUBECONFIG="$HOME/local-control-plane/kubeconfig"

Deploy Calico CNI to local-control-plane.

Because we have set KUBECONFIG in the above step our kubectl commands will run against the local-control-plane cluster. If you want to be certain, add --kubeconfig="$HOME/local-control-plane/kubeconfig" to the commands.

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
#Example with kubeconfig specified
#kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml --kubeconfig="$HOME/local-control-plane/kubeconfig"

Now we will initialise the local-control-plane cluster as our cluster-api management cluster.

clusterctl init --infrastructure vsphere

Revert back to the kubeconfig for the bootstrap cluster, migrate management components from it to the local-control-plane cluster, and finally delete.

unset KUBECONFIG
clusterctl move --to-kubeconfig="$HOME/local-control-plane/kubeconfig"
kind delete cluster --name bootstrap

Set the local-control-plane kubeconfig as our default

cp ~/local-control-plane/kubeconfig ~/.kube/config

You will now be able to perform cluster management operations. Some examples below.

#Retrieve related vSphere VMs
kubectl get vspheremachine
#Retrieve related machines
kubectl get machine
#Retrieve additional machine details
kubectl describe machine <machine-name>
#Retrieve additional cluster information
kubectl describe cluster local-control-plane
#Get machinedeployment (replicaset equivalent for machines)
kubectl get machinedeployment
#Scale worker nodes machinedeployment
kubectl scale machinedeployment local-control-plane-md-0 --replicas=3
#Get various machine resources at once
kubectl get cluster,machine,machinesets,machinedeployment,vspheremachine
#Get most/all Cluster API resources
kubectl get clusters,machinedeployments,machinehealthchecks,machines,machinesets,providers,kubeadmcontrolplanes,machinepools,haproxyloadbalancers,vsphereclusters,vspheremachines,vspheremachinetemplates,vspherevms

The next article will cover deploying a workload cluster that can be consumed by your development teams.

Next Article in this Series: Cluster API Workload Cluster (vSphere)

Reference: The Cluster API project

vRO Email Templates with Dynamic Data

Wed, 03 Jun 2020 00:00:00 GMT

There are quite often occasions when you need to utilise vRO for sending out your email, there are a couple of ways of doing this, such as creating the HTML yourself with JavaScript code - or you can utilise Resources within vRO and store a HTML file.

This is the method we are going to walk through today. We are obtaining information from two places, the VC:VirtualMachine object, and also from its related vCAC:VirtualMachine - as the VMs in question have been deployed from vRA.

For the sake of this article, we are only retrieving simple information, but you can get it from anywhere - a payload directly from a vRA EBS event, an external system, etc - the method for replacing data remains the same.

#Create the HTML template

This is going to be a relatively simple HTML template, in this scenario we need to email a user with specific information from their VM.

<html>
<body>
  <p>================</p>
  <p>Please find your VM details below;</p>
 
  <ul>
    <li>VM Name = {{vmName}}</li>
    <li>Expiry Date = {{expiryDate}}</li>
    <li>IP Address = {{ipAddress}}</li>
    <li>Memory = {{memory}}</li>
    <li>CPU = {{cpu}}</li>
    <li>Guest OS = {{guestOs}}</li>
  </ul>

  <p>================</p>
</body>

</html>

Once your template has been created, save it as VM-Details.html upload it to the Resources section within vRO, under a folder called Email-Templates.

#Create the Workflow

Next, create your workflow - you will need two items in it. A script element, which we will add the below code in to, and we will make use of the Send notification workflow, which is found out of the box, under Library > Mail.

Create one input in to the script element, name vm of type VC:VirtualMachine and one output called emailContent of type string. Copy in the below code.

/*
  - Input: vm [VC:VirtualMachine]
  - Output: emailContent [string]
*/

var vmName = vm.name;
var ipAddress = vm.ipAddress;
var memory = vm.memory;
var cpu = vm.cpu;
var guestOs = vm.guestOS;

var vcacVm = Server.findAllForType("vCAC:VirtualMachine", "ExternalReferenceId eq '" + vm.id + "'")[0];
var expiryDate = vcacVm.expires;
if (expiryDate) {
	expiryDate = expiryDate.toDate();
}

var htmlTemplate = fetchEmailTemplate("VM-Details.html");
		
var fieldKeyValues = new Properties();
fieldKeyValues.put("{{vmName}}",vmName);
fieldKeyValues.put("{{ipAddress}}",ipAddress);
fieldKeyValues.put("{{guestOs}}",guestOs);
fieldKeyValues.put("{{cpu}}",cpu);
fieldKeyValues.put("{{memory}}",memory);
fieldKeyValues.put("{{expiryDate}}",expiryDate);

for each (field in fieldKeyValues.keys) {
	htmlTemplate = updateContent(field,fieldKeyValues.get(field),htmlTemplate);
}

System.debug("==== Email Content ==== \n" + htmlTemplate);
emailContent = htmlTemplate;

function fetchEmailTemplate(elementName) { 
	var categoryPath = "Email-Templates";
	var category = Server.getResourceElementCategoryWithPath(categoryPath);
	for each (var resourceElement in category.resourceElements) {
		if (resourceElement.name.toLowerCase() === elementName.toLowerCase()) {
			var mime = resourceElement.getContentAsMimeAttachment()
			return mime.content;
		}
	}
}

function updateContent(key,value,content) {
	content = content.replace(key,value);
	return content;
}

#What does the code do?

In lines 6 to 16 we are retrieving all of the data we need for the email.

Line 18 we retrieve our email template from our resource element, this uses a function within the script (lines 35 to 44).

In lines 20 to 26 we are putting these bits of information in to a Properties object, for the key we are specifying the variable that needs to be replaced within the HTML - for example {{cpu}} is the key name, which relates to line 11 in the HTML template.

Finally in lines 28 to 30, we iterate over each key within the Properties element, find a match within the HTML content and replace it with the relevant value (this uses the function found on lines 46 to 49).

Your email should look something like the below

#How can I utilise this?

Once you have all of the data you need, modify the HTML template to have variables that you need to replace.

Add matching variables in to your Properties object, so a matching key with the correct value/data, and that is it!

Dont forget to map the required inputs and outputs in to the Send notification workflow!

A little side note, if you are using the same variable multiple times within the HTML template, you might need to change the method for replacing the content - you should be able to use a Regex with the /g flag

vRA 8.1 Code Stream - Add GitHub Endpoint

Tue, 02 Jun 2020 00:00:00 GMT

When adding a GitHub endpoint, I came across a part where the Guided Setup suggested using https://github.com/<name>/<repo-name> but instead you need to use the API endpoint https://api.github.com/<name>/<repo-name>

To access your repo you should also create a personal access token, instead of using your password for the GitHub API: https://github.com/settings/tokens

This screenshot is from the Guided Setup, showing the suggested repo URL format.

The other issue I had with the setup was the variable reference, for some reason when creating one through the Endpoint page it didn’t auto-populate the field - you can manually reference it by using the following format ${var.<varName>} - where <varName> is the name of your variable (without the < >)

Once you have changed the above, you should be able to validate your endpoint successfully.

All of this can be found in the official vRA 8.1 documentation, just be mindful of the info in the Guided Setup

Variable creation and referencing: https://docs.vmware.com/en/vRealize-Automation/8.1/Using-and-Managing-CodeStream/GUID-CF0E3CB7-BD6A-45F5-BA08-82F8C2E0373B.html

GitHub Endpoint URL: https://docs.vmware.com/en/vRealize-Automation/8.1/Using-and-Managing-CodeStream/GUID-06E10810-CFF1-486C-B773-13A48A92D914.html

The Importance Of Standards

Mon, 10 Feb 2020 00:00:00 GMT

Recently I have been working with a financial industry customer who is all in on Automation, however it became apparent quite early on that they haven’t standardised their processes. Now, in day to day running’s and operations of a business – particularly IT – this might not be a problem, but when you want to start automating, a lack of standards begins to complicate things. In the below article we will take a look at a process that hasn’t been standardised, and then compare it to the same process with some standards applied. We will then go on to discuss how much simpler it is to automate a standardised process.

#Non-standardised Process 1 – Creating a new user

This process creates a new user, currently there are three ways someone can request a new account, and as we can see the information can arrive staggered, each of the yellow block. This results in multiple edits and updates before the account is ready for handover.

This staggered process also introduces the risk of error, for example, the request received via the phone might result in the incorrect spelling of a surname which then requires more edits to the user account.

On top of the different ways the process can be run, we also have no naming standards within this process, for example;

A new user request comes in with the first name and last name details of Joe Bloggs
Service Desk Staff 1 generates the username of: jbloggs (first name initial and full last name)
Service Desk Staff 2 generates the username of: bloggj (first five characters from last name and first name initial)

#What is a Standard

Standardisation is a set of rules and agreements that all users or consumers must adhere to, so that all processes are equal and meet set guidelines. This ensures that the end product has consistent quality and that any conclusions made are comparable with all other equivalent items in the same process. Standardisation is achieved by setting generally accepted guidelines with regard to how a product or service is created or supported, as well as to how a business is operated or how certain required processes are governed. The goal of standardisation is to enforce a level of consistency or uniformity to certain practices or operations within the selected environment.

(Reference: https://www.investopedia.com/terms/s/standardization.asp)\ (Reference: http://www.qualityindustries.com/news-and-events/2015/06/30/five-benefits-of-standardized-work.1652673)

#Standardised Process 1 – Creating a new user

This is the same process as we saw earlier, but we have now applied standards. All requests for a new user must come in through a form, and this form has data fields that the requester must complete before submitting.

This now ensures that all data required for a user account arrives at once, and because it is via a typed form, we avoid the issues of misinterpretation that we had earlier with the first & last name spelling.

The new process reduces the number of edits and ultimately results in the requester getting their end product – the user account – much sooner.

In the previous non-standardised process, we also had the differences in naming standards, these have now changed and all users created are in the format of: first initial and full last name.

#Automating Process

When automating a process, we can see how standards simplify the task. Without standards we would need to account for every scenario that the request could follow, allow for breaks in the process, enable human interaction, and provide error correction. With standards we can allow automation to generate the username, validate the inputs and hand over the item to the requester.

When we have standards, estimating the time required for automation also becomes simpler, we have a clearly defined process with a set of inputs (the user details) and an expected outcome (the user account).

Once the process is standardised, and automated, changes to this process become more controlled – we can introduce and enforce a review process which means any changes to the process are agreed within the team and released in an agreed manner.

#How Can You Standardise?

This is a very simplified approach, but it’s a great start for introducing standards.

Identify the core processes owners
- These are the people that run it day-to-day (e.g. service desk staff)
Hold a process review workshop with the process owners
- Include someone who can moderate the meeting and create agreement between the team
Whiteboard the process, get each owner to do it to highlight any differences
- Like the non-standardised flow diagrams above, this will show differences amongst the team
- Remember that because process owners might have their own view on a process, they aren’t wrong
Identify the following information for the process
- Process Inputs (what information is needed for the process to run, this can be from both users and other systems)
- Process Outputs (what is the end product of the process, what does the requester get at the end)
- Error handling (what happens in an error occurs at each of the steps)
For any process inputs, try to identify standards within those
- Naming standards, for example, username generation
As an outcome, you should have one documented process that has been agreed by all process owners, this is your new standard
Try to identify how this will be rolled out to the business and enforced – this is important, otherwise your standard will collapse again!

Kubernetes - CKA Study Resources

Thu, 23 Jan 2020 00:00:00 GMT

I am currently preparing for the CKA exam, here are the resources I have found so far…

Detailed study guide: https://github.com/burkeazbill/cka-studyguide
Knowledge dump with terminal tips: https://gist.github.com/benc-uk/09cd8e8db23ff3f0e901bc04e0417ea4
CKA Study guide: https://github.com/David-VTUK/CKA-StudyGuide
Kubernetes Certified Administrator online resources: https://github.com/walidshaari/Kubernetes-Certified-Administrator
List of tools and resources used, as well as pointers on aliases: https://www.contino.io/insights/the-ultimate-guide-to-passing-the-cka-exam
Long list of CKA resources: https://gist.github.com/strongjz/4c9ad30a12ab715ae94cf72d0e7bbc30
@kubernetesio Concepts: https://kubernetes.io/docs/concepts/
@kubernetesio kubeclt cheat sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
Kubernetes the hard way: https://github.com/kelseyhightower/kubernetes-the-hard-way
@linuxacademyCOM kubectl cheat sheet: https://linuxacademy.com/site-content/uploads/2019/04/Kubernetes-Cheat-Sheet_07182019.pdf
Exam preperation pointers: https://medium.com/faun/preparation-and-resources-for-cka-exam-ca868fc678c9
CKA Ultimate guide: https://medium.com/faun/certified-kubernetes-administrator-cka-ultimate-guide-238710f9ba73
@linuxfoundation Intro to Kubernetes: https://training.linuxfoundation.org/training/introduction-to-kubernetes/
KubeAcademy: https://kube.academy/courses
Kubernet.io Study guide: http://www.kubernet.io/
@teamKatacoda Kubernetes: https://www.katacoda.com/courses/kubernetes

vRealize Orchestrator - Datastore Browser File Query File Types

Thu, 12 Dec 2019 00:00:00 GMT

When performing a Datastore File browser search, the results can contain the following instance types, which files will be categorised under.

The instance types can be used when performing IF statements etc. E.g. if (file instanceof VcVmDiskFileInfo)

VcFolderFileInfo
VcFloppyImageFileInfo
VcIsoImageFileInfo
VcTemplateConfigFileInfo
VcVmNvramFileInfo
VcVmDiskFileInfo
VcFileInfo
VcVmConfigFileInfo
VcVmLogFileInfo
VcVmSnapshotFileInfo

VMware vRealize Automation 8 Podcast

Wed, 27 Nov 2019 00:00:00 GMT

“Automation has come a long way from the days of pen and paper, flowcharts, code and scripts, and now with the release of VMware vRealize Automation (vRA) 8 it’s moved to another level. Easy to deploy, streamlined, user friendly and faster – in this podcast, Xtravirt Cloud Automation consultant Sam Perrin discusses what vRA 8 is all about and why it’s re-ignited his excitement and passion for automation.”

vRealize Automation 7 - Install vRA Agents on CentOS virtual machine

Tue, 01 Oct 2019 00:00:00 GMT

Ensure curl is install and change to the tmp directory for when we download the agent installer

yum install curl -y
cd /tmp

Download and extract the installer and make the script executable. Add the --insecure flag to the curl command if your vRA is behind a self-signed certificate.

Update the value of $VRA_FQDN to be your vRA instance (without the https://)

export $VRA_FQDN=vra01.domain.com
curl --insecure -LO https://$VRA_FQDN/software/download/prepare_vra_template_linux.tar.gz
tar -xvf prepare_vra_template_linux.tar.gz
cd prepare_vra_template_linux
chmod +x prepare_vra_template.sh

Execute the script and follow the prompts. You will need to know the FQDN for the vRA and IaaS boxes.

./prepare_vra_template.sh

Remove udev persistence rules.

/bin/rm -f /etc/udev/rules.d/70*

Enable machines cloned from this template to have their own unique identifiers

/bin/sed -i '/^$HWADDR\|UUID$=/d' /etc/sysconfig/network-scripts/ifcfg-eth0

If you rebooted or reconfigured the reference machine after installing the software bootstrap agent, reset the agent and follow the steps.

/opt/vmware-appdirector/agent-bootstrap/agent_reset.sh

Power down the VM

shutdown -h now

Small, Medium or Large: There Is No One Size Fits All

Tue, 18 Dec 2018 00:00:00 GMT

#Why A T-Shirt & Do I Look Good In This?

Working out the effort required for an automation project can be challenging. Teams may have different abilities and skill-sets causing individuals effort estimations for tasks to differ. The primary advantage to t-shirt sizes is the speed and ease of getting started and they can be a great way of getting used to relative estimating\ By expelling the focus on a numerical score, the development team has the ability and freedom to think more dynamically and abstractly about the effort associated with a request. By limiting the number of t-shirt sizes, you allow for faster estimates and less deliberation around placement. You can use t-shirts without a numerical value associated with them, but with the values you can better identify the differences in size between the t-shirts and use those same values for pricing and velocity calculations.

#What’s In A T-Shirt

A t-shirt is a defined amount of effort, measured in days, hours, points, or anything else you feel appropriate. These types of estimates become more efficient and effective when you have lots of ‘things’ to estimate at the same time, rather than just a single ‘thing.’ Behind a t-shirt size you often have a numerical value associated to it, this can be Fibonacci numbers or hours and days. So for example, if you believe the new requirement will suit a medium t-shirt size, you now have an idea of the effort that will be required. The table below shows an example of days of effort per t-shirt size.

#T-Shirt Sizing Table

Size	Effort (days)
Small	1
Medium	2-3
Large	4.5
X-Large	Estimated Per Requirement!

#So How Do We Get Started?

We create a relative estimation. We do this by comparing the new requirement to a previous requirement
The old and new requirements are evaluated (at a high level) and a decision is made as to if they are equivalent in effort or not
If the old requirement had previously been gauged a ‘medium t-shirt’ and the requirement doesn’t fit this size.. well, you pick the right size t-shirt to fit the body!

#Does It Fit Right

During the development process you will get an idea if the estimate was accurate or not. If it’s way off adjust it accordingly. Just keep in mind that it’s also beneficial to keep with the initial estimate so that it can be reviewed afterwards. When you do, it allows you to ask questions such as

Why was the initial estimate for the request an S,M,L,XL?
When the work was finished, was the request more like a S,M,L,XL?
Why was the request larger than expected?
What made this request an S,M,L,XL instead of a S,M,L,XL? Was there just more work? Was this type of work new?
How can we avoid underestimating this type of request in the future (hint: adopting a working Agile methodology, working in sprints and having planning and review sessions will greatly help with this. A blog post for another time maybe?)

Once you have reviewed your estimates against the reality (post delivery) you can build out or adjust your baseline for each size. New requests from this point will use this baseline. If a request is too big to fit a t-shirt you have two options

Break it down into smaller requirements that do fit within your sizes
As a last resort, place it into a X-Large or XX-Large t-shirt size where you don’t have an estimation of effort (warning: This is a really bad idea and completely devalues all your other sizing efforts. You don’t know how long it will take, amount of resource required, cost.. so what do I do? you ask. See the previous point)

#How Does Xtravirt Use T-Shirt Sizes?

Often the information received during an automation project kickoff, is the name and a brief description of the Service Request (the thing that needs to be automated). The real detail is missing at this stage. Having a small amount of information makes detailed planning difficult, you have questions that need to be answered, you don’t know the risks, full requirements or the even the success criteria; yet you still need to provide two estimates to the customer – a cost estimate and an effort estimate. This is where t-shirt sizes help us.

We work with four sizes; small, medium, large and extra-large. Together as a team we go through the list of Service Requests, discuss what we believe may be required and provide our estimates. These estimates are based on previous experience and so afford us an educated idea on what size to allocate (relative estimation). When the team are agreed on the size of the Service Request we move on to the next until we have an estimate for all the required work. Once all the estimations are completed, we can provide an idea on the total effort, which then leads to an informed cost quote for the customer!

Due to our experiences, we have been able to analyse our decisions, see where we have over or under estimated effort and identify why that happened. This allows us to ensure the original t-shirt still fits. If it doesn’t, we can change it for future estimates. Using this data, we can build out a baseline number of days associated to each t-shirt size.

#The Takeaway

Create effort estimations before commencing any work
Record actual effort
Review actual vs estimated effort to better educate yourself for ‘next time’
Using terminology such as ‘t-shirt’ sizes help abstract away from hours, minutes, days of effort which is often hard to quantify
Continual refinement and education means better estimation of effort and delivery cost as opposed to continually working with ‘Guesstimates’

Behind Automation

Fri, 14 Sep 2018 00:00:00 GMT

I was lucky enough to partake in a podcast recently with colleagues Paul Davey and Stuart Robinson of Xtravirt. If you want to learn more about automation then please check it out!

This episode outlines:

What is automation?
The common reasons businesses are automating business processes.
Some of tools being used in automation – from Puppet and Chef for configuration management to HashiCorp’s options for provisioning, security and the running of applications.
The need for businesses to take a business logic first approach to automation projects.
Some common mistakes to avoid.
How Xtravirt have designed workshops and best practice methodologies to ensure automation projects stay on track and deliver the business outcomes they were intended to.

You can listen to the podcast by clicking here.

Install VestaCP on Google Cloud in 5 steps

Mon, 29 Aug 2016 00:00:00 GMT

This article will provide you with 5 quick steps to get up and running with VestaCP and finish with adding your first domain to the portal.

VestaCP is an Open Source control panel that helps you manage your web server and simplifies a lot of things! https://vestacp.com/

Within the Google Cloud Console, navigate to Networking > Firewall Rules.
- Add a new firewall rule for VestaCP
- Name = vestacp-panel
- Source Filter = allow from any source (0.0.0.0/0)
- Allowed protocols and ports= 8083/tcp
- Target Tags = vestacp-panel
Deploy a new Compute Engine VM Instance
- Define Name and Zone based on your preference
- Choose Machine Type (I have used “micro”)
- Boot Disk = 10gb and CentOS 7
- Firewall = allow HTTP and HTTPS (so your websites can get through when setup)
- Expand “Management, disk, networking, SSH keys”
  - Under Management, in the Tags section, add “vestacp-panel”
  - Under Networking, assign a New Static External IP (or re-use an old one if you have any) – take note of the External IP assigned, you’ll need this later.
- Create the instance
Wait for the new VM to finish deploying, connect to it via SSH and install VestaCP
- cd /tmp/
- curl -O http://vestacp.com/pub/vst-install.sh
- sudo bash vst-install.sh
- Follow the on screen steps
- When it asks for an FQDN, provide the full URL that you will use to access the portal (e.g. panel.example.com) – you will also need to add this as an A record for your domain.
- Take note of password presented to you at the end of the install
Login to your new VestaCP Control Panel
- https://<your_server_ip>:8083 or https://<your_server_URL>:8083
- Username = admin
- Password = provided at the end of Step 3
- If you can’t reach the portal, check your Firewall Rules from Step 1 and ensure your VM has the correct tags, as per Step 2
Configure your VestaCP server
- Edit the Admin user and change the password to something secure
- Edit the VestaCP firewall and modify any existing rules to drop traffic you don’t want/need (POP3, DB, PING etc)
- Add your domain
  - After this you’ll need to point your website to your new VestaCP web server

There are plenty more things to set up within VestaCP, such as setting Private Nameservers, creating Packages etc but the above will get you up and running.

Elasticsearch Cluster Install on CentOS 7

Fri, 24 Jun 2016 00:00:00 GMT

This post wont go in to any detail around setting up CentOS itself, but it will cover getting a simple 3 node Elasticsearch cluster up and running, with a couple of helpful plugins.

Install Java and check the install

yum install java-1.8.0-openjdk.x86_64 -y
java -version

Download Elasticsearch (check the site here for the latest versions)

cd /tmp
wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.2.1/elasticsearch-2.2.1.rpm

Install Elasticsearch

cd /tmp
sudo rpm -ivh elasticsearch-2.2.1.rpm

This results in Elasticsearch being installed in /usr/share/elasticsearch/ with its configuration files placed in /etc/elasticsearch and its init script added in /etc/init.d/elasticsearch

Enable Elasticsearch on Boot

sudo systemctl enable elasticsearch.service

Configure Elasticsearch

vi /etc/elasticsearch/elasticsearch.yml

Add near the top, and replace anything in < > with your details

cluster.name: <name of the cluster>
#below line dynamically sets the node name based on the server hostname
node.name: ${HOSTNAME}
#below binds elasticsearch to the local (127.0.0.1) and site ip address (e.g. 192.168.1.1)
network.host: [_site_, _local_]
http.port: 9200
bootstrap.mlockall: true
#below sets the nodes the cluster should find
discovery.zen.ping.unicast.hosts: ["<node1_ip_or_fqdn>", "<node2_ip_or_fqdn>", "<node3_ip_or_fqdn>"]

Restart and Verify Elasticsearch status

sudo systemctl restart elasticsearch.service
sudo systemctl status elasticsearch.service

You should be able to access the node now, on: http://<node-ip>:9200

Optional: Install Elastic-HQ and Head Plugins

cd /usr/share/elasticsearch/
bin/plugin install mobz/elasticsearch-head
sudo bin/plugin install royrusso/elasticsearch-HQ
sudo systemctl restart elasticsearch.service
sudo systemctl status elasticsearch.service

Once installed, you can access these plugins via:\ Head = http://<node-ip>:9200/_plugin/head HQ = http://<node-ip>:9200/_plugin/hq

Additional Nodes:

Repeat the same process on your other servers, you can add more than 3 nodes to the cluster, but if you do make sure you update discovery.zen.ping.unicast.hosts in your Elasticsearch config to include those additional nodes.

CentOS 7 – Installing pymssql with pip, libgnutls.so.26 error

Tue, 09 Feb 2016 00:00:00 GMT

When trying to install pymssql with pip you may get a requirements error for libgnutls.so.26

To fix this error I took the following steps:

For the below you may need the EPEL repo, install this with the below command

yum install epel-release

Install gcc

yum install gcc

Install python-devel

yum install python-devel

Download freetds-devel and freetds for CentOS 7 and store in /tmp/ (or somewhere else, but remember where you stored it for later)

http://rpm.pbone.net/index.php3/stat/4/idpl/31220521/dir/redhat_el_7/com/freetds-devel-0.95.19-1.el7.x86_64.rpm.html

http://rpm.pbone.net/index.php3/stat/4/idpl/31220520/dir/redhat_el_7/com/freetds-0.95.19-1.el7.x86_64.rpm.html

Install both of them

yum install /tmp/freetds-0.95.19-1.el7.x86_64.rpm
yum install /tmp/freetds-devel-0.95.19-1.el7.x86_64.rpm

Run pip install pymssql again, it should hopefully succeed this time.

FortiGate as Slave DNS with Windows DNS Master

Sat, 24 Oct 2015 00:00:00 GMT

It is possible to setup your FortiGate device so that requests towards specific domains are forwarded to a Windows DNS server. Below are the steps needed to get this working.

On the Windows Master DNS Server

Open DNS Manager and open the zone needed
Find the Start of Authority (SOA) record
Properties > Zone Transfers
Allow Zone Transfers > To Any Server (you can also specify the server)
Open ‘Notify’ and add in the Slave DNS IP address (the IP your FortiGate will be seen as to the Master DNS)

On the FortiGate Slave

Enable the DNS Database feature (System > Config > Features)
Go to System > Network > DNS Servers and Create a new DNS Database.
- Type: Slave
- View: Shadow
- DNS Zone:<dns_zone>
- Domain Name:<dns_zone.local>
- IP of Master:
- Authoritative: Enable/Disable

You will then to add an SRV record, as well as identify the source IP address if your Master DNS is over VPN;

Connect to FortiGate CLI
Add SRV record

config system dns-database
edit "<dns_zone>"
set forwarder "<IP of Master DNS>"
next
end

Specify source IP

config system dns-database
edit "<dns_zone>"
set source-ip "<IP of FortiGate>"
next
end

Add the DNS service to the Interface

Go to System > Network > DNS Servers and Create a new DNS Service.
Interface:<Interface users will connect to>
Mode: Recursive

Testing

You can test the new DNS service works by running a ping through your FortiGate CLI

execute ping-options source <interface IP address>
execute ping <FQDN of a server in the new DNS zone>

Potential Issues

The interface your users connect to may be set to use the “Same as System DNS”. If you use external DNS servers this means your users will not benefit from the changes we have just made.

To fix this change the interface settings to be “Same as Interface IP”, this now means users will get the Interface IP as their DNS server, and will benefit from all DNS Database changes we make on the FortiGate

FortiGate – Change Switch Mode

Tue, 13 Oct 2015 00:00:00 GMT

Out of the box, your FortiGate device will most likely be in Switch mode, this groups the internal interfaces in to a single switch. To change to Interface mode, follow the below steps.

Interface mode allows you to control all of the interfaces separately. To change modes make sure none of the interfaces (lan/internal) are referenced

Enter Interface Mode

config system global
set internal-switch-mode interface
end

Enter Switch Mode:

config system global
set internal-switch-mode switch
end

FortiGate Reports Menu Missing

Thu, 03 Sep 2015 00:00:00 GMT

You might come across an instance where your Reports menu is missing from within the FortiGate GUI. I noticed this today when I went to disable local report emails and wasn’t able to!

We use FortiAnalyzer for our reports, so when transferring to this we disabled Local Disk logging, this in turn removed the reports menu.

It is a quick fix to get the menu back;

Connect to your FortiGate GUI
Log & Report > Log Config > Log Settings
Enable Disk logging
Enable Local Reports
- If you are using FortiAnalyzer you may get an error message about Fortinet recommendations, you can ignore this for the moment
Log out and then Log back in, you will now see the menu
Log & Report > Log Config > Report

In my case we disabled “Email Generated Reports” and then disabled Local Reports and Disk logging. By disabling Local Reports you clear the FortiAnalyzer message you might have come across earlier.

FortiManager Disable SSLv3

Fri, 07 Aug 2015 00:00:00 GMT

This is documented in the latest release notes, but to quickly disable SSLv3 on your FortiManager device, run the following;

config system global
set ssl-protocol tlsv1
end

IPsec VPN with Public IP Subnet’s on a FortiGate

Tue, 23 Jun 2015 00:00:00 GMT

I recently came across a requirement where I had to create a site-to-site IPsec VPN, this is usually not an issue, set your Phase 1 and Phase 2 settings, apply your policies and you are good to go, but the difference this time was those local and remote subnets were Public IP addresses.\ The Public IP address our side was also being used as a VIP

Below I will document the steps in getting this working – the issue I had was where I put my policies, and not enabling NAT on the outgoing policy!

#Create your IPsec tunnel;

FortiGate GUI > VPN > IPsec > Tunnels > Create New

Set your name and chose your template. I used “Custom VPN Tunnel (No Template)”
Fill in your Phase1 settings
Fill in your Phase2 settings;
- Local Subnet – this will be your Public IP/Range
- Remote Subnet – this will be their Public IP/Range
Press OK to create the tunnel

#Add in your new route;

FortiGate GUI > Router > Static Routes > Create New

Destination IP/Mask – this will be the Remote Subnet you entered for your Phase2
Device – this will be the tunnel you have just created
Change any of the other settings if you need to
OK to add the new route

#Create an IP Pool;

FortiGate GUI > Policy & Objects > Objects > IP Pools > Create New

Add a name and comments if required, and set the type (I am using Overload)
External IP Range = the range you set for Local Subnet
Ok to create the IP Pool

#Create your policies;

FortiGate GUI > Policy & Objects > Policy > IPv4 > Create New

Outgoing

Incoming Interface = The internal interface where your server exists
Source Address = An object with the internal IP address of your server
Outgoing Interface = The tunnel you just made
Destination Address = An object with the remote Public IP range
Apply any Schedules and Service restrictions and Action = Accept

You now need to enable NAT, this is the bit I missed at first;

NAT = On
Use Dynamic IP Pool = Select the pool you made in the previous step
Add any other settings and then OK to create your first policy

Incoming

Create the second policy for Tunnel to Internal
Incoming Interface = The tunnel you just made
Source Address = An object with the remote Public IP range
Outgoing Interface = The internal interface where your server exists
Destination Address = The VIP that belongs to the internal server (same as the IP Pool address)
Apply any Schedules and Service restrictions and Action = Accept

On this policy you do not need NAT

NAT = Off
Add any other settings and then OK to create your first policy

Check to see if your Tunnel is Up and try sending some traffic down it\ FortiGate GUI > VPN > Monitor > IPsec Monitor

Disable SSLv3 on FortiGate GUI and SSL VPN

Tue, 09 Jun 2015 00:00:00 GMT

To disable SSLv3 on both the FortiGate GUI and SSL VPN you need to run the below commands via CLI. According to the FortiGuard website, the only reported compatibility issue that may follow with running the below is with IE6

For the HTTPS GUI:

config system global
set strong-crypto enable
end

For SSL VPN:

config vpn ssl settings
set sslv3 disable
end

Set Port Speed for a FortiGate Virtual Switch Interface

Tue, 31 Mar 2015 00:00:00 GMT

You might come across a scenario where you have created a Virtual Switch/Hardware Switch on your FortiGate and you need to set a specific port within that switch to a different speed

In our case we had a 4 port switch, 1 of those ports connected to another appliance, the other device was set as 100mb Full – by default all of the ports on the FortiGate virtual-switch were set to auto, so this 1 port ultimately fell down to 100mb Half

Below are the steps to set a port to the correct speed within a FortiGate switch

config system virtual-switch
edit <switch name>
config port
edit <port>
set speed <speed>
end
end

Speed options:

auto
10full
10half
100full
100half
1000full

The steps needed to set an interface speed for a port that isnotin a virtual-switch are slightly different, for that you use:

config system interface
edit <port>
set speed <speed>
end
end

You can use theshowcommand to show available ports/switches that you can edit

Within the FortiGate web console under Network > Interfaces, if you hover over the Interface image you can see the speed of a port

Upgrading Dell PowerEdge R710 BIOS

Mon, 23 Feb 2015 00:00:00 GMT

You will need to download the required BIOS from the Dell website, in ‘Non-Packaged’ File Format. I also used Rufus to create a bootable USB drive

R710 BIOS 6.4.0 – R710-060400C.exe\ http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverid=4HKX2

Rufus 1.4.12\ http://rufus.akeo.ie/downloads/rufus-1.4.12.exe

Once you have got the above we can begin to create our Bootable USB

Plug in your USB drive and open Rufus.exe
From the Device drop down select your USB drive
Leave the other large drop downs as default
Label the USB drive if required
Check the 2nd, 3rd and 4th check boxes
Ensure FreeDOS is selected from the smaller drop down
Press Start

Rufus Settings

Running the Update

Your new bootable drive will show within Windows Explorer, open it and copy your BIOS .exe download to the root of the drive.\ Rename the .exe to something simpler, such as R710.exe (this will help later).

Insert the USB in to the server and boot, when prompted pressF11to Enter Boot Menu.\ Use the arrow keys to highlight ‘Hard Drive C:’, on the sub-menu select your USB device and press enter.

When at the command prompt type inR710.exe(if you did not rename the file in the earlier steps you will need to use the original name at this point). You will be entered in to the BIOS updater screen. Follow the steps until completion

Forcing a BIOS Update

If you wish to re-flash your BIOS with the same version as already installed, or if you need to replace an OEM BIOS you might need to use/forcetypein order to force a BIOS update.

E.g. – `R710.exe /forcetype`

EqualLogic PS4100 and ESXi 5.5 VMkernel Setup

Wed, 12 Nov 2014 00:00:00 GMT

Read the first blog post here about Dell MEM and ESXi 5.5

As part of the EqualLogic MEM setup.pl script it creates a new vSwitch and also its iSCSI VMkernel’s, but due to that script not currently being compatible with ESXi 5.5 U2 we had to manually create the connections. Below is the setup we have used and a few differences compared to the Dell EqualLogic best practice document.

In the document it states that the heartbeat connection is no longer required for ESXi servers of version 5.1 or above. I had issues with this and found that if a switch that one of the adapters connected to failed, the pings would stop to the storage device and ESXi would report that all paths were down for the EqualLogic array, and therefore lose access to the datastore. I added a heartbeat connection in and was able to keep a path active when an adapter failed. Below will cover the settings I have in place and the tests I did to determine successful failover in various scenarios

The Storage Heartbeat VMkernel port is no longer required for ESX servers running version 5.1 and greater

vSwitch Setup

Create a new Standard VMkernel vSwitch on your host and assign two or more network adapters to it (these are obviously the adapters that connect to your storage network where the EqualLogic sits)
Name the adapter “EQL Heartbeat”
Assign an IP to the EQL Heartbeat VMkernel port
Finish the setup screen
Find the vSwitch in the Networking window and select Properties
Add a new VMkernel port to the vSwitch; press Add, select VMkernel, label it EQL_1
Assign an IP address (same subnet as your heartbeat port), finish the setup
Repeat the process again and label the new VMkernel EQL_2, again this needs to be in the same subnet as the other two VMkernel ports

Once you have done that you should have something that looks similar to this;

vSwitch Layout

We need to make a few more changes to this vSwitch;

Edit each of the items in the screen shot to have an MTU of 9000 (your switch must also support this!)
Edit vSwitch > NIC teaming tab, all adapters should be active
Edit EQL Heartbeat > NIC teaming tab, all adapters should be active (will most likely be grayed out as inherits settings from vSwitch)
Edit EQL_1 > NIC teaming tab, under Failover Order tick Override switch failover adapter, keep one in Active and move the other to unused
Edit EQL_2 > NIC teaming tab, under Failover Order tick Override switch failover adapter, keep one in Active and move the other to unused – this needs to be different to the one used in EQL_1

EQL_1 Properties

EQL_2 Properties

iSCSI Software Adapter

If you don’t yet have an iSCSI software adapter now is the time to add one in, once you have done that select it and press Properties. Under Network Configuration and press Add, select the two adapters you just configured (EQL_1 and EQL_2).

While you are here also add in your EqualLogic’s group IP address in to the Dynamic Discovery tab

Setup VMware Round Robin

The next steps I have documented in the post Dell MEM v1.2 and ESXi 5.5 U2, so I wont cover them again here. These steps setup cover the remaining recommendations from Dell.

Testing Failover

To confirm the above settings I did the followings tests to confirm failover – during these tests you should not have all paths down.\ These tests make the assumption that you have at least two switches, two network adapters and two controllers in your EqualLogic, and that you return to default before running the next test.

Remove a network cable from the host (which connects to the storage network)
Remove the other network cable
Power off 1 switch
Power off other switch
Disconnect controller
Disconnect other controller

If you do encounter an APD scenario double-check your cabling, as per this diagram from Dell, a long with the heartbeat adapter the cabling also caused a problem. This was resolved with the help of this diagram.

PS4100 Cabling

Dell MEM v1.2 and ESXi 5.5 U2

Tue, 04 Nov 2014 00:00:00 GMT

It would seem that the Dell MEM v1.2 will not install on ESXi 5.5 U2 using esxcli. It can apparently be deployed using vMA or VUM but as these aren’t used we had to look for an alternative.

The issue was with Dell who pointed us towards the following best practice document;\ http://en.community.dell.com/cfs-file/__key/telligent-evolution-components-attachments/13-4491-00-00-20-43-46-01/TR1091-Best-Practices-with-EqualLogic-and-VMware-1.1.pdf?forcedownload=true

In summary they recommend the following.\ – Use Round Robin\ – Change from 1000 IO’s per path to 3 IO’s per path\ – Change iSCSI Timeout values from default to 60 seconds\ – Disable Delayed ACK

Dell have provided some handy scripts to get you started, here they are for ESXi 5.x (they can all be found in the PDF attached earlier in the post so please refer to this for full details);

Set all EqualLogic volumes to Round Robin and set IOPS value to 3 – this must be run on all hosts;

esxcli storage nmp satp set --default-psp=VMW_PSP_RR --satp=VMW_SATP_EQL ; for i in `esxcli storage nmp device list | grep EQLOGIC|awk '{print $7}'|sed 's/(//g'|sed 's/)//g'` ; do esxcli storage nmp device set -d $i --psp=VMW_PSP_RR ; esxcli storage nmp psp roundrobin deviceconfig set -d $i -I 3 -t iops ; done

Set a default so that new EQL volumes will inherit the correct settings;

esxcli storage nmp satp rule add -s "VMW_SATP_EQL" -V "EQLOGIC" -M "100E-00" -P "VMW_PSP_RR" -O "iops=3"

The above command will require a restart of the host before it becomes effective, once you have restarted you can verify the correct settings using the below;

esxcli storage nmp device list

The output will be similar to what is shown below, the parts in bold are what you are looking for;

naa.6090a098703e5059e3e2e483c401f002
Device Display Name: EQLOGIC iSCSI Disk
(naa.6090a098703e5059e3e2e483c401f002) 
Storage Array Type: VMW_SATP_EQL
Storage Array Type Device Config: SATP VMW_SATP_EQL does not support device configuration. 
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=iops,iops=3,bytes=10485760,useANO=0;lastPathIndex=3: NumIOsPending=0,numBytesPending=0}

VMW_SATP_EQL (indicates its an EqualLogic)

VMW_PSP_RR (path selection is set to Round Robin)

policy=iops,iops=3 (shows IOPS have been set to 3)

Next up is to change the default iSCSI timeout values. “By default, the MEM configuration script will make an attempt to set each of these timeout values to 60 seconds which is the recommendation.”, so we will copy that with the below;

esxcli iscsi adapter param set --adapter=vmhba## --key=LoginTimeout --value=60

Replace ## with that of your iSCSI software adapter, for example vmhba33.

After you have done the above you will need to disable Delayed ACK, Dell don’t provide a command line for disabling this, so it needs to be done through the vCentre GUI – please refer to Page 10 in the PDF for steps on how to do this.

Dell do list a few other recommendations, around such things as LRO and SIOC but I have not applied these so I wont go in to detail on what they mention.

Hopefully this post will come in handy for a few people, as I know it held us back a bit!

Please check the commands before you apply them and also work with your vendor if you are unsure of the implications of changes some of the settings listed above will have towards other storage arrays, etc. 🙂

I will soon be posting about the VMkernel/NIC setup for this, as we have had some issues relating to that too – events such as APD! I will update this post with a link to that once it is ready

UPDATE: Post relating to iSCSI setup is here! (to be updated)