This is not theory. This is what happens in a live cluster.

What Dual-Stack Really Means in OKE

In a dual-stack Kubernetes cluster, every Pod receives:

• One IPv4 address

• One IPv6 address

Both addresses are active simultaneously.

This enables:

• Backward compatibility with IPv4 systems

• Native IPv6 communication

• Future-proof cloud networking

In OCI, dual-stack works natively with VCN networking. There is no overlay tunnel involved. Pod IPs are directly routable inside the VCN.

That architectural decision makes a significant difference in how traffic behaves.

How a Pod Receives Both IPv4 and IPv6

When a Pod is scheduled onto a node, the OCI VCN-native CNI allocates IP addresses from the configured Pod subnet.

For example, a single Pod may receive:

IPv4  → 100.0.5.102
IPv6  → fd00:100:0:4:0:6ed9:2497:6dd8

Both addresses are bound to the Pod’s network interface.

To confirm this in a running cluster:

PRATIK_BOR@cloudshell:~ (ap-sydney-1)$ kubectl get pods -A \
> -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IPv4:.status.podIPs[0].ip,IPv6:.status.podIPs[1].ip
NS            NAME                                   IPv4          IPv6
default       nginx-568f7cfd68-g4h2c                 100.0.5.102   fd00:100:0:4:0:6ed9:2497:6dd8
default       nginx-568f7cfd68-pvgct                 100.0.5.215   fd00:100:0:4:0:bc9b:fecf:997
default       nginx-568f7cfd68-vprr8                 100.0.5.106   fd00:100:0:4:0:c05c:cfe5:7593
kube-system   coredns-bfd957d6b-ddr79                100.0.5.110   fd00:100:0:4:0:a800:ed61:1c0d
kube-system   coredns-bfd957d6b-slxml                100.0.5.104   fd00:100:0:4:0:3717:f114:d94d
kube-system   coredns-bfd957d6b-w2tgj                100.0.5.122   fd00:100:0:4:0:d046:deaf:a1f0
kube-system   csi-oci-node-rfmkw                     100.0.3.90    fd00:100:0:3:0:d02f:752e:66c7
kube-system   csi-oci-node-s2thk                     100.0.3.117   fd00:100:0:3:0:eab9:9713:9d05
kube-system   csi-oci-node-tc5rc                     100.0.3.174   fd00:100:0:3:0:d4c0:26f3:8990
kube-system   kube-dns-autoscaler-7f9c9ddb55-c4vdd   100.0.5.125   fd00:100:0:4:0:6eee:830e:daf7
kube-system   kube-proxy-5cwj5                       100.0.3.117   fd00:100:0:3:0:eab9:9713:9d05
kube-system   kube-proxy-pb427                       100.0.3.90    fd00:100:0:3:0:d02f:752e:66c7
kube-system   kube-proxy-zldlk                       100.0.3.174   fd00:100:0:3:0:d4c0:26f3:8990
kube-system   proxymux-client-g9bhp                  100.0.3.174   fd00:100:0:3:0:d4c0:26f3:8990
kube-system   proxymux-client-tbqdm                  100.0.3.90    fd00:100:0:3:0:d02f:752e:66c7
kube-system   proxymux-client-w9cdf                  100.0.3.117   fd00:100:0:3:0:eab9:9713:9d05
kube-system   vcn-native-ip-cni-8qfhn                100.0.3.117   fd00:100:0:3:0:eab9:9713:9d05
kube-system   vcn-native-ip-cni-fqdx7                100.0.3.90    fd00:100:0:3:0:d02f:752e:66c7
kube-system   vcn-native-ip-cni-gszjb                100.0.3.174   fd00:100:0:3:0:d4c0:26f3:8990
PRATIK_BOR@cloudshell:~ (ap-sydney-1)$

The output clearly shows both IP versions assigned to the same Pod.

Inside the container:

kubectl exec -it nginx-568f7cfd68-g4h2c  -- hostname -i
fd00:100:0:4:0:6ed9:2497:6dd8 100.0.5.102

You’ll see both inet (IPv4) and inet6 entries.

This confirms that dual-stack is not just enabled at the control plane level it is active at the workload level.

Pod-to-Pod Traffic: What Actually Happens

Same Node Communication

When two Pods run on the same node:

• Traffic stays local.

• Linux networking handles routing internally.

• No VCN-level routing occurs.

Dual-stack does not change this behavior. IPv4 and IPv6 packets are both handled locally by the node’s networking stack.

Latency is minimal because packets never leave the host.

Cross-Node Communication

When Pods are on different nodes, the behavior becomes more interesting.

In OCI’s VCN-native networking:

1. The source Pod sends traffic.

2. The node routes it directly into the VCN.

3. The destination node receives it.

4. The packet is delivered to the target Pod.

There is no overlay tunnel.
There is no encapsulation.
There is no extra hop.

For IPv6 traffic specifically:

• OCI routes IPv6 natively.

• No NAT is required.

• IPv6 packets travel directly using the assigned IPv6 CIDR.

This creates a very clean routing model compared to many overlay-based CNI implementations.

External Traffic Flow (World → Pod)

When exposing a workload using a Kubernetes LoadBalancer service, OCI provisions a native load balancer.

In a dual-stack subnet, that Load Balancer can have:

• Public IPv4

• Public IPv6

Traffic flow looks like this:

Client → OCI Load Balancer → Node → Pod

If the client connects over IPv6:

IPv6 Client → LB IPv6 → Pod IPv6

This enables true end-to-end IPv6 connectivity without translation.

That’s a significant architectural advantage.'

Pod to External World (Egress)

Outbound traffic behaves differently for IPv4 and IPv6.

IPv4 Egress

• Requires NAT Gateway

• Private IPv4 → Translated → Public IPv4

IPv6 Egress

• No NAT required (if using globally routable IPv6)

• Direct routing possible

IPv6 removes an entire NAT layer from the architecture.

This simplifies troubleshooting and reduces stateful dependencies in network design.

Final Thoughts

Dual-stack Kubernetes is no longer experimental.

In OCI, it is stable, practical, and production-ready — provided that:

• Subnets are properly designed

• IPv6 CIDRs are correctly allocated

• Traffic flows are understood

• Observability tools are updated

IPv6 adoption is accelerating globally. Running dual-stack today prepares your Kubernetes platform for what’s coming tomorrow.

In this DIY article, you will learn how to:

• Build a custom RHEL 8/9/10 image using Red Hat Image Builder

• Export the image in QCOW2 format

• Upload it to OCI Object Storage

• Import and use it as a custom image in OCI Compute

This approach ensures consistent builds, faster provisioning, and compliance across cloud environments.

Prerequisites

Before starting, ensure you have:

• A valid Red Hat account with active RHEL subscription or developer access

• Access to Red Hat Hybrid Cloud Console

• Permissions to use Red Hat Image Builder

• An OCI tenancy with access to:

  • Object Storage

  • Compute → Custom Images

• (Optional) OCI CLI configured for automation

Step 1: Log in to Red Hat Hybrid Cloud Console

Open the Red Hat console:

https://console.redhat.com/

Log in with your Red Hat credentials and verify that your account has developer access or an active subscription. Without this, image creation options will not be visible.Step 2: Open Image Builder (Inventory Image Tool)

From the Red Hat Console dashboard:

1. Navigate to Inventory

2. Select Image Builder

Image Builder allows you to create cloud-ready RHEL images for multiple platforms, including OCI.

Step 3: Create a Blueprint for RHEL 8 / 9 / 10

In Image Builder:

1. Click Create Blueprint

2. Select the required RHEL version:

   • RHEL 8

   • RHEL 9

   • RHEL 10

3. Choose architecture (x86_64 is recommended for OCI)

Customize the blueprint:

• Install required RPM packages

• Configure users and SSH public keys

• Enable repositories

• Apply security hardening or baseline configurations

This blueprint acts as your gold image definition.

Step 4: Build the Custom RHEL Image

After saving the blueprint:

1. Click Build Image

2. Select QCOW2 as the image output format

QCOW2 is the recommended and supported format for OCI custom image imports.

The build process will start and may take several minutes. Wait until the status shows Completed.

Step 5: Download the RHEL QCOW2 Image

Once the image build is complete:

• Download the generated QCOW2 image file

• (Optional) Validate checksum or file integrity

Ensure sufficient disk space before downloading, as image sizes can be multiple GBs.

Step 6: Upload the Image to OCI Object Storage

Log in to the OCI Console:

1. Navigate to Object Storage → Buckets

2. Select an existing bucket or create a new one (example: RHEL-Images)

3. Upload the downloaded QCOW2 image

   

Make sure the bucket is in the same region where you plan to import the custom image.

Step 7: Import the Custom RHEL Image into OCI

After uploading the image:

1. Go to Compute → Custom Images

2. Click Import Image

3. Select Object Storage as the source

1. Choose the bucket and QCOW2 image object

2. Set image type to QCOW2

3. Provide a descriptive name (example: RHEL9-Golden-Image)

The import process may take several minutes depending on image size.

Step 8: Launch an OCI Instance Using the Custom Image

Once the image import completes:

1. Go to Compute → Instances

2. Click Create Instance

3. Choose Custom Image

4. Select your imported RHEL image

Configure shape, networking, and SSH access, then launch the instance.

Post-Deployment Validation

After the instance starts:

• Confirm OS version:

  cat /etc/redhat-release

• Verify installed packages and custom configurations

• Validate SSH, networking, and application readiness

• 

Benefits of Using Custom RHEL Images on OCI

• Faster VM provisioning

• Consistent OS builds across environments

• Improved security and compliance

• Reduced configuration drift

• Enterprise-ready golden image strategy

• In-place OS conversion during migration.

Conclusion

Exporting a custom RHEL 8/9/10 image from Red Hat Image Builder and importing it into Oracle Cloud Infrastructure is a reliable way to standardize Linux deployments. By using QCOW2 images and OCI Custom Images, organizations can achieve scalable, secure, and repeatable infrastructure builds.

Malware protection on Linux servers is often overlooked, especially in enterprise environments where systems host critical applications, databases, and shared storage. While Linux is inherently secure, it is not immune to malware, ransomware, or infected files introduced through users, shared mounts, or file transfers.

ClamAV is a widely adopted open-source antivirus engine designed for Linux and Unix-based systems. This article provides a step-by-step, production-ready guide to installing and configuring ClamAV on Oracle Linux 9 (OL9) using the upstream RPM, with on-access scanning, SELinux support, automatic updates, quarantine handling, and scheduled scans.

What is ClamAV and Why Use It?

ClamAV is an open-source antivirus toolkit primarily used on Linux systems to:

• Detect malware, trojans, and ransomware

• Scan uploaded or shared files

• Provide on-access (real-time) malware protection

• Protect Linux servers that interact with Windows clients

Key Features of ClamAV

• Signature-based malware detection

• On-demand and on-access scanning

• Automatic virus definition updates (freshclam)

• Lightweight and server-friendly

• SELinux-compatible

• CLI-driven (ideal for automation and cron jobs)

In enterprise Linux environments, ClamAV is commonly deployed to:

• Scan /home directories

• Protect shared mounts

• Meet compliance and security baseline requirements

• Prevent malware propagation across platforms

Architecture Overview

A standard ClamAV deployment consists of:

This guide configures all components correctly for OL9 with SELinux enforcing.

Prerequisites

Before starting, verify:

• Oracle Linux 9.x

• SELinux in Enforcing mode

• Root or sudo access

• Internet connectivity for virus updates

Verify:

cat /etc/os-release
getenforce

Step 1: Download and Install ClamAV (Upstream RPM)

Oracle Linux repositories often lag behind upstream ClamAV releases. For security and stability, install the official upstream RPM.

cd /tmp
wget https://www.clamav.net/downloads/production/clamav-1.5.1.linux.x86_64.rpm
sudo dnf install -y ./clamav-1.5.1.linux.x86_64.rpm

Verify installation:

rpm -q clamav

Step 2: Create Required System Users

The upstream RPM does not create service users automatically.
ClamAV separates responsibilities using two system accounts:

• clamscan → scanning daemon

• clamupdate → virus database updates

Create users:

sudo useradd -r -s /sbin/nologin -d /var/lib/clamav clamscan
sudo useradd -r -s /sbin/nologin -d /var/lib/clamav clamupdate

Verify:

id clamscan
id clamupdate

Step 3: Create Required Directories

Create directories for:

• Virus databases

• Runtime sockets

• PID files

• Logs

sudo mkdir -p \
  /usr/local/share/clamav \
  /var/log/clamav \
  /run/clamd \
  /run/clamav

Set ownership:

sudo chown -R clamupdate:clamupdate /usr/local/share/clamav /run/clamav
sudo chown -R clamscan:clamscan /run/clamd

Set permissions:

sudo chmod 755 /var/log/clamav /run/clamd

⚠️ Incorrect permissions are the most common reason ClamAV fails to start.

Step 4: Create Log Files Manually (CRITICAL)

ClamAV will not create log files automatically.

sudo touch /var/log/clamav/clamd.log
sudo touch /var/log/clamav/freshclam.log

Set ownership:

sudo chown clamscan:clamscan /var/log/clamav/clamd.log
sudo chown clamupdate:clamupdate /var/log/clamav/freshclam.log

Set permissions:

sudo chmod 640 /var/log/clamav/*.log
sudo chmod 755 /var/log/clamav

Step 5: Configure freshclam (Virus Updates)

Copy the sample configuration:

sudo cp /usr/local/etc/freshclam.conf.sample /usr/local/etc/freshclam.conf
sudo vi /usr/local/etc/freshclam.conf

Use only the following content:

DatabaseDirectory /usr/local/share/clamav
UpdateLogFile /var/log/clamav/freshclam.log
PidFile /run/clamav/freshclam.pid
DatabaseMirror database.clamav.net

Important:
Remove the Example line completely.

Set ownership:

sudo chown clamupdate:clamupdate /usr/local/etc/freshclam.conf

Step 6: Configure clamd (Scanning Daemon)

Copy sample file:

sudo cp /usr/local/etc/clamd.conf.sample /usr/local/etc/clamd.conf
sudo vi /usr/local/etc/clamd.conf

Minimal Production Configuration

DatabaseDirectory /usr/local/share/clamav

LogFile /var/log/clamav/clamd.log
LogTime yes

LocalSocket /run/clamd/clamd.sock
LocalSocketMode 666
PidFile /run/clamd/clamd.pid

User root
Foreground yes

# On-access scanning
OnAccessIncludePath /home
OnAccessExcludeRootUID yes
OnAccessPrevention yes

# Mandatory exclusions
OnAccessExcludePath ^/proc
OnAccessExcludePath ^/sys
OnAccessExcludePath ^/run
OnAccessExcludePath ^/dev
OnAccessExcludePath ^/var/lib
OnAccessExcludePath ^/var/log
OnAccessExcludePath ^/tmp

# Performance
MaxQueue 200
MaxThreads 20
OnAccessMaxThreads 10

Remove the Example line.

Set ownership:

sudo chown clamscan:clamscan /usr/local/etc/clamd.conf

Step 7: SELinux Configuration (MANDATORY)

Allow antivirus scanning in SELinux enforcing mode:

sudo restorecon -Rv /var/log/clamav /run/clamd
sudo setsebool -P antivirus_can_scan_system 1

Step 8: Download Virus Definitions (First Time)

sudo -u clamupdate /usr/local/bin/freshclam

Verify:

ls -lh /usr/local/share/clamav

total 108M
-rw-r--r--. 1 clamupdate clamupdate 8.9K Dec 17 21:18 bytecode-339.cvd.sign
-rw-r--r--. 1 clamupdate clamupdate 276K Dec 17 21:18 bytecode.cvd
-rw-r--r--. 1 clamupdate clamupdate 8.9K Dec 17 21:18 daily-27853.cvd.sign
-rw-r--r--. 1 clamupdate clamupdate  23M Dec 17 21:17 daily.cvd
-rw-r--r--. 1 clamupdate clamupdate   90 Dec 17 21:17 freshclam.dat
-rw-r--r--. 1 clamupdate clamupdate 8.9K Dec 17 21:18 main-63.cvd.sign
-rw-r--r--. 1 clamupdate clamupdate  85M Dec 17 21:18 main.cvd

You should see:

• daily.cvd

• main.cvd

• bytecode.cvd

Step 9: Create systemd Service Files

clamd.service

sudo vi /etc/systemd/system/clamd.service

[Unit]
Description=ClamAV Daemon
After=network.target

[Service]
Type=simple
User=clamscan
Group=clamscan
ExecStart=/usr/local/sbin/clamd --config-file=/usr/local/etc/clamd.conf --foreground
Restart=on-failure
RestartSec=10
RuntimeDirectory=clamd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

Clamav-freshclam.service

sudo vi /etc/systemd/system/clamav-freshclam.service

[Unit]
Description=ClamAV Virus Database Updater
After=network.target

[Service]
Type=oneshot
User=clamupdate
Group=clamupdate
ExecStart=/usr/local/bin/freshclam

[Install]
WantedBy=multi-user.target

Clamonacc.service (On-Access Scanner)

sudo vi /etc/systemd/system/clamonacc.service

[Unit]
Description=ClamAV On-Access Scanner
After=clamd.service
Requires=clamd.service

[Service]
Type=simple
ExecStart=/usr/local/sbin/clamonacc --foreground --fdpass
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Step 10: Enable and Start Services

sudo systemctl daemon-reexec
sudo systemctl daemon-reload

sudo systemctl enable clamd clamav-freshclam clamonacc
sudo systemctl start clamd
sudo systemctl start clamav-freshclam
sudo systemctl start clamonacc

Step 11: Verify Services

systemctl is-active clamd
systemctl is-active clamonacc
systemctl status clamav-freshclam

Expected:

• clamd → active

• clamonacc → active

• freshclam → inactive (0/SUCCESS)

Step 12: Malware Validation (EICAR Test)

[opc@#### ~]$ wget https://secure.eicar.org/eicar_com.zip
--2025-12-17 22:27:37--  https://secure.eicar.org/eicar_com.zip
Resolving secure.eicar.org (secure.eicar.org)... 89.238.73.97, 2a00:1828:1000:2497::2
Connecting to secure.eicar.org (secure.eicar.org)|89.238.73.97|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 184 [application/zip]
Saving to: ‘eicar_com.zip’

eicar_com.zip                    100%[=========================================================>]     184  --.-KB/s    in 0s

2025-12-17 22:27:38 (3.26 MB/s) - ‘eicar_com.zip’ saved [184/184]

[opc@#### ~]$ unzip eicar_com.zip
error:  cannot open zipfile [ eicar_com.zip ]
        Operation not permitted
unzip:  cannot find or open eicar_com.zip, eicar_com.zip.zip or eicar_com.zip.ZIP.
[opc@OHS ~]$

Expected behavior:

• Root can read the file (expected)

• Non-root users are blocked

• Detection logged in clamd.log

Check logs:

[opc@#### ~]$ journalctl -u clamonacc | tail
Dec 17 22:04:10 OHS clamonacc[4297]: ERROR: ClamClient: Could not connect to clamd, Could not connect to server
Dec 17 22:04:10 OHS clamonacc[4297]: ERROR: Clamonacc: daemon is local, but a connection could not be established
Dec 17 22:04:10 OHS systemd[1]: clamonacc.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 17 22:04:10 OHS systemd[1]: clamonacc.service: Failed with result 'exit-code'.
Dec 17 22:04:15 OHS systemd[1]: clamonacc.service: Scheduled restart job, restart counter is at 3.
Dec 17 22:04:15 OHS systemd[1]: Stopped ClamAV On-Access Scanner.
Dec 17 22:04:15 OHS systemd[1]: Started ClamAV On-Access Scanner.
Dec 17 22:04:16 OHS clamonacc[4726]: ClamInotif: watching '/home' (and all sub-directories)
Dec 17 22:05:44 OHS clamonacc[4726]: /home/opc/eicar_com.zip: Eicar-Test-Signature FOUND
Dec 17 22:27:45 OHS clamonacc[4726]: /home/opc/eicar_com.zip: Eicar-Test-Signature FOUND


[opc@#### ~]$ sudo tail -f /var/log/clamav/clamd.log
Wed Dec 17 22:04:11 2025 -> SWF support enabled.
Wed Dec 17 22:04:11 2025 -> HTML support enabled.
Wed Dec 17 22:04:11 2025 -> XMLDOCS support enabled.
Wed Dec 17 22:04:11 2025 -> HWP3 support enabled.
Wed Dec 17 22:04:11 2025 -> OneNote support enabled.
Wed Dec 17 22:04:11 2025 -> Self checking every 600 seconds.
Wed Dec 17 22:05:44 2025 -> /home/opc/eicar_com.zip: Eicar-Test-Signature FOUND
Wed Dec 17 22:15:44 2025 -> SelfCheck: Database status OK.
Wed Dec 17 22:25:44 2025 -> SelfCheck: Database status OK.
Wed Dec 17 22:27:45 2025 -> /home/opc/eicar_com.zip: Eicar-Test-Signature FOUND

Step 13: Quarantine and Scheduled Scans

Create quarantine directory:

sudo mkdir -p /var/quarantine/clamav
sudo chmod 700 /var/quarantine/clamav

Daily scan example:

/usr/local/bin/clamscan -r --infected \
  --move=/var/quarantine/clamav \
  --log=/var/log/clamav/daily_scan.log \
  /home /Data

Conclusion

This DIY ClamAV setup provides enterprise-grade malware protection on Oracle Linux 9, including:

• Real-time scanning

• SELinux enforcement

• Automated updates

• Safe quarantine handling

When deployed correctly, ClamAV becomes a silent, reliable security layer that protects Linux servers without impacting performance.

• “How can I get a list of all EC2 instances across all AWS regions?”

• “How do I build a Lambda function that scans every region safely?”

• “Why do I get AuthFailure errors when calling DescribeInstances?”

…then this DIY AWS Lambda tutorial is exactly what you need.

In this article, we will walk through creating a fully working Python 3.14 AWS Lambda function that scans every AWS region, safely skips restricted regions, and returns a clean JSON list of all EC2 instances.

This guide is written for operations engineers, cloud admins, DevOps teams, and AWS learners who want a practical scenarios.

Prerequisites

Before you start, make sure you have:

• An AWS account

• IAM permissions to create and run Lambda functions

• Basic knowledge of Python and AWS Console

• Access to CloudWatch Logs

Step 1: Create the Lambda Function

1. Go to AWS Console → Lambda

2. Click Create Function

3. Choose:

   • Author from scratch

   • Runtime: Python 3.14

   • Architecture: x86_64 or ARM64

4. Click Create Function

Step 2: Add Required IAM Permissions

Your Lambda role MUST include these permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeRegions"
      ],
      "Resource": "*"
    }
  ]
}

Additionally, CloudWatch logging permissions:

{
  "Effect": "Allow",
  "Action": [
    "logs:CreateLogGroup",
    "logs:CreateLogStream",
    "logs:PutLogEvents"
  ],
  "Resource": "*"
}

These policies allow Lambda to read EC2 information and write logs.

Step 3: Paste the Python 3.14 EC2-Scanning Lambda Code

This version safely handles restricted regions, preventing common errors like:

AuthFailure: AWS was not able to validate the provided access credentials``UnauthorizedOperation

Fully Working Python 3.14 Code

import json
import boto3
from botocore.exceptions import ClientError

def lambda_handler(event, context):
    ec2list = []
    ec2 = boto3.client('ec2')

    # Get all AWS regions
    regions = ec2.describe_regions(AllRegions=True).get('Regions', [])

    for region in regions:
        reg = region['RegionName']
        print(f"* Checking region -- {reg}")

        try:
            client = boto3.client('ec2', region_name=reg)
            paginator = client.get_paginator('describe_instances')

            for page in paginator.paginate():
                for reservation in page.get("Reservations", []):
                    for instance in reservation.get("Instances", []):
                        ec2list.append({
                            "InstanceId": instance.get("InstanceId"),
                            "Region": reg
                        })

        except ClientError as e:
            # Skip restricted or disabled regions
            if "AuthFailure" in str(e):
                print(f"Skipping region {reg}: Not enabled for this account.")
                continue
            else:
                print(f"Error in region {reg}: {e}")
                continue

    return {
        "statusCode": 200,
        "body": json.dumps(ec2list)
    }

This is currently the best and safest multi-region EC2 discovery Lambda code for Python 3.14.

Step 4: Test the Lambda Function

1. Click Test

2. Choose Create Test Event

3. Use this simple test JSON:

{}

4. Run the test.

You will see logs such as:

Test Event Name
hello-world

Response
{
  "statusCode": 200,
  "body": "[]"
}

Function Logs
START RequestId: c036f21d-83e5-4fa7-b26c-5b286a55ac74 Version: $LATEST
* Checking region -- eu-north-1
* Checking region -- eu-west-3
* Checking region -- eu-west-2
* Checking region -- eu-west-1
* Checking region -- ap-northeast-3
* Checking region -- ap-northeast-2
* Checking region -- me-south-1

Finally, your output will be if any instance is running:

[
  {
    "InstanceId": "i-1234567890abcd",
    "InstanceType": "t3.micro",
    "State": "running",
    "Region": "us-east-1"
  }
]

If your account has no instances, it will return:

[]

Troubleshooting

Error: AuthFailure

This means the region is not enabled for your AWS account.
The provided code already skips these regions safely.

Error: UnauthorizedOperation

You are missing IAM permissions.
Add:

ec2:DescribeInstances
ec2:DescribeRegions

Timeout Errors

Increase Lambda timeout to 30–60 seconds:

Lambda → Configuration → General → Edit → Timeout

How Full Stack DR Creates and Stores Backups

Full Stack DR uses OCI Container Instances to manage both backup and restore operations:

• During backup, a Container Instance is created in the primary region, and a backup container runs inside it to capture Kubernetes resources, images, and metadata.

• The generated backup artifacts are stored securely in OCI Object Storage.

• A log file is also created for every operation, enabling auditability and troubleshooting.

For a Restore Operation:

• A Container Instance is created in the standby region, where a restore container retrieves the previously stored data from Object Storage and restores the cluster state.

This isolated container-driven approach ensures that backup/restore tasks are efficient, secure, and do not interfere with your live cluster.

Scheduling Backup Operations

DR plans support flexible backup frequency options:

• Hourly

• Daily

• Weekly

• Monthly

These schedules enable businesses to design RPO/RTO values that match their compliance and operational needs.

Image Replication for DR

Full Stack DR automatically replicates private OCIR images used by workloads in the primary OKE cluster:

• Only private images can be replicated.

• Public Docker Hub or public OCIR images cannot be replicated as part of Full Stack DR.

• Users may choose to supply a custom image replication secret stored in OCI Vault instead of the default.

This ensures that all required images exist in the standby region, enabling applications to start smoothly after failover.

Node Pool Scaling During DR

Full Stack DR allows you to define node pool scaling actions, which can automatically adjust the number of nodes during failover or switchover events.
You can scale up or down each node pool based on the expected workload load in the standby region.

Instance Jump Host (API Access Instance)

Full Stack DR requires access to the OKE cluster’s API endpoint to execute backup or recovery tasks.

• You can specify an existing Instance Jump Host, which must have network access to the cluster's public or private API endpoint.

• If you do not provide an instance, Full Stack DR will automatically create an ephemeral Container Instance to handle API operations.

• Using a dedicated jump host is recommended for stable and controlled access.

This ensures consistent connectivity during DR operations, especially in private or restricted networks.

Load Balancer Mapping

If your workloads use the OCI Native Ingress Controller, Full Stack DR requires mapping each primary region Load Balancer with a corresponding Load Balancer in the standby region.

This establishes consistent routing in the restored environment and ensures services remain accessible post-failover.

Vault Mapping for Secrets

If your applications store Kubernetes secrets in OCI Vault, Full Stack DR supports:

• Mapping primary-region vaults to standby-region vaults

• Enabling vault replication

• Or manually copying secrets to the standby vault

This keeps all secret data synchronized so restored applications function without manual fixes.

Namespace Backup Policies

You have flexible options for selecting namespaces during backup:

• Include all namespaces

• Include specific namespaces

• Exclude selected namespaces

A maximum of 32 namespaces can be selected for fine-grained control.

Conclusion

Oracle Full Stack DR for OKE provides a comprehensive, container-driven approach to Kubernetes disaster recovery. By leveraging Container Instances, Object Storage, image replication, load balancer mapping, and vault replication, it ensures a smooth and predictable failover process.

From scheduled backups to automated scaling and image synchronization, Full Stack DR eliminates complexity and helps organizations maintain business continuity with confidence.

In this article, i used Skopeo a powerful image management tool to migrate entire repositories, including all tags, from ECR to OCIR. This guide also includes a practical script that uses Docker login + Skopeo for secure image transfer.

What is Skopeo?

Skopeo is an open-source CLI tool that performs operations on container registries without needing Docker or Podman running in the background.

Why Skopeo?

• No local image pull needed – copies images directly from registry to registry

• Fast layer transfers – only missing layers are pushed

• Daemonless – reduces overhead, avoids Docker dependency

• Supports all major registries – AWS ECR, OCIR, GCR, DockerHub, Quay, GitHub Container Registry, etc.

• Secure authentication – works with AWS IAM token and OCIR auth token

Because of these advantages, Skopeo is the best tool for migrating images across cloud providers.

Migrating ECR to OCIR Using Skopeo

Below is the method I used:
✔ Authenticate to AWS ECR using Docker
✔ Authenticate to OCIR using Docker
✔ Get all tags from the ECR repository
✔ Loop through each tag
✔ Copy from ECR → OCIR using Skopeo

This is a recommended and production-ready approach.

1. Authenticate to AWS ECR

AWS ECR uses temporary auth tokens. We log in using Docker:

aws ecr get-login-password --region $AWS_REGION | \
docker login --username AWS --password-stdin $ECR_DOMAIN

• $AWS_REGION – Example: us-east-1

• $ECR_DOMAIN – Example: 896077038029.dkr.ecr.us-east-1.amazonaws.com

2. Authenticate to OCIR

OCIR requires an auth token, not your console password.

docker login $OCI_REGISTRY \
  --username "${OCI_NAMESPACE}/oracleidentitycloudservice/${OCIR_EMAIL}" \
  --password "${OCIR_TOKEN}"

Where:

• $OCI_REGISTRY → ap-sydney-1.ocir.io

• $OCI_NAMESPACE → Your tenancy namespace

• $OCIR_EMAIL → Your login email

• $OCIR_TOKEN → OCIR auth token

3. Fetch All Image Tags from the ECR Repository

TAGS=$(aws ecr list-images \
    --region $AWS_REGION \
    --repository-name $ECR_REPO \
    --query 'imageIds[*].imageTag' \
    --output text)

This captures all tags so every version of your image is copied.

4. Loop Through Tags and Copy Images Using Skopeo

Below is your exact logic, written in clean article format:

for TAG in $TAGS; do
    if [ "$TAG" == "None" ]; then
        echo "Skipping untagged image in $ECR_REPO"
        continue
    fi

    SRC="docker://${ECR_DOMAIN}/${ECR_REPO}:${TAG}"
    DEST="docker://${OCI_REGISTRY}/${OCI_NAMESPACE}/${OCI_REPO}:${TAG}"

    echo "Copying $SRC → $DEST"

    skopeo copy --all \
      --src-creds "AWS:$(aws ecr get-login-password --region $AWS_REGION)" \
      --dest-creds "$OCIR_CREDS" \
      "$SRC" "$DEST"
done

What happens here?

• --src-creds authenticates to AWS ECR using fresh token

• --dest-creds authenticates to OCIR

• The --all flag ensures all image manifests and multi-arch data are copied

• Skopeo moves layers directly registry → registry without local storage

Why This Approach Is Effective

This method is commonly used in cloud migrations, DR planning, and multi-cloud container strategies.

Conclusion

Migrating container images from AWS ECR to OCI Container Registry (OCIR) is simple and efficient when using Skopeo. Its ability to copy images directly between registries combined with Docker authenticationmakes it perfect for large migrations where reliability, speed, and automation are essential.

The OKE Cluster Autoscaler supports two authentication methods: Instance Principals and Workload Identity Principals. Instance Principal means the autoscaler uses the identity of the OCI compute instance it runs on, requiring no secrets and offering the simplest, most secure setup.

Workload Identity Principal uses the identity of a Kubernetes workload, allowing more granular access but requiring additional configuration.

In this article, we configured and tested the autoscaler using Instance Principals.
With this setup, your OKE cluster can automatically scale up and down based on workload demand, improving efficiency while reducing operational overhead.

Scales Up (adds nodes) when:

• There are pending pods that cannot be scheduled
  (e.g., not enough CPU/memory on existing nodes).

• Autoscaler requests OCI to create new worker nodes in the node pool.h

Scales Down (removes nodes) when:

• Nodes are under-utilized for a long time
  (default = 10 minutes unless changed).

• No critical pods are running on that node.

• Autoscaler safely drains the node and deletes it from OCI.

STEP 1 — Create Dynamic Group for OKE Nodes

Go to: IAM → Dynamic Groups → Create Dynamic Group

Example: oke-nodepool-dg

Rule (recommended):

Any {instance.compartment.id = '<COMPARTMENT_OCID>'}

STEP 2 — Create IAM Policies

Go to: IAM → Policies → Create Policy

Choose the compartment where your OKE node pool exists.

Paste the required policy:

Allow dynamic-group <dynamic-group-name> to manage cluster-node-pools in compartment <compartment-name>
Allow dynamic-group <dynamic-group-name> to manage instance-family in compartment <compartment-name>
Allow dynamic-group <dynamic-group-name> to use subnets in compartment <compartment-name>
Allow dynamic-group <dynamic-group-name> to read virtual-network-family in compartment <compartment-name>
Allow dynamic-group <dynamic-group-name> to use vnics in compartment <compartment-name>
Allow dynamic-group <dynamic-group-name> to inspect compartments in compartment <compartment-name>

This allows nodes to scale their own node pool.

STEP 3 — Deploy OKE Cluster Autoscaler using Add-On

• Go to OKE → Cluster → Add-Ons

• Select Cluster Autoscaler → Enable

• Set Replicas = 3

• Add node pool scaling config:

    <min>:<max>:<NODEPOOL_OCID>
  

  Example:

    3:5:ocid1.nodepool.oc1...
  

• Save the Add-On

• Verify pods:

    [root@jump-host ~]# kubectl -n kube-system get pods | grep autoscaler
    cluster-autoscaler-64bf849b78-hxqm7   1/1     Running   0          11d
    cluster-autoscaler-64bf849b78-jd8vs   1/1     Running   0          11d
    cluster-autoscaler-64bf849b78-t4fzh   1/1     Running   0          11d
  

  Deploy OKE Cluster Autoscaler (Manual YAML Method)

• In a text editor, create a file called cluster-autoscaler.yaml with the following content:

•   ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        k8s-addon: cluster-autoscaler.addons.k8s.io
        k8s-app: cluster-autoscaler
      name: cluster-autoscaler
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: cluster-autoscaler
      labels:
        k8s-addon: cluster-autoscaler.addons.k8s.io
        k8s-app: cluster-autoscaler
    rules:
      - apiGroups: [""]
        resources: ["events", "endpoints"]
        verbs: ["create", "patch"]
      - apiGroups: [""]
        resources: ["pods/eviction"]
        verbs: ["create"]
      - apiGroups: [""]
        resources: ["pods/status"]
        verbs: ["update"]
      - apiGroups: [""]
        resources: ["endpoints"]
        resourceNames: ["cluster-autoscaler"]
        verbs: ["get", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["watch", "list", "get", "patch", "update"]
      - apiGroups: [""]
        resources:
          - "pods"
          - "services"
          - "replicationcontrollers"
          - "persistentvolumeclaims"
          - "persistentvolumes"
        verbs: ["watch", "list", "get"]
      - apiGroups: ["extensions"]
        resources: ["replicasets", "daemonsets"]
        verbs: ["watch", "list", "get"]
      - apiGroups: ["policy"]
        resources: ["poddisruptionbudgets"]
        verbs: ["watch", "list"]
      - apiGroups: ["apps"]
        resources: ["statefulsets", "replicasets", "daemonsets"]
        verbs: ["watch", "list", "get"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses", "csinodes", "volumeattachments"]
        verbs: ["watch", "list", "get"]
      - apiGroups: ["batch", "extensions"]
        resources: ["jobs"]
        verbs: ["get", "list", "watch", "patch"]
      - apiGroups: ["coordination.k8s.io"]
        resources: ["leases"]
        verbs: ["create"]
      - apiGroups: ["coordination.k8s.io"]
        resourceNames: ["cluster-autoscaler"]
        resources: ["leases"]
        verbs: ["get", "update"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["watch", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["csidrivers", "csistoragecapacities"]
        verbs: ["watch", "list"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: cluster-autoscaler
      namespace: kube-system
      labels:
        k8s-addon: cluster-autoscaler.addons.k8s.io
        k8s-app: cluster-autoscaler
    rules:
      - apiGroups: [""]
        resources: ["configmaps"]
        verbs: ["create","list","watch"]
      - apiGroups: [""]
        resources: ["configmaps"]
        resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
        verbs: ["delete", "get", "update", "watch"]
  
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: cluster-autoscaler
      labels:
        k8s-addon: cluster-autoscaler.addons.k8s.io
        k8s-app: cluster-autoscaler
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-autoscaler
    subjects:
      - kind: ServiceAccount
        name: cluster-autoscaler
        namespace: kube-system
  
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: cluster-autoscaler
      namespace: kube-system
      labels:
        k8s-addon: cluster-autoscaler.addons.k8s.io
        k8s-app: cluster-autoscaler
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: cluster-autoscaler
    subjects:
      - kind: ServiceAccount
        name: cluster-autoscaler
        namespace: kube-system
  
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: cluster-autoscaler
      namespace: kube-system
      labels:
        app: cluster-autoscaler
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: cluster-autoscaler
      template:
        metadata:
          labels:
            app: cluster-autoscaler
          annotations:
            prometheus.io/scrape: 'true'
            prometheus.io/port: '8085'
        spec:
          serviceAccountName: cluster-autoscaler
          containers:
            - image: iad.ocir.io/oracle/oci-cluster-autoscaler:{{ image tag }}
              name: cluster-autoscaler
              resources:
                limits:
                  cpu: 100m
                  memory: 300Mi
                requests:
                  cpu: 100m
                  memory: 300Mi
              command:
                - ./cluster-autoscaler
                - --v=0
                - --stderrthreshold=info
                - --cloud-provider=oci
                - --scale-down-enabled=true
                - --scale-down-delay-after-add=10m
                - --scale-down-delay-after-delete=10s
                - --scale-down-delay-after-failure=3m
                - --scale-down-unneeded-time=10m
                - --scale-down-unready-time=20m
                - --scale-down-utilization-threshold=0.5
                - --scale-down-non-empty-candidates-count=30
                - --scale-down-candidates-pool-ratio=0.1
                - --scale-down-candidates-pool-min-count=50
                - --scan-interval=10s
                - --max-nodes-total=0
                - --cores-total=0:320000
                - --memory-total=0:6400000
                - --max-graceful-termination-sec=600
                - --max-total-unready-percentage=45
                - --ok-total-unready-count=3
                - --max-node-provision-time=15m
                - --nodes=3:5:{{ node pool ocid 1 }}
                - --emit-per-nodegroup-metrics=false
                - --estimator=binpacking
                - --expander=random
                - --ignore-daemonsets-utilization=false
                - --ignore-mirror-pods-utilization=false
                - --write-status-configmap=true
                - --status-config-map-name=cluster-autoscaler-status
                - --max-inactivity=10m
                - --max-failing-time=15m
                - --balance-similar-node-groups=false
                - --unremovable-node-recheck-timeout=5m
                - --expendable-pods-priority-cutoff=-10
                - --daemonset-eviction-for-empty-nodes=false
                - --daemonset-eviction-for-occupied-nodes=true
                - --cordon-node-before-terminating=false
                - --record-duplicated-events=false
                - --max-nodes-per-scaleup=1000
                - --new-pod-scale-up-delay=0s
                - --max-scale-down-parallelism=10
                - --max-bulk-soft-taint-count=10
                - --max-pod-eviction-time=2m0s
                - --debugging-snapshot-enabled=false
                - --enforce-node-group-min-size=false
                - --skip-nodes-with-system-pods=true
                - --skip-nodes-with-local-storage=true
                - --min-replica-count=0
                - --skip-nodes-with-custom-controller-pods=true
              imagePullPolicy: "Always"
  

  Important Note

  • --scale-down-unneeded-time=10m

  • --scan-interval=10s

  • --scale-down-delay-after-add=10m

After a new node is added, the autoscaler waits 10 minutes before checking for scale-down.
Then the node must remain unused for another 10 minutes before being removed.

• Worst-case scale-down time:

  10m (delay-after-add) + 10m (unneeded-time) = 20 minutes.

STEP 4 — Deploy Test Workload & Trigger Autoscaler Scaling

1️⃣ Create the NGINX deployment

Apply the following manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: syd.ocir.io/#######/ocir-repo:nginx-latest
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "500Mi"
        imagePullPolicy: Always

Apply it:

kubectl apply -f nginx.yaml

2️⃣ Scale the deployment to create load

Increase replicas to 50 to force scheduling pressure:

kubectl scale deploy nginx-deployment --replicas=50

This will create pending pods → autoscaler should scale UP your node pool.

3️⃣ Observe Autoscaler events (scale up / scale down)

Run this command to watch autoscaler decisions live:

for p in $(kubectl get pods -n kube-system -l app=cluster-autoscaler -o name); do
  echo "=== $p ==="
  kubectl logs -n kube-system $p --since=1h | grep -Ei 'scale[- ]?up|scale[- ]?down' || true
done

Observe Autoscaler events (scale up)

Observe Autoscaler events (scale Down)

Scale the deployment down to 3 replicas to intentionally create scheduling pressure.

kubectl scale deploy nginx-deployment --replicas=3

In this article, we configured the OKE Cluster Autoscaler, deployed test workloads, and validated scale-up/scale-down events. With this setup, your cluster will continuously maintain the right size while reducing manual operational tasks.

Unlike traditional CNIs (like Flannel, Calico, or Weave) that rely on iptables for packet forwarding and filtering, Cilium uses eBPF to dynamically inject logic into the kernel’s networking stack. This enables faster performance, fine-grained visibility, and deep network security enforcement all without the limitations of legacy Linux networking mechanisms.

Cilium can completely replace:

• Flannel :- as the CNI plugin for pod networking

• kube-proxy :- as the Kubernetes Service load-balancer

• NetworkPolicy :- engines — by enforcing security policies using eBPF

• Monitoring tools :- through its built-in observability layer, Hubble

Prerequisites

• Access to an OKE cluster (v1.27 or later recommended)

• kubectl configured to access your Jump Server or CloudsShell

• helm installed (v3.8+)

• Administrator privileges in the cluster

Step 0 – Disable OKE Addon Flannel

OKE automatically provisions Flannel as the default CNI.
Before installing Cilium, you must disable this addon and remove the existing DaemonSet.

Run the following commands:

# Disable Flannel addon from OKE console

# Remove the Flannel DaemonSet
kubectl delete ds flannel -n kube-system

This ensures Flannel pods are deleted and networking configuration is ready for Cilium to take over.

Step 1 – Add the Cilium Helm Repository

Add and update the official Cilium Helm chart repository:

helm repo add cilium https://helm.cilium.io/
helm repo update

You can verify available versions with:

helm search repo cilium/cilium --versions

Step 2 – Install Cilium CNI

Install latest Cilium version 1.18.3 using Helm:

helm install cilium cilium/cilium \
  --version 1.18.3 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set kubeProxyReplacementHealthzBindAddr="0.0.0.0:10256" \
  --set ipam.mode=kubernetes \
  --set ipv4.enabled=true \
  --set cluster.name=oke-cilium \
  --set k8sServiceHost=$(kubectl config view \
      --minify -o jsonpath='{.clusters[0].cluster.server}' \
      | sed 's#https://##;s#:.*##') \
  --set k8sServicePort=6443 \
  --set cluster.podCIDRList="{10.230.0.0/16}" \
  --set cluster.serviceCIDR="10.96.0.0/16"

Explanation

After installation, Cilium will replace both Flannel (CNI) and kube-proxy at the dataplane level.

Step 3 – Enable Hubble Relay, UI, and Metrics

Hubble provides powerful observability for network flows and policies in your cluster.

Upgrade the Helm release to enable Hubble features:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set bandwidthManager.enabled=true \
  --set hubble.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.ui.service.type=LoadBalancer \
  --set hubble.relay.service.type=ClusterIP \
  --set hubble.relay.resources.limits.cpu=200m \
  --set hubble.relay.resources.limits.memory=256Mi \
  --set hubble.relay.resources.requests.cpu=100m \
  --set hubble.relay.resources.requests.memory=128Mi \
  --set policyEnforcementMode=default

Step 4 – Verify Cilium Installation

Check Pod Status

kubectl -n kube-system get pods -l k8s-app=cilium
kubectl -n kube-system get pods -l k8s-app=hubble-relay
kubectl -n kube-system get pods -l k8s-app=hubble-ui

All pods should be in the Running state.

Check Cilium Agent Health

kubectl -n kube-system exec -ti ds/cilium -- cilium status

Expected output (summary):

Kubernetes:              Ok         1.34 (v1.34.1) [linux/arm64]
Cilium:                  Ok   1.18.3 (v1.18.3-c1601689)
Cilium health daemon:    Ok
Proxy Status:            OK, ip 10.230.1.88, 0 redirects active on ports 10000-20000, Envoy: external
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 49.54   Metrics: Ok

Step 5 – Verify Kube-Proxy Replacement

Cilium replaces kube-proxy by handling Kubernetes Service routing via eBPF.
Verify this at both deployment and runtime levels.

Deployment Level (Helm Values)

helm get values cilium -n kube-system | grep kubeProxyReplacement

Expected output:

kubeProxyReplacement: true

Runtime Level (eBPF Map)

kubectl -n kube-system exec -ti ds/cilium -- cilium bpf lb list

If you see ClusterIP, NodePort, or LoadBalancer entries, Cilium’s eBPF load balancer is active.

Example output:

10.96.0.10:53/UDP -> 10.230.0.5:53/UDP
0.0.0.0:30001/TCP -> 10.230.0.15:8080/TCP

Step 6 – Verify Hubble Functionality

Check Hubble components:

kubectl -n kube-system get pods -l k8s-app=hubble-relay
kubectl -n kube-system get pods -l k8s-app=hubble-ui

Check the Hubble UI service:

kubectl get svc -n kube-system | grep hubble-ui

Then open:

http://<LoadBalancer>

You’ll see a real-time topology of all pod-to-pod communication in your cluster.This confirms Cilium is replacing kube-proxy at the kernel level.

Step 7 - Allow Hubble to Capture All Flows

By default, Cilium aggregates network-flow events to reduce load.
To allow Hubble to record every individual flow, disable aggregation:

kubectl -n kube-system set env daemonset/cilium \
  CILIUM_MONITOR_AGGREGATION=none \
  CILIUM_MONITOR_AGGREGATION_INTERVAL=5s

This ensures fine-grained visibility for debugging, audit, and metrics.

Step 8 - Validation Checklist

Overview

n8n (“nodemation”) is an open-source workflow automation tool that lets you visually connect services and automate tasks.

Oracle Kubernetes Engine (OKE) is a managed Kubernetes service on OCI that provides scalable, highly available clusters with integrated networking and load balancing.

In this guide, you’ll:

• Create a namespace and persistent volume for n8n data

• Deploy n8n to OKE

• Expose it through an OCI LoadBalancer

• Fix permission and cookie issues for production readiness

Prerequisites

Before you begin, ensure you have:

• An existing OKE cluster (v1.34.1 or newer)

• kubectl configured to connect to the cluster (kubectl get nodes works)

• One public subnet OCID (for the OCI Load Balancer)

• Basic understanding of Kubernetes YAML manifests

Step 1 — Create the Namespace and Persistent Volume Claim

Create a namespace for organizational isolation and a PVC to persist n8n configuration and workflows.

apiVersion: v1
kind: Namespace
metadata:
  name: n8n
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: n8n-pvc
  namespace: n8n
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Apply it:

kubectl apply -f n8n-storage.yaml

Step 2 — Create Authentication Secret

Create a Kubernetes secret for n8n’s basic authentication.

apiVersion: v1
kind: Secret
metadata:
  name: n8n-secret
  namespace: n8n
type: Opaque
stringData:
  username: admin
  password: strongpassword123

Apply it:

kubectl apply -f n8n-secret.yaml

Step 3 — Deploy n8n Application

Below is a production-ready Deployment that:

• Uses the fully qualified Docker image

• Mounts the PVC for persistence

• Fixes file-permission issues (fsGroup: 1000)

• Disables secure cookie enforcement (for HTTP access via LoadBalancer)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n
  namespace: n8n
spec:
  replicas: 1
  selector:
    matchLabels:
      app: n8n
  template:
    metadata:
      labels:
        app: n8n
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
      containers:
        - name: n8n
          image: docker.io/n8nio/n8n:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5678
          env:
            - name: N8N_BASIC_AUTH_ACTIVE
              value: "true"
            - name: N8N_BASIC_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: n8n-secret
                  key: username
            - name: N8N_BASIC_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: n8n-secret
                  key: password
            - name: N8N_HOST
              value: "n8n"
            - name: N8N_PORT
              value: "5678"
            - name: N8N_PROTOCOL
              value: "http"
            - name: N8N_SECURE_COOKIE
              value: "false"
            - name: GENERIC_TIMEZONE
              value: "Asia/Kolkata"
            - name: TZ
              value: "Asia/Kolkata"
          volumeMounts:
            - name: n8n-data
              mountPath: /home/node/.n8n
      volumes:
        - name: n8n-data
          persistentVolumeClaim:
            claimName: n8n-pvc

Apply it:

kubectl apply -f n8n-deployment.yaml

Step 4 — Expose n8n via OCI LoadBalancer

Now expose n8n externally so you can access it through a public IP.

apiVersion: v1
kind: Service
metadata:
  name: n8n-service
  namespace: n8n
spec:
  type: LoadBalancer
  selector:
    app: n8n
  ports:
    - name: http
      port: 80
      targetPort: 5678
      protocol: TCP

Apply it:

kubectl apply -f n8n-service.yaml

After a few minutes:

kubectl get svc -n n8n
NAME            TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
n8n-service     LoadBalancer   10.96.123.45   140.238.xxx.xxx  80:31234/TCP   2m

Step 5 — Verify and Troubleshoot

Check pod logs:

kubectl logs -f -n n8n deploy/n8n

Expected startup log:

Editor is now accessible:
 ▸  http://0.0.0.0:5678/

If you see permission errors like EACCES: permission denied, confirm your fsGroup: 1000 is present in the YAML.

If you get a “secure cookie” warning, ensure N8N_SECURE_COOKIE=false is set, or enable HTTPS on your LoadBalancer.

Step 6 — (Optional) Add HTTPS with OCI LoadBalancer

For production, you can enable HTTPS by adding these annotations to the Service:

metadata:
  annotations:
    service.beta.kubernetes.io/oci-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/oci-load-balancer-tls-secret: "n8n/n8n-tls"

Then create a TLS secret:

kubectl create secret tls n8n-tls \
  --cert=server.crt --key=server.key -n n8n

Step 7 — (Optional) Use PostgreSQL for Production

For better performance and reliability, you can connect n8n to an external PostgreSQL (or OCI Autonomous Database):

- name: DB_TYPE
  value: "postgresdb"
- name: DB_POSTGRESDB_HOST
  value: "<your-db-host>"
- name: DB_POSTGRESDB_USER
  value: "<your-db-user>"
- name: DB_POSTGRESDB_PASSWORD
  value: "<your-db-password>"
- name: DB_POSTGRESDB_DATABASE
  value: "n8n"

Summary

In this article, we’ll create:

• Single Bucket without lifecycle

• Single Bucket with lifecycle rules

• Multiple Buckets using a Terraform loop

Prerequisites

Before you begin, make sure you have:

• An OCI tenancy with Object Storage access

• Compartment OCID

• Terraform ≥ 1.5

• OCI provider ≥ 7.3.0

• Configured OCI CLI or API keys

1. Single Bucket Without Lifecycle

(main.tf)

provider "oci" {
  tenancy_ocid     = var.tenancy_ocid
  user_ocid        = var.user_ocid
  fingerprint      = var.fingerprint
  private_key_path = var.private_key_path
  region           = var.region
}

resource "oci_objectstorage_bucket" "bucket" {
  compartment_id = var.compartment_ocid
  name           = var.bucket_name
  namespace      = var.namespace
  storage_tier   = var.storage_tier
}

Define Variables (variables.tf)

variable "tenancy_ocid" {}
variable "user_ocid" {}
variable "fingerprint" {}
variable "private_key_path" {}
variable "region" {}

variable "compartment_ocid" {}
variable "bucket_name" {}
variable "namespace" {}
variable "storage_tier" {
  default = "Standard" # Options: Standard or Archive
}

Provide Variable Values (terraform.tfvars)

private_key_path = "C:/Users/Pratik N Borkar/.oci/xxxx.pem"
user_ocid        = "ocid1.user.oc1..aaaaabmza"
fingerprint      = "3a:d9f:71"
tenancy_ocid     = "ocid1.tenancy.ookmjm7dq"
region           = "ap-sydney-1"


compartment_ocid  = "ocid1.compartr2vdqa"
bucket_name      = "example-bucket"
namespace        = "XXXXXXX"

2. Multiple Bucket Without Lifecycle

(main.tf)

provider "oci" {
  tenancy_ocid     = var.tenancy_ocid
  user_ocid        = var.user_ocid
  fingerprint      = var.fingerprint
  private_key_path = var.private_key_path
  region           = var.region
}

data "oci_objectstorage_namespace" "ns" {
  compartment_id = var.compartment_ocid
}

locals {
  bucket_names = [for i in range(1, var.bucket_count + 1) : format("AIS-%03d", i)]
}

resource "oci_objectstorage_bucket" "bucket" {
  for_each = toset(local.bucket_names)

  compartment_id     = var.compartment_ocid
  name               = each.value
  namespace          = data.oci_objectstorage_namespace.ns.namespace
  storage_tier       = var.storage_tier
}

Define Variables (variables.tf)

variable "tenancy_ocid" {
  type = string
}

variable "user_ocid" {
  type = string
}

variable "fingerprint" {
  type = string
}

variable "private_key_path" {
  type = string
}

variable "region" {
  type = string
}

variable "compartment_ocid" {
  type = string
}

variable "storage_tier" {
  type    = string
  default = "Standard"
}

variable "bucket_count" {
  type    = number
  default = 10
}

Provide Variable Values (terraform.tfvars)

private_key_path = "C:/Users/Pratik N Borkar/.oci/xxxx.pem"
user_ocid        = "ocid1.user.oc1..aaaaabmza"
fingerprint      = "3a:d9f:71"
tenancy_ocid     = "ocid1.tenancy.ookmjm7dqxxxxxx"
region           = "ap-sydney-1"


compartment_ocid  = "ocid1.compartr2vdqaxxxxx"
storage_tier     = "Standard"
bucket_count     = 10

3. Single Bucket With Lifecycle

(main.tf)

provider "oci" {
  tenancy_ocid        = var.tenancy_ocid
  user_ocid           = var.user_ocid
  fingerprint         = var.fingerprint
  private_key_path    = var.private_key_path
  region              = var.region
}

resource "oci_objectstorage_bucket" "this" {
  count          = var.object_storage_bucket_deploy ? 1 : 0
  compartment_id = var.object_storage_bucket_compartment_ocid
  name           = var.object_storage_bucket_name
  namespace      = var.object_storage_bucket_namespace

  access_type           = "NoPublicAccess"
  auto_tiering          = var.object_storage_bucket_storage_tier == "Standard" ? "Disabled" : null
  metadata              = var.object_storage_bucket_metadata
  freeform_tags         = var.object_storage_bucket_freeform_tags
  object_events_enabled = var.object_storage_bucket_object_events_enabled
  storage_tier          = var.object_storage_bucket_storage_tier
  versioning            = var.object_storage_bucket_versioning
}

resource "oci_objectstorage_object_lifecycle_policy" "this" {
  count     = length(var.object_storage_bucket_lifecycle_policy_rules) == 0 ? 0 : 1
  bucket    = oci_objectstorage_bucket.this[0].name
  namespace = oci_objectstorage_bucket.this[0].namespace

  dynamic "rules" {
    for_each = var.object_storage_bucket_lifecycle_policy_rules
    content {
      action      = rules.value.action
      is_enabled  = rules.value.is_enabled
      name        = rules.value.name
      time_amount = rules.value.time_amount
      time_unit   = rules.value.time_unit
      target      = rules.value.target

      dynamic "object_name_filter" {
        for_each = (
          length(rules.value.object_name_filter.inclusion_patterns) > 0 ||
          length(rules.value.object_name_filter.exclusion_patterns) > 0 ||
          length(rules.value.object_name_filter.inclusion_prefixes) > 0
        ) ? [1] : []

        content {
          exclusion_patterns = toset(rules.value.object_name_filter.exclusion_patterns)
          inclusion_patterns = toset(rules.value.object_name_filter.inclusion_patterns)
          inclusion_prefixes = toset(rules.value.object_name_filter.inclusion_prefixes)
        }
      }
    }
  }
}

resource "oci_objectstorage_object_lifecycle_policy" "multipart_uploads" {
  bucket    = oci_objectstorage_bucket.this[0].name
  namespace = oci_objectstorage_bucket.this[0].namespace

  rules {
    action      = "ABORT"
    is_enabled  = true
    name        = "Delete uncommitted or failed multipart uploads Rule"
    time_amount = "7"
    time_unit   = "DAYS"
    target      = "multipart-uploads"
  }
}

Define Variables (variables.tf)

variable "tenancy_ocid" {}
variable "user_ocid" {}
variable "fingerprint" {}
variable "private_key_path" {}
variable "region" {}

variable "object_storage_bucket_compartment_ocid" {}
variable "object_storage_bucket_name" {}
variable "object_storage_bucket_namespace" {}

variable "object_storage_bucket_metadata" {
  type = map(string)
  default = {}
}

variable "object_storage_bucket_freeform_tags" {
  type = map(string)
  default = {}
}

variable "object_storage_bucket_object_events_enabled" {
  type    = bool
  default = false
}

variable "object_storage_bucket_versioning" {
  default = "Enabled"
}

variable "object_storage_bucket_storage_tier" {
  default = "Standard"
}

variable "object_storage_bucket_deploy" {
  type    = bool
  default = true
}

variable "object_storage_bucket_lifecycle_policy_rules" {
  type = list(object({
    action      = string
    is_enabled  = bool
    name        = string
    time_amount = number
    time_unit   = string
    target      = string
    object_name_filter = object({
      inclusion_patterns = list(string)
      exclusion_patterns = list(string)
      inclusion_prefixes = list(string)
    })
  }))
}

Provide Variable Values (terraform.tfvars)

private_key_path = "C:/Users/Pratik N Borkar/.oci/xxxx.pem"
user_ocid        = "ocid1.user.oc1..aaaaabmza"
fingerprint      = "3a:d9f:71"
tenancy_ocid     = "ocid1.tenancy.ookmjm7dqxxxxxx"
region           = "ap-sydney-1"


object_storage_bucket_compartment_ocid = "ocid1.compartmei3ier2vdqa"
object_storage_bucket_name             = "my-logs-bucket"
object_storage_bucket_namespace        = "XXXXXXX"

object_storage_bucket_metadata = {}
object_storage_bucket_freeform_tags = {
  Environment = "Dev"
}
object_storage_bucket_object_events_enabled = false
object_storage_bucket_versioning            = "Enabled"
object_storage_bucket_storage_tier          = "Standard"

object_storage_bucket_lifecycle_policy_rules = [
  {
    action      = "INFREQUENT_ACCESS"
    is_enabled  = true
    name        = "Move Infrequent Access Objects Rule"
    time_amount = 45
    time_unit   = "DAYS"
    target      = "objects"
    object_name_filter = {
      inclusion_patterns = []
      exclusion_patterns = []
      inclusion_prefixes = []
    }
  },
  {
    action      = "ARCHIVE"
    is_enabled  = true
    name        = "Archive Objects Rule"
    time_amount = 90
    time_unit   = "DAYS"
    target      = "objects"
    object_name_filter = {
      inclusion_patterns = []
      exclusion_patterns = []
      inclusion_prefixes = []
    }
  },
  {
    action      = "DELETE"
    is_enabled  = true
    name        = "Delete Objects Rule"
    time_amount = 120
    time_unit   = "DAYS"
    target      = "objects"
    object_name_filter = {
      inclusion_patterns = []
      exclusion_patterns = []
      inclusion_prefixes = []
    }
  },
  {
    action      = "INFREQUENT_ACCESS"
    is_enabled  = true
    name        = "Move Infrequent Access Previous Versions Rule"
    time_amount = 45
    time_unit   = "DAYS"
    target      = "previous-object-versions"
    object_name_filter = {
      inclusion_patterns = []
      exclusion_patterns = []
      inclusion_prefixes = []
    }
  },
  {
    action      = "ARCHIVE"
    is_enabled  = true
    name        = "Archive Previous Versions Rule"
    time_amount = 90
    time_unit   = "DAYS"
    target      = "previous-object-versions"
    object_name_filter = {
      inclusion_patterns = []
      exclusion_patterns = []
      inclusion_prefixes = []
    }
  },
  {
    action      = "DELETE"
    is_enabled  = true
    name        = "Delete Previous Versions Rule"
    time_amount = 240
    time_unit   = "DAYS"
    target      = "previous-object-versions"
    object_name_filter = {
      inclusion_patterns = []
      exclusion_patterns = []
      inclusion_prefixes = []
    }
  }
]

Summary

Using Terraform, you can automate the creation and management of VCNs, ensuring repeatable and consistent infrastructure deployments.

Prerequisites

Before you begin, make sure you have:

1. OCI Account with permissions to create networking resources.

2. Terraform installed (v1.10.0 or higher recommended).

3. OCI Terraform Provider configured.

4. Your OCI credentials:

   • Tenancy OCID

   • User OCID

   • Compartment OCID

   • Fingerprint

   • Private key path

   • Region

Project Structure

Create a working directory, for example:

oci-vcn-terraform/
│
├── main.tf
├── variables.tf
└── terraform.tfvars

Step 1: Define Provider and Resources (main.tf)

# -------------------------------
# Provider
# -------------------------------
provider "oci" {
  tenancy_ocid     = var.tenancy_ocid
  user_ocid        = var.user_ocid
  fingerprint      = var.fingerprint
  private_key_path = var.private_key_path
  region           = var.region
}

# -------------------------------
# VCN
# -------------------------------
resource "oci_core_vcn" "oke_vcn" {
  cidr_block     = var.vcn_cidr
  compartment_id = var.compartment_ocid
  display_name   = "OKE-VCN-Sydney"
  dns_label      = "okevcn"
}

# -------------------------------
# Gateways
# -------------------------------
resource "oci_core_internet_gateway" "igw" {
  display_name   = "Internet-Gateway"
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.oke_vcn.id
  enabled        = true
}

resource "oci_core_nat_gateway" "nat_gw" {
  display_name   = "NAT-Gateway"
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.oke_vcn.id
  block_traffic  = false
}

# -------------------------------
# Subnet Definitions
# -------------------------------
locals {
  subnets = {
    jump_host = { name = "Jump-Host-Subnet", cidr = var.jump_host_cidr, public = true }
    master    = { name = "Master-Subnet", cidr = var.master_cidr, public = false }
    node      = { name = "Node-Subnet", cidr = var.node_cidr, public = false }
    lb        = { name = "LB-Subnet", cidr = var.lb_cidr, public = true }
    pod       = { name = "POD-Subnet", cidr = var.pod_cidr, public = false }
  }
}

# -------------------------------
# Security Lists
# -------------------------------
resource "oci_core_security_list" "sl" {
  for_each = local.subnets

  display_name   = "Security List for ${each.value.name}"
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.oke_vcn.id

  egress_security_rules {
    destination = "0.0.0.0/0"
    protocol    = "all"
  }

  ingress_security_rules {
    source   = "0.0.0.0/0"
    protocol = "all"
  }
}

# -------------------------------
# Route Tables
# -------------------------------
resource "oci_core_route_table" "rt" {
  for_each = local.subnets

  display_name   = "Route Table for ${each.value.name}"
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.oke_vcn.id

  # Attach Internet Gateway for public subnets
  # Attach NAT Gateway for private subnets
  route_rules {
    description       = each.value.public ? "Route to Internet via IGW" : "Route to Internet via NAT"
    destination       = "0.0.0.0/0"
    destination_type  = "CIDR_BLOCK"
    network_entity_id = each.value.public ? oci_core_internet_gateway.igw.id : oci_core_nat_gateway.nat_gw.id
  }
}

# -------------------------------
# Subnets
# -------------------------------
resource "oci_core_subnet" "subnet" {
  for_each = local.subnets

  cidr_block                 = each.value.cidr
  display_name               = each.value.name
  dns_label                  = replace(lower(each.key), "_", "")
  prohibit_public_ip_on_vnic = each.value.public ? false : true
  vcn_id                     = oci_core_vcn.oke_vcn.id
  route_table_id             = oci_core_route_table.rt[each.key].id
  security_list_ids          = [oci_core_security_list.sl[each.key].id]
  compartment_id             = var.compartment_ocid
}

Step 2: Define Variables (variables.tf)

variable "tenancy_ocid" {
  type        = string
  description = "Tenancy OCID"
}

variable "user_ocid" {
  type        = string
  description = "User OCID"
}

variable "fingerprint" {
  type        = string
  description = "API Key Fingerprint"
}

variable "private_key_path" {
  type        = string
  description = "Path to private key"
}

variable "region" {
  type        = string
  description = "OCI region"
}

variable "compartment_ocid" {
  type        = string
  description = "Compartment OCID"
}

variable "existing_vcn_id" {
  type        = string
  description = "Existing VCN OCID"
}

variable "endpoint_subnet_id" {
  type        = string
  description = "Subnet OCID for cluster endpoints"
}

variable "lb_subnet_id" {
  type        = string
  description = "Subnet OCID for load balancer"
}

variable "node_pool_subnet_id" {
  type        = string
  description = "Subnet OCID for node pool"
}

variable "node_pool_ssh_public_key" {
  type        = string
  description = "SSH public key for node pool"
}

variable "nodepool_image_id" {
  type        = string
  description = "OCID of the image for node pool"
}

variable "nodepool_cloud_init" {
  type        = string
  description = "Cloud-init script for node pool"
  default     = ""
}

Step 3: Provide Variable Values (terraform.tfvars)

# -------------------------------
# Provider Authentication
# -------------------------------
tenancy_ocid     = "ocid1.tenancy.oc1.xxx"
user_ocid        = "ocid1.user.oc1..axxx"
fingerprint      = "3a:ds.xx.xx"
private_key_path = "path of key"
region           = "ap-sydney-1"

# -------------------------------
# Networking Details
# -------------------------------
compartment_ocid = "ocid1.compartment.oc1.."

Step 4: Initialize and Apply

terraform init
terraform plan
terraform apply

TerraformConfirm with yes when prompted.
Terraform will provision

| Resource Type    | Count | Purpose                          |
| ---------------- | ----- | -------------------------------- |
| VCN              | 1     | Core network                     |
| Internet Gateway | 1     | Public internet access           |
| NAT Gateway      | 1     | Private outbound internet access |
| Security Lists   | 5     | One per subnet                   |
| Route Tables     | 5     | One per subnet                   |
| Subnets          | 5     | Logical network segments         |

Verification

After the deployment:

1. Log in to the OCI Console.

2. Navigate to Networking → Virtual Cloud Networks.

3. You’ll see the newly created OKE-VCN-Sydney and related resources.

You can also verify via CLI:

oci network vcn list --compartment-id <compartment_ocid>

Clean Up

To remove all resources:

terraform destroy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Create S3 Bucket in my case state.handsonk8s.ga

aws s3 mb s3://state.handsonk8s.ga

Install kops and kubectl In my case Installing Kops on macOS

curl -Lo kops https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-darwin-amd64
chmod +x ./kops
sudo mv ./kops /usr/local/bin/

Installing Kubectl on macOS

curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Add you domain/cluster name is Hosted Zone using Route 53 Make sure TTL should be 60 sec or less than that for urly DNS propagation.

Create public key

Lucifers-MacBook-Pro:.ssh lucifer$ ls
known_hosts
Lucifers-MacBook-Pro:.ssh lucifer$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/lucifer/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/lucifer/.ssh/id_rsa.
Your public key has been saved in /Users/lucifer/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:ZChUKEFJAERAbmpDRsmWi0jC5YWZso8QwgKA+nYyjU0 lucifer@Lucifers-MacBook-Pro.local
The key's randomart image is:
+---[RSA 2048]----+
|^OB+=+.          |
|OX+=o  .         |
|X*+o. . o        |
|Xo  E. o         |
|o+o=    S        |
|..B.+            |
| . +             |
|                 |
|                 |
+----[SHA256]-----+
Lucifers-MacBook-Pro:.ssh lucifer$ 
Lucifers-MacBook-Pro:.ssh lucifer$ ls
id_rsa        id_rsa.pub    known_hosts

Create kops cluster

kops create cluster \
--state "s3://state.handsonk8s.ga" \
--zones "us-east-1a,us-east-1b"  \
--master-count 3 \
--master-size=t2.micro \
--node-count 3 \
--node-size=t2.micro \
--name handsonk8s.ga  \
--yes

--state:- Your S3 bucket

--master-count:- No of Master node count here is set 1

--master-size:- Instance type or size you can set while creating cluster

--node-count:- No of Worker node count here is set 2

--node-size:- Instance type or size you can set while creating cluster

--name:- Your Hosted zone name

Validate kops cluster

kops validate cluster \
       --state "s3://state.handsonk8s.ga" \
       --name handsonk8s.ga

Update kops cluster (Always update cluster after changing Security Groups)

kops update cluster \
       --state "s3://state.handsonk8s.ga" \
       --name cluster.k8s.local  \
       --yes

Distroy kops Cluster

kops delete cluster \
       --state "s3://state.handsonk8s.ga" \
       --name handsonk8s.ga  \
       --yes