How we moved our infrastructure in-house

Why we changed our hosting

In 2025, our cloud bill was growing faster than our client workload. Managed databases, CI runners, logging and staging environments were useful, but we were paying for capacity we rarely used. The monthly total had passed $3,000.

We wanted to host client applications and internal tools without adding a separate cloud subscription for each service. We also needed test environments that resembled production. Running the infrastructure ourselves meant taking responsibility for maintenance and recovery, so repeatable setup was a requirement.

AI-assisted development helped our two-person team write the configuration and troubleshoot it. We still had to understand how the parts worked and test the setup ourselves.

Network separation

A Ubiquiti Dream Machine Pro handles routing and firewall rules, paired with a UniFi 24-port PoE switch. We separate workloads into seven VLANs:

The UDM Pro routes traffic between VLANs. Proxmox attaches each VM to the appropriate virtual bridge; it does not route between networks. Application-to-database traffic passes through the firewall, where rules control access. The guest operating systems have their own security configuration as well.

Virtual machines on Proxmox

Our three-node Proxmox cluster runs on refurbished mini PCs on a shelf in the office. It provides KVM virtual machines, LXC containers, live migration and ZFS storage.

Packer builds the base VM image with Ubuntu Server 24.04, the QEMU guest agent, cloud-init and baseline security settings. Terraform defines each VM’s CPU, memory, storage, host and VLAN. A database VM and a public reverse-proxy VM can share a physical host while remaining on separate networks.

We generally give each service its own VM to make debugging and recovery easier. Terraform lets us create environments with the same configuration and remove their resources when a project ends.

The k3s cluster

The primary application cluster has three control-plane nodes with embedded etcd and a kube-vip virtual IP, plus eight workers. Two workers have dedicated disks for Longhorn storage; three are lighter-weight nodes for less demanding workloads.

The cluster runs on VLAN 60 (10.0.60.0/24). Traefik handles ingress, MetalLB provides load-balancer services in L2/ARP mode, and Longhorn replicates persistent volumes. Applications use ingress routes with automatic TLS termination.

Terraform creates the VMs across the physical Proxmox nodes. Ansible configures them in order:

1. Apply the operating-system baseline.
2. Initialise the first control plane.
3. Deploy kube-vip before the other control planes join.
4. Join the remaining control-plane nodes.
5. Register the workers.
6. Install cluster add-ons.

When a node fails, we can replace its VM and rerun the playbook to rejoin it to the cluster.

Simpler deployments with Coolify

For internal tools, prototypes and small services, we use Coolify on a dedicated VM in VLAN 20. It connects to Docker VMs over SSH, builds containers from Git repositories and handles deployment, SSL certificates, database provisioning and environment variables.

For a web application with Postgres, we can push to Git and let Coolify build and deploy it. We manage the servers and storage ourselves, so we can tell a client which VM holds their data.

Testing Talos and GitOps

We also run a smaller Talos OS cluster for experimental and internal workloads. Talos is a minimal Linux distribution designed for Kubernetes. It is configured through an API and has no SSH access or package manager.

Argo CD applies application configuration from Git. MetalLB, Traefik and cert-manager are defined as Argo CD Applications, with sync waves to order their installation. Configuration changes stay in version control instead of being made manually on a server.

That approach also helps with rebuilding: we apply the bootstrap manifest and sync the root application to restore the declared configuration. Application data still needs its own backups and recovery steps.

Provisioning and recovery

The infrastructure repository contains:

• Packer configurations for base VM images.
• Terraform modules for Proxmox and BinaryLane VPS provisioning.
• Ansible playbooks for operating-system security, Docker, k3s and services.
• Kubernetes manifests and Helm values for cluster add-ons.
• Argo CD Application definitions for Talos.
• RackPeek YAML files recording the hardware layout.

Our documented recovery order is to install Proxmox, restore Terraform state, recreate the VMs, run Ansible and restore database backups. Keeping the configuration in code helps us repeat those steps on replacement hardware.

BinaryLane, an Australian VPS provider, gives us an off-site option. Critical workloads have Terraform configurations for both Proxmox and BinaryLane, and the same Ansible playbooks configure either environment.

Where AI helped

We used Claude, GitHub Copilot and our coding-agent workflows to draft Terraform modules, Ansible roles and Kubernetes manifests. They also helped investigate pods stuck in Pending, review firewall rules and work through recovery sequences.

This reduced the time spent writing configuration and looking up unfamiliar options. It did not remove the need to check network permissions, storage settings or the order in which services start. We remain responsible for reviewing and testing what the tools produce.

Costs after six months

Six months after the move, recurring infrastructure spend was down roughly 70%. The remaining costs were mainly internet bandwidth, hardware depreciation and a small BinaryLane reserve. New client environments took under ten minutes to provision, and tests ran on the same platform as production.

The setup gives us more control over where applications and data live, but it also leaves maintenance and recovery with us. Network separation, repeatable configuration and a documented recovery order are the parts we would prioritise again. The savings need to justify that ongoing work.