Posted on by Steve Bar Yakov Gindi

Stateless AWS AMIs: How to Build a Reboot-Persistent “Detective” Boot

Here is your edited version, fully polished for grammar and flow, with the WordPress Markdown formatting (code blocks and headers) restored so it pastes perfectly into the Block Editor.

Your additions about the “Stateful nightmare,” the “CloudFormation form replacement,” and the “paper vs. real Alpaca trading” are fantastic. They add massive technical credibility specific to AWS Marketplace and algorithmic trading.

The “Detective” Boot: Breaking Through the Kafkaesque Gatekeepers of AWS AMIs

In Franz Kafka’s parable “Before the Law,” a man spends his entire life waiting for a gatekeeper to grant him entry to the Law. He dies waiting for permission that was technically already his. I’ve lived this Kafkaesque nightmare at multiple companies, watching remote, outsourced knowledge-hoarder DevOps act as the “Gatekeeper,” standing in front of the infrastructure and telling everyone “not now” while they clutch a manually created, “perfect” AMI like a holy relic. They twist themselves into impossible positions to protect their tribal knowledge, but I, as the DevOps Rock Star, don’t wait at the door—I built a system that admits itself, fetching secrets and buckets without being manually spoon-fed.

When I built Agent Provost, I engineered a Stateless “Detective” Boot process. The instance doesn’t sit outside the gate waiting for a hoarder to grant it an identity; it performs its own reconnaissance and enters the “Law” of the infrastructure the moment it wakes up.

This logic resides in the Resources > AgentProvostInstance > Properties > UserData section of my cloudformation/alpaca-provost-cf.yml file.

1. The Secure Handshake (IMDSv2)

A self-aware AMI must first be a secure one. I don’t use the legacy IMDSv1 that many “snowflake” images still rely on. My boot script enforces the IMDSv2 token-based handshake.

Pro-Tip: Why use a token if IMDSv1 didn’t need one? IMDSv2 was designed to prevent SSRF (Server-Side Request Forgery) attacks. By requiring a PUT request to generate a session token first, IMDSv2 ensures that only a process with full control over the instance’s network stack can access the metadata.

# Location: cloudformation/alpaca-provost-cf.yml (UserData)

# Generate a secure 6-hour session token
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" \
     -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Use the token to securely fetch the Instance ID and Region
INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" \
     -s http://169.254.169.254/latest/meta-data/instance-id)

REGION=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" \
     -s http://169.254.169.254/latest/meta-data/placement/region)

Note: Querying the local metadata service (169.254.169.254) does not require an IAM role; it is accessible to any process running on the instance.

2. The “Detective” Phase: Dynamic Tag Discovery

This is where the instance stops waiting at the gate. Instead of being “told” its environment by an opaque configuration file, the instance asks the AWS API: “Who created me?” Despite the clear advantages of stateless discovery, some practitioners still insist on hand-crafted, “baked” images to maintain a sense of indispensable control over the environment, much like Kafka’s gatekeeper.

I use the aws ec2 describe-tags command, filtered by the INSTANCE_ID, to find the aws:cloudformation:stack-name tag that AWS automatically applies during deployment.

# Location: cloudformation/alpaca-provost-cf.yml (UserData)

# The instance discovers its own CloudFormation context at runtime
STACK_NAME=$(aws ec2 describe-tags \
    --region "$REGION" \
    --filters "Name=resource-id,Values=$INSTANCE_ID" \
    "Name=key,Values=aws:cloudformation:stack-name" \
    --query "Tags[0].Value" \
    --output text)

The IAM Boundary: Unlike the metadata service in Step 1, the describe-tags command talks to the global EC2 API. For this to work, your InstanceRole must have the ec2:DescribeTags permission. Without this specific IAM grant, the “Detective” boot will fail at the moment of discovery.

3. Reboot-Persistence: The “Detective” Never Sleeps

A common mistake in AWS infrastructure is assuming UserData is enough. Standard UserData only runs once. If the instance reboots for maintenance, the environment variables vanish, and the application fails—or even worse, the EC2 is Stateful and becomes a nightmare to replace or upgrade.

I solved this by using UserData to bootstrap a persistent Systemd Service. This ensures that the “Detective” reconnaissance logic runs every single time the OS boots, not just the first time.

# Location: cloudformation/alpaca-provost-cf.yml (UserData)

# 1. Create the persistent discovery script
cat << 'EOF' > /usr/local/bin/agent-provost-discovery.sh
#!/bin/bash
# [IMDSv2 and Tag Discovery Logic]
# Reconstruct resource names based on the discovered Stack Name
export LOG_BUCKET="agent-provost-logs-${STACK_NAME}"
export SECRET_VAULT="agent-provost-secrets-${STACK_NAME}"
EOF

chmod +x /usr/local/bin/agent-provost-discovery.sh

# 2. Create a Systemd service to run it on every boot
cat << 'EOF' > /etc/systemd/system/agent-provost-discovery.service
[Unit]
Description=Agent Provost Dynamic Resource Discovery
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/agent-provost-discovery.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

systemctl enable agent-provost-discovery.service

Why this is the “Senior” way: By moving the discovery logic into a Systemd service, I’ve made the system self-healing. If AWS migrates the instance or it reboots after a crash, the “Detective” immediately re-runs its reconnaissance, re-maps the S3 buckets and Secrets, and ensures the application starts with the correct context. When you do need to replace the stateless EC2, you can simply select to replace the image from the bottom of the CloudFormation form in the AWS Marketplace, and this Systemd Service will be automatically injected into the new instance.

The Result: Anti-Fragile Infrastructure

Manual AMIs are fragile because they rely on human memory and static “baking,” which inevitably leads to undocumented “crap” being baked into the image. By building a reboot-persistent “Detective” boot process, I’ve created an Anti-Fragile, Stateless system.

If a resource is renamed or moved, or you need to change from paper Alpaca trading to real trading, the AMI doesn’t break—it simply finds the new configuration on the next boot. This is the level of engineering I brought to Agent Provost. It’s about building infrastructure that is self-aware, secure, and entirely automated.