AWS deploy

Note

Synced verbatim from deploy/aws/README.md at build time; edit it there.

The same process model as a laptop, on one EC2 instance: cloudd on 127.0.0.1:8080, one hyphae serve per project on loopback, Caddy on plain :80, the data root on one EBS volume. In front, CloudFront terminates TLS (ACM certificate, TLS 1.2+, HTTP/2+3), applies a WAF rate limit, caches nothing, forwards every viewer header, and is the only source the security group admits on the instance’s :80. No Fargate per tenant, no Lambda, no Cognito, no multi-AZ, no HA — and none is claimed.

https://<domain>  ──▶ CloudFront (TLS, WAF)  ──http :80──▶ Caddy ──▶ cloudd 127.0.0.1:8080
   Route 53 alias       cert: ACM us-east-1        SG: CloudFront        │  one hyphae serve per project
   apex (+ www 301)     cache: disabled            origin-facing         │  EBS /var/lib/hyphae-cloud
                        headers: all viewer        prefix list only      └─ blobs → S3 https (SSM static keys)

Deployed: https://infra-hyphae.run — ids, smoke results, and teardown in docs/operate.md, “First AWS smoke”.

tls_at = "caddy" is the alternative without CloudFront: Caddy holds the certificate itself and 80/443 open to the world.

What is here:

File	Purpose
terraform/	main.tf: security group (:80 from the CloudFront origin-facing prefix list; or 80/443 open with tls_at = caddy), S3 buckets (blobs, artifacts), an IAM user carrying only the blobs object policy (its key is yours to create), an instance role limited to the tarball, the SSM prefix, and the same object policy, EBS gp3 volume (encrypted, prevent_destroy), Debian 12 instance (IMDSv2), Elastic IP. Default VPC unless vpc_id is given. edge.tf: ACM certificate in us-east-1 validated in the Route 53 zone, WAF web ACL (CLOUDFRONT scope: rate rule blocks at waf_rate_limit/5 min/IP, AWS common rule set in count), CloudFront Function for www → apex, the distribution (CachingDisabled + AllViewer managed policies, all methods, redirect-to-https, origin http-only to the EIP’s DNS name, 60 s read timeout for SSE), A/AAAA aliases. Applied for infra-hyphae.run on 2026-09-15.
user-data.sh	Rendered by Terraform. Installs Caddy, awscli, nftables, the official SSM agent (Debian AMIs have none), Node 22 LTS from nodejs.org (Debian’s node 18 lacks --permission); formats and mounts the EBS volume at /var/lib/hyphae-cloud; downloads the dist tarball from the artifacts bucket, verifies SHA256SUMS, runs install.sh; reads the secrets key and the S3 keys from SSM into /etc/hyphae-cloud/secrets.env (0600); fills the S3 endpoint/bucket/region; blocks IMDS for the cloudd uid; enables hyphae-cloud and caddy.
install.sh	Inside the tarball. Copies cloudd, hcloud, the pinned hyphae 3.0.0 (verified by version --json), the Studio bundle, the unit, the Caddyfile; creates user cloudd and the data layout. Never writes a secret, never starts anything.
hyphae-cloud.service	systemd unit: After=network-online.target, RequiresMountsFor=/var/lib/hyphae-cloud, EnvironmentFile=/etc/hyphae-cloud/env + secrets.env, KillMode=mixed, hardened.
Caddyfile	{$CLOUDD_DOMAIN::80} → reverse_proxy 127.0.0.1:8080, no buffering (SSE, long invokes), 40 MB bodies. Domain set → automatic HTTPS. Domain empty → plain :80, smoke test only, not a real deploy.
env.example	/etc/hyphae-cloud/env: bind, roots, HYPHAE_BIN, CLOUDD_TRUST_PROXY=1 + CLOUDD_PROXY_NETS=127.0.0.1/32, CLOUDD_BLOB_BACKEND=s3 with endpoint/bucket/region filled by user data.

Blobs: S3 over HTTPS, static keys from SSM, IMDS still blocked

cloudd’s S3 client speaks https:// to the regional endpoint (https://s3.<region>.amazonaws.com, path-style by default), signs with SigV4 including x-amz-security-token when credentials are temporary, and finds keys in this order: CLOUDD_S3_ACCESS_KEY/CLOUDD_S3_SECRET_KEY, then AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY(/AWS_SESSION_TOKEN), then ~/.aws/credentials, then IMDSv2. With CLOUDD_BLOB_BACKEND=s3 and none of those, it refuses to start.

On this host the choice is static keys in SSM: user data blocks the instance metadata service for the cloudd uid so that project functions (which run as that uid) can never pick up the instance role — and that same rule means cloudd cannot use the role either. So Terraform creates a dedicated IAM user (<name>-blobs) carrying only the object policy for the blobs bucket; you create its access key and put both halves in SSM (step 2); user data writes them into /etc/hyphae-cloud/secrets.env (0600) next to the secrets key, and fills CLOUDD_S3_ENDPOINT, _BUCKET, _REGION in /etc/hyphae-cloud/env. The instance role keeps the tarball and SSM permissions (used by root at boot) plus the same object policy, so a future IMDS-allowed helper could replace the static keys without a Terraform change. Functions still cannot reach IMDS.

The S3 code path has been exercised against MinIO over plain http:// and, for the https:// + session-token + credential-chain pieces, against unit tests and a loopback IMDSv2 stub. It has not been run against AWS S3 from this repository (no AWS API calls are made here); the first real PUT happens when you deploy.

Operator steps

Everything below is run from your workstation with the AWS CLI configured. Nothing in this repository calls AWS on its own.

0. Build the tarball on a glibc that the instance accepts

The instance is Debian 12 (glibc 2.36). Build there, or in the matching container:

make dist-bookworm      # podman/docker: rust:1-bookworm → dist/hyphae-cloud-0.1.0-linux-x86_64.tar.gz
# or, on a Debian 12 / older-glibc box:
make dist
make dist-check

make dist on a newer distribution produces a tarball whose binaries need that distribution’s glibc; make dist prints the glibc it was built on.

1. Provision

cd deploy/aws/terraform
cp terraform.tfvars.example terraform.tfvars      # region, name, domain, hosted_zone_name, tls_at
tofu init
tofu plan -out=plan
tofu apply plan

If your AWS CLI signs in with aws login (session credentials in ~/.aws/login), the provider cannot read them; prefix each tofu command with eval "$(aws configure export-credentials --format env)" &&.

The plan for a fresh deploy is 30 resources. ACM validation takes a few minutes and the CloudFront distribution 5–10 more; Terraform waits. Do step 2 and step 3 before the instance’s first boot (a targeted apply of the IAM user and buckets first, -target=aws_iam_user_policy_attachment.blobs -target=aws_s3_bucket_public_access_block.artifacts -target=aws_s3_bucket_public_access_block.blobs -target=aws_s3_bucket_server_side_encryption_configuration.blobs, then the rest) or replace the instance afterwards (tofu apply -replace=aws_instance.host; the data volume survives).

Outputs: public_ip, artifacts_bucket, blobs_bucket, ssm_parameter, data_volume_id, teardown.

2. Put the secrets in SSM (SecureString) — never in the AMI, image, or state

PREFIX="$(terraform output -raw ssm_prefix)"
# Function-secrets key. Keep a copy offline: rotating it orphans every function secret.
aws ssm put-parameter --region "$REGION" --name "$PREFIX/secrets_key" \
  --type SecureString --value "$(openssl rand -hex 32)"
# S3 keys for the blobs IAM user (the only credentials cloudd will use for S3).
KEY_JSON="$(aws iam create-access-key --user-name "$(terraform output -raw blobs_iam_user)")"
aws ssm put-parameter --region "$REGION" --name "$PREFIX/s3_access_key" --type SecureString \
  --value "$(echo "$KEY_JSON" | python3 -c 'import sys,json;print(json.load(sys.stdin)["AccessKey"]["AccessKeyId"])')"
aws ssm put-parameter --region "$REGION" --name "$PREFIX/s3_secret_key" --type SecureString \
  --value "$(echo "$KEY_JSON" | python3 -c 'import sys,json;print(json.load(sys.stdin)["AccessKey"]["SecretAccessKey"])')"
unset KEY_JSON

Rotate the S3 key by creating a second access key, updating both parameters, restarting hyphae-cloud after re-running the SSM fetch (or editing secrets.env by hand), then deleting the old key.

Accounts are invite-gated (CLOUDD_SIGNUP_MODE=invite in env.example): POST /v0/signup needs a code from /etc/hyphae-cloud/invite-codes, which user data writes from a fourth parameter. One code per line (8–128 bytes), optional label after a space, # comments allowed:

printf 'team-code-2026 QA\npilot-acme-7f3k Acme pilot\n' > invite-codes.txt
aws ssm put-parameter --region "$REGION" --name "$PREFIX/invite_codes" \
  --type SecureString --value "$(cat invite-codes.txt)" --overwrite
rm invite-codes.txt

To add or remove codes: put-parameter --overwrite, then on the box re-run the fetch (aws ssm get-parameter … > /etc/hyphae-cloud/invite-codes as cloudd:cloudd 0600) or reboot, and systemctl restart hyphae-cloud. The daemon refuses to start if the file is missing, empty, or readable by anyone else; codes are hashed on load and never logged. hcloud signup --invite-code CODE … from the terminal; the Studio asks for the code.

3. Upload the tarball

aws s3 cp dist/hyphae-cloud-0.1.0-linux-x86_64.tar.gz \
  "s3://$(terraform output -raw artifacts_bucket)/hyphae-cloud-0.1.0-linux-x86_64.tar.gz"

Do 2 and 3 before the instance’s first boot, or reboot it afterwards (aws ec2 reboot-instances) — user data runs once per launch. Without the tarball the box comes up mounted and configured but with no cloudd; scp the tarball in and run sudo ./install.sh, then sudo systemctl enable --now hyphae-cloud caddy.

4. DNS

With tls_at = cloudfront Terraform creates the A/AAAA aliases (apex and www) in hosted_zone_name; nothing to do. With tls_at = caddy, create an A record for the domain pointing at terraform output -raw public_ip; Caddy retries ACME until the name resolves.

5. Check

DOMAIN=cloud.example.com
curl -sS https://$DOMAIN/healthz          # {"ok":true}
curl -sS https://$DOMAIN/readyz           # {"ok":true,"store":true,"engine":"unchecked","projects":0}
curl -sSI https://$DOMAIN/studio/ | head -1

journalctl -u hyphae-cloud | grep 'S3 credentials resolved' should say source="static (CLOUDD_S3_ACCESS_KEY)" and blob backend ready backend="s3"; the startup HEAD Bucket is the first request the keys must pass.

Then the Studio in a browser, or CLOUDD_URL=https://$DOMAIN hcloud signup ….

Shell access: aws ssm start-session --target "$(tofu output -raw instance_id)" or aws ssm send-command --document-name AWS-RunShellScript … (no SSH rule exists unless you set allow_ssh_cidr; the SSM agent is installed by user data). Logs: journalctl -u hyphae-cloud -u caddy, user data in /var/log/hyphae-cloud-user-data.log (the secrets key is excluded from its trace).

Backups on this host

Stopped daemon, then the kit:

sudo systemctl stop hyphae-cloud
sudo -u cloudd hyphae-cloud-backup-all --data-root /var/lib/hyphae-cloud/data \
  --out /var/lib/hyphae-cloud/kits/$(date -u +%Y%m%dT%H%M%SZ) \
  --hyphae-bin /usr/local/lib/hyphae/3.0.0/bin/hyphae
sudo systemctl start hyphae-cloud

Blob bytes are in the bucket, so the kit records "blobs": "external" when CLOUDD_BLOB_BACKEND=s3 is exported (source /etc/hyphae-cloud/env first); back the bucket up with S3 versioning or replication, out of band. Copy the kit off the volume (aws s3 cp --recursive, or EBS snapshots of the whole volume — a crash-consistent image of the disk, not a Native-verified backup).

Upgrading

Upload the new tarball, then on the box: systemctl stop hyphae-cloud, unpack, ./install.sh, systemctl start hyphae-cloud. install.sh never overwrites /etc/hyphae-cloud/env.

Teardown

cd deploy/aws/terraform
# The data volume is protected; drop `prevent_destroy` in main.tf only when you mean it.
tofu destroy
aws ssm delete-parameters --names "$(tofu output -raw ssm_prefix)/secrets_key" "$(tofu output -raw ssm_prefix)/s3_access_key" "$(tofu output -raw ssm_prefix)/s3_secret_key"
aws s3 rm "s3://<blobs bucket>" --recursive && aws s3 rb "s3://<blobs bucket>"
aws s3 rm "s3://<artifacts bucket>" --recursive && aws s3 rb "s3://<artifacts bucket>"

What this is not

• Not TLS in cloudd (Caddy holds the certificate), not CLOUDD_PUBLIC (the daemon stays on loopback).
• Not HA, not multi-AZ, not autoscaled: one instance, one volume. The EBS volume is the only copy of every tenant until you take backups.
• Not an isolation boundary beyond what docs/product/claims.md states: tenants share this host’s kernel, CPU, disk, and network; Node functions run as trusted project code with this host’s network — anything the security group’s egress allows. The one thing user data takes away from them is the instance metadata service (an nftables rule for the cloudd uid), so a function cannot pick up the instance role’s credentials.
• Not instance-role S3 from cloudd: IMDS is blocked for its uid on purpose; the static IAM-user key in SSM is the credential.
• Not a WAF that blocks on content: only the per-IP rate rule blocks; the managed common rule set counts. Read its metrics before changing that.
• Not multi-region: CloudFront is global, but the origin is one instance in one AZ.