> ## Documentation Index
> Fetch the complete documentation index at: https://tesser.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Run boxes in your own AWS account

> Run an org's boxes in your AWS account while Tesserc hosts the control plane

By default boxes run in Tesserc's AWS account and the org is charged per
awake second, plus a few cents a day for each sleeping box. An org can instead run its boxes in its own account: Tesserc
still hosts the control plane, the CLI, and the dashboard, but every EC2
instance is yours, in your VPC, on your AWS bill. Tesserc bills only its
upcharge per awake hour, from \$0.01, and nothing while a box sleeps (the [pricing page](https://tesser.sh/pricing)
has the rates). Running in your own account is part of the Team plan.

A self-hosted box:

* **has no inbound SSH port.** Laptops reach sshd through AWS Session
  Manager, so the security group admits nothing from outside the org's own
  boxes, which reach each other on TCP 14650.
* **never runs its user-data.** The user-data is three fields of JSON: the
  org id, box id, and control plane URL. None of it is secret. The image
  turns cloud-init's user-data processing off, so nothing in it is
  executed. On first boot a step baked into the image reads it from the
  instance metadata service, refuses to go on unless every field is well
  formed (an https control plane URL), fetches boxd, and installs it only if
  its release signature verifies.
* **holds no Tesserc credential.** boxd proves which instance it runs on
  with a short-lived token that AWS STS signs for the box's role
  (`sts:GetWebIdentityToken`, [outbound identity
  federation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_outbound.html)).
  The control plane accepts it only when the token names this box's
  instance in your account, addressed to the control plane.
* **launches only the image your role allows.** By default that is
  Tesserc's published image: your role may launch images owned by Tesserc's
  publishing account and no one else's. To own the image outright, copy a
  release into your account and pin the role to the copy
  ([Own the image](#own-the-image)).

## Set it up

You need to be an admin of the org, and the org needs to have no boxes: a
box cannot move between clouds, so `tesser rm` any it has first.

The setup is a [Pulumi](https://www.pulumi.com/docs/iac/languages-sdks/javascript/)
program you copy into your own infrastructure repository; it needs a
`@pulumi/aws` 7 release with `aws.iam.OutboundWebIdentityFederation`. It is two files:
`tesser-policy.ts` holds the IAM documents, and `index.ts` makes the
network, the roles, and the instance profile.

Two values come from Tesserc:

```sh theme={"theme":"css-variables"}
tesser cloud template --json | jq -r '.orgId, .imageOwner'
```

The first is the org id, the second the AWS account that publishes
Tesserc's box image. Set them on the stack, in us-east-1 or us-west-2 (the
regions Tesserc publishes its image to):

```sh theme={"theme":"css-variables"}
pulumi config set aws:region us-east-1
pulumi config set tesser:orgId org_0123456789ab
pulumi config set tesser:imageOwner <image owner account>
pulumi config set tesser:parameterStore false
pulumi up
```

```ts tesser-policy.ts theme={"theme":"css-variables"}
type Statement = {
  Sid: string;
  Effect: "Allow";
  Action: string | string[];
  Resource: string | string[];
  Condition?: Record<string, Record<string, string | string[]>>;
};

export function trustPolicy(issuerHost: string, accountId: string, orgId: string) {
  return {
    Version: "2012-10-17",
    Statement: [
      {
        Sid: "TesserOrg",
        Effect: "Allow",
        Principal: { Federated: `arn:aws:iam::${accountId}:oidc-provider/${issuerHost}` },
        Action: "sts:AssumeRoleWithWebIdentity",
        Condition: { StringEquals: { [`${issuerHost}:aud`]: "sts.amazonaws.com", [`${issuerHost}:sub`]: `org:${orgId}` } },
      },
    ],
  };
}

export function permissionsPolicy(t: {
  region: string;
  accountId: string;
  orgId: string;
  boxRole: string;
  image: { id: string } | { owner: string };
  subnetId?: string;
  securityGroupId?: string;
}) {
  if ("owner" in t.image && !/^\d{12}$/.test(t.image.owner)) throw new Error(`tesser's image owner must be a 12-digit AWS account id, not "${t.image.owner}"`);
  const ec2 = (resource: string) => `arn:aws:ec2:${t.region}:${t.accountId}:${resource}`;
  const image: Statement =
    "id" in t.image
      ? { Sid: "LaunchImage", Effect: "Allow", Action: "ec2:RunInstances", Resource: `arn:aws:ec2:${t.region}::image/${t.image.id}` }
      : {
          Sid: "LaunchImage",
          Effect: "Allow",
          Action: "ec2:RunInstances",
          Resource: `arn:aws:ec2:${t.region}::image/*`,
          Condition: { StringEquals: { "ec2:Owner": t.image.owner } },
        };
  const statements: Statement[] = [
    {
      Sid: "Describe",
      Effect: "Allow",
      Action: ["ec2:DescribeInstances", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:DescribeRouteTables", "servicequotas:GetServiceQuota", "iam:GetOutboundWebIdentityFederationInfo"],
      Resource: "*",
    },
    {
      Sid: "LaunchOrgInstances",
      Effect: "Allow",
      Action: "ec2:RunInstances",
      Resource: ec2("instance/*"),
      Condition: { StringEquals: { "aws:RequestTag/tesser:org": t.orgId } },
    },
    image,
    {
      Sid: "LaunchInto",
      Effect: "Allow",
      Action: "ec2:RunInstances",
      Resource: [ec2(`subnet/${t.subnetId ?? "*"}`), ec2(`security-group/${t.securityGroupId ?? "*"}`), ec2("network-interface/*"), ec2("volume/*")],
    },
    {
      Sid: "TagAtLaunch",
      Effect: "Allow",
      Action: "ec2:CreateTags",
      Resource: ec2("instance/*"),
      Condition: { StringEquals: { "ec2:CreateAction": "RunInstances" } },
    },
    {
      Sid: "ManageOrgInstances",
      Effect: "Allow",
      Action: ["ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances"],
      Resource: ec2("instance/*"),
      Condition: { StringEquals: { "aws:ResourceTag/tesser:org": t.orgId } },
    },
    {
      Sid: "RetagOrgInstances",
      Effect: "Allow",
      Action: "ec2:CreateTags",
      Resource: ec2("instance/*"),
      Condition: {
        StringEquals: { "aws:ResourceTag/tesser:org": t.orgId },
        "ForAllValues:StringEquals": { "aws:TagKeys": ["tesser:org", "tesser:pool", "tesser:class", "tesser:service", "tesser:sha"] },
      },
    },
    {
      Sid: "PassBoxRole",
      Effect: "Allow",
      Action: "iam:PassRole",
      Resource: `arn:aws:iam::${t.accountId}:role/${t.boxRole}`,
      Condition: { StringEquals: { "iam:PassedToService": "ec2.amazonaws.com" } },
    },
  ];
  return { Version: "2012-10-17", Statement: statements };
}

export function boxIdentityPolicy(cellUrl: string) {
  return {
    Version: "2012-10-17",
    Statement: [
      {
        Sid: "BoxIdentity",
        Effect: "Allow",
        Action: "sts:GetWebIdentityToken",
        Resource: "*",
        Condition: {
          "ForAllValues:StringEquals": { "sts:IdentityTokenAudience": cellUrl },
          NumericLessThanEquals: { "sts:DurationSeconds": "300" },
        },
      },
    ],
  };
}

export function parameterStorePolicies(t: { region: string; accountId: string; orgId: string }) {
  const resource = `arn:aws:ssm:${t.region}:${t.accountId}:parameter/tesser/${t.orgId}/*`;
  const doc = (sid: string, actions: string[]) => ({ Version: "2012-10-17", Statement: [{ Sid: sid, Effect: "Allow", Action: actions, Resource: resource }] });
  return { control: doc("WriteOrgEnv", ["ssm:PutParameter", "ssm:DeleteParameters"]), box: doc("ReadOrgEnv", ["ssm:GetParametersByPath"]) };
}
```

```ts index.ts theme={"theme":"css-variables"}
import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import { boxIdentityPolicy, parameterStorePolicies, permissionsPolicy, trustPolicy } from "./tesser-policy";

const config = new pulumi.Config("tesser");
const orgId = config.require("orgId");
const parameterStore = config.requireBoolean("parameterStore");
const issuerHost = "api.tesser.sh";
const meshPort = 14650;
const zones: Record<string, string> = { "us-east-1": "use1-az1", "us-west-2": "usw2-az1" };

const region = aws.config.requireRegion();
const accountId = aws.getCallerIdentityOutput().accountId;
const tags = { Name: `tesser-${orgId}`, "tesser:org": orgId };
const boxRoleName = `tesser-${orgId}-box`;

const vpc = new aws.ec2.Vpc("tesser", { cidrBlock: "10.73.0.0/16", enableDnsSupport: true, enableDnsHostnames: true, tags });
const gateway = new aws.ec2.InternetGateway("tesser", { vpcId: vpc.id, tags });
const subnet = new aws.ec2.Subnet("tesser", {
  vpcId: vpc.id,
  cidrBlock: "10.73.0.0/20",
  availabilityZoneId: zones[region],
  mapPublicIpOnLaunch: true,
  tags,
});
const routes = new aws.ec2.RouteTable("tesser", { vpcId: vpc.id, routes: [{ cidrBlock: "0.0.0.0/0", gatewayId: gateway.id }], tags });
new aws.ec2.RouteTableAssociation("tesser", { subnetId: subnet.id, routeTableId: routes.id });

const group = new aws.ec2.SecurityGroup("tesser", { vpcId: vpc.id, description: `Tesserc boxes of org ${orgId}`, tags });
new aws.vpc.SecurityGroupIngressRule("mesh", {
  securityGroupId: group.id,
  referencedSecurityGroupId: group.id,
  ipProtocol: "tcp",
  fromPort: meshPort,
  toPort: meshPort,
  description: "box to box",
});
new aws.vpc.SecurityGroupEgressRule("out", { securityGroupId: group.id, ipProtocol: "-1", cidrIpv4: "0.0.0.0/0" });

const provider = new aws.iam.OpenIdConnectProvider("tesser", { url: `https://${issuerHost}`, clientIdLists: ["sts.amazonaws.com"] });

const boxRole = new aws.iam.Role("tesser-box", {
  name: boxRoleName,
  assumeRolePolicy: JSON.stringify({
    Version: "2012-10-17",
    Statement: [{ Effect: "Allow", Principal: { Service: "ec2.amazonaws.com" }, Action: "sts:AssumeRole" }],
  }),
  tags,
});
new aws.iam.RolePolicyAttachment("tesser-box-ssm", { role: boxRole.name, policyArn: aws.iam.ManagedPolicy.AmazonSSMManagedInstanceCore });
new aws.iam.OutboundWebIdentityFederation("tesser", {});
new aws.iam.RolePolicy("tesser-box-identity", { role: boxRole.id, policy: JSON.stringify(boxIdentityPolicy(`https://${issuerHost}`)) });
const profile = new aws.iam.InstanceProfile("tesser-box", { name: boxRoleName, role: boxRole.name, tags });

const image: pulumi.Output<{ id: string } | { owner: string }> = pulumi.output({ owner: config.require("imageOwner") });

const role = new aws.iam.Role(
  "tesser",
  { name: `tesser-${orgId}`, assumeRolePolicy: accountId.apply((a) => JSON.stringify(trustPolicy(issuerHost, a, orgId))), tags },
  { dependsOn: [provider] },
);
new aws.iam.RolePolicy("tesser-boxes", {
  role: role.id,
  policy: pulumi
    .all([accountId, subnet.id, group.id, image])
    .apply(([a, subnetId, securityGroupId, image]) =>
      JSON.stringify(permissionsPolicy({ region, accountId: a, orgId, boxRole: boxRoleName, image, subnetId, securityGroupId })),
    ),
});

if (parameterStore) {
  const policies = accountId.apply((a) => parameterStorePolicies({ region, accountId: a, orgId }));
  new aws.iam.RolePolicy("tesser-env", { role: role.id, policy: policies.apply((p) => JSON.stringify(p.control)) });
  new aws.iam.RolePolicy("tesser-box-env", { role: boxRole.id, policy: policies.apply((p) => JSON.stringify(p.box)) });
}

export const roleArn = role.arn;
export const subnetId = subnet.id;
export const securityGroupId = group.id;
export const instanceProfile = profile.name;
export const connect = pulumi.interpolate`tesser cloud connect --account ${accountId} --region ${region}${parameterStore ? " --env parameter-store" : ""}`;
```

When `pulumi up` finishes, run the command it exports as `connect`, with
the `~/.aws` profile members' laptops use for
[Session Manager](#session-manager):

```sh theme={"theme":"css-variables"}
tesser cloud connect --account 123456789012 --region us-east-1 --aws-profile acme
```

`--account` finds the role (`tesser-<org id>`), the boxes' instance profile
(`tesser-<org id>-box`), and the subnet and security group tagged with the
org id. The account id can be pasted with dashes, the way the AWS console
shows it. From then on every
`tesser make`, `dev`, and `exec` launches in your account. Boxes sleep,
wake, and are removed exactly as before.

The program makes, in the region you run it in:

* **A VPC** (`10.73.0.0/16`) with one public subnet, an internet gateway,
  and a route to it. The subnet sits in a zone that offers every box size:
  `use1-az1` in us-east-1, `usw2-az1` in us-west-2. Boxes need that egress
  to reach Session Manager and `api.tesser.sh`.
* **A security group**: TCP 14650 from the group itself so boxes reach each
  other, all outbound, and nothing else inbound. No port 22.
* **The identity provider** for `https://api.tesser.sh`. An account holds
  one per URL; when a second org sets up in the same account, drop the
  `OpenIdConnectProvider` from its program and look up the existing one
  with `aws.iam.OpenIdConnectProvider.get`.
* **A role** named `tesser-<org id>` that only this org can assume, with
  the permissions policy from `tesser-policy.ts`.
* **Outbound identity federation**, switched on for the account, so the
  boxes' role can ask STS for identity tokens. It is one account-wide
  setting; a second org in the same account drops it from its program.
* **The boxes' role and instance profile**, `tesser-<org id>-box`, with
  `AmazonSSMManagedInstanceCore`, `sts:GetWebIdentityToken` for tokens
  addressed to `https://api.tesser.sh` only and, with
  `tesser:parameterStore`, read access to the org's parameters.

None of it costs anything. Boxes cost what the same EC2 instances and EBS
volumes cost anyone else. Each box's root volume includes extra space
equal to the box's memory, so the box can hibernate when it sleeps. `pulumi destroy` removes everything it made;
disconnect the org first.

To use an existing VPC, a private subnet behind a NAT (the CLI then reaches
boxes at their private IP with `--private`), or your own role names, keep
`tesser-policy.ts` and change `index.ts`, then connect with the ids
(`--role`, `--subnet`, `--sg`, `--instance-profile`) instead of
`--account`. `tesser cloud template --account <id> --region <region>`
prints the same policies with your values filled in. `PassBoxRole` names
the IAM role inside the boxes' instance profile, not the profile. The
program gives the role and the profile the same name, and `template`
assumes that: it uses the org's instance profile name (or
`tesser-<org id>-box` before the org is connected). If your role's name
differs from the profile's, or has a path, pass it as `boxRole` (in the
program) or `--box-role path/name` (to `template`).

## Own the image

By default boxes launch Tesserc's published image, and the role may launch
any image Tesserc's publishing account owns. To take that choice away from
Tesserc, copy an image release into your account and pin the role to the
copy. Tesserc then launches exactly that image and no other, and a new
release reaches your boxes only when you copy it and move the pin.

Every image build of
[github.com/tesser-sh/ami](https://github.com/tesser-sh/ami) is a
[release](https://github.com/tesser-sh/ami/releases) named `build-<run id>`.
Its tag points at the commit that records the build's AMI ids in
`amis.json`, one per region: `{ "us-east-1": "ami-...", "us-west-2": "ami-..." }`.
Pin that commit, never `main`: the sha is what pins the release, and it is
the tree to review. This prints the latest release's commit:

```sh theme={"theme":"css-variables"}
gh api repos/tesser-sh/ami/commits/$(gh release view -R tesser-sh/ami --json tagName -q .tagName) --jq .sha
```

Replace the `image` line of `index.ts` with:

```ts theme={"theme":"css-variables"}
const amisSha = "<commit of the tesser-sh/ami release you reviewed>";
const amis = pulumi.output(
  fetch(`https://raw.githubusercontent.com/tesser-sh/ami/${amisSha}/amis.json`).then((r) => r.json() as Promise<Record<string, string>>),
);
const copy = new aws.ec2.AmiCopy("tesser-box", {
  name: `tesser-box-${amisSha.slice(0, 12)}`,
  sourceAmiRegion: region,
  sourceAmiId: amis.apply((m) => m[region]!),
  tags,
});
const image: pulumi.Output<{ id: string } | { owner: string }> = copy.id.apply((id) => ({ id }));
export const imageId = copy.id;
```

Move `amisSha` to a newer release's commit to take a new image.

The permissions policy's `LaunchImage` statement becomes
`arn:aws:ec2:<region>::image/<your copy>` with no owner condition, and
`tesser:imageOwner` is no longer needed. Copy a release soon after you pin
its sha: Tesserc keeps only its two newest images public, and an older
source can no longer be copied. Then tell the control plane which image to
launch:

```sh theme={"theme":"css-variables"}
tesser cloud connect --image ami-0123456789abcdef0
```

`--image tesser` goes back to the published image; move the policy back
first. The image can change while boxes run: running and sleeping boxes
keep the image they launched from, and new boxes take the new one. Boxes
hibernate when they sleep only on images from release `build-40566052647`
or later; boxes on an older image are powered down instead, and their dev
server is started again when they wake.

## Serve shared boxes on your own domain

A share gateway makes every shared box answer at `https://box-<id>.<host>/`
for anyone on your network, with no Tesserc login: a teammate, a designer, or
a phone on the VPN opens the URL. The first port the box's service declares
is served there, and every other port at `https://box-<id>--<port>.<host>/`,
so each port is its own site the way `localhost:<port>` is. It is one
process, `tesser gateway serve`, on a small instance in the boxes' VPC,
behind an internal Application Load Balancer that terminates TLS with an ACM
certificate for `*.<host>`. Opening a sleeping box's URL wakes it, with a
page that shows the progress and turns into the app when it answers. The
gateway proxies HTTP and WebSockets to the box's mesh port and keeps the box
awake while traffic flows. A box is served only while it has a live share:
`tesser share <box>` prints the box's gateway URL next to the review link, and
`tesser share <box> --revoke`, or the share expiring, takes it off within 15
seconds.

The gateway does no user auth. Whoever can reach it reaches every shared
box, so admit only your VPN or an SSO proxy to it.

It proves itself to Tesserc with its IAM role's AWS web identity token
(`sts:GetWebIdentityToken`, the same outbound identity federation the boxes
use) and holds no Tesserc secret.

Add this to `index.ts`, with the host the boxes hang off and the CIDR your
people reach the VPC from:

```sh theme={"theme":"css-variables"}
pulumi config set tesser:gatewayHost tesser.example.com
pulumi config set tesser:gatewayFrom 10.0.0.0/8
```

ACM issues the certificate once `<host>`'s public DNS carries a validation
record: `pulumi up` exports it as `certificateValidation`. Add it to your DNS
once and leave it there, and ACM renews the certificate on its own. A `<host>`
with no public DNS can't get an ACM certificate; import one into ACM instead
and pass its ARN with `pulumi config set tesser:gatewayCertificate <arn>`.

```ts gateway.ts theme={"theme":"css-variables"}
const gatewayHost = config.require("gatewayHost");
const gatewayFrom = config.require("gatewayFrom");
const gatewayPort = 8080;
const secondZones: Record<string, string> = { "us-east-1": "use1-az2", "us-west-2": "usw2-az2" };

const certificate = config.get("gatewayCertificate") === undefined
  ? new aws.acm.Certificate("tesser-gateway", { domainName: `*.${gatewayHost}`, validationMethod: "DNS", tags })
  : undefined;
const certificateArn = certificate?.arn ?? config.require("gatewayCertificate");

const lbSubnet = new aws.ec2.Subnet("tesser-gateway", { vpcId: vpc.id, cidrBlock: "10.73.16.0/20", availabilityZoneId: secondZones[region], tags });
new aws.ec2.RouteTableAssociation("tesser-gateway", { subnetId: lbSubnet.id, routeTableId: routes.id });

const lbGroup = new aws.ec2.SecurityGroup("tesser-gateway-lb", { vpcId: vpc.id, description: `Tesserc share gateway load balancer of org ${orgId}`, tags });
const gatewayGroup = new aws.ec2.SecurityGroup("tesser-gateway", { vpcId: vpc.id, description: `Tesserc share gateway of org ${orgId}`, tags });
new aws.vpc.SecurityGroupIngressRule("gateway-in", { securityGroupId: lbGroup.id, cidrIpv4: gatewayFrom, ipProtocol: "tcp", fromPort: 443, toPort: 443 });
new aws.vpc.SecurityGroupEgressRule("lb-gateway", { securityGroupId: lbGroup.id, referencedSecurityGroupId: gatewayGroup.id, ipProtocol: "tcp", fromPort: gatewayPort, toPort: gatewayPort });
new aws.vpc.SecurityGroupIngressRule("gateway-from-lb", { securityGroupId: gatewayGroup.id, referencedSecurityGroupId: lbGroup.id, ipProtocol: "tcp", fromPort: gatewayPort, toPort: gatewayPort });
new aws.vpc.SecurityGroupEgressRule("gateway-https", { securityGroupId: gatewayGroup.id, cidrIpv4: "0.0.0.0/0", ipProtocol: "tcp", fromPort: 443, toPort: 443 });
new aws.vpc.SecurityGroupEgressRule("gateway-mesh", { securityGroupId: gatewayGroup.id, referencedSecurityGroupId: group.id, ipProtocol: "tcp", fromPort: meshPort, toPort: meshPort });
new aws.vpc.SecurityGroupIngressRule("mesh-gateway", {
  securityGroupId: group.id,
  referencedSecurityGroupId: gatewayGroup.id,
  ipProtocol: "tcp",
  fromPort: meshPort,
  toPort: meshPort,
  description: "share gateway to box",
});

const lb = new aws.lb.LoadBalancer("tesser-gateway", {
  internal: true,
  loadBalancerType: "application",
  subnets: [subnet.id, lbSubnet.id],
  securityGroups: [lbGroup.id],
  idleTimeout: 3600,
  tags,
});
const targets = new aws.lb.TargetGroup("tesser-gateway", {
  vpcId: vpc.id,
  port: gatewayPort,
  protocol: "HTTP",
  targetType: "instance",
  healthCheck: { path: "/__tesser/health" },
  tags,
});
new aws.lb.Listener("tesser-gateway", {
  loadBalancerArn: lb.arn,
  port: 443,
  protocol: "HTTPS",
  sslPolicy: "ELBSecurityPolicy-TLS13-1-2-2021-06",
  certificateArn,
  defaultActions: [{ type: "forward", targetGroupArn: targets.arn }],
  tags,
});

const gatewayRole = new aws.iam.Role("tesser-gateway", {
  name: `tesser-${orgId}-gateway`,
  assumeRolePolicy: JSON.stringify({
    Version: "2012-10-17",
    Statement: [{ Effect: "Allow", Principal: { Service: "ec2.amazonaws.com" }, Action: "sts:AssumeRole" }],
  }),
  tags,
});
new aws.iam.RolePolicy("tesser-gateway-token", { role: gatewayRole.id, policy: JSON.stringify(boxIdentityPolicy(`https://${issuerHost}`)) });
new aws.iam.RolePolicyAttachment("tesser-gateway-ssm", { role: gatewayRole.name, policyArn: aws.iam.ManagedPolicy.AmazonSSMManagedInstanceCore });
const gatewayProfile = new aws.iam.InstanceProfile("tesser-gateway", { name: gatewayRole.name, role: gatewayRole.name, tags });

const al2023 = aws.ec2.getAmiOutput({ owners: ["amazon"], mostRecent: true, filters: [{ name: "name", values: ["al2023-ami-2023.*-arm64"] }] });
const gatewayInstance = new aws.ec2.Instance("tesser-gateway", {
  ami: al2023.id,
  instanceType: "t4g.small",
  subnetId: subnet.id,
  vpcSecurityGroupIds: [gatewayGroup.id],
  iamInstanceProfile: gatewayProfile.name,
  metadataOptions: { httpTokens: "required" },
  userDataReplaceOnChange: true,
  userData: pulumi.interpolate`#!/bin/sh
set -eu
export HOME=/root
curl -fsSL https://tesser.sh/install | sh
cat > /etc/systemd/system/tesser-gateway.service <<EOF
[Unit]
After=network-online.target
[Service]
ExecStart=/root/.tesser/bin/tesser gateway serve --org ${orgId} --port ${gatewayPort}
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now tesser-gateway
`,
  tags: { Name: `tesser-${orgId}-gateway` },
});
new aws.lb.TargetGroupAttachment("tesser-gateway", { targetGroupArn: targets.arn, targetId: gatewayInstance.id, port: gatewayPort });

const zone = new aws.route53.Zone("tesser-gateway", { name: gatewayHost, vpcs: [{ vpcId: vpc.id }], tags });
new aws.route53.Record("tesser-gateway", {
  zoneId: zone.zoneId,
  name: `*.${gatewayHost}`,
  type: "A",
  aliases: [{ name: lb.dnsName, zoneId: lb.zoneId, evaluateTargetHealth: false }],
});

export const certificateValidation = certificate?.domainValidationOptions.apply((options) =>
  options.map((o) => `${o.resourceRecordName} ${o.resourceRecordType} ${o.resourceRecordValue}`),
);
export const gatewaySet = pulumi.interpolate`tesser gateway set --url https://${gatewayHost} --role ${gatewayRole.arn}`;
```

After `pulumi up`, run the command it exports as `gatewaySet`. The load
balancer's group admits HTTPS from `gatewayFrom` and sends only to the
gateway's port; the gateway's group admits only the load balancer. The
gateway's egress is HTTPS (Tesserc, STS, Session Manager) and the boxes' mesh
port, which the boxes' group admits from it. The instance carries no `tesser:org` tag, so Tesserc's
role can neither stop nor sweep it. The hosted zone is private to the VPC:
forward `<host>` to it from your VPN's DNS (a Route 53 Resolver inbound
endpoint, or Tailscale split DNS), or point a wildcard record in your own
DNS at the load balancer.

The dev server sees what it sees on a laptop: the gateway presents each
request as `http://localhost:<port>` (with a prefix in front of the box's
label kept, as `<prefix>.localhost`), so Vite's `allowedHosts` and Next's
`allowedDevOrigins` need nothing added. A redirect the app builds to
`http://localhost:<port>` goes back to the browser as the gateway URL, and
the gateway adds a small script to each page that sends the page's own
`fetch`, `XMLHttpRequest`, `WebSocket`, and `EventSource` calls to
`localhost:<port>` through the gateway, so a frontend that calls its API at
`localhost:5000` works unchanged. Links, forms, and image and script
sources that point at `localhost:<port>` are rewritten too, whether they
come in the page's HTML or are added by its scripts (a script-added link or
form is fixed when it is clicked or submitted). The gateway also serves the
ports the box reaches through its `[deps]`, and the ports of the boxes behind
them, as long as each of those boxes belongs to the person who shared the
box or is one of the org's shared instances; any other port answers 404.
Each port's
site keeps its own cookies, where `localhost` shares them across ports, and a
third-party sign-in only returns to the gateway URL if the provider allows
it as a callback. Only HTTP and WebSockets cross the gateway.

## What connect checks

`connect` tests the account end to end and stores nothing if any check
fails. Each failure names what is wrong and how to fix it:

| Check | What it tests |
| - | - |
| region | Tesserc publishes its image in the region, unless the org launches its own copy. |
| role | Tesserc can assume the role: the identity provider exists and the trust policy names this org. |
| subnet, security group | Both exist in the region and share a VPC. |
| addressing | A subnet reached at public IPs assigns them. |
| zone | The subnet's zone offers every box size (m7a). us-east-1's `use1-az3` does not. |
| internet | The subnet routes `0.0.0.0/0` to an internet gateway (public) or a NAT (private), so boxes can reach `api.tesser.sh` and Session Manager. |
| ssh | The group does not admit TCP 22. An open port 22 is a warning: nothing needs it. |
| mesh | The group allows TCP 14650 from itself. Without it, service dependencies fail. |
| launch | A dry run of `RunInstances` for every box size, with the image, instance profile, and tags a real launch carries. A size the policy refuses is a warning; the default size is a failure. |
| manage | Once the org has a box: a dry run of stopping, starting, terminating, and tagging it. A refusal here means boxes cannot sleep or be removed. |
| quota | How many on-demand vCPUs the account may run at once. New accounts are often capped low. |
| federation | The account has outbound identity federation on, so boxes can get the identity token they prove themselves with. |
| parameter store | With `--env parameter-store`, the role can write and delete under the org's prefix. |
| snapshots | With `--snapshots-bucket`, always skipped: Tesserc's role has no access to the bucket, so `connect` cannot test it. Each snapshot build checks the boxes' access before it runs `setup`. |
| s3 route | With `--snapshots-bucket`, how boxes reach S3. A private subnet with no S3 gateway endpoint is a warning: the NAT charges for every snapshot byte it carries. |

`tesser cloud check` runs them again at
any time without changing anything. `tesser cloud` shows the config, and AWS
errors from the last seven days: a call AWS keeps refusing, such as a
`StopInstances` a tightened policy no longer allows, stays listed until it
works again. A dry run cannot test stopping or terminating an instance that
does not exist yet, so those only surface there.

## What the control plane can do

Your code, the disks, the dev server logs, and the traffic between boxes
stay in your account. [Where your code goes](https://tesser.sh/docs/where-your-code-goes.md) lists
every hop and everything the control plane receives. What the control plane
can still do to your boxes, through the role and boxd's WebSocket:

* **Start, stop, and terminate** the org's instances, and launch new ones
  from the image the role allows, into your subnet and group.
* **Start and stop the dev server** declared in your manifest. A start
  names only the service; boxd reads the command from the manifest on the
  box's own disk.
* **Set its env.** boxd drops keys that load a program (`NODE_OPTIONS`,
  `LD_PRELOAD`, `BASH_ENV`, `PROMPT_COMMAND`, the `GIT_` and
  `NPM_CONFIG_` families, and more). The list is a denylist, so it covers
  the known hooks, not every program's. With Parameter Store the control
  plane can write values but not read them.
* **Update boxd.** An update makes boxd fetch a new bundle from the control
  plane, and boxd installs it only if it carries Tesserc's release
  signature; first boot checks the same signature. The control plane can
  pick which signed release runs, not write its own.
* **Replace `authorized_keys`.** sshd has no network path except Session
  Manager, which your IAM gates, so a key alone reaches nothing.

## How Tesserc reaches your account

Tesserc never holds a credential for your account. The control plane is an
OpenID Connect issuer, and the role trusts it for exactly one org id. Each
time the org cell needs EC2 it signs a token that lives five minutes, trades
it for one hour of temporary credentials with `AssumeRoleWithWebIdentity`,
and uses those. Another org's cell cannot assume your role: the trust policy
names the org in `sub`, and the token carries the org that signed it. The
permissions are fenced the same way: the role can only launch instances
tagged with the org id, and only stop, start, or terminate instances that
carry it.

## Session Manager

Every ssh, sync, and daemon tunnel runs `tesser ssm-proxy` as its
ProxyCommand: tesser's own Session Manager client, so no AWS CLI and no
session-manager-plugin on the laptop. The host key is filed under the
instance id.

What it needs:

* **The instance profile** carries `AmazonSSMManagedInstanceCore` (the SSM
  agent ships in the Tesserc image), and the role tesser assumes has
  `iam:PassRole` on its role. The program above does both.
* **Every org member** needs an AWS profile in `~/.aws/config` (or
  `~/.aws/credentials`) for the account, with `ssm:StartSession` and
  `ssm:TerminateSession` on the boxes. Any profile kind works (SSO, static
  keys, `source_profile` + `role_arn`, `credential_process`); tesser reads
  the same files and SSO token cache the AWS CLI does, so `aws sso login --profile <name>` is still how an SSO profile signs in. `--aws-profile <name>` on `cloud connect` is the org's default profile name, the one every
  member's laptop is expected to have; a laptop whose profile is named
  differently overrides it with `tesser cloud aws-profile <name>`, kept in
  that laptop's tesser config. With `--sso-start-url <portal> --sso-region <region> --sso-role <permission set>` on `cloud connect`,
  members don't have to set anything up: a laptop missing the profile, with
  nothing else for the account, gets it appended to `~/.aws/config` (account
  from the role ARN) the first time it makes or wakes a box, and the SSO
  sign-in below takes it from there. A profile the laptop already has is
  never touched.
* **Agents and CI** don't need the profile. When it is missing or will not
  resolve, tesser uses the default AWS credential chain (env keys, an
  instance or container role, the default profile) once STS says those
  credentials are for the org's account; credentials for any other account
  are never used. When neither works (SSO session expired, keys rejected),
  every ssh-driven verb fails immediately and the
  daemon holds its tunnels until it works again. For an SSO profile the verb
  also opens `/__tesser/aws?profile=<name>` on the daemon's port: the daemon
  starts the device-code sign-in `aws sso login --use-device-code` does, shows the code, links to
  the AWS approval page, and writes the token to `~/.aws/sso/cache` when it is
  approved, so no AWS CLI is needed. The tunnels resume on their own. The
  verb's error tells an agent to run `tesser aws-wait <name>` once, in the
  background, and to rerun the command once it says signed in. `tesser aws-wait`
  returns once the code is approved and fails with the reason if it is denied
  or expires. If it fails, the error tells the agent to ask the person at the
  laptop instead of starting another wait. If nobody answers, the daemon opens the sign-in page at most three
  times for a profile and then stops opening it. From then on the verb's error
  says to finish signing in at that page and to rerun the command only once
  the page says signed in. The count starts over when someone uses the page
  (its approval link or **Try again**) or runs a verb in an interactive
  terminal.

That access as an Identity Center permission set, in Pulumi:

```ts theme={"theme":"css-variables"}
export function createDevboxPermissionSet(
  ssoInstanceArn: pulumi.Input<string>,
  provider: aws.Provider
): aws.ssoadmin.PermissionSet {
  const permissionSet = new aws.ssoadmin.PermissionSet(
    'devbox-permission-set',
    {
      instanceArn: ssoInstanceArn,
      name: 'DevboxAccess',
      description: 'Open a Session Manager session to a managed dev box, and nothing else',
      sessionDuration: 'PT12H',
    },
    { provider }
  );

  new aws.ssoadmin.PermissionSetInlinePolicy(
    'devbox-inline-policy',
    {
      instanceArn: ssoInstanceArn,
      permissionSetArn: permissionSet.arn,
      inlinePolicy: JSON.stringify({
        Version: '2012-10-17',
        Statement: [
          {
            Sid: 'StartSessionOnDevBoxes',
            Effect: 'Allow',
            Action: 'ssm:StartSession',
            Resource: 'arn:aws:ec2:*:*:instance/*',
            Condition: {
              StringEquals: { 'ssm:resourceTag/tesser:managed': 'true' },
            },
          },
          {
            Sid: 'StartSessionWithSSHDocument',
            Effect: 'Allow',
            Action: 'ssm:StartSession',
            Resource: 'arn:aws:ssm:*::document/AWS-StartSSHSession',
          },
          {
            Sid: 'OpenOwnDataChannel',
            Effect: 'Allow',
            Action: 'ssmmessages:OpenDataChannel',
            Resource: 'arn:aws:ssm:*:*:session/${aws:userid}-*',
          },
          {
            // the users are federated so we have to use this StringLike method
            // instead of a simpler aws:username based method
            Sid: 'ManageOwnSessions',
            Effect: 'Allow',
            Action: ['ssm:TerminateSession', 'ssm:ResumeSession'],
            Resource: '*',
            Condition: {
              StringLike: {
                'ssm:resourceTag/aws:ssmmessages:session-id': '${aws:userid}*',
              },
            },
          },
          {
            Sid: 'SessionDiscovery',
            Effect: 'Allow',
            Action: [
              'ssm:DescribeInstanceInformation',
              'ssm:DescribeSessions',
              'ssm:GetConnectionStatus',
            ],
            Resource: '*',
          },
        ],
      }),
    },
    { provider }
  );

  return permissionSet;
}
```

`ManageOwnSessions` is the statement to get right. Scoping it by session ARN
the way [AWS leads with](https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-restrict-access-examples.html#restrict-access-example-user-sessions-username)
only matches an IAM user: a federated caller's `${aws:userid}` is
`AROAEXAMPLEID:you@example.com` while the session id is
`you@example.com-<suffix>`, so every terminate is denied, and silently, since
tesser ignores a failed close, leaving sessions Active until SSM times them
out. The [session-tag form](https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-restrict-access-examples.html#restrict-access-example-user-sessions-tags)
above works either way, with the same scoping: your own sessions only.

Laptop-to-box transfers (sync, git packs) run at roughly link speed through
Session Manager: tesser's client sends 32 KiB frames with a 4 MiB window,
where the official plugin paces itself to \~0.7 MB/s. Box-to-laptop is paced
by the SSM agent on the instance at about 0.7 MB/s, which `logs` and
`exec` output never approach.

## Env in your Parameter Store

By default `tesser env set` stores a service's values in the org, and the
control plane injects them when an instance starts. With
`--env parameter-store` they live in your account instead, one SecureString
per key under `/tesser/<org id>/<service>/<KEY>` in the org's region,
encrypted with the account's `aws/ssm` key:

```text theme={"theme":"css-variables"}
tesser cloud connect --account 123456789012 --region us-east-1 --env parameter-store
```

Who touches them:

* **`tesser env set` / `unset`** go through the control plane, which writes
  the value into your Parameter Store and keeps only the key's name, so
  `tesser env ls` and the `env.required` check need no AWS call. The role
  it assumes needs `ssm:PutParameter` and `ssm:DeleteParameters` on
  `parameter/tesser/<org id>/*` and nothing else: no `ssm:GetParameter*`,
  no `ssm:DescribeParameters`. That one statement is what you audit. A
  parameter changed or removed outside `tesser env` is not noticed by
  `env ls` or the `env.required` check; boxes see it the next time they
  read.
* **Each box** reads the values itself with the instance profile's
  credentials: when its dev server starts (a service box reads its
  service's, a workbench every service of its repository, as with
  cell-held env) and, for shells, on connect and after every `env set`.
  That profile's role needs `ssm:GetParametersByPath` on the same prefix;
  the Pulumi program adds it with `tesser:parameterStore`. Any box in the org can read any of the
  org's services' env, which is the same trust boundary as the mesh.

`connect` proves the role can write and delete under the prefix before
storing anything, then moves the values the org already holds into
your Parameter Store and forgets them. A value is at most 4 KB (the standard
tier), and may not be blank; unset a key to remove it. If a box cannot read
its env at start, the run fails with the AWS error in `tesser dev`'s output
rather than starting without it. Moving `--env` back to `tesser`, or
`disconnect`, leaves the parameters in your account and the org with no
values: set them again. If you later move back to `parameter-store`, boxes
read whatever is still under the prefix, including values the org no longer
lists; delete the stale ones or set them again. `tesser service rm` deletes the service's
parameters. Because these are your parameters, the control plane can also
set them, the same way it can start the dev server; what it cannot do is
read one.

## Snapshots

[Snapshots](https://tesser.sh/docs/boxes.md#snapshots) of a connected org live in an S3 bucket in
your account. Until you name one, the org has no snapshots and
`tesser snapshot create` fails. Make a bucket in the org's region with public
access blocked, default encryption SSE-S3 (or SSE-KMS), no versioning, and no
lifecycle rule, since Tesserc deletes the snapshots it no longer needs. Then
connect with it:

```sh theme={"theme":"css-variables"}
tesser cloud connect --snapshots-bucket acme-tesser-snapshots
```

Boxes read and write the bucket themselves, with the credentials of the
boxes' role from their instance profile. The role Tesserc assumes has no S3
or KMS access to it, so snapshot data stays out of the control plane's IAM.
Boxes also delete the snapshots Tesserc no longer keeps. `tesser cloud
template` prints the two policies the bucket needs:

* **The boxes' role** gets `s3:GetObject`, `s3:PutObject`, and
  `s3:DeleteObject` on `<bucket>/<org id>/*`, and `s3:ListBucket` on the
  bucket for keys under `<org id>/`. For an SSE-KMS bucket, pass
  `--snapshots-kms-key <arn>` to `template` and it adds
  `kms:GenerateDataKey` and `kms:Decrypt` on that key, through S3 only.
* **The bucket** gets a policy that denies any request not made over TLS.

Because the access belongs to the boxes' role, any box of the org can read
and rewrite the org's snapshots. That is the same trust boundary as the
org's env in [Parameter Store](#env-in-your-parameter-store). Tesserc still
runs boxd and the image on every box, so this keeps snapshot bytes out of
the control plane's IAM, not out of Tesserc's reach.

Give the subnet's route table an S3 gateway endpoint, which is free. Behind
a NAT, every snapshot byte is billed as NAT data processing.
`--snapshots-bucket none` stops using snapshots.

## Changing or leaving

`tesser cloud connect --max-boxes <n>`, `--env <store>`, `--image <ami>`,
`--aws-profile <name>`, and `--no-crash-output` (boxd keeps a crashed dev
server's last lines on the box instead of sending them with its exit) work
at any time; the other flags default to the
stored config. Where boxes run cannot change while any exist. Remove them,
then `connect` again with the new ids, or `tesser cloud disconnect` to
return the org to Tesserc's cloud.

Disconnecting deletes nothing in your account. `pulumi destroy` removes the
network, the roles, and any image copy when you no longer need them.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.