Bastion server

Logging into servers through the bastion

To improve the security of our infrastructure it's not possible to connect directly to a production server with SSH. Instead, all connections must come from a small server called the "bastion", which only allows connections from a few whitelisted networks and logs any connection attempt.

To log into a server through the bastion you can use SSH's -J flag:

ssh -J bastion.infra.rust-lang.org servername.infra.rust-lang.org

It's also possible to configure SSH to always jump through the bastion when connecting to a host. Add this snippet to your SSH configuration file (usually located in ~/.ssh/config):

Host servername.infra.rust-lang.org
    ProxyJump bastion.infra.rust-lang.org

Please remember the bastion server only allows connections from a small list of IP addresses. Infra team members with AWS access can change the whitelist, but it's good practice to either have your own bastion server or a static IP address.

The SSH keys authorized to log into each account are stored in the simpleinfra repository. Additionally, people with sensitive 1password access can use the master key stored in the vault to log into every account, provided their connection comes from any whitelisted IP.

Common maintenance procedures

Adding a new user to the bastion server

To add a new user to the bastion you need to add its key to a file named <username>.pub in ansible/roles/common/files/ssh-keys, and change the Ansible playbook adding the user to the list of unprivileged users. Please leave a comment clarifying which servers the user will have access to.

Once that's done apply the playbook and add a new whitelisted IP address.

Adding a whitelisted IP

Due to privacy reasons, all the static IP addresses of team members with access to the bastion are stored on AWS SSM Parameter Store instead of public git repositories. To add an IP address you can run this command (taking care of replacing USERNAME and IP_ADDRESS with the proper values):

aws ssm put-parameter --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32"

You'll also need to add the username to the list in terraform/services.tf (key allowed_users in the service_bastion module). Once you made all the needed changes you wanted you need to apply the Terraform configuration.

Updating a whitelisted IP

Due to privacy reasons, all the static IP addresses of team members with access to the bastion are stored on AWS SSM Parameter Store instead of public git repositories. To update an IP address you can run this command (taking care of replacing USERNAME and IP_ADDRESS with the proper values):

aws ssm put-parameter --overwrite --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32"

Once you made all the needed changes you wanted you need to apply the Terraform configuration.

Removing a whitelisted IP

Due to privacy reasons, all the static IP addresses of team members with access to the bastion are stored on AWS SSM Parameter Store instead of public git repositories. To remove an IP address you can run this command (taking care of replacing USERNAME with the proper value):

aws ssm delete-parameter --name "/prod/bastion/allowed-ips/USERNAME"

You'll also need to remove the username from the list in terraform/services.tf (key allowed_users in the service_bastion module). Once you made all the needed changes you wanted you need to apply the Terraform configuration.