SSH Over AWS Systems Manager: Ditching Key Pairs for IAM Authentication

Sat, Nov 1, 2025
7-minute read

If you’ve ever managed SSH keys for EC2 instances at scale, you know the pain. Keys get lost, rotated, shared inappropriately, or worse - committed to Git repos. What if I told you that you could use all your familiar SSH tooling - ssh, scp, rsync, port forwarding, and more - without ever touching an SSH key pair? Enter AWS Systems Manager Session Manager as an SSH proxy.

Why SSH Over SSM?

The traditional SSH model requires:

Managing SSH key pairs (creation, distribution, rotation, revocation)
Exposing port 22 to the internet (or VPN) in security groups
Bastion hosts or jump boxes for private instances
Manual key distribution to team members

With SSM as an SSH proxy, you get:

IAM-based authentication: Access control through IAM policies, not key management
No exposed ports: No need to open port 22 in security groups
Session logging: All sessions can be logged to S3 and CloudWatch for compliance
Works with private instances: Direct access to instances in private subnets with no bastion and no IPSEC VPN tunnel required
Standard SSH tools: Use scp, rsync, ssh-add, and all the tools you already know

Don’t forget that SSM is not restricted to just AWS. You can run the SSM agent on premise, in Azure, in GCP, or realistically anywhere you’re running an operating system. While other cloud providers may have similar features, there are benefits to using a consistent solution across all your virtual machines.

Prerequisites

Before we begin, you’ll need:

AWS CLI installed and configured (aws configure)
Session Manager plugin for AWS CLI (installation guide)
An EC2 instance with the SSM Agent installed (Amazon Linux 2023, Ubuntu, and most modern AMIs have it pre-installed)
IAM permissions for Session Manager
The instance needs an IAM instance profile with AmazonSSMManagedInstanceCore policy

Installing the Session Manager Plugin

The Session Manager plugin is what makes the magic happen. It acts as a proxy between your SSH client and the SSM service. The most up-to-date instructions can be found in the AWS documentation.

Verify the installation:

session-manager-plugin

You should see usage information if it’s installed correctly.

Configuring SSH to Use SSM

The beauty of this approach is that it requires minimal configuration changes. We’ll add a configuration to your ~/.ssh/config file that tells SSH to proxy connections through SSM.

Basic SSH Config

Add this to your ~/.ssh/config:

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

That’s it! This configuration tells SSH to use the ProxyCommand for any host that matches the pattern i-* or mi-* (EC2 instance IDs).

Now you can SSH directly to an instance using its instance ID:

ssh ec2-user@i-1234567890abcdef0

Advanced Configuration

For a more robust setup with multiple AWS accounts and regions:

# SSH over Session Manager - Production Account
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile production --region us-east-1"
    User ec2-user
    IdentityFile /dev/null
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null

# SSH over Session Manager - Development Account
host dev-i-* dev-mi-*
    ProxyCommand sh -c "aws ssm start-session --target $(echo %h | sed 's/^dev-//') --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --profile development --region us-west-2"
    User ubuntu
    IdentityFile /dev/null
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null

The second configuration uses a prefix (dev-) to distinguish between accounts, so you can do:

ssh dev-i-1234567890abcdef0

What About Host Key Verification?

You might have noticed StrictHostKeyChecking no in the advanced config. Since each SSM session creates a new endpoint, traditional host key checking doesn’t work well. If this makes you uncomfortable (and it should!), you have a few options:

Use SSM session logging to S3/CloudWatch for audit trails
Implement session tags and CloudTrail logging for compliance
Use AWS IAM policies to restrict which users can access which instances

The IAM-based authentication is actually more secure than SSH keys in most scenarios because:

You can use MFA requirements in IAM policies
You can implement time-based access restrictions
You can use AWS Organizations SCPs for guardrails
Sessions are automatically audited in CloudTrail

IAM Permissions Required

Your IAM user or role needs these permissions to start SSM sessions:

Note: The JSON below includes comments for clarity - remove them before using in AWS as JSON doesn’t support comments.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ssm:StartSession"],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*", // You should limit this in practice to specific instance IDs or use tags
        "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["ssm:TerminateSession", "ssm:ResumeSession"],
      "Resource": ["arn:aws:ssm:*:*:session/${aws:username}-*"]
    }
  ]
}

Using SCP and Other SSH Tools

Once SSH is configured, all your standard tools work seamlessly:

SCP (Secure Copy)

# Copy file to instance
scp myfile.txt ec2-user@i-1234567890abcdef0:/home/ec2-user/

# Copy file from instance
scp ec2-user@i-1234567890abcdef0:/var/log/application.log ./

Rsync

# Sync directory to instance
rsync -avz --progress ./local-dir/ ec2-user@i-1234567890abcdef0:/remote-dir/

# Sync from instance
rsync -avz --progress ec2-user@i-1234567890abcdef0:/remote-dir/ ./local-dir/

SSH Port Forwarding

# Forward remote port to local
ssh -L 8080:localhost:80 ec2-user@i-1234567890abcdef0

# Now access http://localhost:8080 to reach port 80 on the instance

Git Over SSH

If your instances host Git repositories:

git clone ec2-user@i-1234567890abcdef0:/path/to/repo.git

VSCode Remote SSH

Add your instance to SSH config with a friendly name:

host my-dev-server
    HostName i-1234567890abcdef0
    User ec2-user
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Then in VSCode, use Remote-SSH extension and connect to my-dev-server.

Instance Configuration

Your EC2 instances need minimal configuration:

1. IAM Instance Profile

Attach an instance profile with the AmazonSSMManagedInstanceCore managed policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:UpdateInstanceInformation",
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": "*"
    }
  ]
}

2. SSM Agent

Most modern AMIs come with SSM Agent pre-installed. To verify:

# Amazon Linux 2023
sudo systemctl status amazon-ssm-agent

# Ubuntu
sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent

If it’s not installed, follow the AWS documentation for your OS.

3. Network Configuration

The beauty of SSM is that you don’t need port 22 open! Your security group can be:

**Outbound:**

- HTTPS (443) to 0.0.0.0/0 (for SSM endpoints)

**Inbound:**

- No SSH rules needed!

For private subnets, you’ll need either:

VPC endpoints for SSM, SSM Messages, and EC2 Messages, OR
NAT Gateway/Instance for internet access

Troubleshooting

“TargetNotConnected” Error

This means the instance isn’t registered with Systems Manager. Check:

SSM Agent is running on the instance
Instance has the correct IAM instance profile
Instance can reach SSM endpoints (check security groups and NACLs)
Instance is in a supported region

List your managed instances:

aws ssm describe-instance-information

Permission Denied Errors

Verify your IAM permissions allow ssm:StartSession for the target instance and document.

Connection Timeout

Check that:

SSM Agent is running
Instance has network access to SSM endpoints
Your AWS CLI is configured with the correct region

Session Logging and Compliance

SSM offers built-in session logging. You can enable it with:

S3 Logging: Store session logs in S3 for long-term retention
CloudWatch Logs: Stream session data to CloudWatch for real-time monitoring
CloudTrail: All StartSession API calls are logged

Configure session logging in Session Manager preferences:

aws ssm update-document \
    --name "SSM-SessionManagerRunShell" \
    --content file://session-logging-config.json \
    --document-version '$LATEST'

Terraform Example

Here’s a complete Terraform example to set up an instance with SSM access:

# IAM role for the instance
resource "aws_iam_role" "ssm_role" {
  name = "ec2-ssm-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

# Attach SSM managed policy
resource "aws_iam_role_policy_attachment" "ssm_policy" {
  role       = aws_iam_role.ssm_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

# Instance profile
resource "aws_iam_instance_profile" "ssm_profile" {
  name = "ec2-ssm-profile"
  role = aws_iam_role.ssm_role.name
}

# Security group (no SSH port needed!)
resource "aws_security_group" "ssm_sg" {
  name        = "ssm-managed-instance"
  description = "Security group for SSM-managed instance"
  vpc_id      = var.vpc_id

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# EC2 instance
resource "aws_instance" "app_server" {
  ami           = data.aws_ami.amazon_linux_2023.id
  instance_type = "t3.micro"

  iam_instance_profile   = aws_iam_instance_profile.ssm_profile.name
  vpc_security_group_ids = [aws_security_group.ssm_sg.id]

  tags = {
    Name        = "SSM-Managed-Instance"
    Environment = "development"
  }
}

Conclusion

SSH over AWS Systems Manager Session Manager is a game-changer for managing EC2 instances. By leveraging IAM for authentication, you eliminate the complexity and security risks of SSH key management while gaining powerful audit and compliance capabilities.

The best part is that it works with all your existing SSH-based tools and workflows. No need to learn new commands or change your muscle memory. Just configure your SSH client once and enjoy keyless, secure access to your instances.

Whether you’re managing a handful of instances or thousands across multiple AWS accounts, this approach scales beautifully and integrates seamlessly with your existing AWS security posture.

Give it a try, and you’ll wonder how you ever managed without it. Your security team will thank you and you’ll never have to hunt down another lost SSH key.

Additional Resources

Questions? Thoughts? Feel free to reach out at mike@graywind.org.

aws security aws systems-manager ssh iam security