I am trying to automate a workflow where an AWS Lambda function triggers, connects to a source EC2 instance via SSH, reads a specific configuration file, modifies the text, and then uploads the result to a destination EC2 instance. I know I should probably use Boto3 and the Paramiko library for SSH, but I’m confused about how to handle the private keys securely and manage the security group rules. Could someone share a Pythonic approach to handle this cross-instance transfer without using S3 as an intermediary?
3 answers
The most robust way to do this in Python is using the Paramiko library to establish SSH connections. Since Lambda doesn't come with Paramiko by default, you'll need to create a Lambda Layer. In your script, you would use paramiko.SSHClient() to connect to Instance A, read the file using an SFTP client, perform your string manipulations in memory, and then open a second connection to Instance B to write the file. For security, never hardcode your .pem key; instead, store it in AWS Secrets Manager and have your Lambda fetch it at runtime. Also, ensure your Lambda is inside the same VPC and the destination EC2 security groups allow inbound traffic on Port 22 from the Lambda's security group.
That is a comprehensive setup, but have you considered the execution timeout limits for Lambda? If the files are large or the network latency between instances is high, the function might time out before completing the second upload. Would it be more efficient to use AWS Systems Manager (SSM) Run Command instead of raw SSH to avoid managing private keys and open port 22 entirely?
I’ve implemented a similar pattern for log processing. One tip: make sure your Lambda has a long enough timeout (at least 1 minute) and enough memory to hold the file contents in a variable if you aren't using a temporary /tmp/ directory.
I agree with Brandon. The /tmp/ directory in Lambda is great for this—you can write the modified file locally before the final upload to the second EC2. I used this exact logic last year for a Quality Management project where we had to sanitize logs across different environments, and using the local temp storage made the process much more stable
Ryan, using SSM is actually the more "AWS-native" way to handle this in modern Cloud Technology workflows. With the ssm.send_command() method in Boto3, you can execute shell commands to cat a file or move it via SCP without needing Paramiko layers. However, the original user asked for a direct Python manipulation method. If the file is small, the Paramiko route gives you more granular control over the data transformation logic within the Lambda script itself before it ever touches the second instance.