[ Shell Script ] Mulit AWS Instance 생성 후 코드 실행

[ 시나리오 ]

Master EC2에서 Shell Script로 

10개의 EC2를 생성해 S3를 이용한 'A' 라는 작업을 실행. 

작업실행 후 10개의 EC2 종료

[ 실행예제 ]

1. EC2 생성

2. S3를 가져와 A작업을 하는 _run.sh 코드작성

#!/bin/bash

if [ $# -ne 2 ]; then
 echo "Usage: $0 yyyyMMdd filename"
 exit -1
fi

DATE=$1
FILE=$2

AWS_REGION=s3.ap-northeast-2.amazonaws.com
AWS_ACCESS_KEY_ID=AKAKAKAAKAAAAA
AWS_SECRET_ACCESS_KEY=080sdfsdf(((D0sdf/3333

S3_INPUT_PATH="s3://bucket/input/${DATE}/${FILE}"
S3_OUTPUT_PATH="s3://bucket/input/${DATE}/"
LOCAL_INPUT_PATH="./data/input"
LOCAL_OUTPUT_PATH="./data/output"

echo "[INFO] CP INPUT From S3 To Local"
aws s3 cp ${S3_INPUT_PATH} "${LOCAL_INPUT_PATH}/"

echo "[INFO] Run A process"

...
A작업 실행코드
...

echo "[INFO] CP OUTPUT From Local To S3"
aws s3 cp "${LOCAL_OUTPUT_PATH}/output_${FILE}.csv" ${S3_OUTPUT_PATH}

echo "[INFO] FINISH JOB & SHUTDOWN"
shutdown

3. AMI 이미지 생성 후 EC2 종료

4. Master EC2가 Aws환경설정이 안되어있다면

#!/bin/sh

# curl을 사용하여 aws cli 패키지다운로드
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip

# 설치 진행
sudo ./aws/install -i /usr/local/aws-cli -b /usr/local/bin

# 버전확인
aws --version

5. Master EC2에서 run_instance.sh 코드 작성

#!/bin/sh

# 변수값 정의
DATE=$1
IMAGE_ID=ami-00000000id
INSTANCE_TYPE=t3.small
SUBNET_ID=subnet-77777aa
SECURITY_GROUP_ID=sg-0ffffffff00000
KEY_NAME=your_aws_key
AWS_ACCESS_KEY_ID=AKAKAKAKAKAKAKAAK
AWS_SECRET_ACCESS_KEY=88OOOO888OOOOO

START_NUMBER="00000"
FINISH_NUMBER="00009"
    
/usr/local/bin/aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID"
/usr/local/bin/aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY"
/usr/local/bin/aws configure set default.region "ap-northeast-2"
	
run_instance(){
    S3_FILE=part-$1
	
    /usr/local/bin/aws ec2 run-instances --image-id ${IMAGE_ID}  --count 1 --instance-type ${INSTANCE_TYPE} \
    --subnet-id ${SUBNET_ID} --security-group-ids ${SECURITY_GROUP_ID} --instance-initiated-shutdown-behavior terminate \
    --key-name ${KEY_NAME} --user-data \
    "
    #!/bin/bash
    cd /home/centos/exec
    sh _run.sh ${DATE} ${S3_FILE}
    " > /dev/null
}

echo "[INFO] START JOB : ${DATE}"
for CURR_NUMBER in $(seq -f "%05g" $START_NUMBER $FINISH_NUMBER) 
    do
        run_instance ${CURR_NUMBER}
    done
fi
    
echo "[INFO] FINISH JOB"
exit 0

5. run_instance.sh 실행 후, EC2 10개가 생겼다가 _run.sh실행 후 종료, S3적제