[ Glue ] Pycharm, Notebook 개발환경 설정

[ 사전 설치 ]

1. Docker 

2. Pycharm

3. aws configure 설정

[ Dokcer ]

#이미지 다운로드 

docker pull amazon/aws-glue-libs:glue_libs_1.0.0_image_01

# 참고 문서 : https://aws.amazon.com/ko/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/

# 실행 - Mac / Linux

docker run -itd -p 8888:8888 -p 4040:4040 -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_1.0.0_image_01 /home/jupyter/jupyter_start.sh

# 실행 - Windows 

docker run -itd -p 8888:8888 -p 4040:4040 -v c://user_path//.aws:/root/.aws:rw --name glue_jupyter amazon/aws-glue-libs:glue_libs_1.0.0_image_01 /home/jupyter/jupyter_start.sh

# 제플린이라면 

8888:8888 -> 8080:8080

/home/jupyter/jupyter_start.sh -> /home/zeppelin/bin/zeppelin.sh

#아래 이미지와 같이 해당 container 생성 확인

# bash 셀 사용

docker exec -it glue_jupyter bash

[ 노트북 사용 ]

# notebook or zeppelin

http://localhost:<port_on_client>에 접속

[ Pycharm 설정 ]

아래 참조 url에 따라 실행

https://aws.amazon.com/ko/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/

* 주의할 내용은 Host path값이 C:\User\path\.aws라면

C://User//path//.aws로 치환해야 Pycharm에서 인식한다.

* .aws 폴더가 없다면 aws configure설정했는지 확인