amazon web services - How to run glue script from Glue Dev Endpoint

Question

I have a glue script (test.py) written say in a editor. I connected to glue dev endpoint and copied the script to endpoint or I can store in S3 bucket. Basically glue endpoint is an EMR cluster, now how can I run the script from the dev endpoint terminal? Can I use spark-submit and run it ?

I know we can run it from glue console,but more interested to know if I can run it from glue end point terminal.

Answer 1

You don't need a notebook; you can ssh to the dev endpoint and run it with the gluepython interpreter (not plain python).

e.g.

radix@localhost:~$ DEV_ENDPOINT=glue@ec2-w-x-y-z.compute-1.amazonaws.com
radix@localhost:~$ scp myscript.py $DEV_ENDPOINT:/home/glue/myscript.py
radix@localhost:~$ ssh -i {private-key} $DEV_ENDPOINT
...
[glue@ip-w-x-y-z ~]$ gluepython myscript.py

You can also run the script directly without getting an interactive shell with ssh (of course, after uploading the script with scp or whatever):

radix@localhost:~$ ssh -i {private-key} $DEV_ENDPOINT gluepython myscript.py

If this is a script that uses the Job class (as the auto-generated Python scripts do), you may need to pass --JOB_NAME and --TempDir parameters.