In this post, we are going to learn how to send messages to AWS SQS from spark worker machines using assume role. We will cover send data using Spark Executors and STS assume roles to communicate with AWS SQS service.
Introduction In this post, we are going to learn how to safely launch Apache Spark job using AWS lambda function and SNS topic. We are going to use Assume Role in our lambda function. This way it is much safer
In this post, we will learn how to safely manage AWS security in Apache Spark. Apache Spark provides various filesystem clients (s3, s3n, s3a) for reading and writing to and from Amazon S3. Because s3 block filesystem is deprecated and