In this post, we are going to learn how to send messages to AWS SQS from spark worker machines using assume role. We will cover send data using Spark Executors and STS assume roles to communicate with AWS SQS service.


  • Running Spark cluster on AWS

Spark worker setup for sending data

Setup worker to send data from Executors

To send data from worker machine we need to perform transformation operation by calling a spark function. In this example, we will use mapPartitions(FlatMapFunction<Iterator<String>,String>). Following method creates a new RDD by taking Iterator as input and output. This method will be called once by every Executor that is selected by Spark Master.

JavaRDD input = newRDD.mapPartitions(new FlatMapFunction<Iterator<String>, String>() {
  public Iterator<String> call(Iterator<String> input) throws Exception {
    // send data to SQS code
    return input;

Setup STS assume role to communicate with SQS

Now we need to code for send data to SQS. We will assume role because we need safe communication between spark cluster and AWS SQS. Sample code:

InstanceProfileCredentialsProvider credentialsProvider = new InstanceProfileCredentialsProvider(true);
AWSSecurityTokenService sts = AWSSecurityTokenServiceClientBuilder.standard().withCredentials(credentialsProvider).build();
STSAssumeRoleSessionCredentialsProvider provider = new STSAssumeRoleSessionCredentialsProvider.Builder(roleArn, roleSessionName).withStsClient(sts).build();
AmazonSQS sqs = AmazonSQSClientBuilder.standard().withCredentials(provider).build();
String sqsMessage = "Message from executor"; // message
sqs.sendMessage("SPARK-TEST-QUEUE", sqsMessage);

Source code:


In this post, we covered how to use Spark worker executors to communicate with AWS SQS using STS assume role. This post is an example of applying security and safely communicate with AWS services using assume role. This post also covers how to execute code from Executor Level.

Safely send data to AWS SQS from Spark worker machines
Tagged on:     

Satish Pandey

I am an expert Java Spring Angular developer with 10+ years of rich and varied experience in developing end-to-end Web Applications. I maintain this blog and publish articles in my free time to help the community. Email:

Leave a Reply

Notify of