Safely send data to AWS SQS from Spark worker machines

In this post, we are going to learn how to send messages to AWS SQS from spark worker machines using assume role. We will cover send data using Spark Executors and STS assume roles to communicate with AWS SQS service.

Prerequisites

  • Running Spark cluster on AWS
  • AWS SQS

Spark worker setup for sending data

Setup worker to send data from Executors

To send data from worker machine we need to perform transformation operation by calling a spark function. In this example, we will useĀ mapPartitions(FlatMapFunction<Iterator<String>,String>). Following method creates a new RDD by taking Iterator as input and output. This method will be called once by every Executor that is selected by Spark Master.

JavaRDD input = newRDD.mapPartitions(new FlatMapFunction<Iterator<String>, String>() {
  @Override
  public Iterator<String> call(Iterator<String> input) throws Exception {
    // send data to SQS code
    return input;
  }
});

Setup STS assume role to communicate with SQS

Now we need to code for send data to SQS. We will assume role because we need safe communication between spark cluster and AWS SQS. Sample code:

InstanceProfileCredentialsProvider credentialsProvider = new InstanceProfileCredentialsProvider(true);
AWSSecurityTokenService sts = AWSSecurityTokenServiceClientBuilder.standard().withCredentials(credentialsProvider).build();
STSAssumeRoleSessionCredentialsProvider provider = new STSAssumeRoleSessionCredentialsProvider.Builder(roleArn, roleSessionName).withStsClient(sts).build();
AmazonSQS sqs = AmazonSQSClientBuilder.standard().withCredentials(provider).build();
String sqsMessage = "Message from executor"; // message
sqs.sendMessage("SPARK-TEST-QUEUE", sqsMessage);

Source code: SparkWorkerSQSCommunication.java

Conclusion

In this post, we covered how to use Spark worker executors to communicate with AWS SQS using STS assume role. This post is an example of applying security and safely communicate with AWS services using assume role. This post also covers how to execute code from Executor Level.

Leave a Reply

Your email address will not be published. Required fields are marked *