Safely send data to AWS SQS from Spark worker machines

In this post, we are going to learn how to send messages to AWS SQS from spark worker machines using assume role. We will cover send data using Spark Executors and STS assume roles to communicate with AWS SQS service. Prerequisites Running Spark cluster on AWS AWS SQS Spark worker…

Read more »

Safely Launch Spark job using AWS Lambda and SNS

Introduction In this post, we are going to learn how to safely launch Apache Spark job using AWS lambda function and SNS topic. We are going to use Assume Role in our lambda function. This way it is much safer and no need to use long-term user access credentials. The…

Read more »

Running Systems Manager Run Command from AWS Lambda

In this post, we are going to learn how to execute EC2 run commands from AWS Lambda. Using lambda function as an upper layer to run command can be useful to validate command (if that needs to be run) or apply suitable runtime parameters values. Prerequisites AWS Lambda Systems manager…

Read more »

Launch / Terminate Linux services using AWS Run Command

Introduction In this post, we will learn controlling system services using AWS Run Command. We will see how to control Apache service with an example. The same process can be applied to other services like Mysql, Tomcat, etc. In small IT firms, there are limited team members who know about the…

Read more »

Invoking lambda function using SNS Trigger

Introduction In this post, we are going to learn invoking lambda function using SNS Trigger. We will use SNS topic to display how lambda function captures SNS notifications. Prerequisites SNS topic Lambda function Create SNS Topic Go to SNS management console. Create a new topic with name LambdaSNSTopic. For more information see…

Read more »

Learn AWS Lambda in 5 minutes with Java example

Introduction This post is useful for those who are new to AWS lambda and want a quick configuration and testing. In this post, we will cover both approaches (implement RequestHandler.handleRequest, custom handler) to create a lambda function. Follow this link to know about approaches we cover: Lambda Function Handler. Please visit AWS…

Read more »

Safely Manage AWS Security in Apache Spark

In this post, we will learn how to safely manage AWS security in Apache Spark. Apache Spark provides various filesystem clients (s3, s3n, s3a) for reading and writing to and from Amazon S3. Because s3 block filesystem is deprecated and s3n native is not maintained, I am going to use…

Read more »