Using Spark to load DynamoDB tables with AWS Spark-Dynamodb connector.

-

Monitoring DynamoDB Capacity

Getting a true picture of DynamoDB WCU/RCU capacity is difficult because the default monitors automatically aggregate WCU/RCU metrics by minute. This hides spikes and abstracts away true metrics of WCU/RCU consumption (Cloudwatch also does the same aggregation by minute). In order to get a more accurate picture of these metrics, we decided to use Grafana/Influx stack described in my other post to capture second level metrics for WCU/RCU consumption.



Our Use case

  • 200 TB dataset stored in Parquet on s3
  • Ingest into 20 DynamoDB tables using Spark
  • S3 -> Spark -> DynamoDB using AWS labs emr-dynamodb-hadoop connector
To ingest a dataset this large in a reasonable amount of time we need to make sure DynamoDB is using all possible capacity across all tables, so good monitoring is critical.

The AWS labs Spark connector emr-dynamodb-hadoop has params which let you configure what percentage of your Dynamodb provisioned capacity should be consumed by Spark. The connector even allows you to use over 100% of your Dynamodb WCUs by setting to a value over 1.0 (100%). 

The frustrating part was even after setting 'dynamodb.throughput.write.percent'=1.5 (should allow Spark executors to consume 1.5 times the table WCUs) we still got a WCU graph like below, where consumed WCUs is far below the provisioned WCU limit.



Setup

Connect emr-dynamodb-hadoop table in spark job.
Get the Spark jobconf and set the following parameters:

 var ddbConf = JobConf(spark.sparkContext.hadoopConfiguration)
 ddbConf.set("dynamodb.output.tableName", "output_table_name")
 ddbConf.set("dynamodb.throughput.write.percent", "1.5")
 ddbConf.set("mapred.input.format.class", "org.apache.hadoop.dynamodb.read.DynamoDBInputFormat")
 ddbConf.set("mapred.output.format.class", "org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat")

To get a more accurate monitor I made the following changes to the emr-dynamodb-hadoop connector.

1) Build small wrapper around influxdb. Use the influx-java jar:

<dependency>
    <groupId>org.influxdb</groupId>
    <artifactId>influxdb-java</artifactId>
    <version>2.8</version>
</dependency>

2) Add logging metrics into emr-dynamodb-hadoop








Comments

Post a Comment

Popular posts from this blog

Using Selenium Testing for Electron (Atom shell) Applications

-
Image
Electron (formerly Atom Shell) is a very new way to quickly create javascript applications for multiple platforms. Not a lot of documentation exists about using Selenium to do full-fledged integration testing on Electron Applications. And even less exists about performing such tests in python. This brief tutorial should answer a few questions about Electron Applications and show you how to get started using Selenium to test them. I. Getting started Install and start chromedriver Selenium need this to be able to make calls to the Electron App. Chromedriver acts as a bridge between Selenium and Chrome, it follows Selenium wire protocol. By default, chromium runs on port 9515, you can start on alternate ports, but remember the assigned port this will be passed as an argument to Selenium later. ./chromedriver --port=9515 Install Selenium You'll need to use Selenium's remotewebdriver to interface with chromedriver. Im using a python virtualenv to keep all my python plugi...
Read more

Using Signoz and OpenTelemetry as an alternative to DataDog

-
Image
 Datadog is an essential tool for monitoring large applications, but for hobby projects Sigmoz is a great open source alternative that provides similar functionality. It's also free and easy to setup using docker-compose. Start off by cloning the signoz github repo. git clone https://github.com/SigNoz/signoz cd signoz/deploy If you already have docker and docker-compose installed than you can skip this, but otherwise run the install script. ./install.sh Launch the Sigmoz service. The docker-compose setup includes clickhouse database, Zookeeper service, and a sample application called hotrod. This will bring up everything. docker-compose -f docker/clickhouse-setup/docker-compose.yaml up Then navigate to http://localhost:3301/ and you'll be prompted to setup an account and admin password. Then you'll be able to see the Sigmoz homepage with some example metrics. We can then start on integrating the opentelemetry metrics with our java app. Open telemetry automat...
Read more

Packaging Electron Applications for OSX

-
Image
I want to use Electron to create installable apps for OSX, Precise, and Trusty. I first attempted to use fpm (package distribution across multiple platforms) which works fine for creating rpm and deb packages, but creating the OSX pkg file proved difficult. These steps will summarize how to create an installable .pkg from an Electron Application. NOTE: This tutorial does not cover creating valid packages for the mac app store, which has a whole different set of requirements. But you will end up with a .pkg file which can be distributed and used to install your application. I. Re-Branding Electron Unless you want your Application to be named "Electron" after its installation, you will probably want to rename the application and change the default icon. On OSX, this is done by editing the existing Info.plist file in the Electron.app directory and changing four values to match your Apps name. The Location of the Electron Info.plist file is found under the Electron.app...
Read more