Installing Influxdb and Grafana on EMR

-
I wanted to install recent versions of Grafana/InfluxDB stack onto a EMR cluster node for publishing metrics from Spark. This will allow me to receive metrics coming directly from the Spark listeners.

Used Dockerfile here for configs and setup.



 

Find Amazon Linux version

cat /etc/issue
Amazon Linux AMI release 2018.03
cat /etc/system-release
Amazon Linux AMI release 2018.03

Install Version 5.4.2 of Grafana

sudo yum install https://dl.grafana.com/oss/release/grafana-5.4.2-1.x86_64.rpm

Install Version 1.7.2 on Influx

sudo yum install https://repos.influxdata.com/centos/6/x86_64/stable/influxdb-1.7.2.x86_64.rpm

Configure Grafana/Influx

Move influxdb/config.toml from linked repo to /etc/influxdb/config.toml
Move graphana/config.ini from linked repo to /etc/grafana/config.ini

Add these lines to grafana config file:
type = influxdb
host = localhost:8086
name = grafana
user = grafana
password = grafana

Start Influx

/usr/bin/influxd -config=/etc/influxdb/config.toml &

Launch influx shell

/usr/bin/influx

Create the following users and databases (while in influx shell):
create database data
create database grafana
use data
create user data with password 'data'
use grafana
create user grafana with password 'grafana'

Influx db API starts on : http://localhost:8086

Start Grafana

/usr/sbin/grafana-server --homepath /usr/share/grafana --config /etc/grafana/config.ini

Grafana default username : admin
Grafana default password : admin

Using Dynamic port forwarding and having foxy proxy enable I can see Grafana here: http://ip-10-244-140-82.us-west-2.compute.internal:3000

Command Reference

Command Description Notes
rpm -ql grafana Describe rpm install Used this after I couldnt find how to start grafana-server
drop MEASUREMENT dynamodb Drop measurement in influx Purge Influx after bad metrics



Comments

Post a Comment

Popular posts from this blog

Using Selenium Testing for Electron (Atom shell) Applications

-
Image
Electron (formerly Atom Shell) is a very new way to quickly create javascript applications for multiple platforms. Not a lot of documentation exists about using Selenium to do full-fledged integration testing on Electron Applications. And even less exists about performing such tests in python. This brief tutorial should answer a few questions about Electron Applications and show you how to get started using Selenium to test them. I. Getting started Install and start chromedriver Selenium need this to be able to make calls to the Electron App. Chromedriver acts as a bridge between Selenium and Chrome, it follows Selenium wire protocol. By default, chromium runs on port 9515, you can start on alternate ports, but remember the assigned port this will be passed as an argument to Selenium later. ./chromedriver --port=9515 Install Selenium You'll need to use Selenium's remotewebdriver to interface with chromedriver. Im using a python virtualenv to keep all my python plugi...
Read more

Using Signoz and OpenTelemetry as an alternative to DataDog

-
Image
 Datadog is an essential tool for monitoring large applications, but for hobby projects Sigmoz is a great open source alternative that provides similar functionality. It's also free and easy to setup using docker-compose. Start off by cloning the signoz github repo. git clone https://github.com/SigNoz/signoz cd signoz/deploy If you already have docker and docker-compose installed than you can skip this, but otherwise run the install script. ./install.sh Launch the Sigmoz service. The docker-compose setup includes clickhouse database, Zookeeper service, and a sample application called hotrod. This will bring up everything. docker-compose -f docker/clickhouse-setup/docker-compose.yaml up Then navigate to http://localhost:3301/ and you'll be prompted to setup an account and admin password. Then you'll be able to see the Sigmoz homepage with some example metrics. We can then start on integrating the opentelemetry metrics with our java app. Open telemetry automat...
Read more

Packaging Electron Applications for OSX

-
Image
I want to use Electron to create installable apps for OSX, Precise, and Trusty. I first attempted to use fpm (package distribution across multiple platforms) which works fine for creating rpm and deb packages, but creating the OSX pkg file proved difficult. These steps will summarize how to create an installable .pkg from an Electron Application. NOTE: This tutorial does not cover creating valid packages for the mac app store, which has a whole different set of requirements. But you will end up with a .pkg file which can be distributed and used to install your application. I. Re-Branding Electron Unless you want your Application to be named "Electron" after its installation, you will probably want to rename the application and change the default icon. On OSX, this is done by editing the existing Info.plist file in the Electron.app directory and changing four values to match your Apps name. The Location of the Electron Info.plist file is found under the Electron.app...
Read more