IBM's Big Insights training notes

IBM’s Big Insights: A primer

Recently I had the opportunity to attend a training session about IBM’s Big Insights, in November 2013. Below are my notes about this product.

What is Big Insights in a nutshell?

Big Insights is IBM’s Big Data platform. It is comprised of an all-in-one Big Data infrastructure, with IBM’s flavor of Hadoop and its ecosystem, as well as proprietary tools to query the data like JAQL and AQL, and out-of-the-box connectors and interfaces called accelerators. We’ll review these components in details in the below section.

Big Insights Hadoop infrastructure

Big Insights is composed of a Hadoop infrastructure (independent from vendors like Cloudera). It is using a released version of Hadoop that is well-tested, usually a bit older from trunk. However it differs from the Apache version in some ways also. Big Insights comes integrated with:

-       GPFS (IBM’s version of HDFS) for its file system

-       Adaptive Map Reduce, an enhanced version of MR that attempts to optimize task executions, by way of using automatic job tuning of speculative execution and Task JVM reuses. Map Reduce tasks become aware of the global state of the job they are working in. This helps balance the workload across Map tasks. 

-       Zookeeper, HBase, Hive, Pig

Of note is the fact that Big Insights is not bundled with Cloudera’s CDH anymore; IBM has its own version of Hadoop.

New query language: JAQL

Big Insights offers a language called JAQL, a functional language that can interface will of all the Big Insights tools. It provides API's (or modules) for reaching out to external IBM and 3^rd party tools, such as relational databases, indexing services, text analytics, machine learning etc. JAQL stands for Json Query Language, because it is represented via Json. Similar to Pig, Jaql is automatically taking care managing the complexities of the MapReduce world to optimally perform the work. However it also manages deep level nested semi-structured data.

Jaql can be executed either from its own shell, or from within Eclipse.

Big Insights Applications

Big Insights provides an environment for developing and executing applications. A business user can launch existing applications from the Web console, supply any input parameters and view results.  These applications may be developed using Big Insights’ development tooling which enables programmers to publish completed applications through the Web console.

The BigInsights Eclipse tools include wizards, code generators, context-sensitive help, and a test environment to simplify your development efforts.

Workflow applications are run by Oozie as a workflow job.

Big Sheets

Big Insights also comes with a spreadsheet-like interface to interact with Big data in a manner business users would use Excel. To do so, it presents a familiar interface (e.g. Pivot, Union, Intersection functions) that allows users to gather, filter, combine, explore, and visualize data from various sources. Big Sheets has been designed to be used by non-technical professionals to rapidly gather insight (BigSheets executes work on a simulated environment of sample data first) and analysis from huge amounts of data, and to be able to act on those insights in a timely manner. No need to understand database schemas, no need to understand a query language. And Big Sheets conveniently has a built-in visualization module to chart and publish the results.

Also, the nice thing about it is that Big Sheets is integrated natively with the other Big Insights components, so it’s easy to navigate between the different tools that Big Insights provides; e.g. create an ETL job in Jaql and export the results to Big Sheets..

Big Data Accelerators

Big Insights bundles in some pre-built components for specific solutions to accelerate development on certain specific use cases. The accelerators generally provide business logic, data processing and visualization. An example of this is the Social Data Analytics accelerator, providing  a set of predefined elements as workbooks and dashboards to analyse social data.

Other Big Data tools

The IBM Big Data platform is comprised of Big Sheets, but also other tools like Infosphere Streams for low latency data, and an MPP (Massively Parallel Processing) database. The IBM ecosystem also seems to support Big Data: R is supported in Big Insights, Cognos supports Hive, Netezza integrates with Streams. These systems offer complementary analytical approaches.

IBM offers a free downloadable virtual machine to play with Big Insights.

Overall a good experience, although one can get easily lost by the sea of products IBM offers. On the other hand  tools like Big Sheets and the Accelerators seem very valuable.

Read More

Stinger and Tez: A primer

Recently I had the opportunity to attend a talk from Hortonworks’ own Alan Gates about Tez and Stinger. Here are the notes from this talk.

What is Stinger?

The Stinger initiative aims to redesign Hive to make it what people want today: Hive is currently used for large batch jobs and works great in that sense; but people also want interactive queries, and Hive is too slow today for this. So a big drive is performance, aiming to be 100x faster.

Additionally Hive doesn’t support all of the native SQL features for analytics (e.g. Date calculations, Windowing) so to be able to connect to a BI stack, and give Hive to SQL power users.

Stinger also covers work that is in HDFS and YARN. An example of this is the new data format, ORC. Thus Stinger is not only focused on Hive, it is a cross-project initiative.

This is work done as open source, by Hortonworks, but also SAP, Microsoft, and Facebook.

Why build upon Hive rather than build a new system?

Hive is out there and being used, and works at scale. It s hard to build a new tool that works at scale; it is easy to build a tool that is fast, but works on a small dataset (Gigabyte range).

Hive already manages at scale, and scale is hard. And now the effort is focused on making it faster. It is easier to do this rather than build in scale afterwards..

Also from a vision perspective, users only need one SQL tool, not a fragmented toolkit. Otherwise different tools are going to use different flavors of SQL, different data types, which is going to create an impedance mismatch and going to cause frustration for users. So the focus is to build just one tool, that does both batch and interactive, and have the optimizer make the right decision: is it a query that’s going to run for an hour so I should use batch type operators, or is it going to take 5 sec so I should run it in an interactive way?

Why is SQL compatibility so important?

            SQL is the English of the data world. You can get by pretty much anywhere with English when you are traveling. It is the same way with SQL, it is the language that everyone knows, and there is a workforce of analysts and set of BI tools that speak SQL. It just makes sense that Hadoop can speak SQL.

However Hadoop can not just speak SQL; Hadoop is so much more than that. YARN brings different data processing models: a real time streaming-type models like Storm, a graph processing model like Gyraph, an ETL model like Pig.

Stinger is not only focusing on SQL. Tez and Stinger open up to other tools like Pig, Cascading and make sure everyone can share the benefit.

SQL is how most of the world is going to communicate with Hadoop.

What is Tez and how does it relate to the Stinger initiative?

In Hadoop 1.0, there was only one option for how to execute their data processing: Map Reduce.

YARN

Map Reduce is a great paradigm for a whole set of problems, but in a context of a relational system that is going to stream together a set of relational operators, like Pig, Hive or Cascading, Map Reduce is not the design that you would choose if you starting out from a clean slate. That’s when Hadoop 2.0 comes in with YARN, which separates out resource management from execution. The execution moves from the system to user space, where the user can write an application manager that controls how the job is executed, so that is the opportunity for Pig and Hive to write an app manager tailored to them: Tez. Tez is application that runs on YARN, that builds the underlying execution for a parallel data processing engine for running relational operators like Group by, Join, sort. Tez is being evolved from Map Reduce, with strengths like recoverability that are baked in, and still scales and can handle faults. The underplaying execution layer can handle multiple inputs, which is needed for joins. Map Reduce 1.0 only accepts one input, and additional code is needed to shoehorn joins into it. In Tez, joins are supported natively in the execution layer.

Data movement

Also it frees you up in how you move data; in MR you write data to disk in intermediary stages for persistence in HDFS; this is great for large batch jobs, and you don’t want to give that up, but for quick interactive jobs, you want to be able to move data into memory or via sockets so you can stream in data very quickly. The goal behind Tez is to enable both of those, where the optimizer of the application (like Hive, Pig) can makes the choice: should I use socket-based streaming because I know this is going to be really fast, or should I use HDFS-based communication because this job is going to take an hour, and if it blows at the 59^th minute I don’t want to loose everything?

Startup times

Also, Tez is about driving down the job startup times: MR was built where jobs would take an hour on average; so if it took 30 sec to start the job it was just noise. This is the way it was designed. But for interactive jobs, it is important to drive down the startup times from seconds into milliseconds.

Chip architecture optimizations

Also, Tez will be rewritten to add internal optimizers in Hive to take advantage of chip architectures; traditional databases have operated on 1 record at a time. But you can now process multiple records at a time.

Buffer caching

The next Stinger phase will be about buffer caching for Hadoop: Hadoop needs to know what it needs to keep in memory vs. what should be on disk. Hadoop was build around the notion that you scan everything on disk every time , and that is a great fit for batches , but some jobs and data sets you need to hold the data in memory in the cache , if for example you are doing some frequent joins with say dimension tables. Web buffer caching strategies will help in that respect.

Cost-base optimizers

Hive needs a true cost-based optimizer for Hive, instead of the current rules-based optimizer.

For example it almost always make sense to execute a filter before a join statement. But today Hive is complex enough that there are options in how it executes things, and you don’t know what the right answer is, so you will need to make estimates on what is the optimal plan to execute a particular statement.

What Tez means for Pig and other tools?

            Building Tez not just in Hive; Pig: their API’s are fairly different; different tools expose different API’s that are more appropriate for different use cases. Underneath, there are only so many things that you do to data: you can either sort it, join it, group it, filter it, or project it. Having separate implementations for this is wasted effort. Tez optimizes the execution of these operators to make sure it is shareable across the stack to different tools.

Read More

How to install something on multiple machines at once with pssh (parallel ssh)

It is a common practice to need to install and configure something on multiple machines at once. While there are a number of tools to do this, a quick and dirty way to do this is with parallel ssh. i had a bit of a hard time setting this up the first time, so are my notes on how to configure this correctly. This was tested on Ubuntu but may work on other platforms as well. First, install parallel-ssh .

sudo aptitude install pssh

Then you will need to grant passwordless authentication to the remote machines for pssh to work. First, we have to generate an SSH key:œœ

ssh-keygen -t rsa -P ""

The second line will create an RSA key pair with an empty password. Generally, using an empty password is not recommended, but in this case it is needed to unlock the key without your interaction (you don’t want to enter the passphrase every time Hadoop interacts with its nodes). Second, you have to enable SSH access to your local machine with this newly created key. cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys Third, distribute the public key to the other machines:

ssh-copy-id -i $HOME/.ssh/id_rsa.pub root@machine2 ssh-copy-id -i $HOME/.ssh/id_rsa.pub root@machine3

and you are done. pssh is now renamed parallel-ssh. Let's give it a try:

root@tabitha7:~# parallel-ssh -h ./cassandrahosts -i ls [1] 11:23:32 [SUCCESS] x.x.x.71 tomcat [2] 11:23:32 [SUCCESS] x.x.x.74 jdk-6u45-linux-x64.bin

Read More

Quantified self experiment

Please see an earlier blog post of mine about the Quantified Self movement and an experiment here: my life of data

Read More

AWS Hadoop cost infrastructure comparison evaluation

Also, quick sample worksheet to calculate cost on AWS:

	number of hrs	TB	replication	m1.xlarge disk space(GB)	processing space (%)
		1	3	1690	60%
		5
	Numder of instances =		3.029585799		15.14792899
	Rounded no of instances		4		15
	EC2 cost per hour		0.64		0.64
	EMR cost per hour		0.12		0.12
	Number of hrs		4		4
	total per day for compute		12.16		45.6
	S3- daily cost $		3		17
	Total montly cost		467.2		1880
	EBS cost		128		640
	no of Hadoop instances		4		6
	Hadoop license cost		337		339
	EC2 cost monthly		2188.8		3283.2
	Total monthly cost		2653.8		4262.2
			Desc	Cost
			EC2 large	0.32
			Instances	10
			Hrs/month	80
			total	256
			EMR extra cost rate	0.06		Network price; 200Gig
			EMR cost	48
			Total EC2+EMR	304
			EMR Large m/c instances	10
			Hrs/month	80
			Cost per AWS calc	304

Read More

A few things about the HDP Sandbox for Hadoop

The sandbox is really nice to work with;
With that said, a few tidbits that helped me that i want to share:

- There is a shell access from Ambari, the UI, but sometimes you want to access via ssh;

Dont do this:

$ ssh root@127.0.0.1:2222
ssh: Could not resolve hostname 127.0.0.1:2222: nodename nor servname provided, or not known

Do that instead:

urbanlegends-2:~$ ssh -p 2222 root@127.0.0.1

Password should be 'hadoop'.

- If you want to use Hive, and you are installing HDP from scratch, surprise, you cannot use Beeswax (as the time of this writing, Oct, 2013), it is not integrated yet ..
So you will need to install Beeswax separately from Ambari.
Documentation is not complete, and you will need to download (via yum install beeswax).


- adding a jar for a Serde;
Even though you add the jar in the Hue UI File Browser, the jar location may not be picked up properly when using Hive at the command line. And Hue hides the actual path from you ..
Workaround: run your select statement from Beeswax. adding the jar resource in Beeswax. It will then tell you where the jar was added in the log.

I.e. : Added resource: /tmp/hue_3792@sandbox_201310151419_resources/hive-contrib-0.11.0.2.0.5.0-67.jar 

- installation of Hue:

Documentation:

http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP2/HDP-2.0.0.2/bk_installing_manually_book/content/rpm-chap-hue-1.html 

1. After creation of Hue user

(

3. Create a Hue user and either deploy Hue in that user's home directory or under the /usr/share directory.

) documentation omits to say that you need to actually download and install hue..

i.e. this step, mentioned in HDP 1.3 , was forgotten in HDP 2.0:

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.2/bk_installing_manually_book/content/rpm-chap-hue-3.html

Also this: http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.2/bk_installing_manually_book/content/rpm-chap-hue-4.html

2. After running the daemon,  via /usr/lib/hue/build/env/bin/supervisor

The IP address needs to remain 0.0.0.0 and the port needs to be a free port (check via netstat). Then the daemon should say something like:

“Starting beeswax server on port <port>, talking back to Desktop at <host>

“

and you can check the UI on the browser. “Desktop” refers to the Hue server (generally the same management node as Ambari).

A few notes:

- Installing g++ : you actually need to install gcc-c++.

i.e. yum install gcc-c++ .

- You can install multiple yum packages at once (in fact, all of the ones listed in the HDP doc) but putting their name all on the same yum install line.

But actually

http://gethue.tumblr.com/tagged/release 

Hue Integration: as of HDP 2.0, Ambari and Hue are not integrated together. Therefore their users need to be duplicated in each system. You can integrate Hue and Ambari with LDAP(Active directory) , if that is done enterprise users who have access to  have sso in ambari and hue.

 linux boxes will be able to have sso in ambari and hue.

Hue Security: You need to ensure all users created in Hue have access to create Hive jobs. If not, It could be because you do not have /user/<username> directories in HDFS. You have to create user in hdfs before you can use hue , as you need .staging directory for executing map reduce jobs.

Beeswax settings: If there is a specific serde jar which you have to use every time and by all user , you can put same in /usr/lib/hive/lib and restart hue. It will include the directory in class path while starting beeswax. Check beeswax_server.out for more details.

Read More