IBM’s Big Insights: A primer
Recently I had the opportunity to attend a training session
about IBM’s Big Insights, in November 2013. Below are my notes about this
product.
What is Big Insights in a nutshell?
Big Insights is IBM’s Big Data platform. It is comprised of
an all-in-one Big Data infrastructure, with IBM’s flavor of Hadoop and its
ecosystem, as well as proprietary tools to query the data like JAQL and AQL,
and out-of-the-box connectors and interfaces called accelerators. We’ll review
these components in details in the below section.
Big Insights Hadoop infrastructure
Big Insights is composed of a Hadoop infrastructure
(independent from vendors like Cloudera). It is using a released version of
Hadoop that is well-tested, usually a bit older from trunk. However it differs
from the Apache version in some ways also. Big Insights comes integrated with:
-
GPFS (IBM’s version of HDFS) for its file system
-
Adaptive Map Reduce, an enhanced version of MR
that attempts to optimize task executions, by way of using automatic job tuning
of speculative execution and Task JVM reuses. Map Reduce tasks become aware of
the global state of the job they are working in. This helps balance the
workload across Map tasks.
-
Zookeeper, HBase, Hive, Pig
Of note is the fact that Big Insights is not bundled with
Cloudera’s CDH anymore; IBM has its own version of Hadoop.
New query language: JAQL
Big Insights offers a language called JAQL, a functional
language that can interface will of all the Big Insights tools. It provides
API's (or modules) for reaching out to external IBM and 3rd party
tools, such as relational databases, indexing services, text analytics, machine
learning etc. JAQL stands for Json Query Language, because it is represented
via Json. Similar to Pig, Jaql is automatically taking care managing the
complexities of the MapReduce world to optimally perform the work. However it
also manages deep level nested semi-structured data.
Jaql can be executed either from its own shell, or from
within Eclipse.
Big Insights Applications
Big Insights provides an environment for developing and
executing applications. A business user can launch existing applications from
the Web console, supply any input parameters and view results. These applications may be developed using Big
Insights’ development tooling which enables programmers to publish completed
applications through the Web console.
The BigInsights Eclipse tools include wizards, code
generators, context-sensitive help, and a test environment to simplify your
development efforts.
Workflow
applications are run by Oozie as a workflow job.
Big Sheets
Big Insights also comes with a spreadsheet-like interface to
interact with Big data in a manner business users would use Excel. To do so, it
presents a familiar interface (e.g. Pivot, Union, Intersection functions) that
allows users to gather, filter, combine, explore, and visualize data from
various sources. Big Sheets has
been designed to be used by non-technical professionals to rapidly gather
insight (BigSheets executes work on a simulated environment of sample data
first) and analysis from huge amounts of data, and to be able to act on those
insights in a timely manner. No need to understand database schemas, no need to
understand a query language. And Big Sheets conveniently has a built-in
visualization module to chart and publish the results.
Also, the
nice thing about it is that Big Sheets is integrated natively with the other
Big Insights components, so it’s easy to navigate between the different tools
that Big Insights provides; e.g. create an ETL job in Jaql and export the
results to Big Sheets..
Big Data Accelerators
Big Insights bundles in some pre-built components for
specific solutions to accelerate development on certain specific use cases. The
accelerators generally provide business logic, data processing and
visualization. An example of this is the Social Data Analytics accelerator,
providing a set of predefined elements
as workbooks and dashboards to analyse social data.
Other Big Data tools
The IBM Big Data platform is comprised of Big Sheets, but
also other tools like Infosphere Streams for low latency data, and an MPP
(Massively Parallel Processing) database. The IBM ecosystem also seems to
support Big Data: R is supported in Big Insights, Cognos supports Hive, Netezza
integrates with Streams. These systems offer complementary analytical
approaches.
IBM offers a free downloadable virtual machine to play with
Big Insights.
Overall a good experience, although one can get easily lost by
the sea of products IBM offers. On the other hand tools like Big Sheets and the Accelerators
seem very valuable.
Really a valuable content, keep sharing post like this. It will be helpful to many like me in finding the institute for Hadoop training chennai velachery
ReplyDeleteGenuinely a critical substance, keep sharing post like this. It will be valuable to various like me in finding the association forhadoop training in chennai | hadoop training in chennai
ReplyDeleteHi admin thanks for sharing informative article on hadoop technology. In coming years, hadoop and big data handling is going to be future of computing world. This field offer huge career prospects for talented professionals. Thus, taking Hadoop Training in Chennai will help you to enter big data technology.
ReplyDeleteIt was really a wonderful article and I was really impressed by reading this blog. We are giving all software and Database Course Online Training. Oracle Training in Chennai is one of the reputed Training institute in Chennai. They give professional and real time training for all students.
ReplyDeleteOracle Training in chennai
Informatica Training in chennai
ReplyDeleteThis information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..
Pega Training in Chennai
ReplyDeleteBrilliant article. The information I have been searching precisely. It helped me a lot, thanks. Keep coming with more such informative article. Would love to follow them.
QTP Training in Chennai
ReplyDeleteThank you for the informative post. It was thoroughly helpful to me. Keep posting more such articles and enlighten us.
There are lots of information about latest technology and how to get trained in them, like Hadoop Training in Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies Hadoop Training in Chennai By the way you are running a great blog. Thanks for sharing this..
ReplyDeleteI found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing..
ReplyDeleteGreens Technologies In Chennai
I was looking about the Oracle Training in Chennai for something like this ,Thank you for posting the great content..I found it quiet interesting, hopefully you will keep posting such blogs…
ReplyDeleteGreens Technologies In Chennai
Who wants to learn Informaticawith real-time corporate professionals. We are providing practical oriented best Informatica training institute in Chennai. Informatica Training in chennai
ReplyDeletehey nice site..learn Oracle Training we provided by Oracle Certified Experts. Best Oracle Training institute in Chennai with Job Placement. Oracle Training in chennai
ReplyDeleteJump Start Your Career & Get Ahead. Choose sas training method that works for you. We offer an extensive list of courses in a variety of formats that make learning as easy as possible. SAS Training in Chennai
ReplyDeleteAwesome blog if our training additional way as an SQL and PL/SQL trained as individual, you will be able to understand other applications more quickly and continue to build your skill set which will assist you in getting hi-tech industry jobs as possible in future courese of action..visit this blog Green Technologies In Chennai
ReplyDeleteNice site....Please refer this site also Our vision succes!Training are focused on perfect improvement of technical skills for Freshers and working professional. Our Training classes are sure to help the trainee with COMPLETE PRACTICAL TRAINING and Realtime methodologies. Green Technologies In Chennai
ReplyDeleteLooking for real-time training institue.Get details now may if share this link visit Oracle Training in chennai
ReplyDeleteThis site has very useful inputs related to qtp.This page lists down detailed and information about QTP for beginners as well as experienced users of QTP. If you are a beginner, it is advised that you go through the one after the other as mentioned in the list. So let’s get started QTP Training in Chennai
ReplyDeleteIt is really very helpful for us and I have gathered some important information from this blog.
ReplyDeleteOracle Training In Chennai
Oracle Training in Chennai is one of the best oracle training institute in Chennai which offers complete Oracle training in Chennai by well experienced Oracle Consultants having more than 12+ years of IT experience.
ReplyDeleteThere are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this.
ReplyDeleteGreat post and informative blog.it was awesome to read, thanks for sharing this great content to my vision.Informatica Training In Chennai
ReplyDeleteA Best Pega Training course that is exclusively designed with Basics through Advanced Pega Concepts.With our Pega Training in Chennai you’ll learn concepts in expert level with practical manner.We help the trainees with guidance for Pega System Architect Certification and also provide guidance to get placed in Pega jobs in the industry.
ReplyDeleteOur HP Quick Test Professional course includes basic to advanced level and our QTP course is designed to get the placement in good MNC companies in chennai as quickly as once you complete the QTP certification training course.
ReplyDeleteThanks for sharing this nice useful informative post to our knowledge, Actually SAS used in many companies for their day to day business activities it has great scope in future.
ReplyDeleteGreens Technologies Training In Chennai
ReplyDeleteExcellent information with unique content and it is very useful to know about the information based on blogs.
Greens Technology offer a wide range of training from ASP.NET , SharePoint, Cognos, OBIEE, Websphere, Oracle, DataStage, Datawarehousing, Tibco, SAS, Sap- all Modules, Database Administration, Java and Core Java, C#, VB.NET, SQL Server and Informatica, Bigdata, Unix Shell, Perl scripting, SalesForce , RedHat Linux and Many more.
ReplyDeleteVery nice articles,thanks for sharing this useful information.
ReplyDeleteDataStage Training
ETL Testing Training
Hadoop Training
Very nice articles,thanks for sharing this useful information.
ReplyDeleteOracle Golden Gate Online Training
Ruby On Rails Online Training
SAP XI Online Training
OBIEE Online Training
There are lots of information about latest technology and how to get trained in them, like Best Hadoop Training In Chennai in Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies Hadoop Training in Chennai By the way you are running a great blog. Thanks for sharing this blogs..
ReplyDeleteThis information is impressive..I am inspired with your post writing style & how continuously you describe this topic. After reading your post,thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic
ReplyDeleteAndroid Training In Chennai In Chennai
This comment has been removed by the author.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteSAP Training in Chennai
ReplyDeleteThis post is really nice and informative. The explanation given is really comprehensive and informative..
Oracle Training in chennai
ReplyDeleteThanks for sharing such a great information..Its really nice and informative..
Selenium Training in Chennai
ReplyDeleteWonderful blog.. Thanks for sharing informative blog.. its very useful to me..
Data warehousing Training in Chennai
ReplyDeleteI am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly..
Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly, but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..
ReplyDeleteWebsphere Training in Chennai
Oracle DBA Training in Chennai
ReplyDeleteThanks for sharing this informative blog. I did Oracle DBA Certification in Greens Technology at Adyar. This is really useful for me to make a bright career..
hai,i have to learned to lot of information about java Gain the knowledge and hands-on experience you need to successfully design, build and deploy applications with java.
ReplyDeleteJava Training in Chennai
hybernet is a framework Tool which helps in Functional and Regression testing of an application. If you are interested in hybernet training, our real time working.
ReplyDeleteHibernate Training in Chennai,
Looking for real-time training institue.Get details now may if share this link visit
ReplyDeleteSpring Training in chennai
oraclechennai.in:
Nice site.... refer this site .if Our vision succes!Training are focused on perfect improvement of technical skills for Freshers and working professional. Our Training classes are sure to help the trainee with Realtime methodologies.
ReplyDeleteOracle Rac Training Chennai
haddoop:
Job oriented form_reports training in Chennai is offered by our institue is mainly focused on real time and industry oriented. We provide training from beginner’s level to advanced level techniques thought by our experts.
ReplyDeleteforms-reports Training in Chennai
This is really an awesome article. Thank you for sharing this.It is worth reading for everyone. Visit us:Oracle Training in Chennai
ReplyDeletevery nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.
ReplyDeleteOracle DBA Training in Chennai
Really awesome blog. Your blog is really useful for me. Thanks for sharing this informative blog. Keep update your blog.
ReplyDeleteSAP Training in Chennai
Execellent ! I am truly impressed that there is so much about this subject that has been revealed and you did it so nicely
ReplyDeletesas online training