A Role of Spark in Big Data introduction

Big Data:

Batch    and    real time

For downloading you have to do it by latest which linux flavor

Ubantu.15.10 downloads this then vm player

Ubuntu-> http://www.ubuntu.com/download/desktop

Vmplayer

vmware –

http://www.vmware.com/my_maintenance.html#desktop_end_user_computing/vmware_player/7_0

Select ram as around 6GB in vmplayer

Then update ubantu 64 bit .iso file

Where data is using in traditional systems

RDBMS

This is only pure for business  profit oriented process like transactional data

Web + IOT (internet of things )

Web can generate huge data and the devices to connect internet then generate data this is called as IOT data

80% of data comes from IOT

Batch processing:  it’s processing defining series of jobs

For example in bank transactions having batches some interval all the NEFT/RTGS transactions triggers   

Input data can collect over time of period its non-continues process data

Batch window: log analysis and send back the response then do the recommendation this is going to be delay process. For systems performance also can do

In the log analysis they can do the KPI (key Performance indicators)

This is purely at batch

Another example of – billing applications

Backups – not critical hours they are going to trace backups

For this also batch jobs

We require fast response

Batch window

Main frames is the best batch processing system

Offer recommendations – past data generated and giving the recommendations

Batch processing is complex when its use while produce faster results

If data is large its won’t be process faster

Real-time Data:  changes the data continually

Small differences like mille seconds

Latency: time interval b/w request and response this is called as latency

  Low Latency:  High response time

Air Traffic

Hospital

Bank ATM

Currently this doing real time

Example:

APP nexus

This is 3^rd party  app vendor to give the recommendation based on the credit of users ..how this going to do

AppNexus having the customer credit data how is bit his customer having good credit score based on that they do recommendation pops in the websites

In-Memory Processing:

Bigdata -- > batch Dataà Hadoop

Bigdataà real time à Storm – can do real time not batch jobs

Hadoop for disk computations

Sparkà can do to both batch and real time processing

Data pipeline:

If you want build any data pipe line you can use do the following things

Any big data process

1>      Injecting – Flume,Kafka,Sqoop

2>      Storage  -- HDFS,Hbase,Cassandra,

3>      Processing – Pre processing , Post Processing MR,Hive,PIG,Sprak

4>      Schedule -- OOZIE

5>      Model (post processing Tools) – Post processing – BI Tools QlickView,tableu..so on

For doing this requires lot of integration

MLLIB – Machine learning libraries

Mahout – machine learning libraries

Bases projects of spark like

è  Injecting the data

è  Kafka

è  Spark streaming

è  Store data into Cassandra

è  Spark jobs

Applying regular expressions

Iterative algorithms

Apache Spark – core

Spark SQL

Spark Streaming

MLIB

Hadoop written in java

Spark written by Scala

Scala have lot of advantages

Object oriented and functional programming language

2000 lines in java you can make this as 100 lines

One function can call other functions

Many of the inline functions

Fun1(fun2)...like this

First we need to learn scala then spark

Spark – memory computations

Hadoop – data processing computation engine

1.5 version use phython/scala

2015 – data frames

100% better than Hadoop

Word count in java here 2 lines in scala

is disk lost in spark we can take the back up using linear graph

Hadoop can run

Standalone

Psudeo distributed

Fully distributed

Spark run on below

Standalone     -----   YARN    - ---  MESOS

RDD – in memory of machines 

Few More Questions in Hadoop Interview

      1)        Explain about the current roles and responsibilities ?
2)        Difference between Hadoop1 and hadoop2?
3)        How much size of the cluster your using currently?
4)        Difference between writable and writeomparable in map reduces programming?
5)        Did you combiners in map reduce how to use?
6)        Did you use counters in map reduce programming?
7)        How to write code portions in map reduce programming?
8)        How to do serialize and deserialization the data in map reduce progrmming?
9)        How to swap two data columns date by 11/11, 11/12 in hive.
10)     Did you use d-link how to connect with the hive?
11)     How to do bucking in map reduce?
12)     How to merge two tables eg small and big table using pig or hive?
13)     What is purpose of the spark and what is RDD and context?
14)     Did you write UDF’s in Hive what purpose did you write?
15)     Difference between orderby and sort by in hive?
16)     What is the hive metastore and purpose of this?
17)     How to clean the data with special characters in PIG?
18)     What is kafka and how manage this into traditional way?
19)     What are the main components in STORM?
20)     Solr usage and why you used this into your project?

Thanks

Narasimha

Hadoop QA/QE Roles and Migrate notes Basics

Hadoop has 2 noteworthy Distributions

Cloudera and Hortonworks 

significant center parts : 

HDFS(Data capacity) and MAP reduce(processing u can use phython/scala/java) and major roles which company's are looking underneath below

1. Hadoop Admin 

2. Hadoop Development 

3. Hadoop QA/QE 

Hadoop Admin - is equivalent to framework/web/system administrator 

Hadoop development - for any java,Scala, Phython software developers

Hadoop QA - for the analyzer who has min experience at least basics on java/scala 

i. How relavent it is for QA ? 

Answer: 

in the event that you learn hadoop you have the splendid future. let say you can discover the occupation like the prerequisites 

* Plan, code, and execute computerized testing (with some manual testing if mechanization is impractical). This is the fundamental center of the position. 

* Analyze issues, report absconds, and propose relapse tests to find re events. 

* Work intimately with numerous improvement groups on end-to-end testing. 

* Check for information consistency over numerous sources. 

* Manufacture reenacted full generation situations for complex testing situations. 

* Manufacture and convey custom testing structures for REST-based API's utilizing JSON information positions. 

back ground of this : you ought to know junit testing, manual , Automation and REST web services testing which we have parcel of occupations going on you need to do end to end testing. 

you can discover the ETL utilizing a percentage of the apparatus like Talend..etc 

ii. On the off chance that I learn Haddop now, Where do I begin (which postition) my profession ? Would I be able to begin from where I was remaining in QA ? 

yes you will begin anyplace like same which you working at present in QA. Hadoop QA lead, Hadoop QA Engineering, HAdoop QA Automation..etc 

nothing change what your are currently working however this is extra and new innovation. 

I trust you get clear picture now. 

********* 

presently part of innovation came into picture like 

Spark - this is new Hadoop highlight executed by Scala(functional+object oriented Language) in memory calculation instead of disk. 

IOT + Big Data - attempt to concentrate on this is amazing element on the off chance that you can discover the you-tube you can get more thought what is IOT. 

It would be ideal if you survey and let me know whether you require any illumination require on this..

Why we use Kafka if Flume can do same things!!

Lets speak more about this appears to be numerous individuals confronting this issue in meeting.

Kafka will be used to bridge the architectural gaps between Flume and Storm we found 3 as of now

a)        The Flume sink uses a push mechanism for sending events, while Storm spout uses the pull mechanism for the events.

b)        Flume sink uses a push mechanism; hence it must know in advance where events need to be pushed to particular node to run the topology in different stages.

c)        Kafka is based on the publish-subscribe programming paradigm here producers (flume) push mechanism and consumers (storm) have pull mechanism in our architecture.

Interview Questions in Hadoop Experienced

Howdy All,

on the off chance that you take after these inquiry you without a doubt break the meeting I wager on it. Kindly make it clear comprehend the inquiry and give or set up your own particular responses for this.I wanna share all these ongoing inquiry question which I am confronting day by day.

1)      Explain about your architecture?

2)      What are the challenges did you face at least 3 ?

3)      What are the main storm cluster components?

4)      What happens if nimbus node dead?

5)      How to do versioning in Cassandra?

6)      What is kafka and difference between kafka and flume?

7)      What is the signature of mapper ?

8)      Map reduce phases and importance?

9)      How to give the own input to the mapper directly if main method defined?

10)   What are the demons in Hadoop cluster?

11)   Difference between hadoop1 and hadoop2?

12)   If you migrated hadoop2 did you see any differences?

13)   What are the versions are you using in your project?

14)   What is the format of the data?

15)   What is the data size your handling daily?

16)   How many events you fire in a second?

17)   What is the rate yourself while building same kind of cluster?

18)   How to make custom input to your mapper?

19)   How many mapper and reducer will going to run if particular size of data?

20)   How to run command line mapper program syntax?

21)   Did you work on oozie in your current project?

22)   Did you use splunk and what is your role in splunk?

23)   What is the difference between set and list?

24)   Struts flow?

25)   Difference between hash code and equals what happens if you override ?

26)   Difference between compare and comparable interfaces?

27)   What is custom UDF and what are the UDFS did you written?

28)   How to use the udf in your projects?

29)   Which version of java are you using currently?

30)   Did you involve any data conversion like .csv or json format?

31)   What are the limitations of hive?

32)   If I have 1000 records how to make insert/delete using Sqoop?

Kafka Cluster more precise in true

· Apache Kafka is an appropriated distribute subscribe informing framework publicly released by LinkedIn

· It accompanies straight versatility, elite and information is recreated and divided

· One must be inquiring as to why Kafka is in the outline when flume gives comparable usefulness

· Kafka will be utilized to connect the structural holes in the middle of Flume and Storm.

· Example of 3 holes why Kafka

o The Flume sink utilizes a push component for sending occasions, while Storm spout utilizes the an instrument for the occasions

o work submitted to Storm is known as a topology. Different phases of Storm topologies (MR sort employments) keeps running on distinctive hubs and Storm chiefs focus the hubs in the bunch to run diverse phases of a topology Essentially, the rundown of hubs where a specific topology will run is not known ahead of time

o The Flume sink utilizes a push system, subsequently it must know ahead of time where occasions should be pushed to, which is not the situation here

o In the POC, we didn't utilize Kafka as we added to a custom flume sink to push occasions specifically to Storm as just single hub bunch was being utilized. This is not going to work once the Storm bunch has more than one hub

· Kafka, then again, depends on the distribute subscribe programming worldview. Makers push messages and customers force messages. In our outline, there is a push framework (Flume) toward one side and force framework (Storm) on the flip side. Henceforth, Kafka is an impeccable answer for extension this hole and determines both issues said b

Apache Solr Architecture In Real World

What is Solr?

Apache Solr is a fast open-source Java search server. Solr enables you to easily create search

engines which searches websites, databases and files.

Solr is the popular, blazing fast open source enterprise search platform from the Apache

Lucene project. Solr is powered by Lucene, a powerful open-source full-text search library,

under the hood.

• Doug Cutting created Lucene in 1999.Recognized as a top level Apache Software

Foundation project in 2005

• Yonik Seeley created Solr in 2004.Recognized as a top level Apache Software

Foundation project in 2007

Its major features include powerful full-text search, hit highlighting, faceted search,

dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and

geospatial search.

Solr is highly scalable, providing distributed search and index replication, and it powers

the search and navigation features of many of the world's largest internet sites.

Solr is written in Java and runs as a standalone full-text search server within a servlet

container such as Jetty.

Solr uses the Lucene Java search library at its core for full-text indexing and search, and

has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming

language.

Solr's powerful external configuration allows it to be tailored to almost any type of

application without Java coding, and it has an extensive plugin architecture when more

advanced customization is required.

       

      

Solr makes it easy to add the capability to search through the online store through the following

steps:

 Define a schema. The schema tells Solr about the contents of documents it will be

indexing. In the online store example, the schema would define fields for the product

name, description, price, manufacturer, and so on. Solr's schema is powerful and

flexible and allows you to tailor Solr's behavior to your application. See Documents,

Fields, and Schema Design for all the details.

 Deploy Solr to your application server.

 Feed Solr the document for which your users will search.

 Expose search functionality in your application.

Solr is able to achieve fast search responses because, instead of searching the text

directly, it searches an index instead.

This is like retrieving pages in a book related to a keyword by scanning the index at the back of

a book, as opposed to searching every word of every page of the book.

This type of index is called an inverted index, because it inverts a page-centric data structure

(page->words) to a keyword-centric data structure (word->pages).

Solr stores this index in a directory called index in the data directory.

How Solr represents data

In Solr, a Document is the unit of search

An index consists of one or more Documents, and a Document consists of one or more Fields.

In database terminology, a Document corresponds to a table row, and a Field

corresponds to a table column.

When data is added to Solr, it goes through a series of transformations before being added to

the index. This is called the analysis phase. Examples of transformations include lower-casing,

removing word stems etc. The end result of the analysis are a series of tokens which are then

added to the index. Tokens, not the original text, are what are searched when you perform a

search query.

indexed fields are fields which undergo an analysis phase, and are added to the index.If a field

is not indexed, it cannot be searched on.

Solr Features

 Keyword Searching – queries of terms and boolean operators

 Ranked Retrieval – sorted by relevancy score (descending order)

 Snippet Highlighting – matching terms emphasized in results

 Faceting – ability to apply filter queries based on matching fields

 Paging Navigation – limits fetch sizes to improve performance

 Result Sorting – sort the documents based on field values

 Spelling Correction – suggest corrected spelling of query terms

 Synonyms – expand queries based on configurable definition list

 Auto-Suggestions – present list of possible query terms

 More Like This – identifies other documents that are similar to one in a

 result set

 Geo-Spatial Search – locate and sort documents by distance

 Scalability – ability to break a large index into multiple shards and

 distribute indexing and query operations across a cluster of nodes

A complete Architecture 

Indexing Process