Big Data

Compute centric

hdfs dfs -put file.txt file.txt
hdfs dfs -ls
hdfs dfs -ls /
hdfs dfs -mkdir /tmp/test
hdfs dfs -rm -r -f /tmp/test
hdfs dfs -cat file.txt
hdfs dfs -get fichero.txt
yarn jar application.jar DriverClass input output
yarn application -list
yarn logs -applicationId applicationId
yarn application -kill applicationId
yarn jar job.jar DriverClass input output
mapred job -list
mapred job -kill [jobid]
PYSPARK_DRIVER_PYTHON=ipython pyspark

          [jlopez@login7 ~]$ PYSPARK_DRIVER_PYTHON=ipython pyspark
          >>>  from pyspark.sql import Row
          >>>  Person = Row('name', 'surname')
          >>>  data = []
          >>>  data.append(Person('Joe', 'MacMillan'))
          >>>  data.append(Person('Gordon', 'Clark'))
          >>>  data.append(Person('Cameron', 'Howe'))
          >>>  df = sqlContext.createDataFrame(data)
          >>>  df.show()
            +-------+---------+
            |   name|  surname|
            +-------+---------+
            |    Joe|MacMillan|
            | Gordon|    Clark|
            |Cameron|     Howe|
            +-------+---------+
        
[jlopez@login6 ~]$ sparkR

          [jlopez@login7 ~]$ sparkR
          > df <- createDataFrame(sqlContext, faithful)
          > head(df)
            eruptions waiting
            1     3.600      79
            2     1.800      54
            3     3.333      74
            4     2.283      62
            5     4.533      85
            6     2.883      55
        

        # client mode
        spark-submit --master yarn \
          --name testWC test.py input output
        # cluster mode
        spark-submit --master yarn --deploy-mode cluster \
          --name testWC test.py input output
        

        # client mode
        spark-submit --master yarn --name testWC \
          --class es.cesga.hadoop.Test test.jar \
          input output
        # cluster mode
        spark-submit --master yarn --deploy-mode cluster \
          --name testWC \
          --class es.cesga.hadoop.Test test.jar \
          input output
        

--num-executors NUM    Number of executors to launch (Default: 2)
--executor-cores NUM   Number of cores per executor. (Default: 1)
--driver-cores NUM     Number of cores for driver (cluster mode)
--executor-memory MEM  Memory per executor (Default: 1G)
--queue QUEUE_NAME     The YARN queue to submit to (Default: "default")
--proxy-user NAME      User to impersonate
        

            ROW FORMAT DELIMITED
            FIELDS TERMINATED BY ':'
          

        sqoop list-tables \
          --username ${USER} -P \
          --connect jdbc:postgresql://${SERVER}/${DB}
        

        sqoop import \
          --username ${USER} --password ${PASSWORD} \
          --connect jdbc:postgresql://${SERVER}/${DB} \
          --table mytable \
          --target-dir /user/username/mytable \
          --num-mappers 1
        

        sqoop import \
          --username ${USER} --password ${PASSWORD} \
          --connect jdbc:postgresql://${SERVER}/${DB} \
          --table mytable \
          --target-dir /user/username/mytable \
          --num-mappers 1 \
          --hive-import
        

        sqoop create-hive-table \
          --username ${USER} --password ${PASSWORD} \
          --connect jdbc:postgresql://${SERVER}/${DB} \
          --table mytable
        

        sqoop export \
          --username ${USER} --password ${PASSWORD} \
          --connect jdbc:postgresql://${SERVER}/${DB} \
          --table mytable \
          --export-dir /user/username/mytable \
          --input-fields-terminated-by '\001' \
          --num-mappers 1
        
ssh -C2qTnNf -D 9876 @login.hdp.cesga.es
ssh @login.hdp.cesga.es
start_jupyter
The Jupyter Notebook is running at: http://1.2.3.4:8888/
git clone https://github.com/bigdatacesga/mr-wordcount

              # Download sources and javadoc
              mvn dependency:sources
              mvn dependency:resolve -Dclassifier=javadoc
              # Update the existing Eclipse project
              mvn eclipse:eclipse
              # Or if you using Intellij IDEA
              mvn idea:idea
            
mvn compile
mvn test
mvn package

            javac -classpath $(hadoop classpath) *.java
            jar cvf wordcount.jar *.class
          

public class Driver {
  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf);
    job.setJarByClass(Driver.class);
    job.setJobName("Word Count");
    job.setMapperClass(WordMapper.class);
    job.setCombinerClass(SumReducer.class);
    job.setReducerClass(SumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));    
    boolean success = job.waitForCompletion(true);
    System.exit(success ? 0 : 1);
  }
}
        

public class WordMapper
    extends Mapper<LongWritable, Text, Text, IntWritable> {
	private final static IntWritable one = new IntWritable(1);
	private Text word = new Text();
	@Override
	public void map(LongWritable key, Text value, Context context)
			throws IOException, InterruptedException {
		String line = value.toString();
		for (String field : line.split("\\W+")) {
			if (field.length() > 0) {
				word.set(field);
				context.write(word, one);
			}
		}
	}
}
        

public class SumReducer
  extends Reducer<Text, IntWritable, Text, IntWritable> {
	@Override
	public void reduce(
            Text key, Iterable<IntWritable> values, Context context)
			throws IOException, InterruptedException {
		int wordCount = 0;

		for (IntWritable value : values) {
			wordCount += value.get();
		}
		context.write(key, new IntWritable(wordCount));
	}
}

Framework	Lines	Lines of Boilerplate
MPI+Python	52	20+
Spark+Python	28	2
Chapel	20	1

BD|CESGA

Big Data

What?

The 3V model

Data Centric

MPI Shortcomings

MPI Boilerplate

BD|CESGA

Hardware

Evolution

Hardware Infrastructure

Hardware Master Nodes

Hardware Slave Nodes

BD|CESGA

Software

Platforms available

Hadoop Platform

PaaS Platform

BD|CESGA Portal

hadoop.cesga.es

Front Page

General info

Tools available

WebUI

Tutorials

Hadoop Platform

Your ready-to-use Hadoop ecosystem

Ecosystem

Core Components

HDFS

The Hadoop Distributed Filesystem

HDFS Architecture

HDFS Replicas

Upload a file to HDFS

List files

Working with directories

Working with files

Web File Explorer

Monitoring

YARN

Yet Another Resource Negotiator

YARN Architecture

Launching an application

List running jobs

See application logs

Kill an application

Capacity Scheduler Queues

WebUI

YARN UI

MapReduce

MapReduce

Launching a job

List MR jobs

Cancelling a job

Monitoring

Job History

Quick how-to

Spark

A fast and general engine for large-scale data processing

Speed

Easy

Generality

Language Selection

PySpark

PySpark Basics

Running pyspark interactively

Example

SparkR

SparkR Basics

Running SparkR interactively

Example

spark-submit

Submit job to queue

Spark Components

spark-submit Python

spark-submit Scala/Java

spark-submit options

Hive

SQL-like interface

Hive