What is Big Data Technologies

Big Data and  Processing Frameworks

Content of Big Data:

• What is Big Data?

• Big Data Processing Frameworks (Hadoop, Spark)

• Distributed Data Storage (HDFS, S3)

• Distributed Data Processing (MapReduce, Spark)

What is Big Data?

Big Data refers to the vast amount of structured and unstructured data that is generated from various sources such as social media, sensors, logs, and transactions. It is challenging to analyze and evaluate this data using conventional data processing techniques because of its immense volume, velocity, and variety.Therefore, specialized Big Data technologies and frameworks have been developed to handle and process this data efficiently.

Big Data Processing Frameworks:

Hadoop is a popular Big Data processing framework that provides distributed storage and processing capabilities for large datasets. It is based on the MapReduce programming model, which allows developers to write distributed processing jobs that can be executed across a cluster of commodity hardware.

Apache Spark is another Big Data processing framework that provides fast, in-memory data processing capabilities. It is designed to handle a wide range of Big Data processing tasks such as batch processing, real-time streaming, machine learning, and graph processing.

    Distributed Data Storage:

Hadoop Distributed File System (HDFS) is a distributed file system that provides reliable and scalable storage for Big Data applications. It is designed to store large files across a cluster of commodity hardware and provides built-in fault tolerance and data replication capabilities.

Amazon S3 is a cloud-based storage service that provides reliable and scalable storage for Big Data applications. It is designed to store and retrieve large amounts of data from anywhere in the world and provides a pay-as-you-go pricing model.

    Distributed Data Processing:

Large datasets can be processed in a distributed fashion using the MapReduce programming style.. It allows developers to write parallel processing jobs that can be executed across a cluster of commodity hardware. MapReduce consists of two phases: the Map phase, which processes input data and generates key-value pairs, and the Reduce phase, which aggregates the key-value pairs and generates the final output.

Apache Spark provides an alternative to MapReduce that is faster and more efficient. It is designed to handle complex data processing tasks such as real-time stream processing, machine learning, and graph processing. Spark provides a distributed data processing engine that can be run on a cluster of commodity hardware and provides built-in support for SQL, streaming, and machine learning.

Example code for running a Spark job:

python code

from pyspark import SparkContext

from pyspark.sql import SparkSession

Create a SparkContext

sc = SparkContext(appName="myApp")

Create a SparkSession

spark = SparkSession.builder.appName("myApp").getOrCreate()

Read data from HDFS

data = spark.read.csv("hdfs://localhost:9000/data.csv")

Perform some data processing

Write output to HDFS

output.write.csv("hdfs://localhost:9000/output")

This is a good starting code for running a Spark job, but there are some parts missing, such as the actual data processing steps. Here's a more complete example that includes reading data from HDFS, processing it, and writing the output back to HDFS:

python code

from pyspark import SparkContext

from pyspark.sql import SparkSession

# Create a SparkContext

sc = SparkContext(appName="myApp")

# Create a SparkSession

spark = SparkSession.builder.appName("myApp").getOrCreate()

# Read data from HDFS

data = spark.read.csv("hdfs://localhost:9000/data.csv")

# Perform some data processing

processed_data = data.selectExpr("_c0 as id", "_c1 as name", "_c2 as age")

processed_data = processed_data.filter(processed_data.age > 18)

# Write output to HDFS

processed_data.write.csv("hdfs://localhost:9000/output")

# Stop the SparkContext

sc.stop()

In this example, we read a CSV file from HDFS using SparkSession's read.csv method. We then perform some data processing, which in this case involves selecting specific columns and filtering out rows where age is less than or equal to 18. Finally, we write the processed data back to HDFS using the write.csv method. Note that the output directory must not exist beforehand, or else an error will occur. Finally, we stop the SparkContext to free up resources.

To Main (Topics of Data Science)

                                            Continue to (Data Visualization and Communication)