Machine learning using spark

MACHINE LEARNING USING SPARK

The following topics will be covered in our

Machine Learning Using Spark Online Training:

2

What is Machine Learning? â&#x2013;Ş Machine learning Using Spark-Spark MLlib is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs

that can access data and use it learn for themselves.

3

Into to Machine Learning Using Spark • MLlib is Spark’s machine learning (ML) library. Its goal is to make practical machine learning scalable and easy. At a high level, it provides tools such as: • ML Algorithms: common learning algorithms such as classification, regression, clustering, and collaborative filtering • Featurization: feature extraction, transformation, dimensionality reduction, and selection • Pipelines: tools for constructing, evaluating, and tuning ML Pipelines

4

Tools â&#x20AC;˘ This course will be delivered using Scala and PYTHON API. For explaining statistical concept, R language will also be using. Visualization part will be covered using Bokeh/ggplot library.

5

Introduction to Apache Spark ▪ Spark Programming model ▪ RDD and Data Frame ▪ Transformation and Action ▪ Broadcast and Accumulator ▪ Running HDP on local machine ▪ Launching Spark Cluster Copyright @ 2015 Learntek. All Rights Reserved.

6

Basic Statistics •

Mean, Mode, Media, Range, Variance,

Standard Deviation, Quartiles, Percentiles • Sampling

• Sampling Methods • Sampling Errors • Probability Distributions

•

Normal distribution, t-distribution, Chi-

square, F • Margin of Error, Confidence Interval, Significance level, Degree of Freedom

• Hypothesis concept, Type I and Type II error • P-value, t-Test, Chi-square Test • Correlation Coefficient

7

Machine Learning Using Spark • Introduction to Spark MLlib • Data types: Vector, Labeled Point • Feature Extraction

• Feature Transformation, Normalization • Feature Selectors • Locality Sensitive Hashing(LSH)

8

Regression Analysis with Spark • Types of Regression Models

• Gradient Descent • Linear Regression, Generalized Linear Regression

9

Classification Model with Spark • Linear Models, Naives Bayes Model, Decision Tree

• Training Classification Models • Accuracy and prediction error

• Logistic Regression

• Precision and Recall

• Linear Support Vector Machine

• ROC curve and AUC

• Random Forest

• Cross validation

• Gradient-Boosted Trees

10

Clustering â&#x20AC;˘ Hierarchical clustering â&#x20AC;˘ K-mean clustering

11

Dimensionality Reduction • Principal Component Analysis • Singular Value Decomposition • Clustering as dimensionality reduction • Training a dimensionality reduction model • Evaluating dimensionality reduction models

12

Recommendation Engine ▪ Content based filtering ▪ Collaborative based filtering ▪ Overview of Movie Lens data ▪ Training a recommendation model ▪ Using the recommendation model ▪ Performance Evaluation Copyright @ 2015 Learntek. All Rights Reserved.

13

Text Processing

•Feature Hashing

•TF-IDF Weightings

•TF-IDF model

•Training a TF-IDF model

•Tokenization

•Usage of TF-IDF model

•Stop words

•Evaluating TF-IDF models

14

Prerequisites : â&#x2013;Ş Prior understanding of exploratory data analysis and data visualization will help immensely in learning machine learning concept and This

include basic

applications.

statistical technique for data analysis. Having some

knowledge of R programming or some Python packages like sci-kit, numpy will be useful. However , we are going to cover basic statistics technique as part of this course before going deep into machine learning . This will help

everyone to gain maximum from this course.

15

16

Turn static files into dynamic content formats.

Create a flipbook