# AWS Machine Learning Services


## Amazon Comprehend

- Amazon Comprehend is a natural language processing (NLP) service that uses machine learning to find insights and relationships in a text.

- Amazon Comprehend can identify the language of the text; extract key phrases, places, people, brands, or events; understand how positive or negative the text is; and automatically organize a collection of text files by topic.

- Amazon Comprehend can also identify spam or other unwanted content in a large set of documents.

- Amazon Comprehend can be used to analyze a wide range of text, including social media posts, websites, and documents.

| Feature   | Description                                                                                       |
| --------- | ------------------------------------------------------------------------------------------------- |
| Entity    | identify entities such as people, places, brands, or events in a text.      |
| Languages | identify the language of the text.                                          |
| Sentiment | determine the sentiment (positive, negative, neutral) of the text.          |
| Syntax    | analyze the syntax of the text, including parts of speech and syntax trees. |



## Amazon Translate


- Amazon Translate is a neural machine translation service that delivers fast, high-quality, and affordable language translation.

- Amazon Translate can translate text between languages, and it can also be used to localize websites and applications.


## Amazon Rekognition



- Amazon Rekognition is a deep learning-based image and video analysis service that can identify objects, people, text, scenes, and activities in images and videos, as well as detect any inappropriate content.


- Amazon Rekognition can be used to analyze images and videos in a wide range of use cases, including media, advertising, and public safety.




![](attachments/Pasted%20image%2020240309162847.png)


![](attachments/Pasted%20image%2020240309162910.png)
## Amazon Forecast



- Amazon Forecast is a fully managed service that uses machine learning to deliver highly accurate forecasts.


- Amazon Forecast can be used to forecast demand, sales, and other time-series data, and it can also be used to forecast other types of data, such as weather and financial data.

### Amazon forecast Algorithms

  

#### CNN-QR (Convolutional Neural Network – Quantile Regression)

- Best for: Large datasets with hundreds of time series
- Features: Accepts related historical time series data & metadata

#### DeepAR+

- Model Type: Recurrent Neural Network
- Best for: Large datasets
- Features: Accepts related forward-looking time series & metadata

#### gProphet

- Model Type: Additive model with non-linear trends and seasonality
- Features: Suitable for various time series forecasting scenarios

#### NPTS (Non-Parametric Time Series)

- Best for: Sparse data
- Variants: Available for seasonal/climatological forecasts

#### ARIMA (Autoregressive Integrated Moving Average)

- Commonly used for: Simple datasets (<100 time series)

#### ETS (Exponential Smoothing)

- Commonly used for: Simple datasets (<100 time series)



### AWS LEX


- Amazon Lex is a service for building conversational interfaces into any application using voice and text.

> its what I used to build my senior project 



### Amazon Polly Overview

Amazon Polly is a powerful Neural Text-to-Speech (TTS) service that offers a wide range of voices and supports multiple languages. It provides various features to enhance the quality and customization of synthesized speech.

#### Neural Text-To-Speech

Amazon Polly utilizes Neural TTS technology to produce natural and lifelike speech. It offers a diverse selection of voices in different languages, enabling users to create engaging and dynamic audio content.

#### Lexicons

Lexicons in Amazon Polly allow users to tailor the pronunciation of specific words and phrases. This customization ensures accurate and contextually appropriate rendering of terms, such as replacing "W3C" with "World Wide Web Consortium."

#### SSML (Speech Synthesis Markup Language)

SSML is an integral part of Amazon Polly, providing an alternative to plain text input. This markup language gives users fine-grained control over various aspects of speech synthesis, including emphasis, pronunciation, breathing, whispering, speech rate, pitch, and pauses.

#### Speech Marks

Amazon Polly supports the encoding of speech marks, indicating the start and end of sentences or words in the audio stream. This feature is particularly useful for applications like lip-synching animation, where precise synchronization with speech is crucial.

Sure, here's a summary about AWS Personalize in Markdown format:

### Personalize

AWS Personalize is a machine learning service provided by Amazon Web Services (AWS) that enables developers to build intelligent user personalization capabilities into their applications. It uses advanced machine learning techniques to generate personalized recommendations, personalized search results, and personalized direct marketing.


#### Key Features
1. **Personalized Recommendations**: AWS Personalize can provide personalized recommendations for various use cases, such as product recommendations, content recommendations, and personalized marketing.

2. **Personalized Search**: With Personalize, you can improve search relevance by incorporating user preferences and behavior into search results.

3. **Personalized Direct Marketing**: Personalize helps create more engaging and targeted marketing campaigns by generating personalized messaging and offers for individual users.

4. **Real-time Personalization**: Personalize can generate real-time personalized recommendations based on a user's current behavior and context.

5. **Automatic Model Training**: Personalize automatically trains and optimizes machine learning models based on your data, freeing you from the complexities of building and maintaining personalization models.

6. **Integrations**: Personalize integrates seamlessly with other AWS services like Amazon S3, Amazon SageMaker, and AWS Lambda, making it easy to incorporate personalization into your existing applications.

#### How it Works

1. **Data Preparation**: You provide AWS Personalize with your historical user interaction data, such as user behavior, preferences, and item metadata.

2. **Model Training**: Personalize automatically trains and optimizes machine learning models based on your data.

3. **Real-time Inference**: Your application can then send real-time user events and context to Personalize, which generates personalized recommendations or search results in response.

4. **Continuous Learning**: Personalize continuously updates and improves its models as it receives more data from your application, providing increasingly accurate personalization over time.

AWS Personalize simplifies the process of building and deploying personalization capabilities, allowing developers to focus on their core application while providing their users with highly personalized experiences.


## Amazon Textract


- Amazon Textract is a machine learning service that automatically extracts text and data from scanned documents, forms, and tables.


## Amazon TorchServe and Neuron

- Amazon TorchServe is an open-source model serving library for PyTorch models, and Amazon Neuron is a hardware-accelerated inference engine for AWS Inferentia-based EC2 instances.

- Amazon Neuron is a hardware-accelerated inference engine for AWS Inferentia-based EC2 instances. It is designed to deliver high-performance, low-latency inference for deep learning models in production.



## Amazon Panorama

- Amazon Panorama is a machine learning appliance and software development kit (SDK) that allows you to add computer vision (CV) to existing on-premises cameras to make them "smart" and capable of running computer vision models at the edge.

  
## AWS DeepComposer


- AWS DeepComposer is a machine learning-enabled musical keyboard and a cloud-based service that allows developers to create original music using machine learning models.



## Amazon CodeGuru

- Amazon CodeGuru is a machine learning-based service that provides intelligent recommendations for improving code quality and identifying potential performance issues in your applications.


## Find more information on

1. [Architecting-on-AWS](Architecting-on-AWS.md)

AWS ML Services

Amazon Sage Maker

Architecting-on-AWS

# Machine Learning Notes Website 

This website aims to serve as a repository of notes from AWS machine learning training conducted at BIBF (Bahrain Institute of Banking and Finance) through a partnership program with Tamkeen. 


![](attachments/1701972657370_Page_4_Image_0001.png)
  
  

The goal is to create an open platform where trainees can contribute and access a knowledge base to supplement their learning during the AWS ML certification course.




## Contents

The website will host machine learning tutorials, examples, explanations of concepts, summaries of sessions, handy tips and tricks, etc. related to the curriculum.

- Session-wise notes from lectures
- Cheat sheets  
- Revision notes
- Tips for certification


## Recommend Path 

1. [Cloud Essential](Cloud%20Essential.md)
1. [Architecting-on-AWS](Architecting-on-AWS.md)
1. [Exploratory Data Analysis](Exploratory%20Data%20Analysis.md)
1.  [Data Analytics Fundamentals](Data%20Analytics%20Fundamentals.md)
1. [Data Engineering](Data%20Engineering.md)
1. [Building Data Analytics](Building%20Data%20Analytics.md)
1. [Machine Learning Pipeline on AWS](Machine%20Learning%20Pipeline%20on%20AWS.md)
1.  [MLOPS Engineering](MLOPS%20Engineering.md)
1.  [Amazon Sage Maker](Amazon%20Sage%20Maker.md)




 


BIBF-AI



# Building Batch Data Analytics Solutions on AWS

In this guide, we'll explore how to build batch data analytics solutions using Amazon EMR (Elastic MapReduce). Amazon EMR is a cloud-based big data platform that allows you to process large datasets efficiently. We'll cover various modules to help you understand the key components and best practices for designing and implementing batch analytics solutions.

## Module 1: Introduction to Amazon EMR

### What is Amazon EMR?

Amazon EMR is a managed big data platform that simplifies the deployment and management of distributed data processing frameworks such as Apache Hadoop, Apache Spark, and Apache Hive. It allows you to process large amounts of data using scalable compute resources.

### Key Features of Amazon EMR:

- **Scalability**: EMR automatically scales resources based on workload requirements.
- **Managed Clusters**: EMR handles cluster provisioning, scaling, and termination.
- **Integration with AWS Services**: EMR integrates seamlessly with other AWS services like S3, DynamoDB, and Redshift.
- **Security and Access Control**: EMR provides security features such as encryption, IAM roles, and VPC integration.

### File system options with EMR

- HDFS (Hadoop Distributed File System) - default file system for Hadoop-based workloads
- EMRFS (EMR File System) - allows EMR clusters to read and write data directly to Amazon S3

### Metadata storage solutions for Amazon EMR

- Internal metastore – By default, Hive records metastore information in a MySQL database on the primary node's file system. When a cluster terminates, all cluster nodes shut down, including the primary node. When this happens, local data is lost because node file systems use ephemeral storage.

- External Hive metastore – To use an external MySQL database or
  Amazon Aurora as your Hive metastore, you override the default configuration
  values for the metastore in Hive to specify the external database location,
  either on an Amazon RDS MySQL instance or an Amazon Aurora PostgreSQLinstance.

- AWS Glue Data Catalog – With Amazon EMR version 5.8.0 or later, you can
  configure Hive to use the AWS Glue Data Catalog as its metastore. This
  configuration meets requirements for a persistent metastore or a metastore
  shared by different clusters, services, applications, or AWS accounts. You
  AWS CLI or API, you use the configuration classification for Hive to specify
  the Data Catalog.

## Module 2: Data Analytics Pipeline Using Amazon EMR: Ingestion and Storage

![](attachments/Pasted%20image%2020240214033128.png)
### Data Ingestion:

- Use Amazon S3 to store raw data files.
- Set up EMR clusters to ingest data from S3 buckets.
- Consider using AWS Glue for ETL (Extract, Transform, Load) tasks.

### Data Storage:

- Leverage Amazon S3 for durable and scalable storage.
- Use EMRFS (EMR File System) for efficient data access from EMR clusters.

### Format Supported

![](attachments/Pasted%20image%2020240214033246.png)



#### What is Parquet?

Parquet is a columnar storage file format that provides efficient storage and encoding of data. It is designed to work well with complex data types and nested data structures, making it suitable for big data processing frameworks like Apache Spark and Apache Hive.

> Note imagine a table with 10k  columns, and you only need 50 columns, parquet will only read the 50 columns you need, and not the entire table, which is a huge performance gain.

### Data Compression

![](attachments/Pasted%20image%2020240214033700.png)

#### what to consider when choosing a compression format?

- storage cost
- bandwidth cost
- Compression ratio: The amount of compression achieved by the format.
- Compression speed: The time it takes to compress data.
- Decompression speed: The time it takes to decompress data.
- Compatibility with big data processing frameworks.








## Module 3: High-Performance Batch Data Analytics Using Apache Spark on Amazon EMR

![](attachments/Pasted%20image%2020240214034417.png)

### Spark on EMR: use cases

- ** Interactive Data Analysis**: Spark provides an interactive shell for ad-hoc data analysis and exploration.

- **ETL and Data Transformation**: Use Spark to process and transform large datasets for downstream analytics.

- **Machine Learning**: Leverage Spark's machine learning libraries for building and training predictive models.

- **Graph Processing**: Analyze and process graph data using Spark's graph processing capabilities.

### Apache Spark Overview:

- Spark is an open-source, in-memory data processing framework.
- EMR supports Spark for high-performance batch analytics.
- Use Spark for data transformation, machine learning, and graph processing.

### Benefits of Spark on EMR:

- **In-Memory Processing**: Spark utilizes in-memory caching for faster data processing.

- **Distributed Computing**: Spark distributes data processing tasks across a cluster of nodes.

- **Developer Productivity**: Spark provides high-level APIs for Python, Scala, and Java.


### Spark features on Amazon EMRo

- **EMRFS s3-optimized committer**: The EMRFS S3-optimized committer is a new output committer for Apache Spark that improves write performance when writing to S3. 

- **Spark web UI**: The Spark web UI is a web-based tool for monitoring and debugging Spark applications. You can use the Spark web UI to view information about your Spark application, such as the application's executors, jobs, stages, and tasks.


- **Aamazon EMR on EKS**: Amazon EMR on EKS is a deployment option for Amazon EMR that allows you to run Apache Spark on Amazon Elastic Kubernetes Service (Amazon EKS). Amazon EMR on EKS runs the open-source big data frameworks on Amazon EKS to take advantage of the scalability, elasticity, and flexibility of Kubernetes.

- **AWS Glue Data Catalog**: The AWS Glue Data Catalog is a fully managed, Apache Hive-compatible metadata catalog that makes it easy to discover, prepare, and store data. The AWS Glue Data Catalog is used by AWS Glue to store table  which are
for Spark SQL queries, and is used by Amazon EMR to store table metadata for Spark SQL queries.


### Spark Components:

![](attachments/Pasted%20image%2020240214035324.png)

1. **Spark Core**: The foundation of the Spark platform, providing distributed task dispatching, scheduling, and basic I/O functionalities.

1. **Resilient Distributed Datasets (RDDs)**: A fundamental data structure in Spark that represents a collection of elements distributed across multiple nodes.

1. **Spark SQL**: Provides a DataFrame API for working with structured data and SQL queries.

1. **Spark Streaming**: Enables real-time stream processing and integration with other data sources.


1. **MLlib (Machine Learning Library)**: A scalable machine learning library for building and training machine learning models.


1. **GraphX**: A graph processing library for analyzing and processing graph data.


1. **connectors**: Spark provides connectors for integrating with various data sources such as Amazon S3, HDFS, JDBC, and more.



![](attachments/Pasted%20image%2020240214035606.png)


### Setting Up Spark on EMR:

- Create an EMR cluster with Spark installed.
- Optimize Spark configurations for performance.
- Develop Spark applications using PySpark or Scala.

![](attachments/Pasted%20image%2020240214035656.png)

## Module 4: Processing and Analyzing Batch Data with Amazon EMR and Apache Hive

### Apache Hive:

- Hive is a data warehousing and SQL-like query language for Hadoop.
- Use Hive for batch processing and querying large datasets.
- Define tables, partitions, and views in Hive.

### Integrating Hive with EMR:

- Configure Hive on your EMR cluster.
- Write HiveQL queries to analyze data.
- Optimize Hive performance by tuning settings.


### Batch Data Processing with Hive SQL:

- Use Hive to process and analyze large datasets.
- Write SQL-like queries to perform data transformations.
- Leverage Hive's partitioning and bucketing for efficient processing.

### Benefits of Hbase on EMR:

- **Fast Query Execution**: Hive optimizes queries for efficient execution on EMR clusters.

- **Scalability**: Hive can handle large-scale data processing and analytics.

- **Fault Tolerance**: EMR provides fault-tolerant infrastructure for Hive processing.


## Module 5: Serverless Data Processing

You can use serverless services, practices, and strategies to build more agile applications that help you innovate and respond to change faster. AWS handles infrastructure management tasks, such as capacity provisioning and patching, so you can focus on writing code that serves your customers. This module explores how serverless technology can support data processing, cataloging, and analytics at scale.

- Move from idea to market faster – By eliminating operational overhead, your teams can release quickly, get feedback, and iterate to get to market faster.
-  Adapt at scale – With technologies that automatically scale from zero to peak demands, you can adapt to customer needs faster than ever.
-  Build better applications with less effort – Serverless applications have built-in service integrations. You can focus on building your application instead of configuring it.
-  Pay for use – Serverless has a pay-for-value billing model. You never pay for over-provisioning, and resource use is optimized on your behalf.


![](attachments/Pasted%20image%2020240215072022.png)
## Module 6: Security and Monitoring of Ama zon EMR Clusters

### Security Best Practices:

- Secure EMR clusters using IAM roles and policies.
- Enable encryption for data at rest and in transit.
- Implement network security using VPCs and security groups.

### Monitoring and Logging:

![](attachments/Pasted%20image%2020240215072315.png)



- Use Amazon CloudWatch for monitoring EMR clusters.
- Monitor resource utilization, job progress, and failures.
- Set up logging to track cluster activities.

## Module 7: Designing Batch Analytics Solutions

### Architectural Considerations:

- Choose the right EMR instance types based on workload.
- Design fault-tolerant and scalable architectures.
- Consider data partitioning and bucketing for efficient processing.



![](attachments/Pasted%20image%2020240215072413.png)

![](attachments/Pasted%20image%2020240215072428.png)

![](attachments/Pasted%20image%2020240215072454.png)
## Module B: Developing Modern Data Architectures on AWS

### Beyond EMR:

- Explore other AWS services like Amazon Redshift, Amazon Athena, and AWS Glue.
- Combine EMR with serverless services for comprehensive data solutions.

---

Feel free to expand on each section with additional details, examples, and practical use cases. If you have any specific questions or need further clarification, feel free to ask! 🚀🔍


Building Data Analytics

## Utilizing AWS: Unveiling its Significance

The instructors shed light on the essential role of Amazon Web Services (AWS) and its profound importance. They narrated compelling stories to underscore the significance of embracing cloud computing, emphasizing two key aspects:

### 1. Disaster Recovery
The instructors highlighted how AWS serves as a robust platform for disaster recovery, enabling businesses to swiftly recover and resume operations in the face of unforeseen disruptions.

### 2. Temporary Jobs Across the Web
They shared anecdotes illustrating how AWS facilitates the execution of temporary tasks on the web efficiently, demonstrating its flexibility and scalability.

## Unveiling EC2: An Overview

In the exploration of Amazon's Elastic Compute Cloud (EC2), the instructors provided a foundational introduction:

### Naming of EC2 Instances

The session delved into the nomenclature and identification of EC2 instances, ensuring a comprehensive understanding of the basic components.

## Dedicated Instances vs. Hosts: Decoding the Distinctions

The discussion extended to the nuances between dedicated instances and hosts, highlighting key differentiators:

### 1. Dedicated Instances
   - Changes Servers on Reboot
   - Retains UUID and Configuration, ensuring consistency even after restarts

### 2. Host Instances
   - Maintains server identity across reboots
   - UUID and configuration persist, ensuring a seamless and stable hosting environment

## Reserved EC2 Instances: Unraveling the Changes

The instructors clarified the status of reserved EC2 instances, noting that they are no longer available, replaced by a discounted membership program, providing enhanced cost benefits.



## Cloud Computing

Ally learned that instead of using computers at home or work, you can use "cloud computing". This means using computers somewhere else called "the cloud".

In the cloud there are lots of computers called "servers" that you can use. You don't have to wait a long time for new computers or pay for computers you are not using. You only pay for what you use!

## Virtual Machines

In the cloud there are "virtual machines". These are like pretend computers that act like real computers but are not real. Each virtual machine has its own pretend operating system like Windows or MacOS.

You can make as many virtual machines as you want and they can be turned off and on quickly. This is better than using real computers which take a long time to turn on and off.

## Containers

Containers are like virtual machines but even better. They only have your apps and things your apps need, without a whole operating system. This makes them faster to start than virtual machines.

Containers also make it easy for developers to move their apps from their computers to real computers without problems.

## AWS Services

AWS has lots of cloud computing services like:

- EC2 for virtual machines
- ECS and EKS to organize containers
- S3 for storage
- DynamoDB for databases
- Lambda for running code briefly

You only pay for what you use of each service, like paying per second for Lambda. This is better than buying your own computers.

## Tips for the Test

- Read the question carefully
- Answer based on what's asked
- Know when to use ECS vs EKS for containers
- Pick the simplest option unless told otherwise

###  Virtual Private Cloud (VPC): Building Isolated Environments

In the discussion about VPC, the instructors elucidated the concept of creating isolated environments within AWS:

- **Secure Isolation:** VPCs enable the creation of private and secure networks, allowing businesses to control communication and access within their cloud infrastructure.
    
- **Customization:** Users can tailor VPCs to their specific needs, defining IP address ranges, subnets, and configuring route tables to suit their organizational requirements.
    

###  IP Addresses: Navigating the Digital Landscape

The session transitioned to understanding IP addresses within the AWS ecosystem:

- **Resource Identification:** IP addresses play a pivotal role in uniquely identifying resources within a VPC, facilitating seamless communication between instances.
    
- **Elastic IPs:** Instructors explained the flexibility of Elastic IP addresses, allowing users to associate a static IP with their instances, ensuring consistent accessibility.
    

### Load Balancer: Distributing Workloads Effectively

The discussion then shifted to the significance of Load Balancers in optimizing performance:

- **Even Workload Distribution:** Load Balancers efficiently distribute incoming traffic across multiple instances, promoting load balancing and preventing overload on specific resources.

- **Enhanced Availability:** By distributing traffic across various instances, Load Balancers contribute to increased availability and fault tolerance within the system.
    

###  Direct Connect: Establishing Dedicated Connections

Instructors introduced the concept of Direct Connect as a means of establishing dedicated connections to AWS:

- **Private Network Connection:** Direct Connect provides a private, dedicated connection between an organization's on-premises data center and AWS, enhancing data transfer security and reliability.

- **Predictable Performance:** With a dedicated connection, organizations can achieve more predictable network performance compared to internet-based connections.


### VPN (Virtual Private Network): Securing Communication Channels

The session concluded with insights into VPNs as a mechanism for securing communication channels:

- **Encrypted Communication:** VPNs ensure secure and encrypted communication between on-premises networks and AWS, safeguarding sensitive data during transit.
    
- **Cost-Effective Connectivity:** VPNs offer a cost-effective solution for establishing secure connections, especially for businesses with varying workloads and communication needs.

Cloud Essential

# Data Analytics Fundamentals



## Introduction
Data analytics is the science of analyzing raw data in order to make
conclusions about that information. Many of the techniques and processes of
data analytics have been automated into mechanical processes and algorithms
that work over raw data for human consumption. Data analytics techniques can
reveal trends and metrics that would otherwise be lost in the mass of
information. 


### Benefits of Data Analytics

   ![](attachments/Pasted%20image%2020240205091147.png)


### Components of a Data analytics process
![](attachments/Pasted%20image%2020240205091737.png)

### 5Vs of Big Data

- **Volume**: The amount of data.
- **Velocity**: The speed of data processing.
- **Variety**: The different types of data.
- **Veracity**: The quality of the data.
- **Value**: The value of the data.
  

  ![](attachments/Pasted%20image%2020240205093506.png)
  

## Volume

![](attachments/Pasted%20image%2020240205094811.png)





## Amazon S3 concepts

To get the most out of Amazon S3, you need to understand a few simple concepts. First, Amazon S3 stores data as objects within buckets.

An object is composed of a file and any metadata that describes that file. To store an object in Amazon S3, you upload the file you want to store into a bucket. When you upload a file, you can set permissions on the object and add any metadata.

Buckets are logical containers for objects. You can have one or more buckets in your account and can control access for each bucket individually. You control who can create, delete, and list objects in the bucket. You can also view access logs for the bucket and its objects and choose the geographical region where Amazon S3 will store the bucket and its contents



### Data Lakes

![](attachments/Pasted%20image%2020240205095742.png)

![](attachments/Pasted%20image%2020240205100014.png)

#### Benefits of a data lake on AWS

- Are a cost-effective data storage solution. You can durably store a nearly unlimited amount of data using Amazon S3.

- Implement industry-leading security and compliance. AWS uses stringent data security, compliance, privacy, and protection mechanisms.
- Allow you to take advantage of many different data collection and ingestion tools to ingest data into your data lake. These services include Amazon Kinesis for streaming data and AWS Snowball appliances for large volumes of on-premises data.

- Help you to categorize and manage your data simply and efficiently. Use AWS Glue to understand the data within your data lake, prepare it, and load it reliably into data stores. Once AWS Glue catalogs your data, it is immediately searchable, can be queried, and is available for ETL processing.

- Help you turn data into meaningful insights. Harness the power of purpose-built analytic services for a wide range of use cases, such as interactive analysis, data processing using Apache Spark and Apache Hadoop, data warehousing, real-time analytics, operational analytics, dashboards, and visualizations.
  
  ![](attachments/Pasted%20image%2020240205100300.png)
  

#### Data Lake on AWS

- Collect and store any type of data, at any scale, and at low cost

- Secure the data and prevent unauthorized access

- Catalog, search, and find the relevant data in the central repository

- Quickly and easily perform new types of data analysis

- Use a broad set of analytic engines for one-time analytics, real-time streaming, predictive analytics, AI, and machine learning

### Data Lake vs Data Warehouse

1. **Data Lake**:
   - **Definition**: A **data lake** is like a real lake, collecting data from multiple sources (referred to as "rivers"). It stores various types of data, including structured, semi-structured, and unstructured data.
   - **Purpose**: Data lakes are designed to capture and store large amounts of raw data. This data can be used for machine learning, artificial intelligence (AI) algorithms, and business purposes. After processing, data from a data lake can also be transferred to a data warehouse.
   - **Data Type**: Raw data of all types, regardless of its source or structure.
   - **Schema Position**: After data storage, offering agility and easy data capture.
   - **Users**: Data scientists and those who need in-depth analysis and tools (such as predictive modeling).
   - **Accessibility**: Accessible and easy to update.
   - **History**: Relatively new for big data¹.

2. **Data Warehouse**:
   - **Definition**: A **data warehouse** stores structured data that has been processed for specific purposes. It serves as a repository for already structured data, which can be queried and analyzed.
   - **Purpose**: Data warehouses are organized systems used by business professionals for operations. They provide security and high performance.
   - **Data Type**: Processed data stored according to metrics and attributes.
   - **Schema Position**: Before data storage, ensuring security and performance.
   - **Users**: Business professionals who need structured analytics.
   - **Accessibility**: More complicated to make changes.
   - **History**: The concept has been around for decades.

In summary, data lakes are versatile and accommodate raw data, while data warehouses focus on structured data for specific analytical purposes. Both play crucial roles in managing and utilizing big data effectively. 🌊🏢

For a deeper introduction to data lakes, you can watch this informative video from Google: [What is a data lake?](https://www.youtube.com/watch?v=0YgQzqjCz3E)


### Redshift Spectrum
Amazon Redshift Spectrum is a feature of Amazon Redshift that allows you to run complex SQL queries against exabytes of data in Amazon S3. With Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond the data that is stored on local disks. Redshift Spectrum directly queries the data in your Amazon S3 data lake. You can use Redshift Spectrum to run SQL queries against your data lake and get results in seconds.

###  Data Warehouses
![](attachments/Pasted%20image%2020240205101712.png)


## Data processing

Data processing means the collection and manipulation of data to produce meaningful information. Data collection is divided into two parts, data collection and data processing.

![](attachments/Pasted%20image%2020240205102501.png)

![](attachments/Pasted%20image%2020240205111558.png)


### Type of Processing

![](attachments/Pasted%20image%2020240205112454.png)



![](attachments/Pasted%20image%2020240205123954.png)


### Data  Structeures
![](attachments/Pasted%20image%2020240205124602.png)


![](attachments/Pasted%20image%2020240205124550.png)

![](attachments/Pasted%20image%2020240205124632.png)



### Varitey

![](attachments/Pasted%20image%2020240205133420.png)

#### Indexing

![](attachments/Pasted%20image%2020240205133539.png)


## The problem of Veracity

Data changes over time. As it is transferred from one process to another, and through one system and another, there are opportunities for the integrity of the data to be negatively impacted. You must ensure that you maintain a high level of certainty that the data you are analyzing is trustworthy.


## integrity 

![](attachments/Pasted%20image%2020240205142023.png)

![](attachments/Pasted%20image%2020240205142120.png)

![](attachments/Pasted%20image%2020240205145850.png)




![](attachments/Pasted%20image%2020240205152220.png)


Certainly! Let’s delve into the differences between the **ACID** and **BASE** database transaction models:

1. **ACID (Atomicity, Consistency, Isolation, Durability)**:
    
    - **Atomicity**: Each transaction is either fully executed or completely rolled back. If any part of a transaction fails, the entire transaction is aborted, ensuring data consistency.
    - **Consistency**: Transactions maintain the structural integrity of the database. Valid states are preserved, and no invalid data is allowed.
    - **Isolation**: Transactions are isolated from each other. They cannot interfere with one another while in progress.
    - **Durability**: Completed transactions persist even in the face of network or power failures. Data remains intact.
    - **Use Case Example**: Financial institutions heavily rely on ACID databases for secure money transfers, where atomicity ensures that funds are either fully debited or credited without discrepancies. [Common ACID-compliant databases include MySQL, PostgreSQL, Oracle, SQLite, and Microsoft SQL Server](https://phoenixnap.com/kb/acid-vs-base)[1](https://phoenixnap.com/kb/acid-vs-base).
2. **BASE (Basically Available, Soft state, Eventually consistent)**:
    
    - **Basically Available**: BASE prioritizes high availability over immediate consistency. It acknowledges that temporary inconsistencies may occur during distributed system operations.
    - **Soft State**: Unlike ACID, BASE allows transient inconsistencies. The system can be in an intermediate state during updates or failures.
    - **Eventually Consistent**: Over time, the system converges to a consistent state. It doesn’t guarantee immediate consistency but ensures eventual correctness.
    - **Use Case Example**: NoSQL databases often follow the BASE model. These databases handle large-scale data, trading off strict consistency for scalability and availability. [Examples include Apache CouchDB and IBM Db2](https://phoenixnap.com/kb/acid-vs-base)[1](https://phoenixnap.com/kb/acid-vs-base).



### Types of Data Analytics

![](attachments/Pasted%20image%2020240205160637.png)


![](attachments/Pasted%20image%2020240205160658.png)


![](attachments/Pasted%20image%2020240205160719.png)


Data Analytics Fundamentals

# Data Engineering

## S3

### S3 Storage Classes

- **S3 Standard**: For general-purpose storage of frequently accessed data; designed for durability of 99.999999999% of objects over a given year.
- **S3 Intelligent-Tiering**: Designed to optimize costs by automatically moving data to the most cost-effective access tier, without performance impact or operational overhead.
- **S3 Standard-IA**: For long-lived, infrequently accessed data that needs to be accessed rapidly when needed.
- **S3 One Zone-IA**: For long-lived, infrequently accessed, non-critical data.
- **S3 Glacier**: For long-term archive and digital preservation.
- **S3 Glacier Deep Archive**: For long-term archive and digital preservation with retrieval times in hours.

### Lifecycle Policies

- **Transition Actions**: Define when objects transition to another storage class. For example, you might choose to transition objects to the S3 Standard-IA storage class 30 days after you created them, or archive objects to the S3 Glacier storage class one year after creating them.

- **Expiration Actions**: Define when objects expire. For example, you might choose to expire objects in the S3 Standard storage class 365 days after creating them.

- **Noncurrent Version Transition Actions**: Define when noncurrent object versions transition to another storage class. For example, you might choose to transition noncurrent object versions to the S3 Standard-IA storage class 30 days after creating them, or archive noncurrent object versions to the S3 Glacier storage class one year after creating them.

- **Noncurrent Version Expiration Actions**: Define when noncurrent object versions expire. For example, you might choose to expire noncurrent object versions in the S3 Standard storage class 365 days after creating them.

### S3 Encryption

- **S3-Managed Keys (SSE-S3)**: Amazon S3 encrypts each object with a unique key. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates. Amazon S3 server-side encryption uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.

- **AWS Key Management Service (SSE-KMS)**: With AWS Key Management Service (AWS KMS), you can centrally create, manage, and delete the keys used for server-side encryption. AWS KMS is integrated with several AWS services to make it easy to encrypt the data you store in these services.

- **Customer-Provided Keys (SSE-C)** : You manage the encryption process, the encryption keys, and related tools. You are responsible for protecting the encryption keys you use to protect your data.

### S3 Versioning

- **Versioning**: Versioning is a means of keeping multiple variants of an object in the same bucket. You can use versioning to preserve, retrieve, and restore every version of every object stored in your Amazon S3 bucket. With versioning, you can easily recover from both unintended user actions and application failures.

## AWS kinesis Overview

### kinds of Kinesis

- **Kinesis Data Streams**: Kinesis Data Streams is a massively scalable and durable real-time data streaming service. It can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, financial transactions, social media feeds, IT logs, and location-tracking events.

- **Kinesis Data Firehose**: Kinesis Data Firehose is the easiest way to reliably load streaming data into data stores and analytics tools. It can capture, transform, and load streaming data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today.

- **Kinesis Data Analytics**: Kinesis Data Analytics is the easiest way to process and analyze streaming data in real time with standard SQL without having to learn new programming languages or processing frameworks. Kinesis Data Analytics enables you to query streaming data or build entire streaming applications using SQL, so that you can gain actionable insights and respond to your business and customer needs promptly.

- **Kinesis Video Streams**: Kinesis Video Streams makes it easy to securely stream video from connected devices to AWS for analytics, machine learning (ML), and other processing. Kinesis Video Streams automatically provisions and elastically scales all the infrastructure needed to ingest streaming video data from millions of devices. It also durably stores, encrypts, and indexes video data in your streams, and allows you to access your data through easy-to-use APIs.

![](attachments/Pasted%20image%2020240224111007.png)

#### Kinesis Data Streams

**Provisioned mode**: In this mode, you need to specify the number of shards while creating the stream. You can increase or decrease the number of shards in the stream as per your requirement.

- You choose the number of shards when you create the stream.
- Each shard gets a capacity of 1MB/sec data input and 2MB/sec data output.
- You pay per shard-hour and for the amount of data you ingest and the amount of data you read from the stream.

**On-demand mode**: In this mode, you don’t need to specify the number of shards while creating the stream. Kinesis Data Streams automatically scales the number of shards based on the amount of data in the stream.

- You don’t need to specify the number of shards when you create the stream.
- Kinesis Data Streams automatically scales the number of shards based on the amount of data in the stream.
- You pay for the amount of data you ingest and the amount of data you read from the stream.

![](attachments/Pasted%20image%2020240224155019.png)

###### Kinesis Streams Overview

- Streams are divided in ordered Shards / Partitions
- Data retention is 24 hours by default, can go up to 365 days
- Ability to reprocess / replay data
- Multiple applications can consume the same stream
- Once data is inserted in Kinesis, it can’t be deleted (immutability)
- Records can be up to 1MB in size

#### Kinesis Data Firehose

![](attachments/Pasted%20image%2020240224155427.png)

###### Kinesis Firhose Overview

•Fully Managed Service, no administration
•Near Real Time (60 secnds latency minimum for non full batches)
•Data Ingestion into Redshift / Amazon S3 / ElasticSearch / Splunk
•Automatic scaling
•Supports many data formats
•Data Conversions from CSV / JSON to Parquet / ORC (only for S3)
•Data Transformation through AWS Lambda (ex: CSV => JSON)
•Supports compression when target is Amazon S3 (GZIP, ZIP, and SNAPPY)
•Pay for the amount of data going through Firehose

### Data Streams vs Firehose

**Streams**
•Going to write custom code (producer / consumer)
•Real time (~200 ms latency for classic, ~70 m slatency for enhanced fan-out)
•Automatic scaling with On-demand Mode
•Data Storage for 1 to 365 days, replay capability, multi consumers

**Firehose**
•Fully managed, send to S3, Splunk, Redshift, ElasticSearch
•Serverless data transformations with Lambda
•Nearreal time (lowest buffer time is 1 minute)
•Automated Scaling
•No data storage

### Kinesis Data Analytics

**Use cases**

•Streaming ETL: select columns, make simple transformations, on streaming data
•Continuous metric generation:live leaderboard for a mobile game
•Responsive analytics:look for certain criteria and build alerting (filtering)

**Features**
•Pay only for resources consumed (but it’s not cheap)
•Serverless; scales automatically
•Use IAM permissions to access streaming source and destination(s)
•SQL or Flink to write the computation
•Schema discovery
•Lambda can be used for pre-processing

![](attachments/Pasted%20image%2020240224162100.png)

#### Kinesis Data Analytics + Lambda

- Aws Lambda can be a destination aws well

- Allows lots of flexibility for post-processing
  - Ex: send an alert if a certain condition is met
  - Ex: send data to a third party service
  - Ex: store data in a database

![](attachments/Pasted%20image%2020240224162137.png)

### Kinesis Anaylytics

![](attachments/Pasted%20image%2020240224162816.png)

### Kinesis Summary - Machine Learning

- Kinesis Data Streams: real-time data streaming at scale

- Kinesis Data Firehose: load data streams into AWS data stores near real-time

- Kinesis Data Analytics: analyze data streams with SQL or Flink

- Kinesis Video Streams: store and analyze video streams

## Glue Data Catalog

![](attachments/Pasted%20image%2020240301104417.png)

### Glue Data Catalog

Crawlers connect to a data store, progresses through a prioritized list of classifiers to determine the schema for your data, and then creates metadata tables in the Data Catalog.

- Works with S3, RDS, Redshift, and any JDBC data store

### Why not use python scripts and use Glue and ETL ?

You are nike you have 140 TB of data and you want to do ETL how do you do it?

Downloading 140 TB of data does your computer have enough space? how long will it take?

Even if you have enough space, how long will it take to process 140 TB of data?

> That's why you use Glue and ETL because data is already in the cloud because you won't have data center in your office and you are not IBM or Oracle or Microsoft your data is already in the cloud and you want to process it's much faster to use Glue and ETL than to download the data and process it on your computer.

### Glue ETL

- Transforms your data , cleans it, normalizes it, and prepares it for loading into data warehouse
  - Fully managed, cost-effective, serverless, and easy to use
  - Generate ETL code in Python or Scala , you can modify the code


- Bundled with a lot of pre-built transformations
  - Join, filter, etc
  - You can also use your own scripts
  - Map - add fields, remove fields, etc

- Machine Learning Transformations
  - Find matches ML : identify duplicate records in your dataset,even when
   the records don't have a common unique identifier

- **Format Conversion**
  - Convert data to Parquet or ORC for performance
  - Convert data to Apache Avro or JSON for schema evolution
  - Convert data to CSV or XML for schema evolution
  

### AWS Glue DataBrew


- AWS Glue DataBrew is a visual data preparation tool that enables you to clean and normalize data without writing code.

- Reduces ML and analytics data preparation time by up to 80%.



## Database  Options

![](attachments/Pasted%20image%2020240301113923.png)


![](attachments/Pasted%20image%2020240301114324.png)

![](attachments/Pasted%20image%2020240301114428.png)

## Possible Machine Learning Architecture

![](attachments/Pasted%20image%2020240301114303.png)


![](attachments/Pasted%20image%2020240301114533.png)

![](attachments/Pasted%20image%2020240301114542.png)


![](attachments/Pasted%20image%2020240301114558.png)


#### To Read Next 
[Exploratory Data Anaylis](Exploratory%20Data%20Anaylis.md)

Data Engineering

# Data Lake

## Introduction to data lakes

### Why Data Lakes?

1.  Are a great way to store and analyze large volumes of data at a low cost.
1.  Exponential growth of data.
1.  unstructured data.
    -  Text
    - Images 
    - Videos
    - Audio
1. Need analytics faster 
    - Days 
    - Hours
    - Real time


### Benefits of Data Lakes


![](attachments/Pasted%20image%2020240207093527.png)


### Data warehouse vs Data Lake

![](attachments/Pasted%20image%2020240207094222.png)

### Data Lake

![](attachments/Pasted%20image%2020240207095533.png)


### Analytics Functionality vs Aws

![](attachments/Pasted%20image%2020240207100450.png)



![](attachments/Pasted%20image%2020240207100521.png)


### This Course will cover

![](attachments/Pasted%20image%2020240207101104.png)


### AWS Glue

AWS Glue is a fully managed extract, transform, and load (ETL) service provided by Amazon Web Services (AWS). It is designed to simplify the process of preparing and loading data for analysis. AWS Glue automates the discovery, cataloging, and transformation of data from various sources, making it easier to move and combine datasets for analytics.

#### Key Features:

- **Data Catalog:**
  - AWS Glue includes a centralized metadata repository called the Data Catalog, where information about available data sources, transformations, and targets is stored.
  - The Data Catalog provides a unified view of metadata, making it easier to discover and understand the structure of your data.

- **ETL Automation:**
  - With AWS Glue, you can create ETL jobs using a visual interface or by writing custom code in Python or Scala.
  - The service automatically generates ETL code, reducing the need for manual coding and accelerating the ETL process.

- **Serverless Architecture:**
  - AWS Glue is a serverless service, meaning you don't need to provision or manage infrastructure. It scales dynamically based on the processing needs of your jobs.

- **Integration with Other AWS Services:**
  - AWS Glue seamlessly integrates with other AWS services, such as Amazon S3, Amazon Redshift, and Amazon RDS, allowing you to build end-to-end data workflows.

#### Data Catalog

The Data Catalog is a crucial component of AWS Glue, serving as a metadata repository that stores information about your data assets. It acts as a comprehensive catalog that facilitates the management, organization, and discovery of data. Here are some key aspects of the AWS Glue Data Catalog:

- **Metadata Storage:**
  - The Data Catalog stores metadata about tables, schemas, partitions, and transformations. This metadata provides a detailed understanding of the structure and characteristics of your data.

- **Unified View:**
  - By centralizing metadata in the Data Catalog, AWS Glue provides a unified view of your data assets. This makes it easier for data engineers, analysts, and scientists to discover and work with data across various sources.

- **Search and Discovery:**
  - Users can search for and discover relevant datasets using the Data Catalog. The metadata includes information about data sources, formats, and transformations, aiding in the exploration of available data.

- **Access Control:**
  - AWS Glue allows you to define access controls for the Data Catalog, ensuring that only authorized users can view or modify metadata. This helps in maintaining data security and compliance.

- **Integration with External Tools:**
  - The Data Catalog can be seamlessly integrated with external tools and services, enhancing the overall data governance and collaboration within your organization.


## Data ingestion and cataloging


### Storage of datasets
![](attachments/Pasted%20image%2020240207105153.png)

### Difference between CSV and Parquet

#### 1. Storage Format:

- **CSV:** Stores data in plain text format with values separated by commas or other delimiters. It is a human-readable format but may not be as space-efficient as binary formats.
  
- **Parquet:** Utilizes a columnar storage format, storing data in a highly compressed binary format. It organizes data into columns rather than rows, allowing for better compression and efficient storage.

#### 2. Compression:

- **CSV:** Generally has limited built-in compression. The compression is achieved mainly through general-purpose algorithms applied to the entire file.

- **Parquet:** Uses advanced compression techniques, including dictionary encoding and run-length encoding, which can significantly reduce storage space. It allows for more efficient compression by compressing data within each column separately.

#### 3. Schema Evolution:

- **CSV:** Schema evolution (changing the structure of data over time) can be challenging, especially when dealing with large datasets, as the schema is not explicitly defined within the file.

- **Parquet:** Supports schema evolution, making it easier to add, remove, or modify columns without the need for data conversion. This is particularly useful in evolving data architectures.

#### 4. Performance:

- **CSV:** Reading and writing CSV files can be slower, especially when dealing with large datasets, as the entire file must be processed sequentially.

- **Parquet:** Due to its columnar storage and compression, Parquet is designed for high performance, especially for analytical queries that involve reading only specific columns. This makes it suitable for big data processing frameworks like Apache Spark and Apache Hive.

#### 5. Data Types:

- **CSV:** Generally supports basic data types (text, numbers), and the interpretation of data types may depend on how the data is imported or read.

- **Parquet:** Supports a wide range of complex data types, including nested structures and custom types. This allows for more flexibility in representing diverse datasets.

#### 6. Use Cases:

- **CSV:** Suitable for simple, tabular data storage and exchange, especially when human readability is a priority.

- **Parquet:** Ideal for big data processing, analytics, and data warehousing, where storage efficiency, fast query performance, and schema evolution are crucial.

### Difference type of data ingestion in AWS


![](attachments/Pasted%20image%2020240207110849.png)

![](attachments/Pasted%20image%2020240207110914.png)


![](attachments/Pasted%20image%2020240207111006.png)


### Crawl and Catalogue data 

AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog, an ETL engine that automatically generates Python or Scala code, and a flexible scheduler that handles dependency resolution, job monitoring, and retriesr

AWS Glue Data Catalog The AWS Glue Data Catalog provides a uniform repository where disparate systems can store and find metadata to keep track of data in the data lake. The catalog contains metadata describing and referencing the data lake stores and native storage location. When the data is cataloged, it is immediately available for search and query using Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.

AWS Glue Crawler AWS Glue Crawlers can scan data in all kinds of repositories, classify it, extract schema information from it, and store the metadata automatically in the AWS Glue Data Catalog. AWS Glue Studio is designed to work with both tabular and semi-structured data. Crawlers run as AWS Glue jobs and can be triggered by the scheduler component of AWS Glue.

#### Crawler

![](attachments/Pasted%20image%2020240207113804.png)


## Building a data Lake with AWS Lake Formation


![](attachments/Pasted%20image%2020240207131715.png)


![](attachments/Pasted%20image%2020240207132626.png)


![](attachments/Pasted%20image%2020240207141558.png)


![](attachments/Pasted%20image%2020240207141558.png)


## Data processing and analytics

### Data preparation

![](attachments/Pasted%20image%2020240207141924.png)

### Data cleansing and preparation

Data cleansing is the process of identifying and correcting errors or inconsistencies in data to improve its quality. It involves various techniques to clean, transform, and standardize data, making it suitable for analysis and reporting. Here are some common data cleansing tasks:


#### 1. Removing Duplicates:

- **Description:** Identifying and removing duplicate records from a dataset.
- **Example:** A customer database contains multiple entries for the same customer due to data entry errors or system issues.
- **Techniques:** Using SQL queries, data deduplication tools, or custom scripts to identify and eliminate duplicate records.
- **Benefits:** Reducing data redundancy, improving data accuracy, and ensuring consistency in reporting.
- **Challenges:** Identifying duplicates accurately, especially when dealing with large datasets or complex data structures.
- **Tools:** SQL, Python (pandas library), data deduplication software.
- **Best Practices:** Establishing unique identifiers for records, implementing data validation rules, and regularly monitoring data quality.
- **Considerations:** Balancing the need for data deduplication with the risk of losing valuable information.

#### 2. Standardizing Data:

- **Description:** Converting data into a consistent format or structure to facilitate analysis and comparison.
- **Example:** Standardizing date formats, currency symbols, or units of measurement across different datasets.
- **Techniques:** Using data transformation functions, regular expressions, or lookup tables to standardize data elements.
- **Benefits:** Enhancing data uniformity, enabling accurate aggregation and reporting, and simplifying data integration.
- **Challenges:** Dealing with variations in data formats, handling exceptions, and maintaining data standardization over time.
- **Tools:** ETL tools, scripting languages, data quality software.
- **Best Practices:** Creating data standardization guidelines, leveraging automation for repetitive tasks, and documenting data transformation processes.
- **Considerations:** Adapting standardization rules to specific data domains or business requirements.


#### 3. Handling Missing Values:

- **Description:** Identifying and addressing missing or null values in a dataset.
- **Example:** A sales database contains missing entries for customer addresses, product prices, or order dates.
- **Techniques:** Imputing missing values, removing incomplete records, or flagging missing data for further analysis.
- **Benefits:** Improving data completeness, reducing bias in analysis, and ensuring the accuracy of statistical calculations.
- **Challenges:** Determining the cause of missing data, selecting appropriate imputation methods, and avoiding unintended biases.
- **Tools:** Statistical software, machine learning algorithms, data profiling tools.
- **Best Practices:** Understanding the nature of missing data, validating imputation methods, and documenting data quality issues.
- **Considerations:** Balancing the impact of missing data on analysis with the potential risks of imputation.


#### 4. Data Transformation:

- **Description:** Converting data from one format, structure, or representation to another to meet specific requirements.
- **Example:** Converting raw transaction data into a summarized format for reporting or analysis.
- **Techniques:** Using data mapping, aggregation, or normalization to transform data elements.
- **Benefits:** Enabling data integration, supporting analytical queries, and preparing data for downstream processes.
- **Challenges:** Handling complex transformations, maintaining data lineage, and ensuring data consistency.
- **Tools:** ETL tools, scripting languages, data integration platforms.
- **Best Practices:** Defining clear transformation rules, validating transformation logic, and testing data quality after transformation.
- **Considerations:** Addressing the impact of data transformation on data governance, compliance, and security.

## Additional Lake Formation configuration  

### Blueprints
Blueprints structurally sit on top of AWS Glue, using its service to fulfill tasks. Lake Formation is tightly integrated with AWS Glue. The benefits of this integration are observed across features such as blueprints. Blueprints are essentially templates for standard transformation activities that can involve multiple jobs to perform a specified set of actions. From these blueprints, you instantiate Lake Formation workflows. The workflows then perform jobs and tasks as defined.

### Creating workflows

Workflows generate AWS Glue crawlers, jobs, and triggers to orchestrate the loading and updating of data. You can configure a workflow to run on demand or on a schedule. Workflows that you create in Lake Formation are visible in the AWS Glue console. Each node in a workflow is a job, crawler, or trigger. To monitor progress and troubleshoot, you can track the status of each node in the workflow.


![](attachments/Pasted%20image%2020240207151207.png)

## Modern data architecture

![](attachments/Pasted%20image%2020240207154943.png)

Scalable data lakes – This course has focused on the business value of data
lakes and how to build data lakes with AWS Lake Formation. Business insight is
driven by data of all types: structured, semi-structured, and unstructured.
Organizations have large volumes of data generated continuously. To be able to
extract value, data must be trusted and convenient to find by those roles that
need access. Key AWS services include the following: • Amazon Simple Storage
Service (Amazon S3) – Highly durable, scalable, secure, fast, and inexpensive
storage service.

• Lake Formation – Integrated data lake service that makes it convenient to ingest, clean, catalog, transform, and secure data and make it available for analysis and machine learning (ML).

• Amazon Athena – Interactive analytics service that makes it convenient to analyze data in Amazon S3 using Python or standard SQL


![](attachments/Pasted%20image%2020240207155301.png)

Data Lakes

# Deep Learning



## The biological inspiration

- Neurons in your cerebral cortex are connected via axons
- A neuron “fires” to the neurons it’s connected to, when enough of its input signals are activated.
- Very simple at the individual neuron level – but layers of neurons connected in this way can yield learning behavior.
- Billions of neurons, each with thousands of connections, yields a mind

![](attachments/Pasted%20image%2020240306231605.png)


## TensorFlow and Keras: A Brief Overview

### TensorFlow
- **TensorFlow** is an open-source machine learning framework developed by Google.
- It provides a flexible and efficient platform for building and training deep neural networks.
- Key features:
    - **Graph-based computation**: TensorFlow represents computations as directed acyclic graphs (DAGs), allowing efficient execution on CPUs, GPUs, or TPUs.
    - **Automatic differentiation**: TensorFlow computes gradients automatically for optimization during training.
    - **High-level APIs**: TensorFlow offers both high-level and low-level APIs for model development.
    - **Community support**: A large community actively contributes to TensorFlow's ecosystem.

### Keras
- **Keras** is a high-level neural networks API written in Python.

- Originally developed as an independent library, it is now part of TensorFlow.

- Key features:
    - **User-friendly**: Keras provides a simple and intuitive interface for designing neural networks.
    - **Modularity**: You can easily stack layers to create complex architectures.
    - **Extensibility**: Keras allows custom layer creation and model customization.
    - **Compatibility**: It seamlessly integrates with TensorFlow and other deep learning libraries.

## Code Snippets

### Creating a Simple Neural Network in Keras
```python
import tensorflow as tf
from tensorflow.keras import layers, models

# Define a sequential model
model = models.Sequential([
    layers.Dense(64, activation='relu', input_shape=(784,)),
    layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Print model summary
model.summary()
```

### Using TensorFlow's Low-Level API
```python
import tensorflow as tf

# Create a simple computational graph
a = tf.constant(2.0)
b = tf.constant(3.0)
c = a * b

# Execute the graph
with tf.Session() as sess:
    result = sess.run(c)
    print(f"Result: {result}")
```


## Types of Neural Networks


### Feedforward Neural Networks (FNNs)

- They consist of multiple layers of interconnected neurons, with no feedback loops.
- Information flows in one direction, from the input layer through the hidden layers to the output layer.



### Convolutional Neural Networks (CNNs)

- They are designed to process data with grid-like topology, such as images.
- They use a special type of neuron called a convolutional neuron, which applies a convolution operation to the input data.

![](attachments/Pasted%20image%2020240307001302.png)

> Convolution is just  a fancy way of saying that we are going to divide the input data into overlapping chunks and perform the same operation on each chunk.

![](attachments/Pasted%20image%2020240307001323.png)

### Recurrent Neural Networks (RNNs)

- Deals with sequence in time or space.
 - **Sequence-to-sequence**: Translate a sentence from one language to another.q
 - **Sequence-to-vector**: Sentiment analysis.
 - **Vector-to-sequence**: Image captioning.

![](attachments/Pasted%20image%2020240307001339.png)

## Types of Activation function

![](attachments/Pasted%20image%2020240306234600.png)

Activation functions are crucial elements in neural networks as they introduce non-linearity into the computations. Without activation functions, the neural network would essentially behave like a single linear function, severely limiting its ability to learn and model complex relationships between inputs and outputs. By applying non-linear activation functions to the weighted sums from each layer, the network gains the capability to capture intricate patterns and representations, enabling it to tackle sophisticated tasks effectively.


## Neural Network

[play with this](https://playground.tensorflow.org/)

The game is simple just make this


![](attachments/Pasted%20image%2020240307000553.png)

Be like this 

![](attachments/Pasted%20image%2020240307000739.png)

### Checklist in the website to do
- [ ] Press Train
- [ ] Change learning Rate
- [ ] Change Activation layer
- [ ] Regularization 
- [ ] Regularization Rate
- [ ] Problem Type

## Modern Natural Language Processing


![](attachments/Pasted%20image%2020240307001453.png)






## Transfer Learning

![](attachments/Pasted%20image%2020240307001514.png)



### Batch size
![](attachments/Pasted%20image%2020240307001605.png)

> Smaller Batch Size is more Effective


### Confusion Matrix

Go Read 

[Machine Learning Pipeline on AWS](Machine%20Learning%20Pipeline%20on%20AWS.md)


Deep learning and Machine learning

# Exploratory Data Analysis

## Python

### Pandas 
 Pandas is a fast, powerful, flexible and easy to use open source data analysis and data manipulation library built on top of the Python programming language.

  - **Data Frame**: A two-dimensional data structure, i.e., data is aligned in a tabular fashion in rows and columns.
  
  - **Series**: A one-dimensional labeled array capable of holding data of any type (integer, string, float, python objects, etc.). The axis labels are collectively called index.
  
  - **Panel**: A three-dimensional data structure, i.e., data is aligned in a tabular fashion in rows, columns and panels.
    


### Matplotlib

 Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python.

  - **Line Plot**: A line plot is a graph that shows frequency of data along a number line.
  
  - **Bar Plot**: A bar plot is a graph that represents the category of data with rectangular bars.
  
  - **Histogram**: A histogram is a graph showing frequency distributions.
  
  - **Scatter Plot**: A scatter plot is a graph that shows the relationship between two sets of data.
  
  - **Pie Chart**: A pie chart is a circular statistical graphic, which is divided into slices to illustrate numerical proportion.
  
  - **Box Plot**: A box plot is a graph that gives you a good indication of how the values in the data are spread out.
  
  - **Violin Plot**: A violin plot is a method of plotting numeric data and can be considered a combination of the box plot with a kernel density plot.
  
  - **Heat Map**: A heat map is a graphical representation of data where the individual values contained in a matrix are represented as colors.
  
  - **3D Plot**: A 3D plot is a graph that shows the relationship between three sets of data.
  
  - **Contour Plot**: A contour plot is a graphical technique for representing a 3-dimensional surface by plotting constant z slices, called contours, on a 2-dimensional format.
  
  - **Quiver Plot**: A quiver plot is a type of 2D plot that shows vector lines as arrows.
  
  - **Stream Plot**: A stream plot is a type of 2D plot that shows vector field.
  
  - **Polygon Plot**: A polygon plot is a type of 2D plot that shows a filled polygon.
  
  - **Image Plot**: An image plot is a type of 2D plot that shows an image.
  
  - **Polar Plot**: A polar plot is a type of 2D plot that shows data in polar coordinates.
  
  - **3D Surface Plot**: A 3D surface plot is a type of 3D plot that shows a surface.
  
  

## Major Types of Data 

### Numerical  
  - **Discrete**: Discrete data can only take particular values. There may potentially be an infinite number of those values, but each is distinct and there's no grey area in between. 
  - **Continuous**: Continuous data can take any value within a given range.

### Categorical

  - **Quantitative**: Quantitative data is data expressing a certain quantity, amount or range. Usually, there are measurement units associated with the data.

  - **Nominal**: Nominal data is a group of non-parametric variables, that is, variables that do not follow a regular distribution.
  
### Time Series
  
  - **Cross-Sectional**: Cross-sectional data is data collected by observing many subjects (such as individuals, firms, countries, or regions) at the same point of time, or without regard to differences in time.
  - **Time Series**: Time series data is a sequence of data points in chronological order, such as daily closing stock prices or monthly temperatures.
  
  

## Type of data  distribution

![](attachments/Pasted%20image%2020240301214202.png)

## Amazon Athena

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

## Amazon QuickSight

Amazon QuickSight is a fast, cloud-powered business intelligence service that makes it easy to deliver insights to everyone in your organization. As a fully managed service, QuickSight lets you easily create and publish interactive dashboards that include ML Insights.


## For more Details visit 

[Architecting-on-AWS](Architecting-on-AWS.md)

[Data Lakes](Data%20Lakes.md)



## SPICE 

SPICE (Super-fast, Parallel, In-memory Calculation Engine) is a general purpose, distributed, shared-nothing, parallel processing framework for in-memory analytics. SPICE is designed to be a highly available, fault-tolerant, and self-healing system that provides high throughput and low latency for data processing and analytics.

![](attachments/Pasted%20image%2020240305102328.png)


![](attachments/Pasted%20image%2020240305234634.png)


![](attachments/Pasted%20image%2020240305234647.png)


![](attachments/Pasted%20image%2020240305234656.png)




![](attachments/Pasted%20image%2020240305234705.png)

![](attachments/Pasted%20image%2020240305234713.png)

![](attachments/Pasted%20image%2020240305234720.png)



## The Curse of Dimensionality

![](attachments/Pasted%20image%2020240305234843.png)

## Imputing Missing Data : Mean, Median, Mode Replacement

Replacing missing data with the mean, median, or mode is a common method for handling missing data. This method is also known as univariate imputation. Univariate imputation is the process of replacing missing data with a statistical estimate of the missing value.


## Imputing Missing Data : Random Sample Imputation

Random sample imputation is a technique where we take random observations from the dataset and we use this observation to replace the NaN values. It consists of taking random observation from the dataset and we use this observation to replace the NaN values.


## Imputing Missing Data : Capturing NaN values with a new feature

This technique involves replacing the NaN values with a new feature which captures the NaN values. It is a good method when the data is not missing at random.



## Imputing Missing Data : Machine Learning Algorithm

This technique involves replacing the NaN values with a new feature which captures the NaN values. It is a good method when the data is not missing at random.

KNN: Find K “nearest” (most similar) rows and average their values
Assumes numerical data, not categorical There are ways to handle categorical data (Hamming distance), but categorical data is probably better served by…

Deep Learning
Build a machine learning model to impute data for your machine learning model!
Works well for categorical data. Really well. But it’s complicated.

Regression
Find linear or non-linear relationships between the missing feature and other features
Most advanced technique: MICE (Multiple Imputation by Chained Equations)



## unbalanced data

Unbalanced data typically refers to a problem with classification problems where the classes are not represented equally.


**Random Under-Sampling**: Randomly remove samples from the majority class.

**Random Over-Sampling**: Randomly duplicate samples from the minority class.


**SMOTE (Synthetic Minority Over-sampling Technique)**: Generate synthetic samples from the minority class.


**Cluster-Based Over Sampling**: Generate synthetic samples by generating clusters of the minority class.


## Handling outliers

**Removing outliers**: Remove the outliers from the dataset.

![](attachments/Pasted%20image%2020240306000532.png)




![](attachments/Pasted%20image%2020240306000552.png)






![](attachments/Pasted%20image%2020240306000612.png)



![](attachments/Pasted%20image%2020240306000618.png)




## SageMaker Ground Truth



[Machine Learning Pipeline on AWS](Machine%20Learning%20Pipeline%20on%20AWS.md)





## Next Steps
https://rxhl.notion.site/AWS-Machine-Learning-Specialty-4c51dfb5ae1e476284e215305b7a5d5b





Exploratory Data Analysis

# MLOps Engineering


## Objective

### Initial MlOps Engineering

- Explain the benefits of MLOps.
- Compare and contrast DevOps and MLOps.
- Set up experimentation environments for MLOps with Amazon SageMaker.
- Evaluate the security and governance requirements for a machine learning (ML) use case. 
   
### Repeatable MLOps
   
- Explain best practices for MLOps.

- Describe three options for Creating a CI/CD Pipeline for ML.


### Reliable MLOps


- Recommend best practices for monitoring and troubleshooting ML models in production.

- Demonstrate how to use Amazon SageMaker Model Monitor to monitor ML models in production.

- Demonstrate how to automate an ML solution




## Considerations and Challenges Operations ML

> MLOps—a combination of Machine Learning and Operations—combines people, technology, and processes to deliver collaborative ML solutions. 


> This are more important than the technical aspects of MLOps.

![](attachments/Pasted%20image%2020240311140746.png)

### Why does ML workloads not make it to production?

It might be because a lot of the focus when designing and implementing an ML workload is on the function of the model. The focus is on getting the model to work in a development environment. There is often less focus on operating the model and making it accessible and reliable for users after it is trained. 



## ML Lifecycle



![](attachments/Pasted%20image%2020240311145358.png)


For more Explanation Refer back to 

[Machine Learning Pipeline on AWS](Machine%20Learning%20Pipeline%20on%20AWS.md)


### Consideration for MLOps


1. **Consisitency**:  use the same tools and processes for training, deploying, and monitoring models.

1. **Reproducibility**:  ensure that the results of a model can be reproduced.

1. **Scalability**:  scale the model to handle the expected load.

1. **Auditability**:  track the lineage of the model and the data used to train it.


### Why ML governance


![](attachments/Pasted%20image%2020240311154301.png)


### Sage Maker 

![](attachments/Pasted%20image%2020240311161655.png)

> Important to know  BYOC (Bring Your Own Container) and BYOM (Bring Your Own Model) are two important concepts in Amazon SageMaker. BYOC allows you to use your own Docker container to train and deploy models in Amazon SageMaker. BYOM allows you to use your own machine learning model to train and deploy models in Amazon SageMaker.


More Details

[Parctical Data Science](Parctical%20Data%20Science.md)



![](attachments/Pasted%20image%2020240311163037.png)

## Reaptable MLOps Orchestration in AWS

1. **Data Ingestion and Preparation**:
   - **AWS Services**: Amazon S3, AWS Glue, Amazon Athena, AWS Lake Formation
   - Reaptable leverages AWS services to ingest and prepare data from various sources, such as databases, data lakes, and streaming data.
   - AWS Glue is used for data extraction, transformation, and loading (ETL) processes, while Amazon Athena enables querying and analyzing data stored in Amazon S3.
   - AWS Lake Formation simplifies data lake management and governance.

2. **Model Training**:
   - **AWS Services**: Amazon SageMaker, AWS Batch, AWS Lambda
   - Amazon SageMaker is the primary service for building, training, and deploying ML models.
   - AWS Batch is used for running batch training jobs, while AWS Lambda enables serverless model training for specific use cases.
   - Reaptable integrates with SageMaker to manage the training process, including data preprocessing, hyperparameter tuning, and distributed training.

3. **Model Deployment and Serving**:
   - **AWS Services**: Amazon SageMaker, AWS Lambda, Amazon ECS, Amazon EKS
   - Trained models are deployed using Amazon SageMaker or containerized services like AWS Lambda, Amazon ECS, or Amazon EKS, depending on the use case and scalability requirements.
   - Reaptable simplifies the deployment process, including model packaging, scaling, and monitoring.

4. **Model Monitoring and Management**:
   - **AWS Services**: Amazon SageMaker Model Monitor, AWS CloudWatch, AWS CloudTrail
   - Amazon SageMaker Model Monitor is used for monitoring model performance, data drift, and other quality metrics.
   - AWS CloudWatch and CloudTrail provide logging, monitoring, and auditing capabilities for the entire MLOps pipeline.
   - Reaptable integrates with these services to enable continuous monitoring, model retraining, and version control.

5. **Orchestration and Automation**:
   - **AWS Services**: AWS Step Functions, AWS Lambda, Amazon EventBridge
   - AWS Step Functions is a key service for orchestrating the ML workflow, including data processing, model training, deployment, and monitoring.
   - AWS Lambda is used for running serverless functions as part of the workflow.
   - Amazon EventBridge enables event-driven automation and integration with other AWS services.
   - Reaptable provides a user-friendly interface and abstractions for defining and managing ML workflows using these services.

6. **Infrastructure as Code (IaC)**:
   - **AWS Services**: AWS CloudFormation, AWS Cloud Development Kit (CDK)
   - Reaptable leverages IaC principles to provision and manage the required AWS resources for the MLOps pipeline.
   - AWS CloudFormation and AWS CDK are used to define and deploy the infrastructure stack, enabling reproducibility and scalability.

7. **Security and Governance**:
   - **AWS Services**: AWS Identity and Access Management (IAM), AWS Key Management Service (KMS), AWS Security Hub, AWS Config
   - Reaptable integrates with AWS security and governance services to ensure proper access control, data encryption, continuous compliance monitoring, and configuration management.
   - AWS IAM is used for role-based access control, while AWS KMS provides key management and data encryption capabilities.

Reaptable MLOps Orchestration in AWS focuses on leveraging a wide range of AWS services to streamline and automate the entire ML lifecycle, from data ingestion to model deployment and monitoring. It provides a unified platform for managing ML workflows, ensuring scalability, reproducibility, and governance across the ML pipeline.



## Module 5: Reliable MLOps, Scaling, and Testing 



1. **Testing and Monitoring**:
   - Implement comprehensive testing strategies, including unit tests, integration tests, and end-to-end tests.
   - Set up monitoring and logging systems to track the performance, accuracy, and reliability of your ML models in production.
   - Leverage AWS services like Amazon CloudWatch and AWS Lambda for monitoring and automated testing.

2. **Scaling and Performance**:
   - Understand the scalability and performance requirements of your ML workloads.
   - Leverage AWS services like Amazon SageMaker, AWS Batch, and AWS Elastic Kubernetes Service (EKS) for scalable training and inference.
   - Optimize model serving through techniques like batching, caching, and auto-scaling.

3. **Continuous Integration and Deployment (CI/CD)**:
   - Implement CI/CD pipelines for automating the build, test, and deployment processes of your ML systems.
   - Use AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy for seamless integration and deployment.
   - Automate model retraining, evaluation, and deployment to production environments.

4. **Reproducibility and Versioning**:
   - Maintain reproducibility by versioning data, code, and model artifacts using tools like Git and AWS CodeCommit.
   - Leverage AWS CodeArtifact or Amazon Elastic Container Registry (ECR) for storing and managing artifacts.
   - Track model lineage and experiment metadata using tools like MLflow or Amazon SageMaker Model Registry.

5. **Infrastructure as Code (IaC)**:
   - Adopt IaC principles to manage and provision AWS resources consistently and reliably.
   - Use AWS CloudFormation or Terraform to define and manage your ML infrastructure as code.

Suggested Project Structure:

```
my-ml-project/
├── data/
│   ├── raw/
│   ├── processed/
│   └── ...
├── models/
│   ├── model_definitions/
│   ├── model_artifacts/
│   └── ...
├── tests/
│   ├── unit/
│   ├── integration/
│   └── ...
├── src/
│   ├── data/
│   ├── models/
│   ├── utils/
│   └── ...
├── pipelines/
│   ├── train.py
│   ├── deploy.py
│   └── ...
├── infrastructure/
│   ├── cloudformation/
│   ├── terraform/
│   └── ...
├── requirements.txt
├── README.md
└── ...
```

This structure separates concerns and promotes modularity, making it easier to scale and maintain your ML project. The `data/` directory contains raw and processed data, `models/` holds model definitions and artifacts, `tests/` includes unit and integration tests, `src/` contains the source code, `pipelines/` manages training and deployment scripts, and `infrastructure/` defines the infrastructure as code.

Remember, this is a general structure, and you may need to adapt it based on your specific project requirements and best practices.


### AWS Reliable MLOps Monitoring 

1. **Model Drift Detection**: One of the key aspects of AWS Reliable MLOps Monitoring is detecting model drift. Model drift occurs when the statistical properties of the input data or the relationship between the input data and target variable change over time, leading to a degradation in the model's performance. AWS Reliable MLOps Monitoring helps identify these drifts by continuously monitoring the input data, model predictions, and other relevant metrics.
![](attachments/Pasted%20image%2020240311170731.png)
2. **Data Quality Monitoring**: Another crucial aspect is monitoring the quality of the input data used for making predictions. AWS Reliable MLOps Monitoring helps identify issues such as data drift (changes in the distribution of input data), missing values, outliers, and other data quality problems that can negatively impact the model's performance.

3. **Performance Monitoring**: AWS Reliable MLOps Monitoring also involves monitoring the performance of the deployed ML models in production. This includes tracking metrics such as accuracy, precision, recall, and other relevant performance indicators to ensure that the models are meeting the desired performance levels.

4. **Automated Retraining and Deployment**: In cases where model drift or performance degradation is detected, AWS Reliable MLOps Monitoring can trigger automated retraining and deployment processes. This ensures that the ML models are updated and optimized to maintain their performance and accuracy over time.

5. **Explainability and Bias Monitoring**: AWS Reliable MLOps Monitoring also focuses on monitoring the explainability and fairness of ML models. It helps identify potential biases and ensures that the models are making decisions in a transparent and unbiased manner.

6. **Operational Monitoring**: In addition to monitoring the ML models themselves, AWS Reliable MLOps Monitoring also includes operational monitoring of the underlying infrastructure, such as computing resources, storage, and networking components, to ensure the smooth and reliable operation of the ML systems.

The importance of AWS Reliable MLOps Monitoring lies in its ability to maintain the reliability, accuracy, and performance of ML models in production environments. Without proper monitoring and management, ML models can quickly become outdated or biased, leading to suboptimal or even harmful decision-making. By incorporating AWS Reliable MLOps Monitoring into the MLOps workflow, organizations can ensure that their ML systems remain reliable, efficient, and trustworthy over time.


## Summary  Achieving MLOps Maturity with AWS: A Guide to Streamlining Machine Learning Operations

Machine Learning Operations (MLOps) is an emerging discipline that aims to bridge the gap between data science and software engineering, enabling organizations to efficiently deploy, manage, and maintain machine learning (ML) models in production environments. In this blog post, we'll explore the key concepts and best practices for achieving MLOps maturity using Amazon Web Services (AWS) and its powerful tools, such as Amazon SageMaker.

The Journey to MLOps Maturity
AWS outlines four levels of MLOps maturity: initial, repeatable, reliable, and scalable. Each level represents a progressive step towards a more streamlined and efficient ML workflow.

1. Initial MLOps: At this stage, data science and IT teams work in silos, with limited collaboration and cross-training. Data processing, model training, and deployment are primarily manual processes.

2. Repeatable MLOps: This phase marks the beginning of true MLOps adoption. Teams start collaborating with shared goals, automating data pipelines, and establishing defined paths for model experimentation and deployment.

3. Reliable MLOps: Cross-functional teams emerge, with improved data governance, integrated ML pipelines, and automated model validation. Continuous Integration and Continuous Delivery (CI/CD) pipelines are implemented, along with monitoring and logging for models, workloads, and pipelines.

4. Scalable MLOps: At the pinnacle of MLOps maturity, organizations establish well-defined processes, share best practices, and leverage templatization to speed up new solution development. Customized SageMaker project templates enforce organizational standards and automate the creation of secure MLOps environments.

Building MLOps into Your ML Solutions
To raise your organization's MLOps maturity level, AWS recommends the following approaches:

1. Start small and iterate: Gradually introduce MLOps practices into your existing workflows, rather than attempting a complete overhaul.

2. Embrace "everything-as-code": Adopt an infrastructure-as-code approach to ensure repeatability, consistency, and governance across your ML pipelines.

3. Leverage ML-specific inputs, outputs, and artifacts: Maintain lineage for models, data, and pipelines to enable traceability and reproducibility.

4. Integrate SageMaker features and services: Utilize SageMaker's built-in capabilities, such as the Model Registry and Feature Store, to streamline model management and feature engineering.

5. Implement a Model Registry: Centralize model management, versioning, and deployment within your ML pipelines.

6. Explore the Feature Store: Store, share, and manage ML model features for training and inference, improving collaboration and reusability.

By following these guidelines and leveraging AWS's powerful MLOps tools, organizations can achieve a higher level of maturity, streamlining their ML operations and maximizing the value derived from their data and models.

MLOPS Engineering

# Machine Learning Pipeline on AWS



## Course Objectives

 - Select and justify the appropriate ML approach to a problem
 - Build ,train ,evaluate and deploy and fine-tune  and ML model
 - Apply the steps of the Ml pipeline to solve the problem
 - Describe Best practices for training and deploying ML models pipeline in AWS
 - Identify the steps to apply the ML to  business problems

##  Course Outline

![](attachments/Pasted%20image%2020240211104659.png)

 

## Module 1: Introduction to Machine Learning and Ml Pipeline

Data + Algorithm  + Time = Model

Inferencing: Ask a question to the model

### Business Need 

> Your site needs to be able to predict the likelihood of a customer purchasing a product based on their browsing history.



![](attachments/Pasted%20image%2020240211114402.png)


### Classical programming vs Machine Learning

- Classical programming: 
    - Input + Rules = Output
    - Rules are defined by the programmer
    - Rules are fixed
    

- Machine Learning:
    - Input + Output = Rules
    - Rules are learned from the data
    - Rules are flexible

### Feature and Weight
![](attachments/Pasted%20image%2020240211114531.png)

- Feature: A characteristic of the data
- Weight: A measure of the importance of a feature


### Supervised Learning vs Unsupervised Learning vs Reinforcement Learning

- Supervised Learning:
    - Input + Output = Model
    - The model is trained on labeled data
    - The model is used to make predictions on new data

#### Amazon SageMaker labeling jobs 

 Amazon SageMaker offer labeling jobs to help you label your data. You can use the labeling jobs to create labels for your data. You can use the labels to train your machine learning models. You can use the labels to evaluate your machine learning models. You can use the labels to make predictions with.


![](attachments/Pasted%20image%2020240211125856.png)

#### Types of Supervised Learning
 - **Binary Classification**: Predicting one of two classes
    - For example, predicting whether an email is spam or not spam
 - **Multiclass Classification**: Predicting one of many classes
    - For example, predicting the genre of a movie
- **Regression**: Predicting a continuous value
    - For example, predicting the price of a house



### UnSupervised

- Unsupervised Learning:
    - Input = Model
    - The model is trained on unlabeled data
    - The model is used to find patterns in the data

![](attachments/Pasted%20image%2020240211130153.png)


### Reinforcement Learning

- Reinforcement Learning:
    - Input + Output = Model
    - The model is trained on feedback from the environment
    - The model is used to make decisions in the environment

![](attachments/Pasted%20image%2020240211130517.png)


### Deep Learning

- Deep Learning:
    - A type of machine learning
    - Uses neural networks
    - Can learn from unstructured data
    ![](attachments/Pasted%20image%2020240211130722.png)

    > The figure demonstate going from  AI to ML to Deep Learning


![](attachments/Pasted%20image%2020240211131755.png)

#### Deep Learning recap


- Deep learning models 
    - Learn using Artificial Neural Networks (ANN) 
    - Can be trained on raw features 
- Include: 
    -    A subset of ML methods 
    -   Neurons arranged for computational efficiency 
    - Training networks with hundreds of layers that learn and improve the features

![](attachments/Pasted%20image%2020240211132602.png)


### Machine Learning use cases

- **Recommendation Systems**: Predicting what a user will like
- **Fraud Detection**: Predicting whether a transaction is fraudulent
- **Voice Recognition**: Predicting what a user is saying.
- **Image Recognition**: Predicting what is in an image


### The Pipeline of ML Problem for buy kindle 

#### Bussines problem and Problem Formulation
![](attachments/Pasted%20image%2020240211133202.png)

#### Data collection and integration
![](attachments/Pasted%20image%2020240211133220.png)

#### Data preprocessing and visulization
![](attachments/Pasted%20image%2020240211133245.png)


#### Data prepprocessing and visualization
![](attachments/Pasted%20image%2020240211133434.png)


#### Model Training 

![](attachments/Pasted%20image%2020240211133549.png)


#### Model tuning and feature enginnering

![](attachments/Pasted%20image%2020240211133638.png)


#### Model deployment

![](attachments/Pasted%20image%2020240211133720.png)


## Module 2: AWS SageMaker

Amazon SageMaker is a fully-managed service used to build, train and deploy ML models at any scale. You can use part or all of Amazon SageMaker features.



###  Instances Types
![](attachments/Pasted%20image%2020240211143654.png)

#### Algorithm options

![](attachments/Pasted%20image%2020240211143757.png)


#### SageMaker Deploytme

![](attachments/Pasted%20image%2020240211144426.png)




## Module 3 : Problem Formulation

### Problem Formulation

![](attachments/Pasted%20image%2020240212095115.png)

![](attachments/Pasted%20image%2020240212100001.png)

#### Define Success 

- **Model Performance metrics**: 
    - **Example**:  The model should have an accuracy of 90% in predicting whether a customer will buy a Kindle
        - Accuracy
        - Precision
        - Recall
        - F1 score
        - AUC

- **Business Metrics**:
    - **Example**: The model should increase the number of Kindles sold by 10%
        - Revenue
        - Customer satisfaction
        - Customer retention
        - Customer acquisition
        
## Module 4: Data Preoprocessing 
        
![](attachments/Pasted%20image%2020240212101253.png)

![](attachments/Pasted%20image%2020240212101357.png)

### Python libraries for data preprocessing

- **Pandas**: Data manipulation and analysis which provides data structures and functions to manipulate numerical tables and time series data.
 ```python
import pandas as pd
```


- **Numpy**: A library for the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.
```python
import numpy as np
```

- **Matplotlib**: A plotting library for the Python programming language and its numerical mathematics extension NumPy.
```python
import matplotlib.pyplot as plt
```

- **Seaborn**: A Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.
```python
import seaborn as sns
```

### Statistics for data preprocessing

- **Mean**: The average of the numbers
- **Median**: The middle number
- **Mode**: The number that appears most often
- **Standard Deviation**: A measure of the amount of variation or dispersion of a set of values
- **Variance**: A measure of the amount of variation or dispersion of a set of values
- **Correlation**: A measure of the strength and direction of the relationship between two variables
- **Covariance**: A measure of the relationship between two random variables

###  Dealing with outliers

- **Outliers**: An observation that lies an abnormal distance from other values in a random sample from a population
- **Aritificial Outliers**: Outliers that are created by errors in data collection or entry
- **Natural Outliers**: Outliers that are part of the population
    - ** Tranformation**: Logarithmic, square root, cube root, reciprocal, square, cube, exponential, and power transformations
    - ** impute a value**: Replace the outlier with a value that is within the range of the data


- **Methods to detect outliers**:
    - **Boxplot**
    - **Z-score**
    - **IQR score**
    - **Scatter plot**
    - **Histogram**
    - **DBSCAN**
    - **Isolation Forest**
    - **Local Outlier Factor**
    - **Elliptic Envelope**
    - **One-Class SVM**

![](attachments/Pasted%20image%2020240212104430.png)

![](attachments/Pasted%20image%2020240212104453.png)


![](attachments/Pasted%20image%2020240212104529.png)


![](attachments/Pasted%20image%2020240212104911.png)


![](attachments/Pasted%20image%2020240212104934.png)







## Module 5: Model Training 


### Overfitting and Underfitting and Balanced 

![](attachments/Pasted%20image%2020240213095515.png)

- **Overfitting**: The model is too complex and learns the training data too well

- **Underfitting**: The model is too simple and does not learn the training data well

- **Balanced**: The model learns the training data well and generalizes to new data

### Bias and Variance
![](attachments/Pasted%20image%2020240213095643.png)

- **Bias**: The error introduced by approximating a real-world problem, which may be extremely complicated, by a much simpler model

- **Variance**: The error introduced by attempting to approximate a real-world problem by a much simpler model

### Confusion Matrix


![](attachments/Pasted%20image%2020240213100710.png)

- **True Positive (TP)**: The number of positive cases that were correctly predicted

- **True Negative (TN)**: The number of negative cases that were correctly predicted

- **False Positive (FP)**: The number of negative cases that were incorrectly predicted as positive

- **False Negative (FN)**: The number of positive cases that were incorrectly predicted as negative

### Precision, Recall, F1 Score, and AUC

- **Precision**: The number of true positive cases divided by the number of true positive and false positive cases `(TP / (TP + FP))`


![](attachments/Pasted%20image%2020240213101820.png)



- **Accuracy**: The number of true positive and true negative cases divided by the total number of cases `(TP + TN) / (TP + TN + FP + FN)`

![](attachments/Pasted%20image%2020240213101739.png)

- **Recall**: The number of true positive cases divided by the number of true positive and false negative cases `(TP / (TP + FN))`
![](attachments/Pasted%20image%2020240213101852.png)
- **F1 Score**: The harmonic mean of precision and recall `2 * (precision * recall) / (precision + recall)`


- **AUC**: The area under the receiver operating characteristic curve, which is a plot of the true positive rate against the false positive rate

- **ROC Curve**: A plot of the true positive rate against the false positive rate


- *AUC-ROC*  curve is a performance measurement for classification problem at various threshold settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes. Higher the AUC, better the model is at predicting 0s as 0s and 1s as 1s. By analogy, the Higher the AUC, the better the model is at distinguishing between patients with disease and no disease.

#### Types of Problems Score

![](attachments/Pasted%20image%2020240213102541.png)


##### R2

![](attachments/Pasted%20image%2020240213142302.png)


## Module 7: Model Tuning and Feature Engineering


#### Feature Engineering components

- **Feature Selection**: Selecting the most important features

- **Feature Extraction**: Extracting new features from existing features

- **Feature Transformation**: Transforming features to improve model performance

- **Feature Scaling**: Scaling features to improve model performance


![](attachments/Pasted%20image%2020240213143051.png)

### Techniques

![](attachments/Pasted%20image%2020240213143317.png)

###  Why we choose feature engineering

- **Improve model performance**: Feature engineering can improve the performance of a model

- **Reduce overfitting**: Feature engineering can reduce overfitting by removing irrelevant features

- **Reduce underfitting**: Feature engineering can reduce underfitting by adding relevant features

- **Improve interpretability**: Feature engineering can improve the interpretability of a model by creating features that are easier to understand

### Why change numerical to small number

- **Reduce computational cost**: Smaller numbers require less memory and less computation

- ** Make the model more aware of the scale of the data**: The model will be more aware of the scale of the data and will be able to learn more effectively

> **Note** : so the model think that the small number is more important than the large number

### why change numerical to categorical

- make more sense to the model  example:
  Age: 0-18, 19-30, 31-50, 51-70, 71-100 
This will make more sense to the model than the original numerical data


### Scaling transformation techniques


- **Min-Max Scaling**: Scales the data to a fixed range, usually 0 to 1
- **Standardization**: Scales the data to have a mean of 0 and a standard deviation of 1
- **Max Abs Scaling**: Scales the data to a fixed range, usually -1 to 1 or 0 to 1
- **Normalization**: Scales the data to have a magnitude of 1


### one hot encoding

- **One-Hot Encoding**: A method of converting categorical data into a format that can be provided to ML algorithms to do a better job in prediction. It is a process of converting categorical data into a binary matrix format.


![](attachments/Pasted%20image%2020240217190503.png)

sometimes it's not efficient to do this  so you could try 

  - **Group By somthing Similar
  


### Hyperparameter Tuning   


- **Hyperparameters**: Parameters that are set before the model is 

![](attachments/Pasted%20image%2020240217193855.png)


#### Hyperparameter Tuning seearch  methods

- **Grid Search**: A method of hyperparameter tuning that searches through a specified subset of hyperparameters

- **Random Search**: A method of hyperparameter tuning that searches through a random subset of hyperparameters

- **Bayesian Optimization**: A method of hyperparameter tuning that uses a probabilistic model to search through a subset of hyperparameters


![](attachments/Pasted%20image%2020240217194715.png)


![](attachments/Pasted%20image%2020240217194820.png)


## Model 8  Model Deployment

![](attachments/Pasted%20image%2020240217195806.png)

#### Types of inferences 

![](attachments/Pasted%20image%2020240217200753.png)


#### Inferecing vs. Training
![](attachments/Pasted%20image%2020240217200903.png)


1. **Blue-Green Deployment and Canary Deployment**:
    - **Blue-Green Deployment**:
        - **What is it?** Blue-green deployment is a release management strategy used in software deployment. It involves maintaining two identical production environments: one called "Blue" (the active environment) and the other "Green" (the inactive environment).
        - **How does it work?**
            1. **Preparation**: Set up a staging environment (Green) that mirrors the production environment (Blue).
            2. **Deployment**: Deploy the new version of your application to the inactive environment (Green).
            3. **Testing**: Thoroughly test the new version in the inactive environment to ensure stability.
            4. **Switch**: Once confident, switch users to the Green environment, making it the new active production environment.
        - **Goal**: Minimize downtime and reduce risk during software updates.
    - **Canary Deployment**:
        - **What is it?** Canary deployment gradually rolls out new software versions to a controlled subset of users before deploying to everyone.
        - **How does it work?**
            1. **Incremental Rollout**: Deploy the new version to a percentage of users (the "canaries").
            2. **Monitoring**: Monitor canaries' feedback and performance.
            3. **Iterate**: Address any issues that arise. If successful, proceed with a full-scale release.
        - **Goal**: Identify and fix issues early, minimizing widespread problems or outages.

2. **Amazon SageMaker**:
    - **What is it?** Amazon SageMaker is a fully managed machine learning (ML) service provided by AWS.
    - **Features**:
        - **Build, Train, and Deploy**: Data scientists and developers can build, train, and deploy ML models in a production-ready hosted environment.
        - **UI Experience**: SageMaker provides a user-friendly interface for ML workflows across multiple integrated development environments (IDEs).
        - **Managed ML Algorithms**: Efficiently run ML algorithms against large data in a distributed environment.
        - **Flexible Training Options**: Supports custom algorithms and frameworks.
        - **Secure and Scalable Deployment**: Easily deploy models from the SageMaker console.
    - **Use Cases**: SageMaker is ideal for various ML tasks, including image classification, natural language processing, and time series forecasting.
    - **Point Clouds and SageMaker**:
        - **Amazon SageMaker Ground Truth** simplifies labeling objects in 3D point clouds for ML training datasets.
        - It supports sensor fusion of camera and LiDAR data, making it useful for applications like autonomous vehicles.

#### Checkout

https://stanford-cs329s.github.io/syllabus.html


Machine Learning Pipeline on AWS



# Practical Data Science 




![](attachments/Pasted%20image%2020240218103420.png)

![](attachments/Pasted%20image%2020240218103632.png)


### Common ML use cases

 - **Classification**: Predicting a category
 - **Regression**: Predicting a number
 - **Clustering**: Finding groups in data
 - **Anomaly detection**: Finding outliers


### ML model benefits

- **identify patterns**: ML models can identify patterns in data that humans can't

- **Minimize human error**: ML models can make predictions without human intervention

- **Can become more accurate over time**: ML models can learn from new data and improve their predictions

- **variety of data**: ML models can handle a variety of data types, including images, text, and numbers


![](attachments/Pasted%20image%2020240218235532.png)


### How to build a Quality dataset Journey


![](attachments/Pasted%20image%2020240218235758.png)

Once you have a dataset, you can start building a machine learning model. The process of building a machine learning model can be broken down into the following steps:

1. Frame the problem: What are you trying to predict? What data do you have available to make predictions?

2. Prepare the data: Clean the data, and split it into training and test sets.

3. Model Development: Train a machine learning model on the training data.

4. Deploy the model: Use the model to make predictions on new data.

5. Monitor the model: Keep track of how well the model is performing, and update it as needed.


### Correlation Matrix

![](attachments/Pasted%20image%2020240219003357.png)

A correlation matrix is a good tool to quantify the linear relationship among the variables as it conveys both the strong and weak linear relationships among numerical variables. It is a k by k matrix, where k is the number of columns in your data including the target. Each cell is populated with the corresponding correlation between the corresponding column variables.


The matrix shown is for a dataset with 9 features, F0 - F9. The matrix displays the correlation between each pair of features. The closer a value is to 1 or -1, the more linear is the relationship. A value close to or equal to 0 shows no linear relationship.


The analyzed dataset may then be filtered and adjusted to improve the quality for training and evaluating a model. Such actions may be dropping some features when they are highly correlated to others or filtering the dataset so that features and target are more balanced.56 


### Data considerations for model training

- **Data size and quality**: The more data you have, the better your model will be. However, more data also means more time and resources to process it. You should also consider the quality of your data. If your data is noisy or has missing values, your model will be less accurate.


-  **Representativeness**: Your data should be representative of the problem you are trying to solve. For example, if you are trying to predict the price of a house, your data should include information about the size, location, and condition of the house.



### Dission Tree

![](attachments/Pasted%20image%2020240219012152.png)


### XGBoost 

- XGBoost is a popular machine learning library that is used for building decision trees. It is known for its speed and accuracy, and is often used in machine learning competitions.


![](attachments/Pasted%20image%2020240219013014.png)


#### XGBoost Hyperparameters

- **alpha**: L1 regularization term on weights. Increasing this value will make model more conservative.


- **eta**: Step size shrinkage used in update to prevents overfitting. After each boosting step, we can directly get the weights of new features, and eta shrinks the feature weights to make the boosting process more conservative.#### XGBoost Hyperparameters

- **min_child_weight**: Minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the sum of instance weight less than min_child_weight, then the building process will give up further partitioning. In linear regression task, this simply corresponds to minimum number of instances needed to be in each node. The larger, the more conservative the algorithm will be.

- **num_round**: The number of rounds for boosting. This is the number of boosting iterations. The larger, the more accurate the model will be, but also the longer it will take to train.


- **max_depth**: Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit.


![](attachments/Pasted%20image%2020240219014149.png)


### Amazon Sage Maker inference options


- **Real-time inference**: Use a model to make predictions on new data in real time. This is useful when you need to make predictions quickly, such as in a web application.


- **Batch inference**: Use a model to make predictions on a large batch of data. This is useful when you need to make predictions on a large dataset, such as a log file or a database.


- **Asynchronous inference**: Use a model to make predictions on new data, and then store the predictions in a database. This is useful when you need to make predictions on new data, but don't need the predictions immediately.


- **Batch  inference**: Use a model to make predictions on a large batch of data. This is useful when you need to make predictions on a large dataset, such as a log file or a database.


![](attachments/Pasted%20image%2020240219014538.png)



### Choosing model deployment options

![](attachments/Pasted%20image%2020240219014638.png)


### ML lifecycle

![](attachments/Pasted%20image%2020240219014726.png)

	
### Methodology of MLOps

![](attachments/Pasted%20image%2020240219014825.png)




### Workflow and associated roles 

![](attachments/Pasted%20image%2020240219014837.png)






### SageMaker Studio for ML Lifecycle

SageMaker Studio is a web-based IDE for building, training, debugging, deploying, and monitoring ML models. It supports the full ML lifecycle with a unified visual interface.

## Features

- Write and run code in Jupyter notebooks
- Build and train ML models
- Deploy models and monitor performance
- Track and debug ML experiments

## Architecture

SageMaker Studio notebooks have a two-layered architecture:

1. **JupyterApp**: Provides the JupyterLab IDE user experience
2. **Kernel App**: Runs notebook code cells, defines image and compute instance type

This architecture uses a gateway application pattern to connect JupyterApp and Kernel App.

## Key Benefits

- **Fast**: Launch instances in under 2 minutes
- **Convenient**: Amazon SageMaker Python SDK pre-installed, multiple kernels available
- **Comprehensive**: Build, train, debug, track, monitor, and view experiments
- **Sharable**: Read-only copy includes dependencies
- **Scalable**: Fully elastic compute resources
- **Persistent**: View and share notebooks even when instances or kernels are stopped
- **Scheduled**: Run notebooks on demand or on a fixed/custom schedule
- **Flexible**: Select different kernel images and instance types

### SageMaker Data Wrangler

SageMaker Data Wrangler is a tool that helps to complete each of the key ML data preparation workflow tasks. It includes data loading, cleaning, exploring, and visualization with little to no code. 

## Key capabilities of SageMaker Data Wrangler include the following:

- Select data from multiple data sources and semi-structured data formats.
- Automatically verify data quality and detect abnormalities.
- Understand data with visualization templates.
- Transform data with 300+ built-in transformations plus the ability to author custom transformations.
- Quickly estimate ML model accuracy and diagnose issues before models are deployed into production.
- Automate ML data preparation workflows.# SageMaker Data Wrangler

SageMaker Data Wrangler is a tool that helps to complete each of the key ML data preparation workflow tasks. It includes data loading, cleaning, exploring, and visualization with little to no code. 

## Key capabilities of SageMaker Data Wrangler include the following:

- Select data from multiple data sources and semi-structured data formats.
- Automatically verify data quality and detect abnormalities.
- Understand data with visualization templates.
- Transform data with 300+ built-in transformations plus the ability to author custom transformations.
- Quickly estimate ML model accuracy and diagnose issues before models are deployed into production.
- Automate ML data preparation workflows.# SageMaker Data Wrangler

SageMaker Data Wrangler is a tool that helps to complete each of the key ML data preparation workflow tasks. It includes data loading, cleaning, exploring, and visualization with little to no code. 

## Key capabilities of SageMaker Data Wrangler include the following:

- Select data from multiple data sources and semi-structured data formats.
- Automatically verify data quality and detect abnormalities.
- Understand data with visualization templates.
- Transform data with 300+ built-in transformations plus the ability to author custom transformations.
- Quickly estimate ML model accuracy and diagnose issues before models are deployed into production.
- Automate ML data preparation workflows.


### Regression Problem metrics


![](attachments/Pasted%20image%2020240313164312.png)



1. Mean Squared Error (MSE):
   - MSE measures the average squared difference between the predicted values and the actual values.
   - It provides an understanding of the magnitude of the errors made by the model.
   - Lower MSE values indicate better model performance.
   - MSE is sensitive to outliers, as squaring the errors magnifies the effect of large errors.

2. R-squared (R²):
   - R² measures the proportion of the variance in the target variable that is explained by the independent variables in the model.
   - It ranges from 0 to 1, with a value closer to 1 indicating a better fit of the model to the data.
   - R² provides an intuitive interpretation of the model's goodness-of-fit.
   - However, R² tends to increase as more variables are added to the model, even if those variables are not significant, leading to potential overfitting.

3. Adjusted R-squared (Adj. R²):
   - Adjusted R² is a modified version of R² that accounts for the number of independent variables in the model.
   - It aims to penalize the addition of insignificant variables, reducing the risk of overfitting.
   - Adjusted R² only increases if the new variable improves the model's performance more than would be expected by chance.

When evaluating a regression model, it is recommended to consider both MSE and R² (or Adjusted R²) to get a comprehensive understanding of the model's performance and potential overfitting issues.

Python Code:

```python
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.linear_model import LinearRegression
import numpy as np

# Example data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([2, 4, 5, 4, 5])

# Create and fit a linear regression model
model = LinearRegression()
model.fit(X, y)

# Make predictions
y_pred = model.predict(X)

# Calculate mean squared error
mse = mean_squared_error(y, y_pred)
print("Mean Squared Error:", mse)

# Calculate R-squared
r2 = r2_score(y, y_pred)
print("R-squared:", r2)

# Calculate Adjusted R-squared (assuming single feature)
n = X.shape[0]  # Number of observations
p = 1  # Number of features (assuming single feature)
adj_r2 = 1 - (1 - r2) * ((n - 1) / (n - p - 1))
print("Adjusted R-squared:", adj_r2)
```

In this example, we generate some example data, create and fit a linear regression model, and then calculate the mean squared error, R-squared, and Adjusted R-squared using functions from the `sklearn.metrics` module.

Note that the Adjusted R-squared calculation assumes a single feature (independent variable). If you have multiple features, you would need to adjust the formula accordingly.







### More about sage Maker Built in Algorthim

[Amazon Sage Maker](Amazon%20Sage%20Maker.md)

Home

Notes

Quiz

⌘k