Data Pre-processing

Raw Data can be structured or unstructured. Pre-processing prepares the input data required for the analysis. This is one of the important task in the data science that should be done carefully

Dig: Major steps in Data Science

Before Pre-processing:

1. Handling zip files: explanation of steps
   

2. to create zip file using shutil

3. to unzip the input .zip file using pyunpack in Python

4. Create a Check List

5. From the previous project experience you can create a Check list or lists having list of items that you find needed to be performed on raw data before using it for the project
   

6. Items in the list can be to check data quantity, quality, gaps, etc

7. Hence for the new or existing project we can use some or all of the items of the Check list as a litmus test on the new raw data that should be passed before using it in the project
   

8. This can help to speed up pre-prcessing, processing steps

9. This can address hidden problems related to raw data if present in very early stage hence will save time and resources in later stage of development

10. Data and Statistics: (coming soon)
    

11. mean, median, mode, 1st-2nd-3rd quartile, percentiles, standard deviation, range of the data

12. Data distributions and Probability distributions: (coming soon)
    

13. frequency distribution and probability distribution

Some useful Pre-processing techniques:

Dig: Making Raw data ready for the analysis

1. Data Cleaning (Coming soon)

2. Data imputing

3. Useful String operations implementation examples. (Coming soon)

Image Processing:

Some of the useful techniques in image processing:

1. Scale, Blur , Gray image using OpenCV , PIL (coming soon)

2. Image rotation/tilting by x angle using Scipy , imutils and OpenCV functions.

3. Automation of tilt or inclination or rotation angle detection and correction of an image using Canny edges detection and Hough lines detection techniques.

4. Automatic White border or padding detection and cropping in an image

Divide features into training and testing:

1. random.seed(): What does it do? (coming soon)

2. Handle Categorical Features. (coming soon)

3. Feature selection (coming soon)

4. Divide features into training and testing. Get index information of train and test features.

Normalization & Standardization:

1. Why we normalise or scale data for most of the Machine Learning or Deep Learning Algorithms?

2. What is Normalization? Implementation in Python using sklearn. What is L1 and L2 Norms?

3. Comparison of scaling methods Normalization Vs Standardization? When to use which one?

Generating same set of data (train, test)

1. Use of Numpy random seed and random_state in train_test split function

Some useful methods, packages:

Suppress a python warnings using following line of code:

1

2

3

import warnings

warnings.filterwarnings("ignore", message="TYPE THE WARNING MESSAGE THAT YOU ARE GETTING")

Cheatsheet for data wrangling 2

Cheatsheet for data wrangling 1

Cheatsheet for data wrangling 2

Cheatsheet for data wrangling 1