Data Processing

Answer

Data cleaning is the process of identifying and correcting or removing errors and inconsistencies within a dataset so that the data is in a format appropriate for analysis.

Answer

Dirty data is data that contains many errors or has not gone through the data-cleaning process. It may include missing values, duplicates, outliers, wrong values, or inconsistencies.

Answer

Data cleaning is a crucial initial step because data is rarely ideal for analysis when first obtained. Clean data improves analysis, helps produce better decisions, and reduces the chance of incorrect…

Answer

Missing values are parts of a dataset that are incomplete. Large portions of crucial missing data can cause bias in results and weaken analysis.

Answer

df.isnull() returns a boolean DataFrame showing missing values. df.isna() is an alias for df.isnull(). df.isnull().sum() counts missing values in each column.

Answer

df.dropna() removes rows or columns with missing values. df.dropna(axis=1) removes columns with missing values. df.dropna(thresh=n) keeps rows with at least n non-NA values. df.fillna(value) fills mis…

Answer

Outliers are values that are far outside the norm and not representative of the data. They may happen because of a typo or because of exceptional circumstances. True outliers should be distinguished f…

Answer

Outliers can be identified using box plots, scatter plots, Z-score methods, or IQR methods. They can be handled by removing them, capping or flooring them, or transforming the data.