{"id":47338,"date":"2024-11-09T16:21:15","date_gmt":"2024-11-09T16:21:15","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=47338"},"modified":"2024-11-10T09:41:57","modified_gmt":"2024-11-10T09:41:57","slug":"jupyter-notebook-lab-session-1-exploring-dataset-with-pandas-and-numpy","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/jupyter-notebook-lab-session-1-exploring-dataset-with-pandas-and-numpy\/","title":{"rendered":"Jupyter notebook – Lab Session – 1 – Exploring Dataset with Pandas and NumPy"},"content":{"rendered":"\n

Importing Libraries<\/strong><\/p>\n\n\n\n

import pandas as pd
import numpy as np
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Import the essential libraries.<\/p>\n\n\n\n
Loading the Dataset<\/strong><\/p>\n\n\n\n
df = pd.read_csv('\/path_to_your_dataset.csv')
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Load the dataset into a Pandas DataFrame.<\/p>\n\n\n\n
Display First Few Rows<\/strong><\/p>\n\n\n\n
df.head()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Display the first five rows to understand the structure.<\/p>\n\n\n\n
Display Last Few Rows<\/strong><\/p>\n\n\n\n
df.tail()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Display the last five rows of the dataset.<\/p>\n\n\n\n
Dataset Information<\/strong><\/p>\n\n\n\n
df.info()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Get an overview, including data types and null values.<\/p>\n\n\n\n
Descriptive Statistics<\/strong><\/p>\n\n\n\n
df.describe()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Get statistics like mean, median, min, and max for each column.<\/p>\n\n\n\n
Column Names<\/strong><\/p>\n\n\n\n
df.columns
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> List all column names in the dataset.<\/p>\n\n\n\n
Shape of the Dataset<\/strong><\/p>\n\n\n\n
df.shape
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Get the number of rows and columns.<\/p>\n\n\n\n
Check for Null Values<\/strong><\/p>\n\n\n\n
df.isnull().sum()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Count null values in each column.<\/p>\n\n\n\n
Drop Rows with Null Values<\/strong><\/p>\n\n\n\n
df_cleaned = df.dropna()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Remove rows with null values for a cleaner dataset.<\/p>\n\n\n\n
Fill Null Values<\/strong><\/p>\n\n\n\n
df.fillna(value='Unknown', inplace=True)
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Fill null values with a placeholder.<\/p>\n\n\n\n
Unique Values in a Column<\/strong><\/p>\n\n\n\n
df['column_name'].unique()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Display unique values in a specific column.<\/p>\n\n\n\n
Value Counts<\/strong><\/p>\n\n\n\n
df['column_name'].value_counts()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Count the occurrences of each unique value in a column.<\/p>\n\n\n\n
Filter Rows by Condition<\/strong><\/p>\n\n\n\n
df_filtered = df[df['column_name'] > some_value]
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Filter rows based on a condition.<\/p>\n\n\n\n
Selecting Multiple Columns<\/strong><\/p>\n\n\n\n
df[['column1', 'column2']]
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Select and display specific columns.<\/p>\n\n\n\n
Add a New Column<\/strong><\/p>\n\n\n\n
df['new_column'] = df['column1'] + df['column2']
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Add a new column by combining values from other columns.<\/p>\n\n\n\n
Rename Columns<\/strong><\/p>\n\n\n\n
df.rename(columns={'old_name': 'new_name'}, inplace=True)
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Rename columns for better readability.<\/p>\n\n\n\n
Sorting Values<\/strong><\/p>\n\n\n\n
df.sort_values(by='column_name', ascending=False)
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Sort the dataset by a specific column.<\/p>\n\n\n\n
Drop a Column<\/strong><\/p>\n\n\n\n
df.drop('column_name', axis=1, inplace=True)
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Remove a specific column.<\/p>\n\n\n\n
Group By and Aggregate<\/strong><\/p>\n\n\n\n
df.groupby('column_name').sum()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Group by a column and apply an aggregate function like sum.<\/p>\n\n\n\n
Calculate Mean of a Column<\/strong><\/p>\n\n\n\n
df['column_name'].mean()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Calculate the mean of a specific column.<\/p>\n\n\n\n
Calculate Median of a Column<\/strong><\/p>\n\n\n\n
df['column_name'].median()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Calculate the median of a specific column.<\/p>\n\n\n\n
Standard Deviation of a Column<\/strong><\/p>\n\n\n\n
df['column_name'].std()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Calculate the standard deviation of a specific column.<\/p>\n\n\n\n
Detecting Outliers<\/strong><\/p>\n\n\n\n
df[(df['column_name'] > upper_limit) | (df['column_name'] < lower_limit)]
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Detect outliers by specifying upper and lower limits.<\/p>\n\n\n\n
Apply Custom Function<\/strong><\/p>\n\n\n\n
df['new_column'] = df['column_name'].apply(lambda x: x * 2)
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Apply a custom function to each value in a column.<\/p>\n\n\n\n
Pivot Table<\/strong><\/p>\n\n\n\n
df.pivot_table(values='value_column', index='index_column', columns='column_name')
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Create a pivot table to analyze relationships.<\/p>\n\n\n\n
Correlation Matrix<\/strong><\/p>\n\n\n\n
df.corr()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Calculate the correlation matrix for numeric columns.<\/p>\n\n\n\n
Visualizing with Histograms<\/strong><\/p>\n\n\n\n
df['column_name'].hist()
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Plot a histogram for a column to view the distribution.<\/p>\n\n\n\n
Scatter Plot<\/strong><\/p>\n\n\n\n
df.plot.scatter(x='column_x', y='column_y')
<\/code><\/pre>\n\n\n\n
Explanation:<\/em> Create a scatter plot to see relationships between two columns.<\/p>\n\n\n\n
Box Plot<\/strong><\/p>\n\n\n\n
df.boxplot(column='column_name')\n<\/code>\nExplanation:<\/em> Generate a box plot to identify the spread and outliers.<\/pre>\n\n\n\n
Live Example of Data set Attached<\/h2>\n\n\n\n
DOWNLOAD from HERE – CLICK HERE<\/a><\/p>\n\n\n\n