Basic Data Processing & Analysis using Python Libraries

NumPy Architecture, Memory Layout & Array Attributes

Standard Python lists store elements as arrays of pointers referencing independent PyObject entities scattered across the heap. For large-scale numerical computation, this causes severe memory overhead and frequent CPU Cache Misses. NumPy (Numerical Python) solves this by introducing the C-based ndarray (N-dimensional array), which allocates a single, contiguous block of homogeneous memory addresses.

1. Python List vs NumPy Array (Architectural Comparison)

• Contiguous Memory Buffer: All elements in an ndarray reside sequentially in raw memory. The CPU pre-fetches contiguous bytes into high-speed L1/L2 caches, enabling SIMD vectorization.
• Homogeneous Data Types: Every element in an ndarray shares the exact same numeric dtype (e.g., int32, float64), eliminating runtime type checking overhead.
• Vectorization & ufuncs: Element-wise operations execute compiled C loops under the hood without Python interpreter loop overhead.

PYTHON BUILT-IN LIST [10, 20, 30]:
Array of Pointers: [ 0x104A ] ----> [PyObject: Type=int, Value=10, RefCount=1] (28 Bytes)
                   [ 0x89BC ] ----> [PyObject: Type=int, Value=20, RefCount=1] (28 Bytes)
                   [ 0x33DF ] ----> [PyObject: Type=int, Value=30, RefCount=1] (28 Bytes)
===> Memory Address fragmented all over Heap! Massive pointer dereferencing latency!

NUMPY NDARRAY (np.array([10, 20, 30], dtype=np.int32)):
Contiguous Raw Memory Buffer (C-Order):
+---------------+---------------+---------------+
| 00000000 000A | 00000000 0014 | 00000000 001E |  (12 Bytes TOTAL! 4 bytes per int)
+---------------+---------------+---------------+
[Byte Offset 0]  [Byte Offset 4]  [Byte Offset 8]
Metadata Header: shape=(3,), strides=(4,), dtype=int32, data_ptr=0x1000

2. Essential ndarray Attributes (High-Yield for Exam)

An ndarray is an object wrapper over raw memory with the following key attributes:

import numpy as np

# Create a 2x3 2D array of 32-bit integers
arr = np.array([[10, 20, 30], [40, 50, 60]], dtype=np.int32)

print("1. ndim (Dimensions):", arr.ndim)       # 2 (2D Array)
print("2. shape (Rows, Cols):", arr.shape)      # (2, 3)
print("3. size (Total Items):", arr.size)       # 6 elements
print("4. dtype (Data Type):", arr.dtype)       # int32
print("5. itemsize (Bytes/item):", arr.itemsize) # 4 bytes
print("6. nbytes (Total Bytes):", arr.nbytes)   # 6 * 4 = 24 bytes
print("7. strides (Byte steps):", arr.strides)  # (12, 4) -> 12 bytes to next row, 4 to next col

Array Creation, Numerical Ranges & File I/O

NumPy provides several optimized built-in constructors to instantiate arrays without manual loops:

Numerical Ranges: `arange` vs `linspace` vs `logspace`

High-Yield TU Exam Question: "Differentiate between np.arange() and np.linspace() with syntax and output."

# 1. np.arange(start, stop, step):
# Generates evenly spaced values within half-open interval [start, stop)
# Stop value is strictly EXCLUDED!
r1 = np.arange(0, 10, 2)
print("arange:", r1) # [0 2 4 6 8]

# 2. np.linspace(start, stop, num_samples):
# Generates N evenly spaced numbers over closed interval [start, stop]
# Stop value is strictly INCLUDED!
r2 = np.linspace(0, 1, 5)
print("linspace:", r2) # [0.   0.25 0.5  0.75 1.  ]

# 3. np.logspace(start, stop, num):
# Generates numbers spaced evenly on a log scale (10^start to 10^stop)
r3 = np.logspace(1, 3, 3) # [10^1, 10^2, 10^3]
print("logspace:", r3) # [  10.  100. 1000.]

Array Persistence & File I/O

• np.save("matrix.npy", arr): Serializes a single ndarray into fast, uncompressed binary format preserving dtype and shape metadata.
• loaded_arr = np.load("matrix.npy"): Directly restores binary arrays into RAM.
• np.savetxt("data.csv", arr, delimiter=",") & np.loadtxt("data.csv", delimiter=","): Plaintext CSV export and ingestion.

Indexing, Slicing & Boolean Masking

NumPy provides multi-dimensional slicing, integer array indexing (fancy indexing), and conditional boolean masking that operates at hardware speed.

matrix = np.array([
    [10, 20, 30, 40],
    [50, 60, 70, 80],
    [90, 100, 110, 120]
])

# 1. Multi-dimensional Slicing: [row_start:row_end, col_start:col_end]
sub_matrix = matrix[0:2, 1:3]
print("Rows 0-1, Cols 1-2:\n", sub_matrix)
# Output:
# [[20 30]
#  [60 70]]

# 2. Extract entire column or row
col_2 = matrix[:, 2] # 3rd column across all rows: [30, 70, 110]

# 3. Boolean Masking (Vectorized conditional filtering - High Yield)
mask = matrix > 65
filtered_values = matrix[mask]
print("Values > 65:", filtered_values) # [70 80 90 100 110 120]

# 4. In-place Conditional Replacement
matrix[matrix < 50] = 0 # Sets all numbers below 50 to 0

Broadcasting Rules & Vectorized Operations

Broadcasting describes how NumPy treats arrays with different shapes during arithmetic operations (addition, multiplication). Without making unneeded memory copies, NumPy virtually stretches the smaller array across the larger array so that they have compatible shapes.

The Formal Broadcasting Rules (TU Exam Standard)

When operating on two arrays, NumPy compares their shapes element-wise starting from the trailing dimensions (Right to Left). Two dimensions are compatible when:

1. Their numeric lengths are strictly equal, OR
2. One of the dimension lengths is exactly 1 (which is virtually stretched to match the other).

If neither condition is met, NumPy triggers a fatal ValueError: operands could not be broadcast together.

EXAMPLE 1: Matrix (3, 3) + 1D Array (3,)
Array A Shape:  (3, 3)  <--- 3 Rows, 3 Columns
Array B Shape:     (3,)  <--- 1D Array of length 3

STEP 1: Prepend 1 to smaller shape: Array B becomes (1, 3)
STEP 2: Compare from Right to Left:
        Dim 1 (Trailing): A=3, B=3 (Match! Equal dimensions)
        Dim 0 (Leading):  A=3, B=1 (Match! Dimension is 1, stretched to 3)

VIRTUAL EXPANSION IN MEMORY:
+----+----+----+       +----+----+----+       +----+----+----+
| 10 | 20 | 30 |       |  1 |  2 |  3 |       | 11 | 22 | 33 |
+----+----+----+       +----+----+----+       +----+----+----+
| 40 | 50 | 60 |   +   |  1 |  2 |  3 |   =   | 41 | 52 | 63 |  (Result: 3x3)
+----+----+----+       +----+----+----+       +----+----+----+
| 70 | 80 | 90 |       |  1 |  2 |  3 |       | 71 | 82 | 93 |
+----+----+----+       +----+----+----+       +----+----+----+
  Array A (3, 3)         Virtual B (3, 3)       Output Matrix

EXAMPLE 2: INCOMPATIBLE SHAPES (Triggers ValueError):
Array A Shape: (4, 3)
Array B Shape:    (4,) ===> Right-to-Left: 3 vs 4 (Neither is 1, Not Equal!)
====> RESULT: FAILS! Cannot broadcast (4, 3) with (4,)!

A = np.array([[10, 20, 30], [40, 50, 60], [70, 80, 90]]) # shape (3, 3)
B = np.array([1, 2, 3])                                     # shape (3,)

# Broadcasting automatically expands B across all 3 rows
result = A + B
print("Broadcasting Addition:\n", result)

# Scalar Broadcasting (Multiply single integer across entire matrix)
scaled = A * 2 # 2 is broadcast virtually across all elements

Statistical Functions, Sorting & Searching

NumPy provides high-speed vector aggregations. Mastering the axis parameter is mandatory for the TU examination: axis=0 (column-wise reduction) and axis=1 (row-wise reduction).

data = np.array([
    [80, 75, 90],
    [60, 85, 70],
    [95, 90, 85]
])

# 1. Statistical Aggregations
print("Total Mean:", np.mean(data))                        # 81.11
print("Column-wise Mean (axis=0):", np.mean(data, axis=0)) # [78.33, 83.33, 81.67]
print("Row-wise Mean (axis=1):", np.mean(data, axis=1))    # [81.67, 71.67, 90.00]
print("Standard Deviation:", np.std(data))
print("Median:", np.median(data))
print("Percentile (75th):", np.percentile(data, 75))

# 2. Sorting & Index Traversal
flat_sorted = np.sort(data, axis=None) # Flattens array and sorts all items
col_sorted = np.sort(data, axis=0)     # Sorts each column independently
sorted_indices = np.argsort(data[0])   # Returns sorted indices of 1st row

# 3. Vectorized Conditional Search
# np.where(condition, x, y): returns x if condition is True, else y
modified = np.where(data >= 80, "Distinction", "Pass")

# np.count_nonzero() for fast threshold counts
count_above_85 = np.count_nonzero(data > 85)
print("Marks > 85 Count:", count_above_85) # 3

SciPy: Linalg, Integrate, Interpolate, Cluster & IO

SciPy (Scientific Python) is an advanced scientific computing library built directly on top of NumPy. While NumPy supplies raw array data structures and basic vector operations, SciPy provides production-grade mathematical algorithms for engineering and scientific analysis.

+-------------------------------------------------------------------------+
|                  SCIPY SCIENTIFIC COMPUTING ECOSYSTEM                   |
+-------------------------------------------------------------------------+
       |                  |                   |                  |
       v                  v                   v                  v
+--------------+   +--------------+   +---------------+   +--------------+
| scipy.linalg |   |scipy.integrate|   |scipy.interpolate| | scipy.cluster|
| Matrix Inv   |   | quad()       |   | interp1d()    |   | vq.kmeans()  |
| Eigenvalues  |   | dblquad()    |   | Spline fitting|   | Vector quant |
| Det, Solve Ax=b  | ODE Solvers  |   | Curve approx  |   | K-Means Clust|
+--------------+   +--------------+   +---------------+   +--------------+
       |                  |                   |                  |
       +------------------+-------------------+------------------+
                                  |
                                  v
+-------------------------------------------------------------------------+
| scipy.constants (pi, c, G, h)   |   scipy.io (loadmat, savemat, wavfile)|
+-------------------------------------------------------------------------+

Core SciPy Subpackages with Working Code

# 1. scipy.constants: Physical and Mathematical Constants
from scipy import constants
print("Speed of Light (c):", constants.c, "m/s")
print("Planck constant (h):", constants.h)
print("Standard Pi:", constants.pi)

# 2. scipy.linalg: Solve System of Equations (2x + 3y = 8, 4x + 9y = 20)
from scipy import linalg
A = np.array([[2, 3], [4, 9]])
b = np.array([8, 20])
solution = linalg.solve(A, b) # Direct LAPACK solver for Ax = b
print("Solution [x, y]:", solution) # [2., 1.33333333]
print("Determinant:", linalg.det(A))
print("Matrix Inverse:\n", linalg.inv(A))

# 3. scipy.integrate: Definite Numerical Integration (Integral of x^2 from 0 to 3)
from scipy import integrate
def integrand(x):
    return x ** 2
area, error = integrate.quad(integrand, 0, 3) # Exact analytical: x^3 / 3 = 27 / 3 = 9
print(f"Integral Result: {area} (Est. Error: {error})")

# 4. scipy.interpolate: Gap Filling & Curve Fitting
from scipy.interpolate import interp1d
x_known = np.array([0, 1, 2, 3, 4])
y_known = np.array([0, 1, 4, 9, 16])
f_interp = interp1d(x_known, y_known, kind="linear")
print("Interpolated value at x=2.5:", f_interp(2.5)) # 6.5

# 5. scipy.cluster.vq: K-Means Clustering
from scipy.cluster.vq import kmeans, vq
data_pts = np.array([[1.1, 2.3], [1.5, 1.9], [8.2, 9.1], [8.9, 8.8]], dtype=float)
centroids, _ = kmeans(data_pts, 2) # Discover 2 cluster centroids
print("Discovered Centroids:\n", centroids)

Pandas Ecosystem: Series & DataFrames

Pandas is the gold standard for handling structured, tabular, heterogeneous datasets. It provides two foundational data structures:

                        COLUMNS (axis=1: horizontal operations / column headers)
                       +-------------------+--------------------+--------------------+
                       | "name" (str)      | "semester" (int64) | "gpa" (float64)    |
+----------------------+-------------------+--------------------+--------------------+
| Index 0 (Row label)  | "Bikram Karki"    | 3                  | 3.82               |
+----------------------+-------------------+--------------------+--------------------+
| Index 1 (Row label)  | "Prajita Sen"     | 3                  | 3.95               |
+----------------------+-------------------+--------------------+--------------------+
| Index 2 (Row label)  | "Aman Sharma"     | 2                  | 3.40               |
+----------------------+-------------------+--------------------+--------------------+
INDEX / ROWS (axis=0: vertical operations / agg over rows)

Creating DataFrames from Scratch & Inspection

High-Yield TU Exam Question: "Explain four different ways to create a Pandas DataFrame from scratch with syntax."

import pandas as pd
import numpy as np

# Method 1: From Dictionary of Lists
data_dict = {
    "Roll": ["MCA-01", "MCA-02", "MCA-03"],
    "Name": ["Aayush", "Binod", "Chahana"],
    "Score": [88, 94, 76]
}
df1 = pd.DataFrame(data_dict)

# Method 2: From List of Dictionaries (Rows represented as dicts)
rows_list = [
    {"Name": "Deepak", "Age": 24, "City": "Kathmandu"},
    {"Name": "Elina", "Age": 23, "City": "Pokhara"}
]
df2 = pd.DataFrame(rows_list)

# Method 3: From 2D NumPy Array (With custom Index and Column labels)
np_data = np.array([[10, 20], [30, 40], [50, 60]])
df3 = pd.DataFrame(np_data, columns=["Test1", "Test2"], index=["S1", "S2", "S3"])

# Method 4: From Disk Files (CSV / JSON)
# df_csv = pd.read_csv("students.csv")
# df_json = pd.read_json("records.json")

# Essential Inspection Methods:
print(df1.head(2))       # First 2 rows
print(df1.tail(1))       # Last 1 row
print(df1.info())        # Memory usage, dtypes and null counts
print(df1.describe())    # Summary statistics: count, mean, std, min, max
print(df1.shape)         # Dimensions tuple: (rows, cols)

Handling Missing Data (NaN Imputation & Dropping)

Real-world datasets contain missing entries represented in Pandas as NaN (Not a Number) or None. In the TU examination, data cleaning involves 3 primary steps:

1. Detecting Missing Values: df.isna() or df.isnull(). Column null tally: df.isna().sum().
2. Dropping Missing Values: df.dropna().
   • df.dropna(how="any"): Drops entire row if any column contains NaN.
   • df.dropna(subset=["email"]): Drops row only if specific column contains NaN.
3. Imputing / Filling Missing Values: df.fillna().
   • Constant Imputation: df.fillna(0)
   • Statistical Imputation: df["marks"].fillna(df["marks"].mean())
   • Propagation: df.ffill() (forward fill) or df.bfill() (backward fill)

raw_data = {
    "Name": ["Ram", "Shyam", "Hari", "Gita"],
    "Age": [22, np.nan, 24, 21],
    "Score": [85, 90, np.nan, np.nan]
}
df = pd.DataFrame(raw_data)

# 1. Detect missing value counts per column
print("Missing counts:\n", df.isna().sum())

# 2. Mean Imputation: Impute missing numerical score with column mean
mean_score = df["Score"].mean()
df["Score"] = df["Score"].fillna(mean_score)

# 3. Drop rows where critical identifier (Age) is still missing
clean_df = df.dropna(subset=["Age"])
print("Cleaned DataFrame:\n", clean_df)

Slicing & Subsetting: `.loc` vs `.iloc`

Most Frequent TU Trap: "Differentiate between .loc and .iloc with clear examples and endpoint behavior."

students = pd.DataFrame({
    "name": ["Anil", "Bhawana", "Chirag", "Divya"],
    "gpa": [3.6, 3.9, 3.4, 3.8],
    "semester": [1, 2, 3, 4]
}, index=["R101", "R102", "R103", "R104"])

# 1. loc usage: label-based ('R101' to 'R103' BOTH inclusive!)
print(students.loc["R101":"R103", ["name", "gpa"]])

# 2. iloc usage: integer-based (0 to 2 only, index 2 excluded!)
print(students.iloc[0:2, 0:2])

# 3. Boolean Subsetting (Logical Bitwise Filtering)
high_achievers = students[(students["gpa"] >= 3.7) & (students["semester"] >= 2)]
print("GPA >= 3.7:\n", high_achievers)

Merging, Joining & Concatenating DataFrames

When datasets originate from multiple disparate sources, Pandas provides 3 relational operations:

• `pd.concat([df1, df2], axis=0)`: Stacks tables vertically (row binding) or horizontally (axis=1 column binding).
• `pd.merge(df1, df2, on='key', how='...')`: Executes relational SQL joins (inner, left, right, outer) on common key columns.
• `df1.join(df2)`: Combines DataFrames based on their row indexes.

HOW='INNER' (Default):
Intersection only! Only keys existing in BOTH df1 and df2 are retained.

HOW='LEFT':
Preserves ALL rows from Left table (df1). Unmatched columns from df2 filled with NaN.

HOW='RIGHT':
Preserves ALL rows from Right table (df2). Unmatched columns from df1 filled with NaN.

HOW='OUTER' (Full Outer Join):
Union of both! All rows from both sides preserved. Missing values filled with NaN.

# Student Profile Table
students_df = pd.DataFrame({
    "sid": [101, 102, 103, 104],
    "name": ["Kabir", "Laxmi", "Manoj", "Neeta"]
})

# Exam Marks Table
marks_df = pd.DataFrame({
    "sid": [102, 103, 105],
    "subject": ["Python", "Python", "Python"],
    "score": [92, 85, 78]
})

# 1. Inner Join: Intersection only (102, 103)
inner_merged = pd.merge(students_df, marks_df, on="sid", how="inner")

# 2. Left Join: Preserves all students; unattempted exams show NaN score
left_merged = pd.merge(students_df, marks_df, on="sid", how="left")

# 3. Full Outer Join: Union of all records from both tables
outer_merged = pd.merge(students_df, marks_df, on="sid", how="outer")

Full Data Analysis Exam Project: Student Marks Analytics Pipeline

A recurring 10-mark comprehensive question in the TU MCA examination tests candidate ability to combine NumPy vectorization with Pandas ETL workflows: "Write an end-to-end Python program using NumPy and Pandas to ingest student records, clean missing marks via statistical imputation, compute vectorized metrics, filter high achievers with boolean masks, and export clean CSV reports."

import numpy as np
import pandas as pd

def run_student_analytics():
    # 1. Raw Data Ingestion (Simulated CSV DataFrame)
    raw_records = {
        "Roll": ["MCA-01", "MCA-02", "MCA-03", "MCA-04", "MCA-05", "MCA-06"],
        "Name": ["Aayush", "Bipana", "Chirag", "Deepa", "Eshwar", "Farhan"],
        "Theory": [75.0, np.nan, 82.0, 45.0, 92.0, 68.0],
        "Practical": [22.0, 24.0, np.nan, 18.0, 25.0, 21.0],
        "Attendance": [85, 92, 78, 60, 95, 80]
    }
    df = pd.DataFrame(raw_records)
    print("=== 1. Initial Raw Dataset ===")
    print(df)

    # 2. Data Cleaning & Imputation (Missing Data Handling)
    # Fill Theory NaN with mean; fill Practical NaN with default 20.0
    df["Theory"] = df["Theory"].fillna(df["Theory"].mean())
    df["Practical"] = df["Practical"].fillna(20.0)

    # 3. Feature Engineering (Vectorized Calculation)
    df["Total_Score"] = df["Theory"] + df["Practical"]
    df["Percentage"] = (df["Total_Score"] / 125.0) * 100

    # 4. NumPy Statistical Analysis
    scores_array = df["Total_Score"].values
    mean_val = np.mean(scores_array)
    std_val = np.std(scores_array)
    median_val = np.median(scores_array)

    print(f"\n=== 2. Statistical Summary (NumPy Engine) ===")
    print(f"Mean Score: {mean_val:.2f} | Std Deviation: {std_val:.2f} | Median: {median_val:.2f}")

    # 5. Advanced Filtering (Boolean Masking)
    # Students with Distinction (>= 80%) AND Attendance >= 75%
    distinction_students = df[(df["Percentage"] >= 80.0) & (df["Attendance"] >= 75)]
    print("\n=== 3. Distinction Students (Attendance >= 75%) ===")
    print(distinction_students[["Roll", "Name", "Total_Score", "Percentage"]])

    # 6. Export Processed Report
    df.to_csv("tu_final_processed_marks.csv", index=False)
    print("\n[SUCCESS] Pipeline completed. Exported to 'tu_final_processed_marks.csv'.")

if __name__ == "__main__":
    run_student_analytics()