week 8 writing

Reference files/W8 - Learning Objectives on Writing Paper.md

@@ -0,0 +1,45 @@
+## **Remember (Knowledge)**
+
+Students will be able to:
+
+* Recall LaTeX syntax for document structure, figures, citations, and spacing  
+* Identify components of ML research papers (introduction, methods, results, conclusion, limitations)  
+* Recognize standard formatting requirements for academic conferences and journals
+
+## **Understand (Comprehension)**
+
+Students will be able to:
+
+* Describe the purpose and audience for each section of a research paper
+
+## **Apply (Application)**
+
+Students will be able to:
+
+* Format complete research papers in LaTeX with proper figures, tables, and citations  
+* Write clear methodology sections with sufficient detail for reproducibility  
+* Present experimental results using appropriate visualizations and statistical analysis
+
+## **Analyze (Analysis)**
+
+Students will be able to:
+
+* Diagnose LaTeX formatting issues and resolve compilation errors (if applicable)  
+* Examine related work to identify research gaps and position their contributions  
+* Compare their methodology approaches with existing methods
+
+## **Evaluate (Evaluation)**
+
+Students will be able to:
+
+* Critically assess the validity and reliability of their experimental design  
+* Evaluate the clarity and persuasiveness of their written arguments
+
+## **Create (Synthesis)**
+
+Students will be able to:
+
+* Produce research papers  
+* Develop compelling visualizations that effectively communicate complex ML concepts  
+* Synthesize technical knowledge into coherent research narratives
+

Reference files/Week_4_content.txt

@@ -1,630 +0,0 @@
-
-In this course, you'll learn the complete NLP workflow by exploring a fascinating real-world question: Does review length and language relate to reviewer ratings and decisions in academic peer review? If so, how?
-Using data from the International Conference on Learning Representations (ICLR), you'll develop practical NLP skills while investigating how reviewers express their opinions. Each module builds upon the previous one, creating a coherent analytical pipeline from raw data to insight.
-Learning Path
-Data Loading and Initial Exploration: Setting up your environment and understanding your dataset
-Text Preprocessing and Normalization: Cleaning and standardizing text data
-Feature Extraction and Measurement: Calculating metrics from text
-Visualization and Pattern Recognition: Creating insightful visualizations
-Drawing Conclusions from Text Analysis: Synthesizing findings into actionable insights
-Let's begin our exploration of how NLP can provide insights into academic peer review!
-
-Module 1: Initial Exploration
-The Challenge
-Before we can analyze how review length relates to paper evaluations, we need to understand our dataset. In this module, we'll set up our Python environment and explore the ICLR conference data.
-1.1: Set up and get to your data
-The first step in any NLP project is loading and understanding your data. Let's set up our environment and examine what we're working with:
-python
-# Import necessary libraries
-import pandas as pd
-import numpy as np
-import matplotlib.pyplot as plt
-import seaborn as sns
-import string
-from nltk.corpus import stopwords
-from nltk.tokenize import word_tokenize, sent_tokenize
-from wordcloud import WordCloud
-
-# Load the datasets
-df_reviews = pd.read_csv('../data/reviews.csv')
-df_submissions = pd.read_csv('../data/Submissions.csv')
-df_dec = pd.read_csv('../data/decision.csv')
-df_keyword = pd.read_csv('../data/submission_keyword.csv')
-Let's look at the first few rows of each dataset to understand what information we have:
-python
-# View the first few rows of the submissions dataset
-df_submissions.head()
-# View the first few rows of the reviews dataset
-df_reviews.head()
-# View all columns and rows in the reviews dataset
-df_reviews
-# View the first few rows of the keywords dataset
-df_keyword.head()
-1.2: Looking at Review Content
-Let's examine an actual review to understand the text we'll be analyzing:
-python
-# Display a sample review
-df_reviews['review'][1]
-Think about: What kinds of information do you see in this review? What language patterns do you notice?
-1.3: Calculating Basic Metrics
-Let's calculate our first simple metric - the average review score for each paper:
-python
-# Get the average review score for each paper
-df_average_review_score = df_reviews.groupby('forum')['rating_int'].mean().reset_index()
-df_average_review_score
-Key Insight: Each paper (identified by 'forum') receives multiple reviews with different scores. The average score gives us an overall assessment of each paper.
-Module 2: Data Integration
-In this module, we'll merge datasets for later analysis.
-2.1 Understanding the Need for Data Integration
-In many NLP projects, the data we need is spread across multiple files or tables. In our case:
-The df_reviews dataset contains the review text and ratings
-The df_dec dataset contains the final decisions for each paper
-To analyze how review text relates to paper decisions, we need to merge these datasets.
-2.2 Performing a Dataset Merge
-Let's combine our review data with the decision data:
-python
-# Step 1 - Merge the reviews dataframe with the decisions dataframe
-df_rev_dec = pd.merge(
-    df_reviews,              # First dataframe (reviews) 
-    df_dec,                  # Second dataframe (decisions)
-    left_on='forum',         # Join key in the first dataframe
-    right_on='forum',        # Join key in the second dataframe
-    how='inner'              # Keep only matching rows
-)[['review','decision','conf_name_y','rating_int','forum']]  # Select only these columns
-# Display the first few rows of the merged dataframe
-df_rev_dec.head()
-2.3 Understanding Merge Concepts
-Join Key: The 'forum' column identifies the paper and connects our datasets
-Inner Join: Only keeps papers that appear in both datasets
-Column Selection: We keep only relevant columns for our analysis
-How to Verify: Always check the shape of your merged dataset to ensure you haven't lost data unexpectedly
-Try it yourself: How many rows does the merged dataframe have compared to the original review dataframe? What might explain any differences?
-
-Module 3: Basic Text Preprocessing
-In this module, you'll learn essential data preprocessing techniques for NLP projects. We'll standardize text through case folding, clean up categorical variables, and prepare our review text for analysis.
-3.1 Case Folding (Lowercase Conversion)
-A fundamental text preprocessing step is converting all text to lowercase to ensure consistency:
-python
-# Convert all review text to lowercase (case folding)
-df_rev_dec['review'] = df_rev_dec['review'].str.lower()
-# Display the updated dataframe
-df_rev_dec
-Why Case Folding Matters
-Consistency: "Novel" and "novel" will be treated as the same word
-Reduced Dimensionality: Fewer unique tokens to process
-Improved Pattern Recognition: Easier to identify word frequencies and patterns
-Note: While case folding is generally helpful, it can sometimes remove meaningful distinctions (e.g., "US" vs. "us"). For our academic review analysis, lowercase conversion is appropriate.
-
-3.2 Examining Categorical Values
-Let's first check what unique decision categories exist in our dataset:
-python
-# Display the unique decision categories
-df_rev_dec['decision'].unique()
-3.3 Standardizing Decision Categories
-We can see that there are multiple "Accept" categories with different presentation formats. Let's standardize these:
-python
-# Define a function to clean up and standardize decision categories
-def clean_up_decision(text):
-    if text in ['Accept (Poster)','Accept (Spotlight)', 'Accept (Oral)','Accept (Talk)']:
-        return 'Accept'
-    else:
-        return text
-# Apply the function to create a new standardized decision column
-df_rev_dec['decision_clean'] = df_rev_dec['decision'].apply(clean_up_decision) 
-# Check our new standardized decision categories
-df_rev_dec['decision_clean'].unique()
-Why Standardization Matters
-Simplified Analysis: Reduces the number of categories to analyze
-Clearer Patterns: Makes it easier to identify trends by decision outcome
-Better Visualization: Creates more meaningful and readable plots
-Consistent Terminology: Aligns with how conferences typically report accept/reject decisions
-Try it yourself: What other ways could you group or standardize these decision categories? What information might be lost in our current approach?
-
-Module 4: Text Tokenization
-4.1 Introduction to Tokenization
-Tokenization is the process of breaking text into smaller units like sentences or words. Let's examine a review:
-python
-# Display a sample review
-df_reviews['review'][1]
-4.2 Sentence Tokenization
-Let's break this review into sentences using NLTK's sentence tokenizer:
-python
-# Import the necessary library if not already imported
-from nltk.tokenize import sent_tokenize
-# Tokenize the review into sentences
-sent_tokenize(df_reviews['review'][1])
-4.3 Counting Sentences
-Now let's count the number of sentences in the review:
-python
-# Count the number of sentences
-len(sent_tokenize(df_reviews['review'][1]))
-4.4 Creating a Reusable Function
-Let's create a function to count sentences in any text:
-python
-# Define a function to count sentences in a text
-def sentence_count(text):
-    return len(sent_tokenize(text))
-4.5 Applying Our Function to All Reviews
-Now we'll apply our function to all reviews to get sentence counts:
-python
-# Add a new column with the sentence count for each review
-df_rev_dec['sent_count'] = df_rev_dec['review'].apply(sentence_count)
-# Display the updated dataframe
-df_rev_dec.head()
-Key Insight: Sentence count is a simple yet effective way to quantify review length. The number of sentences can indicate how thoroughly a reviewer has evaluated a paper.
-
-Module 5: Visualization of Text Metrics
-5.1 Creating a 2D Histogram
-Let's visualize the relationship between review length (in sentences), rating, and decision outcome:
-python
-# Create a 2D histogram with sentence count, rating, and decision
-ax = sns.histplot(data=df_rev_dec, x='sent_count',
-                  y='rating_int',
-                  hue='decision_clean',
-                  kde=True, 
-                  log_scale=(True,False),
-                  legend=True)
-5.2 Enhancing Our Visualization
-Let's improve our visualization with better labels and formatting:
-python
-# Set axis labels
-ax.set(xlabel='Review Length (# Sentences)', ylabel='Review Rating')
-# Move the legend outside the plot for better visibility
-sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
-# Ensure the layout is properly configured
-plt.tight_layout()
-# Display the plot
-plt.show()
-5.3 Interpreting the Visualization
-This visualization reveals several interesting patterns:
-Length-Rating Relationship: Is there a pattern that entails how length of review is correlated with rating?
-Decision Patterns: Are there visible clusters for accepted vs. rejected papers?
-Density Distribution: Where are most reviews concentrated in terms of length and rating?
-Outliers: Are there unusually long or short reviews at certain rating levels?
-Discussion Question: Based on this visualization, do reviewers tend to write longer reviews when they're more positive or more critical? What might explain this pattern?
-
-Module 6: Additional Text Processing - Tokenization
-Tokenization is the process of breaking text into smaller units (tokens) that serve as the building blocks for natural language processing. In this lesson, we'll explore how to tokenize text, remove stopwords and punctuation, and analyze the results.
-6.1 Text Cleaning
-Before tokenization, we often clean the text to remove unwanted characters. Let's start by removing punctuation:
-python
-# Removing punctuation
-df_rev_dec['clean_review_word'] = df_rev_dec['review'].str.translate(str.maketrans('', '', string.punctuation))
-What's happening here?
-string.punctuation contains all punctuation characters (.,!?;:'"()[]{}-_)
-str.maketrans('', '', string.punctuation) creates a translation table to remove these characters
-df_rev_dec['review'].str.translate() applies this translation to all review texts
-6.2 Word Tokenization
-After cleaning, we can tokenize the text into individual words:
-python
-# Tokenizing the text
-df_rev_dec['tokens'] = df_rev_dec['clean_review_word'].apply(word_tokenize)
-
-# Example: Look at tokens for the 6th review
-df_rev_dec['tokens'][5]
-What's happening here?
-word_tokenize() is an NLTK function that splits text into a list of words
-We apply this function to each review using pandas' apply() method
-The result is a new column containing lists of words for each review
-6.3 Removing Stopwords
-Stopwords are common words like "the," "and," "is" that often don't add meaningful information for analysis:
-python
-# Getting the list of English stopwords
-stop_words = set(stopwords.words('english'))
-
-# Removing stopwords from our tokens
-df_rev_dec['tokens'] = df_rev_dec['tokens'].apply(lambda x: [word for word in x if word not in stop_words])
-What's happening here?
-stopwords.words('english') returns a list of common English stopwords
-We convert it to a set for faster lookup
-The lambda function filters each token list, keeping only words that aren't stopwords
-This creates more meaningful token lists focused on content words
-6.4 Counting Tokens
-Now that we have our cleaned and filtered tokens, let's count them to measure review length:
-python
-# Count tokens for each review
-df_rev_dec['tokens_counts'] = df_rev_dec['tokens'].apply(len)
-
-# View the token counts
-df_rev_dec['tokens_counts']
-What's happening here?
-We use apply(len) to count the number of tokens in each review
-This gives us a quantitative measure of review length after removing stopwords
-The difference between this and raw word count shows the prevalence of stopwords
-6.5 Visualizing Token Counts vs. Ratings
-Let's visualize the relationship between token count, rating, and decision:
-python
-# Create a 2D histogram with token count, rating, and decision
-ax = sns.histplot(data=df_rev_dec, x='tokens_counts',
-                  y='rating_int',
-                  hue='decision_clean',
-                  kde=True, 
-                  log_scale=(True,False),
-                  legend=True)
-
-# Set axis labels
-ax.set(xlabel='Review Length (# Tokens)', ylabel='Review Rating')
-
-# Move the legend outside the plot
-sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
-
-plt.tight_layout()
-plt.show()
-What's happening here?
-We create a 2D histogram showing the distribution of token counts and ratings
-Colors distinguish between accepted and rejected papers
-Log scale on the x-axis helps visualize the wide range of token counts
-Kernel density estimation (KDE) shows the concentration of reviews
-Module 7: Aggregating Data by Paper
-7.1 Understanding Data Aggregation
-So far, we've been analyzing individual reviews. However, each paper (identified by 'forum') may have multiple reviews. To understand paper-level patterns, we need to aggregate our data.
-7.2 Calculating Paper-Level Metrics
-Let's aggregate our review metrics to the paper level by calculating means:
-python
-# Aggregate reviews to paper level (mean of metrics for each paper)
-df_rev_dec_ave = df_rev_dec.groupby(['forum','decision_clean'])[['rating_int','tokens_counts','sent_count']].mean().reset_index()
-What's happening here?
-We're grouping reviews by both 'forum' (paper ID) and 'decision_clean' (accept/reject)
-For each group, we calculate the mean of 'rating_int', 'tokens_counts', and 'sent_count'
-The reset_index() turns the result back into a regular DataFrame
-The result is a paper-level dataset with average metrics for each paper
-Try it yourself: How many papers do we have in our dataset compared to reviews? What does this tell us about the review process?
-Module 8: Visualizing Token Count vs. Rating
-8.1 Creating an Advanced Visualization
-Now let's visualize the relationship between token count and rating at the paper level:
-python
-# Create a 2D histogram with token count, rating, and decision
-ax = sns.histplot(data=df_rev_dec_ave, x='tokens_counts',
-                  y='rating_int',
-                  hue='decision_clean',
-                  kde=True, 
-                  log_scale=(True,False),
-                  legend=True)
-
-# Set axis labels
-ax.set(xlabel='Review Length (# Tokens)', ylabel='Review Rating')
-
-# Move the legend outside the plot
-sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
-
-plt.tight_layout()
-plt.show()
-8.2 Interpreting the Visualization
-This visualization reveals important patterns in our data:
-Decision Boundaries: Notice where the color changes from one decision to another
-Length-Rating Relationship: Is there a correlation between review length and rating?
-Clustering: Are there natural clusters in the data?
-Outliers: What papers received unusually long or short reviews?
-Key Insight: At the paper level, we can see if the average review length for a paper relates to its likelihood of acceptance.
-Module 9: Comparing Token Count and Sentence Count
-9.1 Visualizing Sentence Count vs. Rating
-Let's create a similar visualization using sentence count instead of token count:
-python
-# Create a 2D histogram with sentence count, rating, and decision
-ax = sns.histplot(data=df_rev_dec_ave, x='sent_count',
-                  y='rating_int',
-                  hue='decision_clean',
-                  kde=True, 
-                  log_scale=(True,False),
-                  legend=True)
-
-# Set axis labels
-ax.set(xlabel='Review Length (# Sentences)', ylabel='Review Rating')
-
-# Move the legend outside the plot
-sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
-
-plt.tight_layout()
-plt.show()
-9.2 Comparing Token vs. Sentence Metrics
-By comparing these two visualizations, we can understand:
-Which Metric is More Informative: Do token counts or sentence counts better differentiate accepted vs. rejected papers?
-Different Patterns: Do some papers have many short sentences while others have fewer long ones?
-Consistency: Are the patterns consistent across both metrics?
-Discussion Question: Which metric—tokens or sentences—seems to be a better predictor of paper acceptance? Why might that be?
-Module 10: Word Cloud Visualizations
-10.1 Creating a Word Cloud from Review Text
-Word clouds are a powerful way to visualize the most frequent words in a text corpus:
-python
-# Concatenate all review text
-text = ' '.join(df_rev_dec['clean_review_word'])
-
-# Generate word cloud
-wordcloud = WordCloud().generate(text)
-
-# Display word cloud
-plt.figure(figsize=(8, 6))
-plt.imshow(wordcloud, interpolation='bilinear')
-plt.axis('off')
-plt.show()
-10.2 Visualizing Paper Keywords
-Now let's visualize the primary keywords associated with the papers:
-python
-# Concatenate all primary keywords
-text = ' '.join(df_keyword['primary_keyword'])
-
-# Generate word cloud
-wordcloud = WordCloud().generate(text)
-
-# Display word cloud
-plt.figure(figsize=(8, 6))
-plt.imshow(wordcloud, interpolation='bilinear')
-plt.axis('off')
-plt.show()
-10.3 Visualizing Paper Abstracts
-Finally, let's create a word cloud from paper abstracts:
-python
-# Concatenate all abstracts
-text = ' '.join(df_submissions['abstract'])
-
-# Generate word cloud
-wordcloud = WordCloud().generate(text)
-
-# Display word cloud
-plt.figure(figsize=(8, 6))
-plt.imshow(wordcloud, interpolation='bilinear')
-plt.axis('off')
-plt.show()
-Interpreting Word Clouds
-Word clouds provide insights about:
-Dominant Themes: The most frequent words appear largest
-Vocabulary Differences: Compare terms across different sources (reviews vs. abstracts)
-Field-Specific Terminology: Technical terms reveal the focus of the conference
-Sentiment Indicators: Evaluative words in reviews reveal assessment patterns
-Try it yourself: What differences do you notice between the word clouds from reviews, keywords, and abstracts? What do these differences tell you about academic communication?
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-V1.1 Week 4 - Intro to NLP 
-Course Overview
-In this course, you'll learn fundamental Natural Language Processing (NLP) concepts by exploring a fascinating real-world question: What is the effect of releasing a preprint of a paper before it is submitted for peer review?
-Using the ICLR (International Conference on Learning Representations) database - which contains submissions, reviews, and author profiles from 2017-2022 - you'll develop practical NLP skills while investigating potential biases and patterns in academic publishing.
-Learning Path
-Understanding Text as Data: How computers represent and work with text
-Text Processing Fundamentals: Basic cleaning and normalization
-Quantitative Text Analysis: Measuring and comparing text features
-Tokenization Approaches: Breaking text into meaningful units
-Text Visualization Techniques: Creating insightful visual representations
-From Analysis to Insights: Drawing evidence-based conclusions
-Let's dive in!
-…
-Step 4: Text Cleaning and Normalization for Academic Content
-Academic papers contain specialized vocabulary, citations, equations, and other elements that require careful normalization.
-Key Concept: Scientific text normalization preserves meaningful technical content while standardizing format.
-Stop Words Removal
-Definition: Stop words are extremely common words that appear frequently in text but typically carry little meaningful information for analysis purposes. In English, these include articles (the, a, an), conjunctions (and, but, or), prepositions (in, on, at), and certain pronouns (I, you, it).
-Stop words removal is the process of filtering these words out before analysis to:
-Reduce noise in the data
-Decrease the dimensionality of the text representation
-Focus analysis on the content-bearing words
-In academic text, we often extend standard stop word lists to include domain-specific terms that are ubiquitous but not analytically useful (e.g., "paper," "method," "result").
-python
-# Load standard English stop words
-from nltk.corpus import stopwords
-standard_stop_words = set(stopwords.words('english'))
-
-# Add academic-specific stop words
-academic_stop_words = ['et', 'al', 'fig', 'table', 'paper', 'using', 'used', 
-                      'method', 'result', 'show', 'propose', 'use']
-all_stop_words = standard_stop_words.union(academic_stop_words)
-
-# Apply stop word removal
-def remove_stop_words(text):
-    words = text.split()
-    filtered_words = [word for word in words if word.lower() not in all_stop_words]
-    return ' '.join(filtered_words)
-
-# Compare before and after
-example = "We propose a novel method that shows impressive results on the benchmark dataset."
-filtered = remove_stop_words(example)
-
-print("Original:", example)
-print("After stop word removal:", filtered)
-# Output: "propose novel method shows impressive results benchmark dataset."
-Stemming and Lemmatization
-Definition: Stemming and lemmatization are text normalization techniques that reduce words to their root or base forms, allowing different inflections or derivations of the same word to be treated as equivalent.
-Stemming is a simpler, rule-based approach that works by truncating words to their stems, often by removing suffixes. For example:
-"running," "runs," and "runner" might all be reduced to "run"
-"connection," "connected," and "connecting" might all become "connect"
-Stemming is faster but can sometimes produce non-words or incorrect reductions.
-Lemmatization is a more sophisticated approach that uses vocabulary and morphological analysis to return the dictionary base form (lemma) of a word. For example:
-"better" becomes "good"
-"was" and "were" become "be"
-"studying" becomes "study"
-Lemmatization generally produces more accurate results but requires more computational resources.
-python
-from nltk.stem import PorterStemmer, WordNetLemmatizer
-import nltk
-nltk.download('wordnet')
-
-# Initialize stemmer and lemmatizer
-stemmer = PorterStemmer()
-lemmatizer = WordNetLemmatizer()
-
-# Example words
-academic_terms = ["algorithms", "computing", "learning", "trained", 
-                  "networks", "better", "studies", "analyzed"]
-
-# Compare stemming and lemmatization
-for term in academic_terms:
-    print(f"Original: {term}")
-    print(f"Stemmed: {stemmer.stem(term)}")
-    print(f"Lemmatized: {lemmatizer.lemmatize(term)}")
-    print()
-
-# Demonstration in context
-academic_sentence = "The training algorithms performed better than expected when analyzing multiple neural networks."
-
-# Apply stemming
-stemmed_words = [stemmer.stem(word) for word in academic_sentence.lower().split()]
-stemmed_sentence = ' '.join(stemmed_words)
-
-# Apply lemmatization
-lemmatized_words = [lemmatizer.lemmatize(word) for word in academic_sentence.lower().split()]
-lemmatized_sentence = ' '.join(lemmatized_words)
-
-print("Original:", academic_sentence)
-print("Stemmed:", stemmed_sentence)
-print("Lemmatized:", lemmatized_sentence)
-When to use which approach:
-For academic text analysis:
-Stemming is useful when processing speed is important and approximate matching is sufficient
-Lemmatization is preferred when precision is crucial, especially for technical terms where preserving meaning is essential
-In our ICLR paper analysis, lemmatization would likely be more appropriate since technical terminology often carries specific meanings that should be preserved accurately.
-Challenge Question: How might stemming versus lemmatization affect our analysis of technical innovation in ICLR papers? Can you think of specific machine learning terms where these approaches would yield different results?
-
-
-V1.0 Week 4 - Intro to NLP 
-The Real-World Problem
-Imagine you're part of a small business team that has just launched a new product. You've received hundreds of customer reviews across various platforms, and your manager has asked you to make sense of this feedback. Looking at the mountain of text data, you realize you need a systematic way to understand what customers are saying without reading each review individually.
-Your challenge: How can you efficiently analyze customer feedback to identify common themes, sentiments, and specific product issues?
-Our Approach
-In this module, we'll learn how to transform unstructured text feedback into structured insights using Natural Language Processing. Here's our journey:
-Understanding text as data
-Basic processing of text information
-Measuring text properties
-Cleaning and normalizing customer feedback
-Visualizing patterns in the feedback
-Analyzing words vs. tokens
-Let's begin!
-Step 1: Text as Data - A New Perspective
-When we look at customer reviews like:
-"Love this product! So easy to use and the battery lasts forever."
-"Terrible design. Buttons stopped working after two weeks."
-We naturally understand the meaning and sentiment. But how can a computer understand this?
-Key Concept: Text can be treated as data that we can analyze quantitatively.
-Unlike numerical data (age, price, temperature) that has inherent mathematical properties, text data needs to be transformed before we can analyze it.
-Interactive Exercise: Look at these two reviews. As a human, what information can you extract? Now think about how a computer might "see" this text without any processing.
-Challenge Question: What types of information might we want to extract from customer reviews? List at least three analytical goals.
-Step 2: Basic Text Processing - Breaking Down Language
-Before we can analyze text, we need to break it down into meaningful units.
-Key Concept: Tokenization is the process of splitting text into smaller pieces (tokens) such as words, phrases, or characters.
-For example, the review "Love this product!" can be tokenized into ["Love", "this", "product", "!"] or ["Love", "this", "product", "!"] depending on our approach.
-Interactive Example: Let's tokenize these customer reviews:
-python
-# Simple word tokenization
-review = "Battery life is amazing but the app crashes frequently."
-tokens = review.split()  # Results in ["Battery", "life", "is", "amazing", "but", "the", "app", "crashes", "frequently."]
-Notice how "frequently." includes the period. Basic tokenization has limitations!
-Challenge Question: How might we handle contractions like "doesn't" or hyphenated words like "user-friendly" when tokenizing?
-Step 3: Measuring Text - Quantifying Feedback
-Now that we've broken text into pieces, we can start measuring properties of our customer feedback.
-Key Concept: Text metrics help us quantify and compare text data.
-Common metrics include:
-Length (words, characters)
-Complexity (average word length, unique words ratio)
-Sentiment scores (positive/negative)
-Interactive Example: Let's calculate basic metrics for customer reviews:
-python
-# Word count
-review = "The interface is intuitive and responsive."
-word_count = len(review.split())  # 6 words
-
-# Character count (including spaces)
-char_count = len(review)  # 41 characters
-
-# Unique words ratio
-unique_words = len(set(review.lower().split()))
-unique_ratio = unique_words / word_count  # 1.0 (all words are unique)
-Challenge Question: Why might longer reviews not necessarily contain more information than shorter ones? What other metrics beyond length might better capture information content?
-Step 4: Text Cleaning and Normalization
-Customer feedback often contains inconsistencies: spelling variations, punctuation, capitalization, etc.
-Key Concept: Text normalization creates a standardized format for analysis.
-Common normalization steps:
-Converting to lowercase
-Removing punctuation
-Correcting spelling
-Removing stop words (common words like "the", "is")
-Stemming or lemmatizing (reducing words to their base form)
-Interactive Example: Let's normalize a review:
-python
-# Original review
-review = "The battery LIFE is amazing!!! Works for days."
-
-# Lowercase
-review = review.lower()  # "the battery life is amazing!!! works for days."
-
-# Remove punctuation and extra spaces
-import re
-review = re.sub(r'[^\w\s]', '', review)  # "the battery life is amazing works for days"
-
-# Remove stop words
-stop_words = ["the", "is", "for"]
-words = review.split()
-filtered_words = [word for word in words if word not in stop_words]
-# Result: ["battery", "life", "amazing", "works", "days"]
-Challenge Question: How might normalization affect sentiment analysis? Could removing punctuation or stop words change the perceived sentiment of a review?
-Step 5: Text Visualization - Seeing Patterns
-Visual representations help us identify patterns across many reviews.
-Key Concept: Text visualization techniques reveal insights that are difficult to see in raw text.
-Common visualization methods:
-Word clouds
-Frequency distributions
-Sentiment over time
-Topic clusters
-Interactive Example: Creating a simple word frequency chart:
-python
-from collections import Counter
-
-# Combined reviews
-reviews = ["Battery life is amazing", "Battery drains too quickly", 
-           "Great battery performance", "Screen is too small"]
-
-# Count word frequencies
-all_words = " ".join(reviews).lower().split()
-word_counts = Counter(all_words)
-# Result: {'battery': 3, 'life': 1, 'is': 2, 'amazing': 1, 'drains': 1, 'too': 2, 'quickly': 1, 'great': 1, 'performance': 1, 'screen': 1, 'small': 1}
-
-# We could visualize this as a bar chart
-# Most frequent: 'battery' (3), 'is' (2), 'too' (2)
-Challenge Question: Why might a word cloud be misleading for understanding customer sentiment? What additional information would make the visualization more informative?
-Step 6: Words vs. Tokens - Making Choices
-As we advance in NLP, we face an important decision: should we analyze whole words or more sophisticated tokens?
-Key Concept: Different tokenization approaches have distinct advantages and limitations.
-Word-based analysis:
-Intuitive and interpretable
-Misses connections between related words (run/running/ran)
-Struggles with compound words and new terms
-Token-based analysis:
-Can capture subword information
-Handles unknown words better
-May lose some human interpretability
-Interactive Example: Comparing approaches:
-python
-# Word-based
-review = "The touchscreen is unresponsive"
-words = review.lower().split()  # ['the', 'touchscreen', 'is', 'unresponsive']
-
-# Subword tokenization (simplified example)
-subwords = ['the', 'touch', 'screen', 'is', 'un', 'responsive']
-Challenge Question: For our customer feedback analysis, which approach would be better: analyzing whole words or subword tokens? What factors would influence this decision?
-Putting It All Together: Solving Our Problem
-Now that we've learned these fundamental NLP concepts, let's return to our original challenge: analyzing customer feedback at scale.
-Here's how we'd approach it:
-Collect and tokenize all customer reviews
-Clean and normalize the text
-Calculate key metrics (length, sentiment scores)
-Visualize common terms and topics
-Identify positive and negative feedback themes
-Generate an automated summary for the product team
-By applying these NLP fundamentals, we've transformed an overwhelming mass of text into actionable insights that can drive product improvements!
-Final Challenge: How could we extend this analysis to track customer sentiment over time as we release product updates? What additional NLP techniques might be helpful?
-

Reference files/w6_logistic_regression_lab.py

@@ -1,400 +0,0 @@
-# -*- coding: utf-8 -*-
-"""W6_Logistic_regression_lab
-
-Automatically generated by Colab.
-
-Original file is located at
-    https://colab.research.google.com/drive/1MG7N2HN-Nxow9fzvc0fzxvp3WyKqtgs8
-
-# 🚀 Logistic Regression Lab: Stock Market Prediction
-
-## Lab Overview
-In this lab, we'll use logistic regression to try predicting whether the stock market goes up or down. Spoiler alert: This is intentionally a challenging prediction problem that will teach us important lessons about when logistic regression works well and when it doesn't.
-## Learning Goals:
-
-- Apply logistic regression to real data
-- Interpret probabilities and coefficients
-- Understand why some prediction problems are inherently difficult
-- Learn proper model evaluation techniques
-
-## The Stock Market Data
-
-In this lab we will examine the `Smarket`
-data, which is part of the `ISLP`
-library. This data set consists of percentage returns for the S&P 500
-stock index over 1,250 days, from the beginning of 2001 until the end
-of 2005. For each date, we have recorded the percentage returns for
-each of the five previous trading days,  `Lag1`  through
- `Lag5`. We have also recorded  `Volume`  (the number of
-shares traded on the previous day, in billions),  `Today`  (the
-percentage return on the date in question) and  `Direction`
-(whether the market was  `Up`  or  `Down`  on this date).
-
-### Your Challenge
-**Question**: Can we predict if the S&P 500 will go up or down based on recent trading patterns?
-
-**Why This Matters:** If predictable, this would be incredibly valuable. If not predictable, we learn about market efficiency and realistic expectations for prediction models.
-
-
-To answer the question, **we start by importing  our libraries at this top level; these are all imports we have seen in previous labs.**
-"""
-
-import numpy as np
-import pandas as pd
-from matplotlib.pyplot import subplots
-import statsmodels.api as sm
-from ISLP import load_data
-from ISLP.models import (ModelSpec as MS,
-                         summarize)
-
-"""We also collect together the new imports needed for this lab."""
-
-from ISLP import confusion_table
-from ISLP.models import contrast
-from sklearn.discriminant_analysis import \
-     (LinearDiscriminantAnalysis as LDA,
-      QuadraticDiscriminantAnalysis as QDA)
-from sklearn.naive_bayes import GaussianNB
-from sklearn.neighbors import KNeighborsClassifier
-from sklearn.preprocessing import StandardScaler
-from sklearn.model_selection import train_test_split
-from sklearn.linear_model import LogisticRegression
-
-"""Now we are ready to load the `Smarket` data."""
-
-Smarket = load_data('Smarket')
-Smarket
-
-"""This gives a truncated listing of the data.
-We can see what the variable names are.
-"""
-
-Smarket.columns
-
-"""We compute the correlation matrix using the `corr()` method
-for data frames, which produces a matrix that contains all of
-the pairwise correlations among the variables.
-
-By instructing `pandas` to use only numeric variables, the `corr()` method does not report a correlation for the `Direction`  variable because it is
- qualitative.
-
- ![image.png](attachment:image.png)
-"""
-
-Smarket.corr(numeric_only=True)
-
-"""As one would expect, the correlations between the lagged return  variables and
-today’s return are close to zero.  The only substantial correlation is between  `Year`  and
- `Volume`. By plotting the data we see that  `Volume`
-is increasing over time. In other words, the average number of shares traded
-daily increased from 2001 to 2005.
-
-"""
-
-Smarket.plot(y='Volume');
-
-"""## Logistic Regression
-Next, we will fit a logistic regression model in order to predict
- `Direction`  using  `Lag1`  through  `Lag5`  and
- `Volume`. The `sm.GLM()`  function fits *generalized linear models*, a class of
-models that includes logistic regression.  Alternatively,
-the function `sm.Logit()` fits a logistic regression
-model directly. The syntax of
-`sm.GLM()` is similar to that of `sm.OLS()`, except
-that we must pass in the argument `family=sm.families.Binomial()`
-in order to tell `statsmodels` to run a logistic regression rather than some other
-type of generalized linear model.
-"""
-
-allvars = Smarket.columns.drop(['Today', 'Direction', 'Year'])
-design = MS(allvars)
-X = design.fit_transform(Smarket)
-y = Smarket.Direction == 'Up'
-glm = sm.GLM(y,
-             X,
-             family=sm.families.Binomial())
-results = glm.fit()
-summarize(results)
-
-"""The smallest *p*-value here is associated with  `Lag1`. The
-negative coefficient for this predictor suggests that if the market
-had a positive return yesterday, then it is less likely to go up
-today. However, at a value of 0.15, the *p*-value is still
-relatively large, and so there is no clear evidence of a real
-association between  `Lag1`  and  `Direction`.
-
-We use the `params`  attribute of `results`
-in order to access just the
-coefficients for this fitted model.
-"""
-
-results.params
-
-"""Likewise we can use the
-`pvalues`  attribute to access the *p*-values for the coefficients.
-"""
-
-results.pvalues
-
-"""The `predict()`  method of `results` can be used to predict the
-probability that the market will go up, given values of the
-predictors. This method returns predictions
-on the probability scale. If no data set is supplied to the `predict()`
-function, then the probabilities are computed for the training data
-that was used to fit the logistic regression model.
-As with linear regression, one can pass an optional `exog` argument consistent
-with a design matrix if desired. Here we have
-printed only the first ten probabilities.
-"""
-
-probs = results.predict()
-probs[:10]
-
-"""In order to make a prediction as to whether the market will go up or
-down on a particular day, we must convert these predicted
-probabilities into class labels,  `Up`  or  `Down`.  The
-following two commands create a vector of class predictions based on
-whether the predicted probability of a market increase is greater than
-or less than 0.5.
-"""
-
-labels = np.array(['Down']*1250)
-labels[probs>0.5] = "Up"
-
-"""The `confusion_table()`
-function from the `ISLP` package summarizes these predictions, showing   how
-many observations were correctly or incorrectly classified. Our function, which is adapted from a similar function
-in the module `sklearn.metrics`,  transposes the resulting
-matrix and includes row and column labels.
-The `confusion_table()` function takes as first argument the
-predicted labels, and second argument the true labels.
-"""
-
-confusion_table(labels, Smarket.Direction)
-
-"""The diagonal elements of the confusion matrix indicate correct
-predictions, while the off-diagonals represent incorrect
-predictions. Hence our model correctly predicted that the market would
-go up on 507 days and that it would go down on 145 days, for a
-total of 507 + 145 = 652 correct predictions. The `np.mean()`
-function can be used to compute the fraction of days for which the
-prediction was correct. In this case, logistic regression correctly
-predicted the movement of the market 52.2% of the time.
-
-"""
-
-(507+145)/1250, np.mean(labels == Smarket.Direction)
-
-"""At first glance, it appears that the logistic regression model is
-working a little better than random guessing. However, this result is
-misleading because we trained and tested the model on the same set of
-1,250 observations. In other words, $100-52.2=47.8%$ is the
-*training* error  rate. As we have seen
-previously, the training error rate is often overly optimistic --- it
-tends to underestimate the test error rate.  In
-order to better assess the accuracy of the logistic regression model
-in this setting, we can fit the model using part of the data, and
-then examine how well it predicts the *held out* data.  This
-will yield a more realistic error rate, in the sense that in practice
-we will be interested in our model’s performance not on the data that
-we used to fit the model, but rather on days in the future for which
-the market’s movements are unknown.
-
-To implement this strategy, we first create a Boolean vector
-corresponding to the observations from 2001 through 2004. We  then
-use this vector to create a held out data set of observations from
-2005.
-"""
-
-train = (Smarket.Year < 2005)
-Smarket_train = Smarket.loc[train]
-Smarket_test = Smarket.loc[~train]
-Smarket_test.shape
-
-"""The object `train` is a vector of 1,250 elements, corresponding
-to the observations in our data set. The elements of the vector that
-correspond to observations that occurred before 2005 are set to
-`True`, whereas those that correspond to observations in 2005 are
-set to `False`.  Hence `train` is a
-*boolean*   array, since its
-elements are `True` and `False`.  Boolean arrays can be used
-to obtain a subset of the rows or columns of a data frame
-using the `loc` method. For instance,
-the command `Smarket.loc[train]` would pick out a submatrix of the
-stock market data set, corresponding only to the dates before 2005,
-since those are the ones for which the elements of `train` are
-`True`.  The `~` symbol can be used to negate all of the
-elements of a Boolean vector. That is, `~train` is a vector
-similar to `train`, except that the elements that are `True`
-in `train` get swapped to `False` in `~train`, and vice versa.
-Therefore, `Smarket.loc[~train]` yields a
-subset of the rows of the data frame
-of the stock market data containing only the observations for which
-`train` is `False`.
-The output above indicates that there are 252 such
-observations.
-
-We now fit a logistic regression model using only the subset of the
-observations that correspond to dates before 2005. We then obtain predicted probabilities of the
-stock market going up for each of the days in our test set --- that is,
-for the days in 2005.
-"""
-
-X_train, X_test = X.loc[train], X.loc[~train]
-y_train, y_test = y.loc[train], y.loc[~train]
-glm_train = sm.GLM(y_train,
-                   X_train,
-                   family=sm.families.Binomial())
-results = glm_train.fit()
-probs = results.predict(exog=X_test)
-
-"""Notice that we have trained and tested our model on two completely
-separate data sets: training was performed using only the dates before
-2005, and testing was performed using only the dates in 2005.
-
-Finally, we compare the predictions for 2005 to the
-actual movements of the market over that time period.
-We will first store the test and training labels (recall `y_test` is binary).
-"""
-
-D = Smarket.Direction
-L_train, L_test = D.loc[train], D.loc[~train]
-
-"""Now we threshold the
-fitted probability at 50% to form
-our predicted labels.
-"""
-
-labels = np.array(['Down']*252)
-labels[probs>0.5] = 'Up'
-confusion_table(labels, L_test)
-
-"""The test accuracy is about 48% while the error rate is about 52%"""
-
-np.mean(labels == L_test), np.mean(labels != L_test)
-
-"""The `!=` notation means *not equal to*, and so the last command
-computes the test set error rate. The results are rather
-disappointing: the test error rate is 52%, which is worse than
-random guessing! Of course this result is not all that surprising,
-given that one would not generally expect to be able to use previous
-days’ returns to predict future market performance. (After all, if it
-were possible to do so, then the authors of this book would be out
-striking it rich rather than writing a statistics textbook.)
-
-We recall that the logistic regression model had very underwhelming
-*p*-values associated with all of the predictors, and that the
-smallest *p*-value, though not very small, corresponded to
- `Lag1`. Perhaps by removing the variables that appear not to be
-helpful in predicting  `Direction`, we can obtain a more
-effective model. After all, using predictors that have no relationship
-with the response tends to cause a deterioration in the test error
-rate (since such predictors cause an increase in variance without a
-corresponding decrease in bias), and so removing such predictors may
-in turn yield an improvement.  Below we refit the logistic
-regression using just  `Lag1`  and  `Lag2`, which seemed to
-have the highest predictive power in the original logistic regression
-model.
-"""
-
-model = MS(['Lag1', 'Lag2']).fit(Smarket)
-X = model.transform(Smarket)
-X_train, X_test = X.loc[train], X.loc[~train]
-glm_train = sm.GLM(y_train,
-                   X_train,
-                   family=sm.families.Binomial())
-results = glm_train.fit()
-probs = results.predict(exog=X_test)
-labels = np.array(['Down']*252)
-labels[probs>0.5] = 'Up'
-confusion_table(labels, L_test)
-
-"""Let’s evaluate the overall accuracy as well as the accuracy within the days when
-logistic regression predicts an increase.
-"""
-
-(35+106)/252,106/(106+76)
-
-"""Now the results appear to be a little better: 56% of the daily
-movements have been correctly predicted. It is worth noting that in
-this case, a much simpler strategy of predicting that the market will
-increase every day will also be correct 56% of the time! Hence, in
-terms of overall error rate, the logistic regression method is no
-better than the naive approach. However, the confusion matrix
-shows that on days when logistic regression predicts an increase in
-the market, it has a 58% accuracy rate. This suggests a possible
-trading strategy of buying on days when the model predicts an
-increasing market, and avoiding trades on days when a decrease is
-predicted. Of course one would need to investigate more carefully
-whether this small improvement was real or just due to random chance.
-
-Suppose that we want to predict the returns associated with particular
-values of  `Lag1`  and  `Lag2`. In particular, we want to
-predict  `Direction`  on a day when  `Lag1`  and
- `Lag2`  equal $1.2$ and $1.1$, respectively, and on a day when they
-equal $1.5$ and $-0.8$.  We do this using the `predict()`
-function.
-"""
-
-newdata = pd.DataFrame({'Lag1':[1.2, 1.5],
-                        'Lag2':[1.1, -0.8]});
-newX = model.transform(newdata)
-results.predict(newX)
-
-Smarket
-
-import pandas as pd
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import classification_report, confusion_matrix
-import statsmodels.api as sm
-
-
-# Load the dataset
-data = load_data('Smarket')
-
-# Display the first few rows of the dataset
-print(data.head())
-
-# Prepare the data for logistic regression
-# Using 'Lag1' and 'Lag2' as predictors and 'Direction' as the response
-data['Direction'] = data['Direction'].map({'Up': 1, 'Down': 0})
-X = data[['Lag1', 'Lag2']]
-y = data['Direction']
-
-# Split the data into training and testing sets
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
-
-# Fit the logistic regression model
-log_reg = LogisticRegression()
-log_reg.fit(X_train, y_train)
-
-# Make predictions on the test set
-y_pred = log_reg.predict(X_test)
-
-# Print classification report and confusion matrix
-print(classification_report(y_test, y_pred))
-print(confusion_matrix(y_test, y_pred))
-
-# Visualize the decision boundary
-plt.figure(figsize=(10, 6))
-
-# Create a mesh grid for plotting decision boundary
-x_min, x_max = X['Lag1'].min() - 1, X['Lag1'].max() + 1
-y_min, y_max = X['Lag2'].min() - 1, X['Lag2'].max() + 1
-xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.01),
-                     np.arange(y_min, y_max, 0.01))
-
-# Predict the function value for the whole grid
-Z = log_reg.predict(np.c_[xx.ravel(), yy.ravel()])
-Z = Z.reshape(xx.shape)
-
-# Plot the decision boundary
-plt.contourf(xx, yy, Z, alpha=0.8)
-plt.scatter(X_test['Lag1'], X_test['Lag2'], c=y_test, edgecolor='k', s=20)
-plt.xlabel('Lag1')
-plt.ylabel('Lag2')
-plt.title('Logistic Regression Decision Boundary')
-plt.show()

Reference files/week 7/W7_Lab_KNN_clustering.ipynb

@@ -1,481 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "id": "b1c6a137",
-      "metadata": {
-        "id": "b1c6a137"
-      },
-      "source": [
-        "# Clustering Lab: State Crime Pattern Analysis\n",
-        "\n",
-        "## Lab Overview\n",
-        "\n",
-        "Welcome to your hands-on clustering lab! You'll be working as a policy analyst for the Department of Justice, analyzing crime patterns across US states. Your mission: discover hidden safety profiles that could inform federal resource allocation and crime prevention strategies.\n",
-        "\n",
-        "**Your Deliverable**: A policy brief with visualizations and recommendations based on your clustering analysis.\n",
-        "\n",
-        "---\n",
-        "\n",
-        "## Exercise 1: Data Detective Work\n",
-        "**Time: 15 minutes | Product: Data Summary Report**\n",
-        "\n",
-        "### Your Task\n",
-        "Before any analysis, you need to understand what you're working with. Create a brief data summary that a non-technical policy maker could understand.\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "```python\n",
-        "import numpy as np\n",
-        "import pandas as pd\n",
-        "import matplotlib.pyplot as plt\n",
-        "from statsmodels.datasets import get_rdataset\n",
-        "from sklearn.preprocessing import StandardScaler\n",
-        "from sklearn.cluster import KMeans, AgglomerativeClustering\n",
-        "\n",
-        "# Load the data\n",
-        "USArrests = get_rdataset('USArrests').data\n",
-        "print(\"Dataset shape:\", USArrests.shape)\n",
-        "print(\"\\nVariables:\", USArrests.columns.tolist())\n",
-        "print(\"\\nFirst 5 states:\")\n",
-        "print(USArrests.head())\n",
-        "```"
-      ],
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 106
-        },
-        "id": "mqRVE1hlXK9x",
-        "outputId": "5a1bbd64-15cd-4e1c-9344-64a901d8a396"
-      },
-      "id": "mqRVE1hlXK9x",
-      "execution_count": null,
-      "outputs": [
-        {
-          "output_type": "error",
-          "ename": "SyntaxError",
-          "evalue": "invalid syntax (<ipython-input-1-2035427107>, line 1)",
-          "traceback": [
-            "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-2035427107>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    ```python\u001b[0m\n\u001b[0m    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
-          ]
-        }
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "## Your Investigation\n",
-        "Complete this data summary table:\n",
-        "\n",
-        "| Variable | What it measures | Average Value | Highest State | Lowest State |\n",
-        "|----------|------------------|---------------|---------------|--------------|\n",
-        "| Murder | Rate per 100,000 people | ??? | ??? | ??? |\n",
-        "| Assault | Rate per 100,000 people | ??? | ??? | ??? |\n",
-        "| UrbanPop | Percentage living in cities | ??? | ??? | ??? |\n",
-        "| Rape | Rate per 100,000 people | ??? | ??? | ??? |\n",
-        "\n",
-        "**Deliverable**: Write 2-3 sentences describing the biggest surprises in this data. Which states are not what you expected?\n",
-        "\n",
-        "---\n",
-        "\n",
-        "## Exercise 2: The Scaling Challenge\n",
-        "**Time: 10 minutes | Product: Before/After Comparison**\n",
-        "\n",
-        "### Your Task\n",
-        "Demonstrate why scaling is critical for clustering crime data.\n",
-        "\n"
-      ],
-      "metadata": {
-        "id": "7qkDKTe4XLtG"
-      },
-      "id": "7qkDKTe4XLtG"
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "```python\n",
-        "# Check the scale differences\n",
-        "print(\"Original data ranges:\")\n",
-        "print(USArrests.describe())\n",
-        "\n",
-        "print(\"\\nVariances (how spread out the data is):\")\n",
-        "print(USArrests.var())\n",
-        "\n",
-        "# Scale the data\n",
-        "scaler = StandardScaler()\n",
-        "USArrests_scaled = scaler.fit_transform(USArrests)\n",
-        "scaled_df = pd.DataFrame(USArrests_scaled,\n",
-        "                        columns=USArrests.columns,\n",
-        "                        index=USArrests.index)\n",
-        "\n",
-        "print(\"\\nAfter scaling - all variables now have similar ranges:\")\n",
-        "print(scaled_df.describe())\n",
-        "```"
-      ],
-      "metadata": {
-        "id": "zQ3VowYNXLeQ"
-      },
-      "id": "zQ3VowYNXLeQ",
-      "execution_count": null,
-      "outputs": []
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "### Your Analysis\n",
-        "1. **Before scaling**: Which variable would dominate the clustering? Why?\n",
-        "2. **After scaling**: Explain in simple terms what StandardScaler did to the data.\n",
-        "\n",
-        "**Deliverable**: One paragraph explaining why a policy analyst should care about data scaling.\n",
-        "\n",
-        "---\n",
-        "\n",
-        "## Exercise 3: Finding the Right Number of Groups\n",
-        "**Time: 20 minutes | Product: Recommendation with Visual Evidence**\n",
-        "\n",
-        "### Your Task\n",
-        "Use the elbow method to determine how many distinct crime profiles exist among US states.\n"
-      ],
-      "metadata": {
-        "id": "FnOT700SXLPh"
-      },
-      "id": "FnOT700SXLPh"
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "```python\n",
-        "# Test different numbers of clusters\n",
-        "inertias = []\n",
-        "K_values = range(1, 11)\n",
-        "\n",
-        "for k in K_values:\n",
-        "    kmeans = KMeans(n_clusters=k, random_state=42, n_init=20)\n",
-        "    kmeans.fit(USArrests_scaled)\n",
-        "    inertias.append(kmeans.inertia_)\n",
-        "\n",
-        "# Create the elbow plot\n",
-        "plt.figure(figsize=(10, 6))\n",
-        "plt.plot(K_values, inertias, 'bo-', linewidth=2, markersize=8)\n",
-        "plt.xlabel('Number of Clusters (K)')\n",
-        "plt.ylabel('Within-Cluster Sum of Squares')\n",
-        "plt.title('Finding the Optimal Number of State Crime Profiles')\n",
-        "plt.grid(True, alpha=0.3)\n",
-        "plt.show()\n",
-        "\n",
-        "# Print the inertia values\n",
-        "for k, inertia in zip(K_values, inertias):\n",
-        "    print(f\"K={k}: Inertia = {inertia:.1f}\")\n",
-        "```"
-      ],
-      "metadata": {
-        "id": "zOQrS9lmXpTF"
-      },
-      "id": "zOQrS9lmXpTF",
-      "execution_count": null,
-      "outputs": []
-    },
-    {
-      "cell_type": "markdown",
-      "id": "2e388ef2",
-      "metadata": {
-        "id": "2e388ef2"
-      },
-      "source": [
-        "### Your Decision\n",
-        "Based on your elbow plot:\n",
-        "1. **What value of K do you recommend?** (Look for the \"elbow\" where the line starts to flatten)\n",
-        "2. **What does this mean in policy terms?** (How many distinct types of state crime profiles exist?)\n",
-        "\n",
-        "**Deliverable**: A one-paragraph recommendation with your chosen K value and reasoning.\n",
-        "\n",
-        "---\n",
-        "\n",
-        "## Exercise 4: K-Means State Profiling\n",
-        "**Time: 25 minutes | Product: State Crime Profile Report**\n",
-        "\n",
-        "### Your Task\n",
-        "Create distinct crime profiles and identify which states belong to each category.\n",
-        "\n",
-        "\n",
-        "\n",
-        "\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "```python\n",
-        "# Use your chosen K value from Exercise 3\n",
-        "optimal_k = 4  # Replace with your chosen value\n",
-        "\n",
-        "# Perform K-means clustering\n",
-        "kmeans = KMeans(n_clusters=optimal_k, random_state=42, n_init=20)\n",
-        "cluster_labels = kmeans.fit_predict(USArrests_scaled)\n",
-        "\n",
-        "# Add cluster labels to original data\n",
-        "USArrests_clustered = USArrests.copy()\n",
-        "USArrests_clustered['Cluster'] = cluster_labels\n",
-        "\n",
-        "# Analyze each cluster\n",
-        "print(\"State Crime Profiles Analysis\")\n",
-        "print(\"=\" * 50)\n",
-        "\n",
-        "for cluster_num in range(optimal_k):\n",
-        "    cluster_states = USArrests_clustered[USArrests_clustered['Cluster'] == cluster_num]\n",
-        "    print(f\"\\nCLUSTER {cluster_num}: {len(cluster_states)} states\")\n",
-        "    print(\"States:\", \", \".join(cluster_states.index.tolist()))\n",
-        "    print(\"Average characteristics:\")\n",
-        "    avg_profile = cluster_states[['Murder', 'Assault', 'UrbanPop', 'Rape']].mean()\n",
-        "    for var, value in avg_profile.items():\n",
-        "        print(f\"  {var}: {value:.1f}\")\n",
-        "```"
-      ],
-      "metadata": {
-        "id": "_5b0nE6KXv1P"
-      },
-      "id": "_5b0nE6KXv1P",
-      "execution_count": null,
-      "outputs": []
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "### Your Analysis\n",
-        "For each cluster, create a profile:\n",
-        "\n",
-        "**Cluster 0: \"[Your Creative Name]\"**\n",
-        "- **States**: [List them]\n",
-        "- **Characteristics**: [Describe the pattern]\n",
-        "- **Policy Insight**: [What should federal agencies know about these states?]\n",
-        "\n",
-        "**Deliverable**: A table summarizing each cluster with creative names and policy recommendations.\n",
-        "\n",
-        "---\n",
-        "\n",
-        "## Exercise 5: Hierarchical Clustering Exploration\n",
-        "**Time: 25 minutes | Product: Family Tree Interpretation**\n",
-        "\n",
-        "### Your Task\n",
-        "Create a dendrogram to understand how states naturally group together.\n"
-      ],
-      "metadata": {
-        "id": "J1WVGb_nX4ye"
-      },
-      "id": "J1WVGb_nX4ye"
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "```python\n",
-        "from scipy.cluster.hierarchy import dendrogram, linkage\n",
-        "\n",
-        "# Create hierarchical clustering\n",
-        "linkage_matrix = linkage(USArrests_scaled, method='complete')\n",
-        "\n",
-        "# Plot the dendrogram\n",
-        "plt.figure(figsize=(15, 8))\n",
-        "dendrogram(linkage_matrix,\n",
-        "           labels=USArrests.index.tolist(),\n",
-        "           leaf_rotation=90,\n",
-        "           leaf_font_size=10)\n",
-        "plt.title('State Crime Pattern Family Tree')\n",
-        "plt.xlabel('States')\n",
-        "plt.ylabel('Distance Between Groups')\n",
-        "plt.tight_layout()\n",
-        "plt.show()\n",
-        "```"
-      ],
-      "metadata": {
-        "id": "Y9a_cbZKX7QX"
-      },
-      "id": "Y9a_cbZKX7QX",
-      "execution_count": null,
-      "outputs": []
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "### Your Interpretation\n",
-        "1. **Closest Pairs**: Which two states are most similar in crime patterns?\n",
-        "2. **Biggest Divide**: Where is the largest split in the tree? What does this represent?\n",
-        "3. **Surprising Neighbors**: Which states cluster together that surprised you geographically?\n",
-        "\n",
-        "### Code to Compare Methods"
-      ],
-      "metadata": {
-        "id": "0PaImqZtX6f3"
-      },
-      "id": "0PaImqZtX6f3"
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "```python\n",
-        "# Compare your K-means results with hierarchical clustering\n",
-        "from scipy.cluster.hierarchy import fcluster\n",
-        "\n",
-        "# Cut the tree to get the same number of clusters as K-means\n",
-        "hierarchical_labels = fcluster(linkage_matrix, optimal_k, criterion='maxclust') - 1\n",
-        "\n",
-        "# Create comparison\n",
-        "comparison_df = pd.DataFrame({\n",
-        "    'State': USArrests.index,\n",
-        "    'K_Means_Cluster': cluster_labels,\n",
-        "    'Hierarchical_Cluster': hierarchical_labels\n",
-        "})\n",
-        "\n",
-        "print(\"Comparison of K-Means vs Hierarchical Clustering:\")\n",
-        "print(comparison_df.sort_values('State'))\n",
-        "\n",
-        "# Count agreements\n",
-        "agreements = sum(comparison_df['K_Means_Cluster'] == comparison_df['Hierarchical_Cluster'])\n",
-        "print(f\"\\nMethods agreed on {agreements} out of {len(comparison_df)} states ({agreements/len(comparison_df)*100:.1f}%)\")\n",
-        "```"
-      ],
-      "metadata": {
-        "id": "tJQ-C5GFYBRT"
-      },
-      "id": "tJQ-C5GFYBRT",
-      "execution_count": null,
-      "outputs": []
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "**Deliverable**: A paragraph explaining the key differences between what K-means and hierarchical clustering revealed.\n",
-        "\n",
-        "---\n",
-        "\n",
-        "## Exercise 6: Policy Brief Creation\n",
-        "**Time: 20 minutes | Product: Executive Summary**\n",
-        "\n",
-        "### Your Task\n",
-        "Synthesize your findings into a policy brief for Department of Justice leadership.\n",
-        "\n",
-        "### Code Framework for Final Visualization"
-      ],
-      "metadata": {
-        "id": "dx1fNhu4YD7-"
-      },
-      "id": "dx1fNhu4YD7-"
-    },
-    {
-      "cell_type": "code",
-      "source": [
-        "```python\n",
-        "# Create a comprehensive visualization\n",
-        "fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(15, 12))\n",
-        "\n",
-        "# Plot 1: Murder vs Assault by cluster\n",
-        "colors = ['red', 'blue', 'green', 'orange', 'purple']\n",
-        "for i in range(optimal_k):\n",
-        "    cluster_data = USArrests_clustered[USArrests_clustered['Cluster'] == i]\n",
-        "    ax1.scatter(cluster_data['Murder'], cluster_data['Assault'],\n",
-        "               c=colors[i], label=f'Cluster {i}', s=60, alpha=0.7)\n",
-        "ax1.set_xlabel('Murder Rate')\n",
-        "ax1.set_ylabel('Assault Rate')\n",
-        "ax1.set_title('Murder vs Assault by Crime Profile')\n",
-        "ax1.legend()\n",
-        "ax1.grid(True, alpha=0.3)\n",
-        "\n",
-        "# Plot 2: Urban Population vs Rape by cluster\n",
-        "for i in range(optimal_k):\n",
-        "    cluster_data = USArrests_clustered[USArrests_clustered['Cluster'] == i]\n",
-        "    ax2.scatter(cluster_data['UrbanPop'], cluster_data['Rape'],\n",
-        "               c=colors[i], label=f'Cluster {i}', s=60, alpha=0.7)\n",
-        "ax2.set_xlabel('Urban Population %')\n",
-        "ax2.set_ylabel('Rape Rate')\n",
-        "ax2.set_title('Urban Population vs Rape Rate by Crime Profile')\n",
-        "ax2.legend()\n",
-        "ax2.grid(True, alpha=0.3)\n",
-        "\n",
-        "# Plot 3: Cluster size comparison\n",
-        "cluster_sizes = USArrests_clustered['Cluster'].value_counts().sort_index()\n",
-        "ax3.bar(range(len(cluster_sizes)), cluster_sizes.values, color=colors[:len(cluster_sizes)])\n",
-        "ax3.set_xlabel('Cluster Number')\n",
-        "ax3.set_ylabel('Number of States')\n",
-        "ax3.set_title('Number of States in Each Crime Profile')\n",
-        "ax3.set_xticks(range(len(cluster_sizes)))\n",
-        "\n",
-        "# Plot 4: Average crime rates by cluster\n",
-        "cluster_means = USArrests_clustered.groupby('Cluster')[['Murder', 'Assault', 'Rape']].mean()\n",
-        "cluster_means.plot(kind='bar', ax=ax4)\n",
-        "ax4.set_xlabel('Cluster Number')\n",
-        "ax4.set_ylabel('Average Rate')\n",
-        "ax4.set_title('Average Crime Rates by Profile')\n",
-        "ax4.legend()\n",
-        "ax4.tick_params(axis='x', rotation=0)\n",
-        "\n",
-        "plt.tight_layout()\n",
-        "plt.show()\n",
-        "```"
-      ],
-      "metadata": {
-        "id": "N8bkxURpYHJF"
-      },
-      "id": "N8bkxURpYHJF",
-      "execution_count": null,
-      "outputs": []
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "### Your Policy Brief Template\n",
-        "\n",
-        "**EXECUTIVE SUMMARY: US State Crime Profile Analysis**\n",
-        "\n",
-        "**Key Findings:**\n",
-        "- We identified [X] distinct crime profiles among US states\n",
-        "- [State examples] represent the highest-risk profile\n",
-        "- [State examples] represent the lowest-risk profile\n",
-        "- Urban population [does/does not] strongly correlate with violent crime\n",
-        "\n",
-        "**Policy Recommendations:**\n",
-        "1. **High-Priority States**: [List and explain why]\n",
-        "2. **Resource Allocation**: [Suggest how to distribute federal crime prevention funds]\n",
-        "3. **Best Practice Sharing**: [Which states should learn from which others?]\n",
-        "\n",
-        "**Methodology Note**: Analysis used unsupervised clustering on 4 crime variables across 50 states, with data standardization to ensure fair comparison.\n",
-        "\n",
-        "**Deliverable**: A complete 1-page policy brief with your clustering insights and specific recommendations.\n"
-      ],
-      "metadata": {
-        "id": "rAy_Ye0WYLK0"
-      },
-      "id": "rAy_Ye0WYLK0"
-    }
-  ],
-  "metadata": {
-    "jupytext": {
-      "cell_metadata_filter": "-all",
-      "formats": "Rmd,ipynb",
-      "main_language": "python"
-    },
-    "kernelspec": {
-      "display_name": "Python 3 (ipykernel)",
-      "language": "python",
-      "name": "python3"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.10.4"
-    },
-    "colab": {
-      "provenance": []
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 5
-}

Reference files/week 7/w7_curriculum

@@ -1,178 +0,0 @@
-Unsupervised Learning: K-means and Hierarchical Clustering
-1. Course Overview
-The State Safety Profile Challenge
-In this week, we'll explore unsupervised machine learning through a compelling real-world challenge: Understanding crime patterns across US states without any predetermined categories.
-Unsupervised Learning: A type of machine learning where we find hidden patterns in data without being told what to look for. Think of it like being a detective who examines evidence without knowing what crime was committed - you're looking for patterns and connections that emerge naturally from the data.
-Example: Instead of being told "find violent states vs. peaceful states," unsupervised learning lets the data reveal its own natural groupings, like "states with high murder but low assault" or "urban states with moderate crime."
-Imagine you're a policy researcher working with the FBI's crime statistics. You have data on violent crime rates across all 50 US states - murder rates, assault rates, urban population percentages, and rape statistics. But here's the key challenge: you don't know how states naturally group together in terms of crime profiles.
-Your Mission: Discover hidden patterns in state crime profiles without any predefined classifications!
-The Challenge: Without any predetermined safety categories, you need to:
-●	Uncover natural groupings of states based on their crime characteristics
-●	Identify which crime factors tend to cluster together
-●	Understand regional patterns that might not follow obvious geographic boundaries
-●	Find states with surprisingly similar or different crime profiles
-Cluster: A group of similar things. In our case, states that have similar crime patterns naturally group together in a cluster.
-Example: You might discover that Alaska, Nevada, and Florida cluster together because they all have high crime rates despite being in different regions of the country.
-Why This Matters: Traditional approaches might group states by region (South, Northeast, etc.) or population size. But what if crime patterns reveal different natural groupings? What if some Southern states cluster more closely with Western states based on crime profiles? What if urban percentage affects crime differently than expected?
-Urban Percentage: The proportion of a state's population that lives in cities rather than rural areas.
-Example: New York has a high urban percentage (87%) while Wyoming has a low urban percentage (29%).
-What You'll Discover Through This Challenge
-●	Hidden State Safety Types: Use clustering to identify groups of states with similar crime profiles
-●	Crime Pattern Relationships: Find unexpected connections between different types of violent crime
-●	Urban vs. Rural Effects: Discover how urbanization relates to different crime patterns
-●	Policy Insights: Understand which states face similar challenges and might benefit from shared approaches
-Clustering: The process of grouping similar data points together. It's like organizing your music library - songs naturally group by genre, but clustering might reveal unexpected groups like "workout songs" or "rainy day music" that cross traditional genre boundaries.
-Core Techniques We'll Master
-K-Means Clustering: A method that divides data into exactly K groups (where you choose the number K). It's like being asked to organize 50 students into exactly 4 study groups based on their academic interests.
-Hierarchical Clustering: A method that creates a tree-like structure showing how data points relate to each other at different levels. It's like a family tree, but for data - showing which states are "cousins" and which are "distant relatives" in terms of crime patterns.
-Both K-Means and Hierarchical Clustering are examples of unsupervised learning.
- 
-2. K-Means Clustering
-
-What it does: Divides data into exactly K groups by finding central points (centroids).
-Central Points (Centroids): The "center" or average point of each group. Think of it like the center of a basketball team huddle - it's the point that best represents where all the players are standing.
-Example: If you have a cluster of high-crime states, the centroid might represent "average murder rate of 8.5, average assault rate of 250, average urban population of 70%."
-USArrests Example: Analyzing crime data across 50 states, you might discover 4 distinct state safety profiles:
-●	High Crime States (above average in murder, assault, and rape rates)
-●	Urban Safe States (high urban population but lower violent crime rates)
-●	Rural Traditional States (low urban population, moderate crime rates)
-●	Mixed Profile States (high in some crime types but not others)
-How to Read K-Means Results:
-●	Scatter Plot: Points (states) colored by cluster membership
-○	Well-separated colors indicate distinct state profiles
-○	Mixed colors suggest overlapping crime patterns
-●	Cluster Centers: Average crime characteristics of each state group
-●	Elbow Plot: Helps choose optimal number of state groupings
-Cluster Membership: Which group each data point belongs to. Like being assigned to a team - each state gets assigned to exactly one crime profile group.
-Example: Texas might be assigned to "High Crime States" while Vermont is assigned to "Rural Traditional States."
-Scatter Plot: A graph where each point represents one observation (in our case, one state). Points that are close together have similar characteristics.
-Elbow Plot: A graph that helps you choose the right number of clusters. It's called "elbow" because you look for a bend in the line that looks like an elbow joint.
-Key Parameters:
-python
-# Essential parameters from the lab
-KMeans(
-    n_clusters=4,        # Number of state safety profiles to discover
-    random_state=42,     # For reproducible results
-    n_init=20           # Run algorithm 20 times, keep best result
-)
-Parameters: Settings that control how the algorithm works. Like settings on your phone - you can adjust them to get different results.
-n_clusters: How many groups you want to create. You have to decide this ahead of time.
-random_state: A number that ensures you get the same results every time you run the analysis. Like setting a specific starting point so everyone gets the same answer.
-n_init: How many times to run the algorithm. The computer tries multiple starting points and picks the best result. More tries = better results.
-
-3. Hierarchical Clustering
-What it does: Creates a tree structure (dendrogram) showing how data points group together at different levels.
-Dendrogram: A tree-like diagram that shows how groups form at different levels. Think of it like a family tree, but for data. At the bottom are individuals (states), and as you go up, you see how they group into families, then extended families, then larger clans.
-Example: At the bottom level, you might see Vermont and New Hampshire grouped together. Moving up, they might join with Maine to form a "New England Low Crime" group. Moving up further, this group might combine with other regional groups.
-USArrests Example: Analyzing state crime patterns might reveal:
-●	Level 1: High Crime vs. Low Crime states
-●	Level 2: Within high crime: Urban-driven vs. Rural-driven crime patterns
-●	Level 3: Within urban-driven: Assault-heavy vs. Murder-heavy profiles
-How to Read Dendrograms:
-●	Height: Distance between groups when they merge
-○	Higher merges = very different crime profiles
-○	Lower merges = similar crime patterns
-●	Branches: Each split shows a potential state grouping
-●	Cutting the Tree: Draw a horizontal line to create clusters
-Height: In a dendrogram, height represents how different two groups are. Think of it like difficulty level - it takes more "effort" (higher height) to combine very different groups.
-Example: Combining two very similar states (like Vermont and New Hampshire) happens at low height. Combining very different groups (like "High Crime States" and "Low Crime States") happens at high height.
-Cutting the Tree: Drawing a horizontal line across the dendrogram to create a specific number of groups. Like slicing a layer cake - where you cut determines how many pieces you get.
-Three Linkage Methods:
-●	Complete Linkage: Measures distance between most different states (good for distinct profiles)
-●	Average Linkage: Uses average distance between all states (balanced approach)
-●	Single Linkage: Uses closest states (tends to create chains, often less useful)
-Linkage Methods: Different ways to measure how close or far apart groups are. It's like different ways to measure the distance between two cities - you could use the distance between the farthest suburbs (complete), the average distance between all neighborhoods (average), or the distance between the closest points (single).
-Example: When deciding if "High Crime Group" and "Medium Crime Group" should merge, complete linkage looks at the most different states between the groups, while average linkage looks at the typical difference.
-Choosing Between K-Means and Hierarchical:
-●	Use K-Means when: You want to segment states into specific number of safety categories for policy targeting
-●	Use Hierarchical when: You want to explore the natural structure of crime patterns without assumptions
-Segmentation: Dividing your data into groups for specific purposes. Like organizing students into study groups - you might want exactly 4 groups so each has a teaching assistant.
-Exploratory Analysis: Looking at data to discover patterns without knowing what you'll find. Like being an explorer in uncharted territory - you're not looking for a specific destination, just seeing what interesting things you can discover.
- 
-4. Data Exploration
-Step 1: Understanding Your Data
-Essential Checks (from the USArrests example):
-python
-# Check the basic structure
-print(data.shape)          # How many observations and variables?
-print(data.columns)        # What variables do you have?
-print(data.head())         # What do the first few rows look like?
-
-# Examine the distribution
-print(data.mean())         # Average values
-print(data.var())          # Variability
-print(data.describe())     # Full statistical summary
-Observations: Individual data points we're studying. In our case, each of the 50 US states is one observation.
-Variables: The characteristics we're measuring for each observation. In USArrests, we have 4 variables: Murder rate, Assault rate, Urban Population percentage, and Rape rate.
-Example: For California (one observation), we might have Murder=9.0, Assault=276, UrbanPop=91, Rape=40.6 (four variables).
-Distribution: How values are spread out. Like looking at test scores in a class - are most scores clustered around the average, or spread out widely?
-Variability (Variance): How much the values differ from each other. High variance means values are spread out; low variance means they're clustered together.
-Why This Matters: The USArrests data showed vastly different scales:
-●	Murder: Average 7.8, Variance 19
-●	Assault: Average 170.8, Variance 6,945
-●	This scale difference would dominate any analysis without preprocessing
-Scales: The range and units of measurement for different variables. Like comparing dollars ($50,000 salary) to percentages (75% approval rating) - they're measured very differently.
-Example: Assault rates are in the hundreds (like 276 per 100,000) while murder rates are single digits (like 7.8 per 100,000). Without adjustment, assault would seem much more important just because the numbers are bigger.
-Step 2: Data Preprocessing
-Standardization (Critical for clustering):
-python
-from sklearn.preprocessing import StandardScaler
-
-# Always scale when variables have different units
-scaler = StandardScaler()
-data_scaled = scaler.fit_transform(data)
-Standardization: Converting all variables to the same scale so they can be fairly compared. Like converting all measurements to the same units - instead of comparing feet to meters, you convert everything to inches.
-StandardScaler: A tool that transforms data so each variable has an average of 0 and standard deviation of 1. Think of it like grading on a curve - it makes all variables equally important.
-Example: After standardization, a murder rate of 7.8 might become 0.2, and an assault rate of 276 might become 1.5. Now they're on comparable scales.
-When to Scale:
-●	✅ Always scale when variables have different units (dollars vs. percentages)
-●	✅ Scale when variances differ by orders of magnitude
-●	❓ Consider not scaling when all variables are in the same meaningful units
-Orders of Magnitude: When one number is 10 times, 100 times, or 1000 times bigger than another. In USArrests, assault variance (6,945) is about 365 times bigger than murder variance (19) - that's two orders of magnitude difference.
-Step 3: Exploratory Analysis
-For K-Means Clustering:
-python
-# Try different numbers of clusters to find optimal K
-inertias = []
-K_range = range(1, 11)
-for k in K_range:
-    kmeans = KMeans(n_clusters=k, random_state=42, n_init=20)
-    kmeans.fit(data_scaled)
-    inertias.append(kmeans.inertia_)
-
-# Plot elbow curve
-plt.plot(K_range, inertias, 'bo-')
-plt.xlabel('Number of Clusters (K)')
-plt.ylabel('Within-Cluster Sum of Squares')
-plt.title('Elbow Method for Optimal K')
-Inertias: A measure of how tightly grouped each cluster is. Lower inertia means points in each cluster are closer together (better clustering). It's like measuring how close teammates stand to each other - closer teammates indicate better team cohesion.
-Within-Cluster Sum of Squares: The total distance from each point to its cluster center. Think of it as measuring how far each student sits from their group's center - smaller distances mean tighter, more cohesive groups.
-Elbow Method: A technique for choosing the best number of clusters. You plot the results and look for the "elbow" - the point where adding more clusters doesn't help much anymore.
-For Hierarchical Clustering:
-python
-# Create dendrogram to explore natural groupings
-from sklearn.cluster import AgglomerativeClustering
-from ISLP.cluster import compute_linkage
-from scipy.cluster.hierarchy import dendrogram
-
-hc = AgglomerativeClustering(distance_threshold=0, n_clusters=None, linkage='complete')
-hc.fit(data_scaled)
-linkage_matrix = compute_linkage(hc)
-
-plt.figure(figsize=(12, 8))
-dendrogram(linkage_matrix, color_threshold=-np.inf, above_threshold_color='black')
-plt.title('Hierarchical Clustering Dendrogram')
-AgglomerativeClustering: A type of hierarchical clustering that starts with individual points and gradually combines them into larger groups. Like building a pyramid from the bottom up.
-distance_threshold=0: A setting that tells the algorithm to build the complete tree structure without stopping early.
-Linkage Matrix: A mathematical representation of how the tree structure was built. Think of it as the blueprint showing how the dendrogram was constructed.
-Step 4: Validation Questions
-Before proceeding with analysis, ask:
-1.	Do the variables make sense together? (e.g., don't cluster height with income)
-2.	Are there obvious outliers that need attention?
-3.	Do you have enough data points? (Rule of thumb: at least 10x more observations than variables)
-4.	Are there missing values that need handling?
-Outliers: Data points that are very different from all the others. Like a 7-foot-tall person in a group of average-height people - they're so different they might skew your analysis.
-Example: If most states have murder rates between 1-15, but one state has a rate of 50, that's probably an outlier that needs special attention.
-Missing Values: Data points where we don't have complete information. Like a student who didn't take one of the tests - you need to decide how to handle that gap in the data.
-Rule of Thumb: A general guideline that works in most situations. For clustering, having at least 10 times more observations than variables helps ensure reliable results.
-

app/main.py

@@ -24,6 +24,7 @@ from app.pages import week_4
 from app.pages import week_5
 from app.pages import week_6
 from app.pages import week_7
+from app.pages import week_8
 # Page configuration
 st.set_page_config(
     page_title="Data Science Course App",
@@ -165,6 +166,8 @@ def show_week_content():
         week_6.show()
     elif st.session_state.current_week == 7:
         week_7.show()
+    elif st.session_state.current_week == 8:
+        week_8.show()
     else:
         st.warning("Content for this week is not yet available.")
 
@@ -177,7 +180,7 @@ def main():
         return
     
     # User is logged in, show course content
-    if st.session_state.current_week in [1, 2, 3, 4, 5, 6, 7]:
+    if st.session_state.current_week in [1, 2, 3, 4, 5, 6, 7, 8]:
         show_week_content()
     else:
         st.title("Data Science Research Paper Course")

app/pages/week_8.py

@@ -0,0 +1,564 @@
+import streamlit as st
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+import plotly.express as px
+import plotly.graph_objects as go
+from plotly.subplots import make_subplots
+
+def show():
+    st.title("Week 8: Research Paper Writing and LaTeX")
+    
+    # Introduction
+    st.header("Learning Objectives")
+    st.markdown("""
+    By the end of this week, you will be able to:
+    
+    **Remember (Knowledge):**
+    - Recall LaTeX syntax for document structure, figures, citations, and spacing
+    - Identify components of ML research papers (introduction, methods, results, conclusion, limitations)
+    - Recognize standard formatting requirements for academic conferences and journals
+    
+    **Understand (Comprehension):**
+    - Describe the purpose and audience for each section of a research paper
+    
+    **Apply (Application):**
+    - Format complete research papers in LaTeX with proper figures, tables, and citations
+    - Write clear methodology sections with sufficient detail for reproducibility
+    - Present experimental results using appropriate visualizations and statistical analysis
+    
+    **Analyze (Analysis):**
+    - Diagnose LaTeX formatting issues and resolve compilation errors
+    - Examine related work to identify research gaps and position their contributions
+    - Compare methodology approaches with existing methods
+    
+    **Evaluate (Evaluation):**
+    - Critically assess the validity and reliability of experimental design
+    - Evaluate the clarity and persuasiveness of written arguments
+    
+    **Create (Synthesis):**
+    - Produce research papers
+    - Develop compelling visualizations that effectively communicate complex ML concepts
+    - Synthesize technical knowledge into coherent research narratives
+    """)
+    
+    # Module 1: Research Paper Architecture
+    st.header("Module 1: Research Paper Architecture")
+    
+    st.markdown("""
+    Every section of your paper must answer specific questions that reviewers ask. Think of your paper as a conversation 
+    with skeptical experts who need convincing.
+    """)
+    
+    # Paper Structure Table
+    st.subheader("Research Paper Structure")
+    
+    paper_structure = {
+        "Section": ["🔥 Introduction", "🔬 Methods", "📊 Results", "🎯 Conclusion", "⚠️ Limitations"],
+        "Key Problems/Focus": [
+            "What problem are you solving? Why does it matter? How is your approach different?",
+            "How did you collect data? What analysis techniques? Can others replicate this?",
+            "What concrete findings emerged? How do they address your research questions?",
+            "What's the key takeaway? How does this advance the field? What are practical implications?",
+            "What are honest constraints? What biases might exist? What couldn't you address?"
+        ],
+        "Aim For": [
+            "Compelling motivation",
+            "Rigorous reproducibility",
+            "Clear evidence",
+            "Lasting impact",
+            "Honest transparency"
+        ]
+    }
+    
+    st.dataframe(pd.DataFrame(paper_structure))
+    
+    # Detailed Section Guidelines
+    st.subheader("Detailed Section Guidelines")
+    
+    # Introduction Section
+    with st.expander("🔥 Introduction: Building Compelling Motivation"):
+        st.markdown("""
+        **What is it:** The introduction is your paper's first impression and often determines whether reviewers continue reading.
+        
+        **Why this matters:** A weak introduction leads to immediate rejection, regardless of how brilliant your technical contribution might be.
+        
+        **What to do:**
+        1. Use the "inverted pyramid" approach
+        2. Start with broad context, then narrow to specific problem
+        3. Clearly articulate the gap in existing solutions
+        4. Present your approach as a logical response
+        5. Conclude with explicit contributions (3-4 bullet points)
+        
+        **Example Structure:**
+        ```
+        1. Broad context about the field
+        2. Specific problem you're addressing
+        3. Gap in existing solutions
+        4. Your approach as response to gap
+        5. Explicit contributions
+        ```
+        """)
+    
+    # Methods Section
+    with st.expander("🔬 Methods: Ensuring Rigorous Reproducibility"):
+        st.markdown("""
+        **What is it:** The methods section has evolved from simple description to detailed documentation that enables complete replication.
+        
+        **Why this matters:** Irreproducible research wastes community resources and undermines scientific credibility.
+        
+        **What to document:**
+        - Dataset specifics (exact version, preprocessing steps, train/validation/test splits)
+        - Model architecture details (layer sizes, activation functions, initialization schemes)
+        - Training procedures (optimization algorithm, learning rate schedules, batch sizes)
+        - Computational environment (hardware specifications, software versions, random seeds)
+        
+        **Write as if creating a recipe** that a competent colleague could follow to recreate your exact results.
+        """)
+    
+    # Results Section
+    with st.expander("📊 Results: Presenting Clear Evidence"):
+        st.markdown("""
+        **What is it:** The results section synthesizes your raw findings into compelling evidence for your claims.
+        
+        **Why this matters:** This section proves whether your methodology actually works and answers your research questions.
+        
+        **What to do:**
+        1. Organize results logically (general performance to specific analyses)
+        2. Start with overall model performance using standard metrics
+        3. Include detailed comparisons, ablation studies, and error analysis
+        4. Use clear visualizations with appropriate error bars
+        5. Report negative results honestly
+        6. Connect each finding back to your original research questions
+        """)
+    
+    # Conclusion Section
+    with st.expander("🎯 Conclusion: Creating Lasting Impact"):
+        st.markdown("""
+        **What is it:** The conclusion shapes how the research community understands and remembers your contribution.
+        
+        **Why this matters:** Your technical contribution only matters if others can understand its significance and apply it.
+        
+        **What to do:**
+        1. Begin with concise summary of key findings (2-3 sentences)
+        2. State how findings advance theoretical understanding or practical applications
+        3. Discuss broader implications beyond your specific problem domain
+        4. Suggest concrete directions for future research
+        5. Balance confidence with humility about scope
+        """)
+    
+    # Limitations Section
+    with st.expander("⚠️ Limitations: Demonstrating Honest Transparency"):
+        st.markdown("""
+        **What is it:** Acknowledging limitations shows scientific maturity and helps readers appropriately interpret your findings.
+        
+        **Why this matters:** Every study has constraints, and attempting to hide them makes reviewers suspicious.
+        
+        **Three types of limitations to address:**
+        1. **Scope limitations:** What populations, contexts, or problem types might your results not apply to?
+        2. **Methodological constraints:** Sample size issues, measurement limitations, or experimental design trade-offs
+        3. **Potential biases:** Dataset bias, researcher bias, or systematic errors in your approach
+        
+        **For each limitation:** Explain potential impact and suggest how future work could address it.
+        """)
+    
+    # Quick Reference Framework
+    st.subheader("Quick Reference Framework")
+    st.markdown("""
+    **Title → Problem → Gap → Method → Findings → Impact → Limitations**
+    
+    This progression ensures logical flow and helps readers follow your research narrative from motivation through contribution to appropriate interpretation.
+    """)
+    
+    # Module 2: LaTeX Introduction
+    st.header("Module 2: Introduction to LaTeX")
+    
+    st.markdown("""
+    **What is LaTeX?**
+    
+    Think of LaTeX as a sophisticated word processor that works differently from Microsoft Word or Google Docs. 
+    Instead of clicking buttons to format text, you write commands that tell the computer how to format your document.
+    """)
+    
+    # Why LaTeX
+    st.subheader("Why Learn LaTeX for Academic Writing?")
+    
+    latex_benefits = {
+        "Benefit": [
+            "Professional appearance",
+            "Mathematical notation",
+            "Reference management",
+            "Industry standard"
+        ],
+        "Description": [
+            "LaTeX automatically handles spacing, fonts, and layout to meet academic standards",
+            "Essential for ML papers with equations and formulas",
+            "Automatically formats citations and bibliographies",
+            "Most computer science conferences and journals expect LaTeX submissions"
+        ]
+    }
+    
+    st.dataframe(pd.DataFrame(latex_benefits))
+    
+    # LaTeX Code Examples
+    st.subheader("LaTeX Code Examples")
+    
+    # Basic Structure
+    with st.expander("Basic Document Structure"):
+        st.markdown("**LaTeX Code:**")
+        st.code("""
+\\documentclass{article}
+\\usepackage[utf8]{inputenc}
+\\usepackage{graphicx}
+\\title{Your Research Paper Title}
+\\author{Your Name}
+\\date{\\today}
+\\begin{document}
+\\maketitle
+\\section{Introduction}
+Your introduction text goes here.
+\\section{Methods}
+Your methods section goes here.
+\\section{Results}
+Your results section goes here.
+\\section{Conclusion}
+Your conclusion goes here.
+\\end{document}
+        """, language="latex")
+        
+        st.markdown("**Rendered Output:**")
+        st.markdown("""
+        <div style="border: 1px solid #ccc; padding: 40px; margin: 20px auto; background-color: white; font-family: 'Times New Roman', Times, serif; color: black; box-shadow: 0 0 10px rgba(0,0,0,0.1); max-width: 800px;">
+            <h1 style="text-align: center; font-size: 22px; font-weight: bold; margin-bottom: 10px; color: black;">Your Research Paper Title</h1>
+            <p style="text-align: center; font-size: 16px; margin-bottom: 30px; color: black;"><em>Your Name</em><br><em>Today's Date</em></p>
+            <h2 style="font-size: 18px; font-weight: bold; margin-top: 20px; margin-bottom: 10px; color: black;">1. Introduction</h2>
+            <p style="font-size: 16px; line-height: 1.6; color: black;">Your introduction text goes here.</p>
+            <h2 style="font-size: 18px; font-weight: bold; margin-top: 20px; margin-bottom: 10px; color: black;">2. Methods</h2>
+            <p style="font-size: 16px; line-height: 1.6; color: black;">Your methods section goes here.</p>
+            <h2 style="font-size: 18px; font-weight: bold; margin-top: 20px; margin-bottom: 10px; color: black;">3. Results</h2>
+            <p style="font-size: 16px; line-height: 1.6; color: black;">Your results section goes here.</p>
+            <h2 style="font-size: 18px; font-weight: bold; margin-top: 20px; margin-bottom: 10px; color: black;">4. Conclusion</h2>
+            <p style="font-size: 16px; line-height: 1.6; color: black;">Your conclusion goes here.</p>
+        </div>
+        """, unsafe_allow_html=True)
+    
+    # Sections and Subsections
+    with st.expander("Creating Sections and Subsections"):
+        st.markdown("**LaTeX Code:**")
+        st.code("""
+\\section{Introduction}          % Creates: 1. Introduction
+\\subsection{Background}         % Creates: 1.1 Background  
+\\subsubsection{Deep Learning}   % Creates: 1.1.1 Deep Learning
+
+% Tip: Overleaf shows section structure in the left panel for easy navigation
+        """, language="latex")
+        
+        st.markdown("**Rendered Output:**")
+        st.markdown("""
+        <div style="border: 1px solid #ddd; padding: 20px; background-color: white; font-family: 'Times New Roman', serif; color: black;">
+            <h2 style="color: black; font-size: 18px; font-weight: bold;">1. Introduction</h2>
+            <h3 style="color: black; font-size: 16px; font-weight: bold; padding-left: 20px;">1.1 Background</h3>
+            <h4 style="color: black; font-size: 16px; font-style: italic; padding-left: 40px;">1.1.1 Deep Learning</h4>
+            <p style="color: black; padding-left: 40px; margin-top: 10px;"><em>Tip: Overleaf shows section structure in the left panel for easy navigation</em></p>
+        </div>
+        """, unsafe_allow_html=True)
+    
+    # Figures
+    with st.expander("Adding Figures"):
+        st.markdown("**LaTeX Code:**")
+        st.code("""
+\\begin{figure}[h]
+\\centering
+\\includegraphics[width=0.8\\textwidth]{research_question.jpg}
+\\caption{The cycle of research from practical problem to research answer.}
+\\label{fig:research_cycle}
+\\end{figure}
+
+% Reference it in your text
+Figure~\\ref{fig:research_cycle} shows the relationship between problems, questions, and answers.
+        """, language="latex")
+        
+        st.markdown("**Rendered Output:**")
+        
+        # Center the image using columns
+        col1, col2, col3 = st.columns([1, 2, 1])
+        with col2:
+            st.image("assets/Pictures/research_question.jpg", width=384, caption="Figure 1: The cycle of research from practical problem to research answer.")
+        
+        st.markdown("""
+        <div style="background-color: white; font-family: 'Times New Roman', serif; color: black; padding: 0 20px 20px 20px;">
+             <p style="color: black; text-align: left;">Figure 1 shows the relationship between problems, questions, and answers.</p>
+        </div>
+        """, unsafe_allow_html=True)
+    
+    # Citations and Bibliography
+    with st.expander("Citations and Bibliography"):
+        st.markdown("**LaTeX Code:**")
+        st.code("""
+% In your main document
+\\usepackage{biblatex}
+\\addbibresource{sample.bib}
+
+% Cite a reference
+Our approach builds on recent work \\cite{einstein} and extends it by...
+
+% Print bibliography
+\\printbibliography
+
+% In sample.bib file:
+@article{einstein,
+  title={On the electrodynamics of moving bodies},
+  author={Einstein, Albert},
+  journal={Annalen der Physik},
+  volume={322},
+  number={10},
+  pages={891--921},
+  year={1905}
+}
+        """, language="latex")
+        
+        st.markdown("**Rendered Output:**")
+        st.markdown("""
+        <div style="border: 1px solid #ddd; padding: 20px; background-color: white; font-family: Times, 'Times New Roman', serif; color: black;">
+            <p style="color: black; margin-bottom: 1.5em;">Our approach builds on recent work [1] and extends it by...</p>
+            <h3 style="color: black; margin-bottom: 0.5em; font-weight: bold;">References</h3>
+            <p style="line-height: 1.6; padding-left: 2em; text-indent: -2em;">
+                [1] A. Einstein, "On the electrodynamics of moving bodies," <i>Annalen der Physik</i>, vol. 322, no. 10, pp. 891–921, 1905.
+            </p>
+        </div>
+        """, unsafe_allow_html=True)
+    
+    # Mathematical Equations
+    with st.expander("Mathematical Equations"):
+        st.markdown("**LaTeX Code:**")
+        st.code("""
+% Inline math
+The loss function $L(\\theta)$ is defined as...
+
+% Display math
+\\begin{equation}
+L(\\theta) = \\frac{1}{n} \\sum_{i=1}^{n} (y_i - f(x_i, \\theta))^2
+\\label{eq:loss}
+\\end{equation}
+
+% Reference the equation
+As shown in Equation~\\ref{eq:loss}, the loss function...
+        """, language="latex")
+        
+        st.markdown("**Rendered Output:**")
+        st.markdown("""
+        <div style="border: 1px solid #ddd; padding: 20px; background-color: white; font-family: 'Times New Roman', serif; color: black;">
+            <p style="color: black;">The loss function <em>L(&theta;)</em> is defined as...</p>
+            <div style="display: flex; justify-content: space-between; align-items: center; margin: 20px 0;">
+                <div style="flex-grow: 1; text-align: center;">
+                    <img src="https://latex.codecogs.com/svg.latex?L(\\theta)%20=%20\\frac{1}{n}%20\\sum_{i=1}^{n}%20(y_i%20-%20f(x_i,%20\\theta))^2" />
+                </div>
+                <div style="font-style: italic; color: black;">(1)</div>
+            </div>
+            <p style="color: black;">As shown in Equation (1), the loss function...</p>
+        </div>
+        """, unsafe_allow_html=True)
+    
+    # Tables
+    with st.expander("Creating Tables"):
+        st.markdown("**LaTeX Code:**")
+        st.code("""
+\\begin{table}[h]
+\\centering
+\\begin{tabular}{|l|c|r|}
+\\hline
+\\textbf{Method} & \\textbf{Accuracy} & \\textbf{Time (s)} \\\\
+\\hline
+Baseline & 85.2\\% & 120 \\\\
+Our Method & 89.7\\% & 95 \\\\
+\\hline
+\\end{tabular}
+\\caption{Performance comparison of different methods}
+\\label{tab:results}
+\\end{table}
+
+% Reference the table
+Table~\\ref{tab:results} shows the performance comparison...
+        """, language="latex")
+        
+        st.markdown("**Rendered Output:**")
+        st.markdown("""
+        <div style="border: 1px solid #ddd; padding: 20px; background-color: white; font-family: 'Times New Roman', serif; color: black;">
+            <div style="text-align: center; margin: 20px 0;">
+                <table style="border-collapse: collapse; width: 100%; max-width: 500px; margin: 0 auto;">
+                    <tr style="border: 1px solid #000;">
+                        <th style="border: 1px solid #000; padding: 8px; text-align: left; font-weight: bold;">Method</th>
+                        <th style="border: 1px solid #000; padding: 8px; text-align: center; font-weight: bold;">Accuracy</th>
+                        <th style="border: 1px solid #000; padding: 8px; text-align: right; font-weight: bold;">Time (s)</th>
+                    </tr>
+                    <tr style="border: 1px solid #000;">
+                        <td style="border: 1px solid #000; padding: 8px; text-align: left;">Baseline</td>
+                        <td style="border: 1px solid #000; padding: 8px; text-align: center;">85.2%</td>
+                        <td style="border: 1px solid #000; padding: 8px; text-align: right;">120</td>
+                    </tr>
+                    <tr style="border: 1px solid #000;">
+                        <td style="border: 1px solid #000; padding: 8px; text-align: left;">Our Method</td>
+                        <td style="border: 1px solid #000; padding: 8px; text-align: center;">89.7%</td>
+                        <td style="border: 1px solid #000; padding: 8px; text-align: right;">95</td>
+                    </tr>
+                </table>
+                <p style="margin-top: 10px; font-style: italic; color: black;">Table 1: Performance comparison of different methods</p>
+            </div>
+            <p style="color: black;">Table 1 shows the performance comparison...</p>
+        </div>
+        """, unsafe_allow_html=True)
+    
+    # Interactive LaTeX Practice
+    st.header("Interactive LaTeX Practice")
+    
+    st.markdown("""
+    Let's practice some common LaTeX commands. Try these exercises:
+    """)
+    
+    # Exercise 1: Basic Document
+    with st.expander("Exercise 1: Create a Basic Document"):
+        st.markdown("""
+        **Task:** Create a basic LaTeX document with title, author, and three sections.
+        
+        **Steps:**
+        1. Open Overleaf and create a new project
+        2. Replace the default content with your own
+        3. Add a title and your name
+        4. Create three sections: Introduction, Methods, Results
+        5. Add some placeholder text to each section
+        6. Compile to see your PDF
+        """)
+        
+        st.code("""
+\\documentclass{article}
+\\title{My First LaTeX Document}
+\\author{Your Name}
+\\date{\\today}
+
+\\begin{document}
+\\maketitle
+
+\\section{Introduction}
+This is the introduction section.
+
+\\section{Methods}
+This is the methods section.
+
+\\section{Results}
+This is the results section.
+
+\\end{document}
+        """, language="latex")
+    
+    # Exercise 2: Adding Figures
+    with st.expander("Exercise 2: Adding a Figure"):
+        st.markdown("""
+        **Task:** Add a figure to your document.
+        
+        **Steps:**
+        1. Upload an image to your Overleaf project
+        2. Add the figure code to your document
+        3. Add a caption and label
+        4. Reference the figure in your text
+        """)
+        
+        st.code("""
+\\begin{figure}[h]
+\\centering
+\\includegraphics[width=0.7\\textwidth]{your-image.png}
+\\caption{Description of your figure}
+\\label{fig:example}
+\\end{figure}
+
+As shown in Figure~\\ref{fig:example}, our results demonstrate...
+        """, language="latex")
+    
+    # Exercise 3: Citations
+    with st.expander("Exercise 3: Adding Citations"):
+        st.markdown("""
+        **Task:** Add citations to your document.
+        
+        **Steps:**
+        1. Create a .bib file with your references
+        2. Add the bibliography package to your document
+        3. Add citations in your text
+        4. Include the bibliography at the end
+        """)
+        
+        st.code("""
+% In your main document
+\\usepackage{biblatex}
+\\addbibresource{references.bib}
+
+% Add citations
+Recent work \\cite{smith2023} has shown that...
+
+\\printbibliography
+
+% In references.bib:
+@article{smith2023,
+  title={Recent advances in machine learning},
+  author={Smith, John and Johnson, Jane},
+  journal={Journal of ML Research},
+  year={2023}
+}
+        """, language="latex")
+    
+    # Common LaTeX Issues and Solutions
+    st.header("Common LaTeX Issues and Solutions")
+    
+    issues_solutions = {
+        "Issue": [
+            "Document won't compile",
+            "Figure not appearing",
+            "Citations not showing",
+            "Math equations not rendering",
+            "Bibliography not generating"
+        ],
+        "Common Cause": [
+            "Missing closing brace or bracket",
+            "Wrong filename or path",
+            "Missing \\printbibliography command",
+            "Missing math mode delimiters",
+            "Missing \\addbibresource command"
+        ],
+        "Solution": [
+            "Check for matching braces and brackets",
+            "Verify filename and upload to Overleaf",
+            "Add \\printbibliography at end of document",
+            "Use $ for inline, \\begin{equation} for display",
+            "Add \\addbibresource{filename.bib}"
+        ]
+    }
+    
+    st.dataframe(pd.DataFrame(issues_solutions))
+    
+    # Best Practices
+    st.header("Best Practices for Research Paper Writing")
+    
+    st.markdown("""
+    **Writing Tips:**
+    1. **Start with an outline** - Plan your paper structure before writing
+    2. **Write the methods first** - It's usually the easiest section
+    3. **Use clear, concise language** - Avoid jargon when possible
+    4. **Be specific** - Use concrete numbers and examples
+    5. **Revise multiple times** - Good writing is rewriting
+    
+    **LaTeX Tips:**
+    1. **Compile frequently** - Catch errors early
+    2. **Use meaningful labels** - fig:results is better than fig:1
+    3. **Keep backups** - Version control your LaTeX files
+    4. **Use templates** - Start with conference/journal templates
+    5. **Learn keyboard shortcuts** - Speed up your workflow
+    """)
+    
+    # Additional Resources
+    st.header("Additional Resources")
+    st.markdown("""
+    **LaTeX Resources:**
+    - [Overleaf Documentation](https://www.overleaf.com/learn)
+    - [LaTeX Wikibook](https://en.wikibooks.org/wiki/LaTeX)
+    - [CTAN (Comprehensive TeX Archive Network)](https://ctan.org/)
+    
+    """)