From Words to Meaningful Comparison

• 31%What is text similarity?
  Understanding when two sentences are close, related, or different.
• 32%Why computers cannot directly understand meaning
  Computers need numbers, not raw human language.
• 33%Converting text into numbers
  The first big idea behind NLP models.
• 34%Bag of Words idea
  Represent a sentence by counting words.
• 35%CountVectorizer concept
  Use Python to automatically create word-count vectors.
• 36%What is Cosine Similarity?
  Compare the direction of two vectors.
• 37%Comparing two sentences
  Calculate a similarity score between sentences.
• 38%Mini project
  Build a Similar Sentence Finder.
• 39%Real use cases
  Search, recommendation, doubt matching, chatbot matching.
• 40%Revision and test
  Check whether the student can explain and code the concept.

Cosine similarity compares the direction of two vectors. If two text vectors point in a similar direction, the similarity score is high. If they point in very different directions, the score is low.

cosine similarity = 1      means very similar direction
cosine similarity = 0      means no useful similarity
cosine similarity near 0   means mostly different

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity

sentences = [
    "I like machine learning",
    "I enjoy learning AI",
    "The cat is sleeping"
]

vectorizer = CountVectorizer()
vectors = vectorizer.fit_transform(sentences)

similarity = cosine_similarity(vectors)

print("Vocabulary:")
print(vectorizer.get_feature_names_out())

print("\nVectors:")
print(vectors.toarray())

print("\nSimilarity Matrix:")
print(similarity)

When we compare three sentences with each other, Python gives a matrix. Each row and column represents one sentence.

This example is useful for Programmers Picnic AI-ML Classes. A student asks a question. The program compares the question with stored lesson titles.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity

student_question = "What is Python used for?"

lesson_titles = [
    "Python for AI and Machine Learning",
    "Introduction to HTML and CSS",
    "Sorting algorithms in Python",
    "How to deploy Angular on GitHub Pages",
    "Natural Language Processing basics"
]

all_texts = [student_question] + lesson_titles

vectorizer = CountVectorizer()
vectors = vectorizer.fit_transform(all_texts)

scores = cosine_similarity(vectors[0:1], vectors[1:]).flatten()

best_index = scores.argmax()
best_score = scores[best_index]
best_lesson = lesson_titles[best_index]

print("Student question:")
print(student_question)

print("\nBest matching lesson:")
print(best_lesson)

print("\nSimilarity score:")
print(best_score)

Project Goal

The student enters one sentence. The program compares it with a list of stored sentences. Then it finds the most similar sentence.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity

stored_sentences = [
    "Python is used for artificial intelligence",
    "HTML is used to create web pages",
    "CSS is used to style websites",
    "Machine learning helps computers learn from data",
    "Natural language processing works with human language",
    "Sorting algorithms arrange data in order"
]

user_sentence = input("Enter your sentence: ")

all_sentences = [user_sentence] + stored_sentences

vectorizer = CountVectorizer()
vectors = vectorizer.fit_transform(all_sentences)

similarity_scores = cosine_similarity(vectors[0:1], vectors[1:]).flatten()

best_index = similarity_scores.argmax()
best_match = stored_sentences[best_index]
best_score = similarity_scores[best_index]

print("\nYour sentence:")
print(user_sentence)

print("\nMost similar stored sentence:")
print(best_match)

print("\nSimilarity score:")
print(best_score)

print("\nAll scores:")
for sentence, score in zip(stored_sentences, similarity_scores):
    print(round(score, 3), "->", sentence)

Use the embedded editor below. Paste the code examples, run them, and change the sentences. Try your own lesson titles from Programmers Picnic.

Students should now be able to explain:

• What text similarity means.
• Why text must be converted into numbers.
• What Bag of Words means.
• What CountVectorizer does.
• What cosine similarity means.
• How to compare one sentence with many stored sentences.
• How search and recommendation systems use similarity.