Better Text Search with TF-IDF

• 41%Problem with simple word counting
  Every word gets equal treatment, even weak words.
• 42%What are common words?
  Words that appear everywhere and do not add strong meaning.
• 43%Stop words
  Words like is, am, are, the, a, an, of, to.
• 44%Why all words are not equally important
  Some words carry meaning. Some words only support grammar.
• 45%What is TF?
  Term Frequency means how often a word appears in a document.
• 46%What is IDF?
  Inverse Document Frequency means how rare or special a word is.
• 47%What is TF-IDF?
  TF-IDF gives higher importance to useful and meaningful words.
• 48%TF-IDF Vectorizer in Python
  Use TfidfVectorizer from scikit-learn.
• 49%Better search using TF-IDF + Cosine Similarity
  Rank documents by meaning-heavy words.
• 50%Revision and mini test
  Check whether students can explain and code the idea.

Stop words are very common words that usually do not carry strong topic meaning. They are important for grammar, but often weak for search ranking.

In search ranking, some words should have more importance than others. Below is a beginner-friendly visualization.

This example compares a search query with different documents and finds the best match.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

documents = [
    "Python is used for machine learning",
    "Machine learning is a part of artificial intelligence",
    "HTML is used to create web pages",
    "CSS is used to style web pages"
]

query = "Python machine learning"

all_texts = [query] + documents

vectorizer = TfidfVectorizer(stop_words="english")
vectors = vectorizer.fit_transform(all_texts)

scores = cosine_similarity(vectors[0:1], vectors[1:]).flatten()

best_index = scores.argmax()

print("Query:")
print(query)

print("\nBest matching document:")
print(documents[best_index])

print("\nScore:")
print(scores[best_index])

print("\nAll scores:")
for document, score in zip(documents, scores):
    print(round(score, 3), "->", document)

This example searches through lesson titles from Programmers Picnic AI-ML Classes.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

lesson_titles = [
    "Python basics for beginners",
    "HTML and CSS introduction",
    "Machine Learning with Python",
    "Angular deployment on GitHub Pages",
    "NLP text similarity using cosine similarity",
    "TF IDF search ranking in natural language processing",
    "Sorting algorithms in Python",
    "NumPy arrays for AI and ML"
]

query = input("Search for a lesson: ")

all_texts = [query] + lesson_titles

vectorizer = TfidfVectorizer(stop_words="english")
vectors = vectorizer.fit_transform(all_texts)

scores = cosine_similarity(vectors[0:1], vectors[1:]).flatten()

best_index = scores.argmax()
best_lesson = lesson_titles[best_index]
best_score = scores[best_index]

print("\nYour search:")
print(query)

print("\nBest matching lesson:")
print(best_lesson)

print("\nSimilarity score:")
print(round(best_score, 3))

print("\nRanked results:")

ranked = sorted(
    zip(lesson_titles, scores),
    key=lambda item: item[1],
    reverse=True
)

for title, score in ranked:
    print(round(score, 3), "->", title)

A search system should not only find one result. It should rank all results from most useful to least useful.

Project Goal

The student enters a search query. The program compares it with a list of lesson titles and descriptions. Then it returns the best matching lessons in ranked order.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

lessons = [
    {
        "title": "Python Basics",
        "description": "Learn variables, input, output, arithmetic operators and beginner programming."
    },
    {
        "title": "NumPy for AI and ML",
        "description": "Learn arrays, matrix operations, numerical computing and AI data handling."
    },
    {
        "title": "Machine Learning with Python",
        "description": "Learn supervised learning, training data, prediction and model evaluation."
    },
    {
        "title": "NLP Text Similarity",
        "description": "Learn CountVectorizer, cosine similarity and sentence comparison."
    },
    {
        "title": "TF-IDF Search Ranking",
        "description": "Learn stop words, term frequency, inverse document frequency and better text search."
    },
    {
        "title": "HTML and CSS Web Design",
        "description": "Learn how to create beautiful web pages using HTML and CSS."
    },
    {
        "title": "Angular GitHub Pages Deployment",
        "description": "Learn how to build and deploy Angular apps on GitHub Pages."
    }
]

search_texts = []

for lesson in lessons:
    combined_text = lesson["title"] + " " + lesson["description"]
    search_texts.append(combined_text)

query = input("What do you want to learn? ")

all_texts = [query] + search_texts

vectorizer = TfidfVectorizer(stop_words="english")
vectors = vectorizer.fit_transform(all_texts)

scores = cosine_similarity(vectors[0:1], vectors[1:]).flatten()

ranked_results = sorted(
    zip(lessons, scores),
    key=lambda item: item[1],
    reverse=True
)

print("\nSearch query:")
print(query)

print("\nBest lessons:")

for lesson, score in ranked_results:
    if score > 0:
        print("\nTitle:", lesson["title"])
        print("Description:", lesson["description"])
        print("Score:", round(score, 3))

Paste the Python examples into the embedded editor. Change the documents, lesson titles, and search queries.

Students should now be able to explain:

• Why simple word counting is limited.
• What common words and stop words are.
• Why all words are not equally important.
• What TF means.
• What IDF means.
• What TF-IDF means.
• How to use TfidfVectorizer in Python.
• How to build a better lesson search program.