Building Balita of the Week: Creating an AI-Powered News Quiz Website from Scratch

Recently, I launched Balita of the Week, an app that allows users to answer weekly quizzes on both local and global news. To create the quiz, I initially planned to use AI to identify trending topics, generate questions, and source relevant content. However, I quickly realized that achieving this with a simple prompt was not feasible.

In this article, I will outline how I was able to create a news quiz using a blend of Natural Language Processing (NLP) techniques and Large Language Models (LLMs).

Overview

Below is the diagram showing the final process of the quiz creator.

Basically, here are the steps which I'll discuss in detail:

1. Fetching Headlines: Use a web crawler to gather headlines from articles published in the past week.
2. Topic Modeling: Input these list of headlines to BERTopic, a topic modeling technique that groups these headlines into topics.
3. Topic Grading: Use GPT-4o to evaluate the topics based on - based on impact and relevance, generating a "Topic Score" and search terms for the next step.
4. Relevant Article Search: Using the search terms generated, find related articles using the Bing News API. Select the best article based on criteria like recency and credibility of the source to a name a few (called the "Article Score")
5. Question Generation: Use GPT-4o to create quiz questions, multiple-choice answers, and explanations, along with a Filipino translation.
6. Upload the quiz to the database.

To illustrate, I'll be using the quiz for August 2, 2024 as an example.

Fetching Headlines

The first step is to gather headlines from reputable local news organizations. We focus on headlines because some news sites list all articles created on a given day, including just the headline and category.

Using BeautifulSoup, we extract headlines, their categories, and their published dates. These headlines are then clustered into topics.

Each week, 500-600 headlines are collected. To remove irrelevant topics, puff pieces, and editorials, only headlines from the local news, world news, technology, and sports are selected.

Topic Modeling

Initially, I used DBSCAN (Density-Based Spatial Clustering of Applications with Noise) to cluster articles into topics, followed by c-TF-IDF (Class-based Term frequency-inverse document frequency) to extract key terms from the cluster to create the topic name. However, the results were suboptimal due to insufficient data pre-processing.

So I explored other options and ultimately decided to use BERTopic, a more advanced modeling technique that combines HBSCAN (which uses a hierarchal approach to clustering compared to DBSCAN) with c-TF-IDF. This method produces dense clusters with easily interpretable topics while keeping important words in the topic descriptions. Basically, I utilized BERTopic by:

1. Encoding the headlines into a 384-dim vector space using the all-MiniLM-L6-v2 model
2. Reducing the dimensions of the embeddings using UMAP.
3. Cluster the documents using HDBSCAN.
4. Apply c-TF-IDF to extract the most important words per topic (representative words).

(To learn more about BERTopic's algorithm, you can check out their documentation)

For our use case, we've limited the number of topics to be between 8 and 15. We also set that headlines would only be in English.

Below are the 15 topics generated for the week of August 2, 2024. Take note that the topic with the id -1 includes articles that didn't fit the other topics and this will be disregarded.

We can see that most articles are related to the Olympics, with its opening ceremony held during this week (July 27).

BERTopic also provides visualization options. Below is the graph showing the embedded headlines reduced to two dimensions.

Some key observations:

• The model can be pretty specific in identifying topics. For example, the headlines about the PH Budget for 2025 was split into two distinct topics.
  • 12_2025_dbm_proposed_p6 which contains headlines about the actual budget, the deliberations happening in congress, and some opinions from groups.
  • 13_marcos_054_expenses_p1 which also contains the budget headlines (but with Pres. Marcos included in the headline) as well as some articles about the budget allotted for the president's travel expenses.
• Due to the topic limit, some topics are combined. For instance, the topic 5_philhealth_regulate_approved_cannabis. For the 29 headlines in this topic, the headlines talk about:
  • Cutting down the premiums for PhilHealth (The Universal Healthcare system of the Philippines).
  • The Supreme Court ruling that the FDA is the sole regulator for tobacco products
  • The turnover of PhilHealth funds to other government projects
  • The House approving the Medical Cannabis Act
  • Some other topics about government regulation, diseases, and health advice

BERTopics also visualizes topics over time, which can be insightful for understanding trends.

• We cans see that the 2024 olympics was discussed the whole week, but peaked by July 31 when events with Philippine athletes are commencing.
• 1_duterte_guo_pogo_roque peaked during July 29-30 when Harry Roque was in a Senate Hearing about the POGO issue related to Bamban Mayor Alice Guo and indirectly related to former President Rodrigo Duterte.
• The Venezuelan elections started on Jul 29, so local news sources started focusing on it during that time.

For our purposes, we focus on the following datapoints for grading the topics:

• Topic name
• Headline count
• Representative words
• Representative headlines

Grading and Selecting Topics

To grade the topics, we use the following formula to calculate the Topic Score.

The score is a weighted normalized score based on the number of headlines in the topic ($c_h$) and the normalized impact score ($s_{impact}$).

The impact score ($s_{impact}$) is determined by GPT-4o, which evaluates each topic on a scale of 0-100 based on the following criteria:

• Impact: Prioritizes topics that represent significant changes or shifts in the status quo.
• Relevance: Scores higher for topics that engage a broad audience or are widely discussed.
• Urgency: Gives higher scores to topics addressing immediate issues or crises.
• Rep Docs Type: Scores lower for puff pieces, editorials, listicles, and clickbait.
• Conflict: Give a higher score to more engaging stories.

Here is the final ranking of the topics for August 2, 2024:

(Note that $w_c$ is set at 0.3)

Even though the relevance score for 0_paris_olympic_ceremony_delgaco wasn’t the highest, it ranked first due to the number of headlines. Other top topics include Typhoon Carina and the Gaza conflict.

Along with the impact score, the prompt also generates a 5-6 word search term for finding relevant articles. We select the top 8-10 topics for the quiz, based on a score threshold (currently set at 0.4).

Find Relevant Articles

Although we have representative headlines, they are not the best way to select articles because they come from a limited set of sources. Instead, we use the search terms generated by the LLM to find articles through the Bing News Search API.

We aim to retrieve 5 articles per search term. If fewer than 2 articles are returned, we shorten the search terms until we get at least 3 articles. This usually occurs when there are multiple topics grouped together. For example, the topic 4_carina_india_victims_mudslides with the search term "India mudslide toll Carina NDRRMC landslides" covers both the mudslides/landslides that occur in India and the Philippines (during Typhoon Carina).

Once we have enough articles, we calculate the Article Score ($s_{article}$):

Source Score

$\hat{s}_{source}$ is the weighted score based on the credibility of the news source.

where:

Search Rank Score

$\hat{s}_{search}$ is the normalized weighted score based on the search rank given by the Bing News Search API.

We use rank normalization to transform the article's rank $r_{search}$ into a score between $\frac{1}{|r|}$ to $1$, where $|r|$ is the number of articles returned by the api.

Publish Date Score

$\hat{s}_{date}$ is the normalized weighted score based on the recency of the article. We then normalize this rank to get the date score which is then multiplied to the weight to get $\hat{s}_{date}$.

Weights

Current weights prioritize search rank ($w_{search} = 0.6$), followed by source credibility ($w_{source} = 0.3$), and recency ($w_{date} = 0.1$).

Offsets

Additional factors are considered as offsets ($b$), such as image availability, source icon, and article length.

To fit the UI, we deduct the score when the article has no image ($b_{img}$) or if the news source has no icon provided ($b_{src}$). To assure that we have enough information about the topic for question generation, we reduce the final score if the word count of the article is less than 500 ($b_{len}$).

For example, here are the five articles found for the search term "Paris 2024 Olympic ceremony Pacquiao exit":

The article from philstar.com was selected due to its high score, attributed to being from a credible source and the most recent. In contrast, the article from Manila Times received a negative score since it's shorter than our threshold.

Generating Questions

With the selected articles, we use GPT-4o to generate a multiple-choice question based on the top article. The response includes:

• A question structured with a brief introduction.
• Four answer choices, with only one correct answer.
• A one-paragraph explanation providing context, shown after the user answers.

Using the Olympics-related article as an example, the LLM generated the following question:

The question focuses on the controversy surrounding the Games' Opening Ceremony.

To refine the question generation, I made several adjustments to the prompt, including:

• Specifying the type of questions to avoid opinionated responses.
  • Some examples: "Why is the action relevant?", "How would this make a certain group feel?"
• Ensuring the correct answer isn’t included in the question.
• Providing translation guidelines to improve the Filipino version's quality. For example, if a phrase is commonly used in English, is a proper noun, or is a technical term, use the English word or phrase. (e.g, United States instead of Estados Unidos or Department of Foreign Affairs instead of Kagawaran ng Ugnayang Panlabas)

Uploading Quizzes

The final step is to upload the quiz to the Firebase Firestore Database, where analytics and user feedback are also stored.

Final Product

Once uploaded, users can access the quiz on the Balita of the Week website. Here are some screenshots:

Human in the Loop

I also developed a portal for more controlled quiz creation. This interface allows users to:

• Select topics manually rather than relying solely on the top-ranked topics.

• Modify search terms and choose the best article.

• Generate new questions or add specific instructions for prompt creation.

Conclusion

Through this project, I've gained valuable insights into leveraging AI and LLMs to build amazing apps. While LLMs are powerful tools, they are not the ultimate solution. Combining them with traditional methodologies can greatly enhance prompt responses. Additionally, I utilized React Server Components and NextJS' app router to develop faster and more efficient web applications, significantly improving the overall user experience.

Ready to take the quiz? Take the latest quiz here and let's see how well you know the news!

Want to learn more? Have any questions? Feel free to reach out to me at carl@carldegs.com