Трекер цен на Amazon: Firecrawl, Streamlit, Supabase и GitHub Actions

Title:

URL Source: https://raw.githubusercontent.com/firecrawl/firecrawl/main/examples/blog-articles/amazon-price-tracking/notebook.ipynb

Markdown Content: { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to Build an Automated Amazon Price Tracking Tool in Python For Free\n", "## That sends alerts to your phone and keeps price history" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What Shall We Build in This Tutorial?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is a lot to be said about the psychology of discounts. For example, buying a discounted item even though we don't need it isn't saving money at all. That's walking into the oldest trap sellers use to increase sales. However, there are legitimate cases where waiting for a price drop on items you actually need makes perfect sense.\n", "\n", "The challenge is that e-commerce websites run flash sales and temporary discounts constantly, but these deals often disappear as quickly as they appear. Missing these brief windows of opportunity can be frustrating.\n", "\n", "That's where automation comes in. In this guide, we'll build a Python application that monitors product prices across any e-commerce website and instantly notifies you when prices drop on items you're actually interested in. Here is a sneak peak of the app:\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The app looks pretty dull, doesn't it? Well, no worries because it is fully functional:\n", "- It has a minimalistic UI to add or remove products from the tracker\n", "- A simple dashboard to display price history for each product\n", "- Controls for setting the price drop threshold in percentages\n", "- A notification system that sends Discord alerts when a tracked item's price drops\n", "- A scheduling system that updates the product prices on an interval you specify\n", "- Runs for free for as long as you want\n", "\n", "Even though the title says \"Amazon price tracker\" (full disclosure: I was forced to write that for SEO purposes), the app will work for any e-commerce website you can imagine (except Ebay, for some reason). \n", "\n", "So, let's get started building this Amazon price tracker. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The Toolstack We Will Use" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The app's code will be written fully in Python and its libraries:\n", "\n", "- [Streamlit](streamlit.io) for the UI\n", "- [Firecrawl](firecrawl.dev) for AI-based scraping of e-commerce websites\n", "- [SQLAlchemy](https://www.sqlalchemy.org/) for database management\n", "\n", "Apart from Python, we will use these platforms:\n", "\n", "- Discord for notifications\n", "- GitHub for hosting the app\n", "- GitHub Actions for running the app on a schedule\n", "- Supabase for hosting a free Postgres database instance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Building an Amazon Price Tracker App Step-by-step" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since this project involves multiple components working together, we'll take a top-down approach rather than building individual pieces first. This approach makes it easier to understand how everything fits together, since we'll introduce each tool only when it's needed. The benefits of this strategy will become clear as we progress through the tutorial." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: Setting up the environment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, let's create a dedicated environment on our machines to work on the project:\n", "\n", "```bash\n", "mkdir automated-price-tracker\n", "cd automated-price-tracker\n", "python -m venv .venv\n", "source .venv/bin/activate\n", "```\n", "\n", "These commands create a working directory and activate a virtual environment. Next, create a new script called `ui.py` for designing the user interface with Streamlit.\n", "\n", "```bash\n", "touch ui.py\n", "```\n", "\n", "Then, install Streamlit:\n", "\n", "```bash\n", "pip install streamlit\n", "```\n", "\n", "Next, create a `requirements.txt` file and add Streamlit as the first dependency:\n", "\n", "```bash\n", "touch requirements.txt\n", "echo \"streamlit\" >> requirements.txt\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since the code will be hosted on GitHub, we need to initialize Git and create a `.gitignore` file:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "git init\n", "touch .gitignore\n", "echo \".venv\" >> .gitignore # Add the virtual env folder\n", "git commit -m \"Initial commit\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 2: Add a sidebar to the UI for product input" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the final product one more time:\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It has two sections: the sidebar and the main dashboard. Since the first thing you do when launching this app is adding products, we will start building the sidebar first. Open `ui.py` and paste the following code:\n", "\n", "```python\n", "import streamlit as st\n", "\n", "# Set up sidebar\n", "with st.sidebar:\n", " st.title(\"Add New Product\")\n", " product_url = st.text_input(\"Product URL\")\n", " add_button = st.button(\"Add Product\")\n", "\n", "# Main content\n", "st.title(\"Price Tracker Dashboard\")\n", "st.markdown(\"## Tracked Products\")\n", "```\n", "\n", "The code snippet above sets up a basic Streamlit web application with two main sections. In the sidebar, it creates a form for adding new products with a text input field for the product URL and an \"Add Product\" button. The main content area contains a dashboard title and a section header for tracked products. The code uses Streamlit's `st.sidebar` context manager to create the sidebar layout and basic Streamlit components like `st.title`, `st.text_input`, and `st.button` to build the user interface elements.\n", "\n", "To see how this app looks like, run the following command:\n", "\n", "```bash\n", "streamlit run ui.py\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's add a commit to save our progress:\n", "\n", "```bash\n", "git add .\n", "git commit -m \"Add a sidebar to the basic UI\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 3: Add a feature to check if input URL is valid\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the next step, we want to add some restrictions to the input field like checking if the passed URL is valid. For this, create a new file called `utils.py` where we write additional utility functions for our app:\n", "\n", "```bash\n", "touch utils.py\n", "```\n", "\n", "Inside the script, paste following code:\n", "\n", "```bash\n", "# utils.py\n", "from urllib.parse import urlparse\n", "import re\n", "\n", "\n", "def is_valid_url(url: str) -> bool:\n", " try:\n", " # Parse the URL\n", " result = urlparse(url)\n", "\n", " # Check if scheme and netloc are present\n", " if not all([result.scheme, result.netloc]):\n", " return False\n", "\n", " # Check if scheme is http or https\n", " if result.scheme not in [\"http\", \"https\"]:\n", " return False\n", "\n", " # Basic regex pattern for domain validation\n", " domain_pattern = (\n", " r\"^[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(\\.[a-zA-Z]{2,})+$\"\n", " )\n", " if not re.match(domain_pattern, result.netloc):\n", " return False\n", "\n", " return True\n", "\n", " except Exception:\n", " return False\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above function `is_valid_url()` validates URLs by checking several criteria:\n", "\n", "1. It verifies the URL has both a scheme (`http`/`https`) and domain name\n", "2. It ensures the scheme is specifically `http` or `https`\n", "3. It validates the domain name format using regex to check for valid characters and TLD\n", "4. It returns True only if all checks pass, False otherwise\n", "\n", "Let's use this function in our `ui.py` file. Here is the modified code:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "import streamlit as st\n", "from utils import is_valid_url\n", "\n", "\n", "# Set up sidebar\n", "with st.sidebar:\n", " st.title(\"Add New Product\")\n", " product_url = st.text_input(\"Product URL\")\n", " add_button = st.button(\"Add Product\")\n", "\n", " if add_button:\n", " if not product_url:\n", " st.error(\"Please enter a product URL\")\n", " elif not is_valid_url(product_url):\n", " st.error(\"Please enter a valid URL\")\n", " else:\n", " st.success(\"Product is now being tracked!\")\n", "\n", "# Main content\n", "...\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is what's new:\n", "\n", "1. We added URL validation using the `is_valid_url()` function from `utils.py`\n", "2. When the button is clicked, we perform validation:\n", " - Check if URL is empty\n", " - Validate URL format using `is_valid_url()`\n", "3. User feedback is provided through error/success messages:\n", " - Error shown for empty URL\n", " - Error shown for invalid URL format \n", " - Success message when URL passes validation\n", "\n", "Rerun the Streamlit app again and see if our validation works. Then, return to your terminal to commit the changes we've made:\n", "\n", "```bash\n", "git add .\n", "git commit -m \"Add a feature to check URL validity\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 4: Scrape the input URL for product details" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When a valid URL is entered and the add button is clicked, we need to implement product scraping functionality instead of just showing a success message. The system should:\n", "\n", "1. Immediately scrape the product URL to extract key details:\n", " - Product name\n", " - Current price\n", " - Main product image\n", " - Brand name\n", " - Other relevant attributes\n", "\n", "2. Store these details in a database to enable:\n", " - Regular price monitoring\n", " - Historical price tracking\n", " - Price change alerts\n", " - Product status updates\n", "\n", "For the scraper, we will use [Firecrawl](firecrawl.dev), an AI-based scraping API for extracting webpage data without HTML parsing. This solution provides several advantages:\n", "\n", "1. No website HTML code analysis required for element selection\n", "2. Resilient to HTML structure changes through AI-based element detection\n", "3. Universal compatibility with product webpages due to structure-agnostic approach \n", "4. Reliable website blocker bypass via robust API infrastructure\n", "\n", "First, create a new file called `scraper.py`:\n", "\n", "```bash\n", "touch scraper.py\n", "```\n", "\n", "Then, install these three libraries:\n", "\n", "```bash\n", "pip install firecrawl-py pydantic python-dotenv\n", "echo \"firecrawl-py\\npydantic\\npython-dotenv\\n\" >> requirements.txt # Add them to dependencies\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`firecrawl-py` is the Python SDK for Firecrawl scraping engine, `pydantic` is a data validation library that helps enforce data types and structure through Python class definitions, and `python-dotenv` is a library that loads environment variables from a `.env` file into your Python application.\n", "\n", "With that said, head over to the Firecrawl website and [sign up for a free account](https://www.firecrawl.dev/) (the free plan will work fine). You will be given an API key, which you should copy. \n", "\n", "Then, create a `.env` file in your terminal and add the API key as an environment variable:\n", "\n", "```bash\n", "touch .env\n", "echo \"FIRECRAWL_API_KEY='YOUR-API-KEY-HERE' >> .env\"\n", "echo \".env\" >> .gitignore # Ignore .env files in Git\n", "```\n", "\n", "The `.env` file is used to securely store sensitive configuration values like API keys that shouldn't be committed to version control. By storing the Firecrawl API key in `.env` and adding it to `.gitignore`, we ensure it stays private while still being accessible to our application code. This is a security best practice to avoid exposing credentials in source control.\n", "\n", "Now, we can start writing the `scraper.py`:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "from firecrawl import FirecrawlApp\n", "from pydantic import BaseModel, Field\n", "from dotenv import load_dotenv\n", "from datetime import datetime\n", "\n", "load_dotenv()\n", "\n", "app = FirecrawlApp()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, `load_dotenv()` function reads the `.env` file you have in your working directory and loads the environment variables inside, including the Firecrawl API key. When you create an instance of `FirecrawlApp` class, the API key is automatically detected to establish a connection between your script and the scraping engine in the form of the `app` variable.\n", "\n", "Now, we create a Pydantic class (usually called a model) that defines the details we want to scrape from each product:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "class Product(BaseModel):\n", " \"\"\"Schema for creating a new product\"\"\"\n", "\n", " url: str = Field(description=\"The URL of the product\")\n", " name: str = Field(description=\"The product name/title\")\n", " price: float = Field(description=\"The current price of the product\")\n", " currency: str = Field(description=\"Currency code (USD, EUR, etc)\")\n", " main_image_url: str = Field(description=\"The URL of the main image of the product\")\n", "```\n", "\n", "Pydantic models may be completely new to you, so let's break down the `Product` model:\n", "\n", "- The `url` field stores the product page URL we want to track\n", "- The `name` field stores the product title/name that will be scraped\n", "- The `price` field stores the current price as a float number\n", "- The `currency` field stores the 3-letter currency code (e.g. USD, EUR)\n", "- The `main_image_url` field stores the URL of the product's main image\n", "\n", "Each field is typed and has a description that documents its purpose. The `Field` class from Pydantic allows us to add metadata like descriptions to each field. These descriptions are especially important for Firecrawl since it uses them to automatically locate the relevant HTML elements containing the data we want. \n", "\n", "Now, let's create a function to call the engine to scrape URL's based on the schema above:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "def scrape_product(url: str):\n", " extracted_data = app.scrape_url(\n", " url,\n", " params={\n", " \"formats\": [\"extract\"],\n", " \"extract\": {\"schema\": Product.model_json_schema()},\n", " },\n", " )\n", "\n", " # Add the scraping date to the extracted data\n", " extracted_data[\"extract\"][\"timestamp\"] = datetime.utcnow()\n", "\n", " return extracted_data[\"extract\"]\n", "\n", "\n", "if __name__ == \"__main__\":\n", " product = \"https://www.amazon.com/gp/product/B002U21ZZK/\"\n", "\n", " print(scrape_product(product))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code above defines a function called `scrape_product` that takes a URL as input and uses it to scrape product information. Here's how it works:\n", "\n", "The function calls `app.scrape_url` with two parameters:\n", "1. The product URL to scrape\n", "2. A params dictionary that configures the scraping:\n", " - It specifies we want to use the \"extract\" format\n", " - It provides our `Product` Pydantic model schema as the extraction template as a JSON object\n", "\n", "The scraper will attempt to find and extract data that matches our Product schema fields - the URL, name, price, currency, and image URL.\n", "\n", "The function returns just the \"extract\" portion of the scraped data, which contains the structured product information. `extract` returns a dictionary to which we add the date of the scraping as it will be important later on.\n", "\n", "Let's test the script by running it:\n", "\n", "```bash\n", "python scraper.py\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should get an output like this:\n", "\n", "```python\n", "{\n", " 'url': 'https://www.amazon.com/dp/B002U21ZZK', \n", " 'name': 'MOVA Globe Earth with Clouds 4.5\"', \n", " 'price': 212, \n", " 'currency': 'USD', \n", " 'main_image_url': 'https://m.media-amazon.com/images/I/41bQ3Y58y3L._AC_.jpg', \n", " 'timestamp': '2024-12-05 13-20'\n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output shows that a [MOVA Globe](https://www.amazon.com/dp/B002U21ZZK) costs $212 USD on Amazon at the time of writing this article. You can test the script for any other website that contains the information we are looking (except Ebay):\n", "\n", "- Price\n", "- Product name/title\n", "- Main image URL\n", "\n", "One key advantage of using Firecrawl is that it returns data in a consistent dictionary format across all websites. Unlike HTML-based scrapers like BeautifulSoup or Scrapy which require custom code for each site and can break when website layouts change, Firecrawl uses AI to understand and extract the requested data fields regardless of the underlying HTML structure. \n", "\n", "Finish this step by committing the new changes to Git:\n", "\n", "```bash\n", "git add .\n", "git commit -m \"Implement a Firecrawl scraper for products\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 5: Storing new products in a PostgreSQL database" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we want to check product prices regularly, we need to have an online database. In this case, Postgres is the best option since it's reliable, scalable, and has great support for storing time-series data like price histories.\n", "\n", "There are many platforms for hosting Postgres instances but the one I find the easiest and fastest to set up is Supabase. So, please head over to [the Supabase website](https://supabase.com) and create your free account. During the sign-up process, you will be given a password, which you should save somewhere safe on your machine. \n", "\n", "\n", "Then, in a few minutes, your free Postgres instance comes online. To connect to this instance, click on Home in the left sidebar and then, \"Connect\":\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You will be shown your database connection string with a placeholder for the password you copied. You should paste this string in your `.env` file with your password added to the `.env` file:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "echo POSTGRES_URL=\"THE-SUPABASE-URL-STRING-WITH-YOUR-PASSWORD-ADDED\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, the easiest way to interact with this database is through SQLAlchemy. Let's install it:\n", "\n", "```bash\n", "pip install \"sqlalchemy==2.0.35\" psycopg2-binary\n", "echo \"psycopg2-binary\\nsqlalchemy==2.0.35\" >> requirements.txt\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Note: [SQLAlchemy](https://sqlalchemy.org) is a Python SQL toolkit and Object-Relational Mapping (ORM) library that lets us interact with databases using Python code instead of raw SQL. For our price tracking project, it provides essential features like database connection management, schema definition through Python classes, and efficient querying capabilities. This makes it much easier to store and retrieve product information and price histories in our Postgres database.\n", "\n", "After the installation, create a new `database.py` file for storing database-related functions:\n", "\n", "```bash\n", "touch database.py\n", "```\n", "\n", "Let's populate this script:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "from sqlalchemy import create_engine, Column, String, Float, DateTime, ForeignKey\n", "from sqlalchemy.orm import sessionmaker, relationship, declarative_base\n", "from datetime import datetime\n", "\n", "Base = declarative_base()\n", "\n", "\n", "class Product(Base):\n", " __tablename__ = \"products\"\n", "\n", " url = Column(String, primary_key=True)\n", " prices = relationship(\n", " \"PriceHistory\", back_populates=\"product\", cascade=\"all, delete-orphan\"\n", " )\n", "\n", "\n", "class PriceHistory(Base):\n", " __tablename__ = \"price_histories\"\n", "\n", " id = Column(String, primary_key=True)\n", " product_url = Column(String, ForeignKey(\"products.url\"))\n", " name = Column(String, nullable=False)\n", " price = Column(Float, nullable=False)\n", " currency = Column(String, nullable=False)\n", " main_image_url = Column(String)\n", " timestamp = Column(DateTime, nullable=False)\n", " product = relationship(\"Product\", back_populates=\"prices\")\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "The code above defines two SQLAlchemy models for our price tracking database:\n", "\n", "The Product model represents items we want to track, with the product URL as the primary key. It has a one-to-many relationship with price histories (which means each product in `products` can have multiple price history entry in `price_histories`).\n", "\n", "The `PriceHistory` model stores individual price points over time. Each record contains:\n", "- A unique ID as primary key\n", "- The product URL as a foreign key linking to the `Product`\n", "- The product name\n", "- The price value and currency\n", "- The main product image URL\n", "- A timestamp of when the price was recorded\n", "\n", "The relationship between `Product` and `PriceHistory` is bidirectional, allowing easy navigation between related records. The `cascade` setting ensures price histories are deleted when their product is deleted.\n", "\n", "These models provide the structure for storing and querying our price tracking data in a PostgreSQL database using SQLAlchemy's ORM capabilities." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we define a `Database` class with a singe `add_product` method:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "class Database:\n", " def __init__(self, connection_string):\n", " self.engine = create_engine(connection_string)\n", " Base.metadata.create_all(self.engine)\n", " self.Session = sessionmaker(bind=self.engine)\n", "\n", " def add_product(self, url):\n", " session = self.Session()\n", " try:\n", " # Create the product entry\n", " product = Product(url=url)\n", " session.merge(product) # merge will update if exists, insert if not\n", " session.commit()\n", " finally:\n", " session.close()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "The `Database` class above provides core functionality for managing product data in our PostgreSQL database. It takes a connection string in its constructor to establish the database connection using SQLAlchemy.\n", "\n", "The `add_product` method allows us to store new product URLs in the database. It uses SQLAlchemy's `merge` functionality which intelligently handles both inserting new products and updating existing ones, preventing duplicate entries.\n", "\n", "The method carefully manages database sessions, ensuring proper resource cleanup by using `try`/`finally` blocks. This prevents resource leaks and maintains database connection stability." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's use this method inside the sidebar of our UI. Switch to `ui.py` and make the following adjustments:\n", "\n", "First, update the imports to load the Database class and initialize it:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "import os\n", "import streamlit as st\n", "\n", "from utils import is_valid_url\n", "from database import Database\n", "from dotenv import load_dotenv\n", "\n", "load_dotenv()\n", "\n", "with st.spinner(\"Loading database...\"):\n", " db = Database(os.getenv(\"POSTGRES_URL\"))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code integrates the `Database` class into the Streamlit UI by importing required dependencies and establishing a database connection. The database URL is loaded securely from environment variables using `python-dotenv`. The `Database` class creates or updates the tables we specified in `database.py` after being initialized.\n", "\n", "The database initialization process is wrapped in a Streamlit spinner component to maintain responsiveness while establishing the connection. This provides visual feedback during the connection setup period, which typically requires a brief initialization time." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, in the sidebar code, we only need to add a single line of code to add the product to the database if the URL is valid:\n", "\n", "```python\n", "# Set up sidebar\n", "with st.sidebar:\n", " st.title(\"Add New Product\")\n", " product_url = st.text_input(\"Product URL\")\n", " add_button = st.button(\"Add Product\")\n", "\n", " if add_button:\n", " if not product_url:\n", " st.error(\"Please enter a product URL\")\n", " elif not is_valid_url(product_url):\n", " st.error(\"Please enter a valid URL\")\n", " else:\n", " db.add_product(product_url) # This is the new line\n", " st.success(\"Product is now being tracked!\")\n", "```\n", "\n", "In the final `else` block that runs when the product URL is valid, we call the `add_product` method to store the product in the database.\n", "\n", "Let's commit everything:\n", "\n", "```bash\n", "git add .\n", "git commit -m \"Add a Postgres database integration for tracking product URLs\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 6: Storing price histories for new products" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, after the product is added to the `products` table, we want to add its details and its scraped price to the `price_histories` table. \n", "\n", "First, switch to `database.py` and add a new method for creating entries in the `PriceHistories` table:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "class Database:\n", " ... # the rest of the class\n", "\n", " def add_price(self, product_data):\n", " session = self.Session()\n", " try:\n", " price_history = PriceHistory(\n", " id=f\"{product_data['url']}_{product_data['timestamp']}\",\n", " product_url=product_data[\"url\"],\n", " name=product_data[\"name\"],\n", " price=product_data[\"price\"],\n", " currency=product_data[\"currency\"],\n", " main_image_url=product_data[\"main_image_url\"],\n", " timestamp=product_data[\"timestamp\"],\n", " )\n", " session.add(price_history)\n", " session.commit()\n", " finally:\n", " session.close()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `add_price` method takes a dictionary containing product data (which is returned by our scraper) and creates a new entry in the `PriceHistory` table. The entry's ID is generated by combining the product URL with a timestamp. The method stores essential product information like name, price, currency, image URL, and the timestamp of when the price was recorded. It uses SQLAlchemy's session management to safely commit the new price history entry to the database.\n", "\n", "Now, we need to add this functionality to the sidebar as well. In `ui.py`, add a new import statement that loads the `scrape_product` function from `scraper.py`:\n", "\n", "```python\n", "... # The rest of the imports\n", "from scraper import scrape_product\n", "```\n", "\n", "Then, update the `else` block in the sidebar again:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "with st.sidebar:\n", " st.title(\"Add New Product\")\n", " product_url = st.text_input(\"Product URL\")\n", " add_button = st.button(\"Add Product\")\n", "\n", " if add_button:\n", " if not product_url:\n", " st.error(\"Please enter a product URL\")\n", " elif not is_valid_url(product_url):\n", " st.error(\"Please enter a valid URL\")\n", " else:\n", " db.add_product(product_url)\n", " with st.spinner(\"Added product to database. Scraping product data...\"):\n", " product_data = scrape_product(product_url)\n", " db.add_price(product_data)\n", " st.success(\"Product is now being tracked!\")\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now when a user enters a product URL and clicks the \"Add Product\" button, several things happen:\n", "\n", "1. The URL is validated to ensure it's not empty and is properly formatted.\n", "2. If valid, the URL is added to the products table via `add_product()`.\n", "3. The product page is scraped immediately to get current price data.\n", "4. This initial price data is stored in the price history table via `add_price()`.\n", "5. The user sees loading spinners and success messages throughout the process.\n", "\n", "This gives us a complete workflow for adding new products to track, including capturing their initial price point. The UI provides clear feedback at each step and handles errors gracefully.\n", "\n", "Check that everything is working the way we want it and then, commit the new changes:\n", "\n", "```bash\n", "git add .\n", "git commit -m \"Add a feature to track product prices after they are added\"\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 7: Displaying each product's price history in the main dashboard" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the final product shown in the introduction once again:\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Apart from the sidebar, the main dashboard shows each product's price history visualized with a Plotly line plot where the X axis is the timestamp while the Y axis is the prices. Each line plot is wrapped in a Streamlit component that includes buttons for removing the product from the database or visiting its source URL. \n", "\n", "In this step, we will implement the plotting feature and leave the two buttons for a later section. First, add a new method to the `Database` class for retrieving the price history for each product:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "class Database:\n", " ... # The rest of the code\n", "\n", " def get_price_history(self, url):\n", " \"\"\"Get price history for a product\"\"\"\n", " session = self.Session()\n", " try:\n", " return (\n", " session.query(PriceHistory)\n", " .filter(PriceHistory.product_url == url)\n", " .order_by(PriceHistory.timestamp.desc())\n", " .all()\n", " )\n", " finally:\n", " session.close()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The method queries the price histories table based on product URL, orders the rows in descending order (oldest first) and returns the results. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, add another method for retrieving all products from the `products` table:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "class Database:\n", " ...\n", " \n", " def get_all_products(self):\n", " session = self.Session()\n", " try:\n", " return session.query(Product).all()\n", " finally:\n", " session.close()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The idea is that every time our Streamlit app is opened, the main dashboard queries all existing products from the database and render their price histories with line charts in dedicated components. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To create the line charts, we need Plotly and Pandas, so install them in your environment:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "pip install pandas plotly\n", "echo \"pandas\\nplotly\" >> requirements.txt\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Afterward, import them at the top of `ui.py` along with other existing imports:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "import pandas as pd\n", "import plotly.express as px\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, switch to `ui.py` and paste the following snippet of code after the Main content section:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# Main content\n", "st.title(\"Price Tracker Dashboard\")\n", "st.markdown(\"## Tracked Products\")\n", "\n", "# Get all products\n", "products = db.get_all_products()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, after the page title and subtitle is shown, we are retrieving all products from the database. Let's loop over them:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# Create a card for each product\n", "for product in products:\n", " price_history = db.get_price_history(product.url)\n", " if price_history:\n", " # Create DataFrame for plotting\n", " df = pd.DataFrame(\n", " [\n", " {\"timestamp\": ph.timestamp, \"price\": ph.price, \"name\": ph.name}\n", " for ph in price_history\n", " ]\n", " )\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For each product, we get their price history with `db.get_price_history` and then, convert this data into a dataframe with three columns:\n", "\n", "- Timestamp\n", "- Price\n", "- Product name\n", "\n", "This makes plotting easier with Plotly. Next, we create a Streamlit expander component for each product:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# Create a card for each product\n", "for product in products:\n", " price_history = db.get_price_history(product.url)\n", " if price_history:\n", " ...\n", " # Create a card-like container for each product\n", " with st.expander(df[\"name\"][0], expanded=False):\n", " st.markdown(\"---\")\n", " col1, col2 = st.columns([1, 3])\n", "\n", " with col1:\n", " if price_history[0].main_image_url:\n", " st.image(price_history[0].main_image_url, width=200)\n", " st.metric(\n", " label=\"Current Price\",\n", " value=f\"{price_history[0].price} {price_history[0].currency}\",\n", " )\n", "```\n", "\n", "The expander shows the product name as its title and contains:\n", "\n", "1. A divider line\n", "2. Two columns:\n", " - Left column: Product image (if available) and current price metric\n", " - Right column (