步驟4：生成文檔塊的嵌入

現(xiàn)在，我們使用sencencetransformer將每個文本塊轉(zhuǎn)換為嵌入式。這些嵌入代表了高維矢量空間中文本的含義，這對于以后搜索和檢索相關(guān)文檔很有用。

import json

# Function to scrape and extract data from the URLs
def scrape_data_from_links(links):
    scraped_data = []

    for link in links:
        try:
            response = requests.get(link)
            soup = BeautifulSoup(response.content, 'html.parser')

            # Example: Extract 'title' and 'content' (modify according to your needs)
            title = soup.find('title').get_text()
            content = soup.find('div', class_='content').get_text()  # Adjust selector

            # Store the extracted data
            scraped_data.append({
                'url': link,
                'title': title,
                'content': content
            })
        except requests.exceptions.RequestException as e:
            print(f"Error scraping {link}: {e}")

    return scraped_data

# Scrape data from the extracted links
scraped_contents = scrape_data_from_links(all_extracted_links)

# Save scraped data to a JSON file
with open('/content/scraped_data.json', 'w') as outfile:
    json.dump(scraped_contents, outfile, indent=4)

print("Data scraping complete.")

>步驟5：設(shè)置獵犬

>現(xiàn)在我們配置了獵犬。該組件根據(jù)用戶查詢搜索最相關(guān)的文本塊。它將最類似的文檔塊檢索到查詢中。

import requests
from bs4 import BeautifulSoup
from collections import deque

# Function to extract links using BFS
def bfs_link_extraction(start_url, max_depth=3):
    visited = set()  # To track visited links
    queue = deque([(start_url, 0)])  # Queue to store URLs and current depth
    all_links = []

    while queue:
        url, depth = queue.popleft()

        if depth > max_depth:
            continue

        # Fetch the content of the URL
        try:
            response = requests.get(url)
            soup = BeautifulSoup(response.content, 'html.parser')

            # Extract all links in the page
            links = soup.find_all('a', href=True)
            for link in links:
                full_url = link['href']
                if full_url.startswith('http') and full_url not in visited:
                    visited.add(full_url)
                    queue.append((full_url, depth + 1))
                    all_links.append(full_url)
        except requests.exceptions.RequestException as e:
            print(f"Error fetching {url}: {e}")

    return all_links

# Start the BFS from the homepage
start_url = 'https://www.example.com'  # Replace with the actual homepage URL
all_extracted_links = bfs_link_extraction(start_url)
print(f"Extracted {len(all_extracted_links)} links.")

• 獵犬使用相似性搜索來從矢量商店找到最相關(guān)的塊。
• 參數(shù)k = 6意味著它將返回與查詢最相關(guān)的前6個塊。

>步驟6：創(chuàng)建提示模板

接下來，我們創(chuàng)建一個提示模板，該模板將格式化語言模型的輸入。該模板既包含上下文（檢索到的塊）和用戶的查詢，從而指導(dǎo)模型以僅基于提供的上下文生成答案。

Extracted 1500 links.

• > ChatPromptTemplate格式格式化模型的輸入，以強(qiáng)調(diào)答案僅基于給定上下文的需求。
{context}將被相關(guān)的文本塊替換，{Quartion}將被用戶的查詢替換。

在此步驟中，我們初始化OpenAI GPT模型。該模型將根據(jù)獵犬提供的結(jié)構(gòu)化上下文生成答案。

import json

# Function to scrape and extract data from the URLs
def scrape_data_from_links(links):
    scraped_data = []

    for link in links:
        try:
            response = requests.get(link)
            soup = BeautifulSoup(response.content, 'html.parser')

            # Example: Extract 'title' and 'content' (modify according to your needs)
            title = soup.find('title').get_text()
            content = soup.find('div', class_='content').get_text()  # Adjust selector

            # Store the extracted data
            scraped_data.append({
                'url': link,
                'title': title,
                'content': content
            })
        except requests.exceptions.RequestException as e:
            print(f"Error scraping {link}: {e}")

    return scraped_data

# Scrape data from the extracted links
scraped_contents = scrape_data_from_links(all_extracted_links)

# Save scraped data to a JSON file
with open('/content/scraped_data.json', 'w') as outfile:
    json.dump(scraped_contents, outfile, indent=4)

print("Data scraping complete.")

[
    {
        "url": "https://www.example.com/page1",
        "title": "Page 1 Title",
        "content": "This is the content of the first page. It contains 
        information about integrations and features."
    },
    {
        "url": "https://www.example.com/page2",
        "title": "Page 2 Title",
        "content": "Here we describe the functionalities of the product. 
        It includes various use cases and capabilities."
    }
]

import json

# Load the scraped JSON file containing the web data
with open('/content/scraped_contents_zluri_all_links.json', 'r') as file:
    data = json.load(file)