Backend Development
Python Tutorial
An article will guide you through the urllib library in Python (operating URLs)
An article will guide you through the urllib library in Python (operating URLs)
Jul 25, 2023 pm 02:08 PM
1. Manipulating URLs
urllib provides a series of functions for manipulating URLs. Function. Classify related content.
2. Get()
urllib’s request The module can grab URL content very conveniently, that is, send a GET request to the specified page, and then return an HTTP response:
For example, Crawl the Douban URLhttps://api.growingio.com/v2/22c937bbd8ebd703f2d8e9445f7dfd03/web/pv?stm=1593747087078 and return the response ：##
from urllib import request
with request.urlopen('https://api.growingio.com/v2/22c937bbd8ebd703f2d8e9445f7dfd03/web/pv?stm=1593747087078') as f:
data = f.read()
print('Status:', f.status, f.reason)
for k, v in f.getheaders():
print('%s: %s' % (k, v))
print('Data:', data.decode('utf-8'))You can see the headers and JSON data of the HTTP response:
If you want to simulate the browser sending a GET request, you need to use the Request object by going to RequestBy adding HTTP headers to the object, you can disguise the request as a browser. For example, simulate iPhone 6 to request the Douban homepage:
from urllib import request
req = request.Request('http://www.douban.com/')
req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')
with request.urlopen(req) as f:
print('Status:', f.status, f.reason)
for k, v in f.getheaders():
print('%s: %s' % (k, v))
print('Data:', f.read().decode('utf-8')) In this way, Douban will return the mobile version of the web page suitable for iPhone:
##3. Post()
If you want to send a request via POST, you only need to pass in the parameters data in bytes.
模擬一個微博登錄，先讀取登錄的郵箱和口令，然后按照weibo.cn的登錄頁的格式以username=xxx&password=xxx的編碼傳入：
from urllib import request, parse
print('Login to weibo.cn...')
#電子郵件
email = input('Email: ')
#密碼
passwd = input('Password: ')
#相關(guān)的參數(shù)
login_data = parse.urlencode([
('username', email),
('password', passwd),
('entry', 'mweibo'),
('client_id', ''),
('savestate', '1'),
('ec', ''),
('pagerefer', 'https://passport.weibo.cn/signin/welcome?entry=mweibo&r=http%3A%2F%2Fm.weibo.cn%2F')
])
#網(wǎng)址請求
req = request.Request('https://passport.weibo.cn/sso/login')
req.add_header('Origin', 'https://passport.weibo.cn')
#構(gòu)造User-Agent
req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')
req.add_header('Referer', 'https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=http%3A%2F%2Fm.weibo.cn%2F')
with request.urlopen(req, data=login_data.encode('utf-8')) as f:
print('Status:', f.status, f.reason)
for k, v in f.getheaders():
print('%s: %s' % (k, v))
print('Data:', f.read().decode('utf-8'))如果登錄成功，獲得的響應(yīng)如下：
如果登錄失敗，獲得的響應(yīng)如下：
四、Handler
如果還需要更復(fù)雜的控制，比如通過一個Proxy去訪問網(wǎng)站，需要利用ProxyHandler來處理，示例代碼如下：
import urllib.request
# 構(gòu)建了兩個代理Handler，一個有代理IP，一個沒有代理IP
httpproxy_handler = urllib.request.ProxyHandler({"https": "27.191.234.69:9999"})
nullproxy_handler = urllib.request.ProxyHandler({})
# 定義一個代理開關(guān)
proxySwitch = True
# 通過 urllib.request.build_opener()方法使用這些代理Handler對象，創(chuàng)建自定義opener對象
# 根據(jù)代理開關(guān)是否打開，使用不同的代理模式
if proxySwitch:
opener = urllib.request.build_opener(httpproxy_handler)
else:
opener = urllib.request.build_opener(nullproxy_handler)
request = urllib.request.Request("http://www.baidu.com/")
# 1. 如果這么寫，只有使用opener.open()方法發(fā)送請求才使用自定義的代理，而urlopen()則不使用自定義代理。
response = opener.open(request)
# 2. 如果這么寫，就是將opener應(yīng)用到全局，之后所有的，不管是opener.open()還是urlopen() 發(fā)送請求，都將使用自定義代理。
# urllib.request.install_opener(opener)
# response = urllib.request.urlopen(request)
# 獲取服務(wù)器響應(yīng)內(nèi)容
html = response.read().decode("utf-8")
# 打印結(jié)果
print(html)如果代理成功返回網(wǎng)址的信息。
If the URL is wrong or the proxy address is wrong, return to the interface below.
#5. Summary
? ? Using Python language can help everyone learn Python better. The function provided by urllib is to use programs to execute various HTTP requests. If you want to simulate a browser to complete a specific function, you need to disguise the request as a browser. The method of camouflage is to first monitor the requests sent by the browser, and then camouflage them based on the browser's request header. The User-Agent header is used to identify the browser.
The above is the detailed content of An article will guide you through the urllib library in Python (operating URLs). For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
PHP calls AI intelligent voice assistant PHP voice interaction system construction
Jul 25, 2025 pm 08:45 PM
User voice input is captured and sent to the PHP backend through the MediaRecorder API of the front-end JavaScript; 2. PHP saves the audio as a temporary file and calls STTAPI (such as Google or Baidu voice recognition) to convert it into text; 3. PHP sends the text to an AI service (such as OpenAIGPT) to obtain intelligent reply; 4. PHP then calls TTSAPI (such as Baidu or Google voice synthesis) to convert the reply to a voice file; 5. PHP streams the voice file back to the front-end to play, completing interaction. The entire process is dominated by PHP to ensure seamless connection between all links.
How to use PHP combined with AI to achieve text error correction PHP syntax detection and optimization
Jul 25, 2025 pm 08:57 PM
To realize text error correction and syntax optimization with AI, you need to follow the following steps: 1. Select a suitable AI model or API, such as Baidu, Tencent API or open source NLP library; 2. Call the API through PHP's curl or Guzzle and process the return results; 3. Display error correction information in the application and allow users to choose whether to adopt it; 4. Use php-l and PHP_CodeSniffer for syntax detection and code optimization; 5. Continuously collect feedback and update the model or rules to improve the effect. When choosing AIAPI, focus on evaluating accuracy, response speed, price and support for PHP. Code optimization should follow PSR specifications, use cache reasonably, avoid circular queries, review code regularly, and use X
python seaborn jointplot example
Jul 26, 2025 am 08:11 AM
Use Seaborn's jointplot to quickly visualize the relationship and distribution between two variables; 2. The basic scatter plot is implemented by sns.jointplot(data=tips,x="total_bill",y="tip",kind="scatter"), the center is a scatter plot, and the histogram is displayed on the upper and lower and right sides; 3. Add regression lines and density information to a kind="reg", and combine marginal_kws to set the edge plot style; 4. When the data volume is large, it is recommended to use "hex"
PHP integrated AI emotional computing technology PHP user feedback intelligent analysis
Jul 25, 2025 pm 06:54 PM
To integrate AI sentiment computing technology into PHP applications, the core is to use cloud services AIAPI (such as Google, AWS, and Azure) for sentiment analysis, send text through HTTP requests and parse returned JSON results, and store emotional data into the database, thereby realizing automated processing and data insights of user feedback. The specific steps include: 1. Select a suitable AI sentiment analysis API, considering accuracy, cost, language support and integration complexity; 2. Use Guzzle or curl to send requests, store sentiment scores, labels, and intensity information; 3. Build a visual dashboard to support priority sorting, trend analysis, product iteration direction and user segmentation; 4. Respond to technical challenges, such as API call restrictions and numbers
python list to string conversion example
Jul 26, 2025 am 08:00 AM
String lists can be merged with join() method, such as ''.join(words) to get "HelloworldfromPython"; 2. Number lists must be converted to strings with map(str, numbers) or [str(x)forxinnumbers] before joining; 3. Any type list can be directly converted to strings with brackets and quotes, suitable for debugging; 4. Custom formats can be implemented by generator expressions combined with join(), such as '|'.join(f"[{item}]"foriteminitems) output"[a]|[
python connect to sql server pyodbc example
Jul 30, 2025 am 02:53 AM
Install pyodbc: Use the pipinstallpyodbc command to install the library; 2. Connect SQLServer: Use the connection string containing DRIVER, SERVER, DATABASE, UID/PWD or Trusted_Connection through the pyodbc.connect() method, and support SQL authentication or Windows authentication respectively; 3. Check the installed driver: Run pyodbc.drivers() and filter the driver name containing 'SQLServer' to ensure that the correct driver name is used such as 'ODBCDriver17 for SQLServer'; 4. Key parameters of the connection string
python pandas melt example
Jul 27, 2025 am 02:48 AM
pandas.melt() is used to convert wide format data into long format. The answer is to define new column names by specifying id_vars retain the identification column, value_vars select the column to be melted, var_name and value_name, 1.id_vars='Name' means that the Name column remains unchanged, 2.value_vars=['Math','English','Science'] specifies the column to be melted, 3.var_name='Subject' sets the new column name of the original column name, 4.value_name='Score' sets the new column name of the original value, and finally generates three columns including Name, Subject and Score.
Optimizing Python for Memory-Bound Operations
Jul 28, 2025 am 03:22 AM
Pythoncanbeoptimizedformemory-boundoperationsbyreducingoverheadthroughgenerators,efficientdatastructures,andmanagingobjectlifetimes.First,usegeneratorsinsteadofliststoprocesslargedatasetsoneitematatime,avoidingloadingeverythingintomemory.Second,choos
