PythondeSelenium WirewoShi tsutaWebsukureipingu
konogaidodeha, WebsukureipinguniSelenium WirewoShi Yong suruFang Fa woShuo Ming shi, rikuesutonointa-seputoyaDong De napurokishiro-te-shiyonnadonotopitsukuwoXi imasu.
- Selenium Wiretoha?
- nazeSelenium WirewoShi unoka?
- Selenium WirenoZhu naJi Neng
- rikuesutotoresuponsuhenoakusesu
- rikuesutotoresuponsunointa-seputo
- WebSocketnoJian Shi
- purokishinoGuan Li
- Selenium Wiredenopurokishiro-te-shiyon
- Yao Jian
- sutetsupu1: purokishiworandamuHua suru
- sutetsupu2: purokishiwoShe Ding suru
- sutetsupu3: ta-getsutope-ziniakusesusuru
- sutetsupu4: matometeShi Xing suru
- Bright Data Proxiesniyoruro-te-teingupurokishi
- WebsukureipinguniokeruSeleniumtoSelenium WirenoBi Jiao
- Jie Lun
Selenium Wiretoha?
Selenium Wire ha, SeleniumnoPythonbaindeinguXiang kenoKuo Zhang de, burauzanorikuesutowoZhi Yu dekiruyounishimasu. SeleniumwoShi Yong shinagara, Pythonko-dokaraZhi Jie , rikuesutotoresuponsunoLiang Fang woriarutaimuniinta-seputoshiteBian Geng dekimasu.
Note:
konoraiburarihasudenimentenansusareteimasenga, ikutsukanosukureipinguJi Shu yasukuriputodehaXian Zai moShi Yong sareteimasu.
nazeSelenium WirewoShi unoka?
burauzaniha, WebsukureipinguwoNan shikusuruikutsukanoZhi Xian gaarimasu. tatoeba, Ren Zheng Fu kipurokishiURLwoShe Ding shitari, onzahuraidepurokishiworo-te-shiyonshitaridekimasen. Selenium Wireha, Ren Jian noyu-za-toTong Yang nisaitotoyariQu risurukotode, koreranoZhi Xian woKe Fu surunoniYi Li chimasu.
WebsukureipingudeSelenium WirewoShi Yong subekiLi You noYi Bu haCi notooridesu.
- netsutowa-kutorahuitsukuheZhi Jie akusesudekiru: AJAXrikuesutotoresuponsuwoFen Xi *Jian Shi *Bian Geng shi, Jia Zhi arude-tawoXiao Lu De niChou Chu dekimasu.
- anchibotsutoJian Zhi woHui Bi dekiru:
ChromeDriverha, anchibotsutoshisutemugaJian Zhi niLi Yong suruShi Bie Ke Neng naQing Bao woLu Chu shimasu.undetected-chromedrivernoyounaJi Shu haSelenium WirewoHuo Yong shitekoreranoQing Bao woYin shi, Jian Zhi mekanizumuwoHui Bi shimasu. - burauzanoRou Ruan Xing gaXiang Shang suru: Cong Lai noburauzahaGu Ding noQi Dong She Ding niYi Cun shiteori, Bian Geng surunihaZai Qi Dong gaBi Yao desu. Selenium WirehaakuteibunasetsushiyonNei de, rikuesutohetsuda-yapurokishiShe Ding woriarutaimuniGeng Xin dekirutame, Dong De naWebsukureipinguniZui Shi nasoriyu-shiyontonarimasu.
Selenium WirenoZhu naJi Neng
rikuesutotoresuponsuhenoakusesu
Selenium WirehaburauzakaranoHTTP/HTTPStorahuitsukuwoJian Shi *kiyapuchiyadeki, Ci noZhu Yao Shu Xing heakusesudekimasu.
| Attribute | Description |
|---|---|
driver.requests |
kiyapuchiyasaretarikuesutonorisutowoShi Xi Lie Shun niBao Gao shimasu |
driver.last_request |
Zhi Jin dekiyapuchiyasaretarikuesutowoBao Gao shimasu (koreha driver.requests[-1] woShi uyoriXiao Lu De desu) |
driver.wait_for_request(pat, timeout=10) |
timeout parame-tadeDing Yi saretaShi Jian , pat parame-tadeDing Yi saretapata-n(Bu Fen Wen Zi Lie matahaZheng Gui Biao Xian )niYi Zhi sururikuesutogaJian tsukarumadeDai Ji shimasu. |
driver.har |
Fa Sheng shitaHTTPtoranzakushiyonnoJSONXing Shi noHARa-kaibudesu. |
driver.iter_requests() |
kiyapuchiyasaretarikuesutoniDui suruitere-tawoFan shimasu. |
Selenium Wireno Request obuziekutoniha, Ci noShu Xing gaarimasu.
| Attribute | Description |
|---|---|
body |
rikuesutonoBen Wen ha bytes toshiteTi Shi saremasu. rikuesutoBen Wen ganaiChang He , body haKong ninarimasu(Li : b''). |
cert |
sa-ba-noSSLZheng Ming Shu niGuan suruQing Bao woCi Shu Xing Shi deBao Gao shimasu(Fei HTTPSrikuesutodehaKong desu). |
date |
rikuesutogaXing waretaRi Shi woShi shimasu. |
headers |
rikuesutonohetsuda-noCi Shu raikunaobuziekutowoBao Gao shimasu(Selenium Wirenohetsuda-haDa Wen Zi Xiao Wen Zi woQu Bie sezu, Zhong Fu gaXu Ke sareruDian niZhu Yi shitekudasai). |
host |
rikuesutohosutowoBao Gao shimasu(Li : https://brightdata.jp/). |
method |
HHTPmesotsudo(GET, POST nado)woZhi Ding shimasu |
params |
rikuesutonoparame-tanoCi Shu woBao Gao shimasu(Tong Ming noparame-tagaFu Shu Hui Chu Xian suruChang He , Ci Shu Nei noZhi harisutoninarimasu). |
path |
rikuesutopasuwoBao Gao shimasu. |
querystring |
kueriWen Zi Lie woBao Gao shimasu. |
response |
rikuesutoniGuan Lian Fu keraretaresuponsuobuziekutowoBao Gao shimasu(resuponsuganaiChang He haZhi ga None ninarimasu). |
url |
host, path, querystring woHan muWan Quan narikuesutoURLwoBao Gao shimasu. |
ws_messages |
rikuesutogaWebSocketnoChang He (URLgaYi Ban De ni wss:// noyounaXing Shi ), ws_messages niSong Shou Xin saretaWebSocketmetsuse-zigaHan maremasu. |
Yi Fang de, Response obuziekutohaCi noShu Xing woGong Kai shimasu.
| Attribute | Description |
|---|---|
body |
resuponsunoBen Wen ha bytes toshiteTi Shi saremasu. resuponsuBen Wen ganaiChang He , body haKong ninarimasu(Li : b''). |
date |
resuponsugaShou Xin saretaRi Shi woShi shimasu. |
headers |
resuponsunohetsuda-noCi Shu raikunaobuziekutowoBao Gao shimasu(Selenium Wirenohetsuda-haDa Wen Zi Xiao Wen Zi woQu Bie sezu, Zhong Fu gaXu Ke sareruDian niZhu Yi shitekudasai). |
reason |
OK, Not Found nadonoresuponsunoLi You hure-zuwoBao Gao shimasu. |
status_code |
200, 404 nadonoresuponsusute-tasuwoBao Gao shimasu. |
konoJi Neng wotesutosuruPythonsukuriputowoShu itemimashiyou.
# Initialize the WebDriver with Selenium Wire
driver = webdriver.Chrome()
try:
# Open the target website
driver.get("https://brightdata.jp/")
# Access and print all captured requests
for request in driver.requests:
print(f"URL: {request.url}")
print(f"Method: {request.method}")
print(f"Headers: {request.headers}")
print(f"Response Status Code: {request.response.status_code if request.response else 'No Response'}")
print("-" * 50)
finally:
# Close the browser
driver.quit()
konoko-dohata-getsutosaitowoKai ki, driver.requests woShi Yong shiterikuesutowokiyapuchiyashimasu. sonoHou , forru-pude url, method, headers nadonorikuesutoShu Xing wointa-seputoshiteChu Li shimasu.
Xiang Ding sareruJie Guo haCi notooridesu.
rikuesutotoresuponsunointa-seputo
Selenium Wiredeha, inta-seputa-(burauzawoTong Guo surunetsutowa-kutorahuitsukuniYing zitetoriga-sareruGuan Shu )woShi tsute, rikuesutotoresuponsuwointa-seputooyobiBian Geng dekimasu.
inta-seputa-ha2Zhong Lei arimasu.
driver.request_interceptor: rikuesutowointa-seputoshi, Dan Yi noYin Shu woShou keQu rimasu.driver.response_interceptor: resuponsuwointa-seputoshi, Fa Sheng Yuan norikuesutoYong toresuponsuYong no2tsunoYin Shu woShou keQu rimasu.
Ci ha, rikuesutointa-seputa-noShi iFang woShi suLi desu.
# Define the request interceptor function
def interceptor(request):
# Add a custom header to all requests
request.headers["X-Test-Header"] = "MyCustomHeaderValue"
# Block requests to a specific domain
if "example.com" in request.url:
print(f"Blocking request to: {request.url}")
request.abort() # Abort the request
# Initialize the WebDriver with Selenium Wire
driver = webdriver.Chrome()
# Assign the interceptor function to the driver
driver.request_interceptor = interceptor
try:
# Open a website that makes multiple requests
driver.get("https://brightdata.jp/")
# Print all captured requests
for request in driver.requests:
print(f"URL: {request.url}")
print(f"Headers: {request.headers}")
print("-" * 50)
finally:
# Close the browser
driver.quit()
konosunipetsutonoDong Zuo haCi notooridesu.
- Interceptor function: Song Xin sarerusubetenorikuesutoniDui shiteHu biChu sareruinta-seputa-Guan Shu woZuo Cheng shimasu.
request.headers[]woShi tsute, Song Xin sarerusubetenorikuesutonikasutamuhetsuda-woZhui Jia shimasu. mata,example.comdomeinhenoburauzarikuesutowoburotsukushimasu. - Captures requests: pe-zigaDu miIp maretaHou , Bian Geng saretahetsuda-woHan me, kiyapuchiyasaretasubetenorikuesutogaChu Li saremasu.
Note:
rikuesutonoburotsukuha, Guang Gao , Jie Xi sukuriputo, sa-dopa-teiZhi uizietsutonado, tasukuniBu Yao naZhui Jia riso-suwope-zigaDu miIp muChang He niYou Xiao desu. konoapuro-chiha, Su Du woXiang Shang sase, Dai Yu Fu noXiao Fei woZui Xiao Hua surukotodesukureipinguXiao Lu woGao memasu.
Xiang Ding sareruJie Guo haCi noyouninarimasu.
WebSocketnoJian Shi
Duo kunomodannaWebsaitoha, sa-ba-tonoriarutaimuTong Xin woWei Chi surutameni WebSockets niYi Cun shiteimasu. Cong Lai noHTTPrikuesutotohaYi nari, WebSockets haburauzatosa-ba-Jian niJi Sok De naJie Sok woZuo ri, Zao riFan shinohandoshieikunashideshi-muresunade-taJiao Huan woKe Neng nishimasu.
Zhong Yao nade-tagakoreranochiyaneruwoTong tsuteLiu rerukotogaDuo itame, WebSocket torahuitsukuwointa-seputosureba, riarutaimunosa-ba-resuponsuheZhi Jie akusesudeki, burauzaCe denoChu Li yarendaringunoBi Yao ganakunarimasu.
Yi Xia haSelenium Wireno WebSocket obuziekutonoShu Xing desu.
| Attribute | Description |
|---|---|
content |
metsuse-zinoNei Rong woBao Gao shimasu. str mataha bytes Xing Shi ninarimasu. |
date |
metsuse-zinoRi Shi woShi shimasu. |
headers |
resuponsuhetsuda-noCi Shu raikunaobuziekutowoBao Gao shimasu(Selenium Wirenohetsuda-haDa Wen Zi Xiao Wen Zi woQu Bie sezu, Zhong Fu gaXu Ke sareruDian niZhu Yi shitekudasai). |
from_client |
kuraiantogaSong Xin shitametsuse-zinoChang He ha True, sa-ba-gaSong Xin shitaChang He ha False woFan subooleandesu. |
purokishinoGuan Li
purokishisa-ba-ha, debaisutota-getsutoWebsaitonoJian deZhong Ji toshiteJi Neng shi, IPadoresuwoYin shimasu. IPbe-sunoZhi Xian noHui Bi , re-toZhi Xian niyoruburotsukingunoQing Jian , zioroke-shiyonZhi Xian kontentsuhenoakusesuwoKe Neng nishi, shi-muresunaWebsukureipinguwoZhi Yuan shimasu.
Selenium WiredepurokishiwoShe Ding shitemimashiyou.
options = {
"proxy": {
"http": "
"https": "
}
}
# Initialize the WebDriver with Selenium Wire
driver = webdriver.Chrome(seleniumwire_options=options)
konoShe Ding ha, Su noSeleniumdepurokishiwoShe Ding suruChang He (Chromeno --proxy-server huraguniYi Cun suruBi Yao gaaru)tohaYi narimasu. koreha, Su noSeleniumdehapurokishiShe Ding gaJing De dearukotowoYi Wei shimasu. itsutanpurokishiwoShe Ding suruto, burauzasetsushiyonQuan Ti deYou Xiao ninari, burauzawoZai Qi Dong shinaiXian riBian Geng dekimasen. konoZhi Yue ha, Te niDong De napurokishiro-te-shiyongaBi Yao naChang He niZhi Xian tonariDe masu.
koreniDui shite, Selenium WirehaTong Yi burauzainsutansuNei depurokishiwoDong De niBian Geng dekiruRou Ruan Xing woTi Gong shimasu. koreha proxy Shu Xing nookagedeKe Neng ninarimasu.
driver.proxy = {
"http": "
"https": "
}
sarani, Chromeno --proxy-server huraguha, URLniRen Zheng Qing Bao woHan mupurokishiwosapo-toshiteimasen.
protocol://username:password@host:port
sonoDai wari, Selenium WirehaRen Zheng Fu kipurokishiwoWan Quan nisapo-toshiteirutame, WebsukureipingunihayoriLiang iXuan Ze Zhi tonarimasu.
Selenium Wiredenopurokishiro-te-shiyon
purokishiro-te-shiyonnotameniSelenium Wirepuroziekutowosetsutoatsupushimashiyou. koreniyori, rikuesutogotoniTui Chu IPwoBian Geng dekiruyouninarimasu.
Yao Jian
konogaidonokonoBu Fen niCong uniha, Ci noQian Ti Tiao Jian gaBi Yao desu.
- Python 3.7Yi Shang
- Dui Ying shiteiruWebburauza
mazu, Jia Xiang Huan Jing deirekutoriwoZuo Cheng shimasu.
You Xiao Hua suruniha, WindowsdehaCi woShi Xing shimasu.
macOS/LinuxdehaCi woShi Xing shimasu.
Ci niSelenium Wirewoinsuto-rushimasu(Yi Cun Guan Xi toshiteSeleniumgaZi Dong De niinsuto-rusaremasu).
sutetsupu1: purokishiworandamuHua suru
mazu, You Xiao napurokishiURLnorisutogaBi Yao desu. free proxies norisutowoShi Yong dekimasu. koreraworisutoniZhui Jia shi, random.choice() woShi tsuterandamunaYao Su woXuan Ze shimasu.
proxies = [
"http://PROXY_1:PORT_NUMBER_X",
"http://PROXY_2:PORT_NUMBER_Y",
"http://PROXY_3:PORT_NUMBER_Z",
# ...
]
# Randomize the list
return random.choice(proxies)
Hu biChu sareruto, konoGuan Shu harisutokararandamunapurokishiURLwoFan shimasu.
Dong Zuo saserutameni, random noimportwoWang renaidekudasai.
import random
sutetsupu2: purokishiwoShe Ding suru
get_random_proxy() Guan Shu woHu biChu shitepurokishiURLwoQu De shimasu.
burauzainsutansuwoChu Qi Hua shi, Xuan Ze shitapurokishiwoShe Ding shimasu.
seleniumwire_options = {
"proxy": {
"http": proxy,
"https": proxy
}
}
# Browser configuration
chrome_options = Options()
chrome_options.add_argument("--headless") # Run the browser in headless mode
# Initialize a browser instance with the given configurations
driver = webdriver.Chrome(service=Service(), options=chrome_options, seleniumwire_options=seleniumwire_options)
Shang Ji sunipetsutoniha, Ci noimportgaBi Yao desu.
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
burauzasetsushiyonZhong nipurokishiwoDong De niBian Geng suruniha, Dai wariniCi noko-dowoShi Yong shimasu.
"http": proxy,
"https": proxy
}
sutetsupu3: ta-getsutope-ziniakusesusuru
ta-getsutoWebsaitoniakusesushi, Chu Li woChou Chu shiteburauzawoBi zimasu.
# Visit the target page
driver.get("https://httpbin.io/ip")
# Extract the page output
body = driver.find_element(By.TAG_NAME, "body").text
print(body)
except Exception as e:
# Handle any errors that occur with the browser or the proxy
print(f"Error with proxy {proxy}: {e}")
finally:
# Close the browser
driver.quit()
Dong Zuo saserutameni, Seleniumkara By woimportshitekudasai.
konoLi deha, Wan Xian pe-zihaHTTPBinpuroziekutono /ip endopointodesu. konope-zihaHu biChu shiYuan noIPadoresuwoFan shimasu. subetegaQi Dai dooriniDong Zuo sureba, sukuriputohaShi Xing gotonipurokishirisutokaraYi naruIPwoChu Li suruhazudesu.
sutetsupu4: matometeShi Xing suru
Ci ha, selenium_wire.py huairuniRu rerubekiSelenium Wirenopurokishiro-te-shiyonrozitsukuQuan Ti desu.
from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
def get_random_proxy():
proxies = [
"http://PROXY_1:PORT_NUMBER_X",
"http://PROXY_2:PORT_NUMBER_Y",
"http://PROXY_3:PORT_NUMBER_Z",
# Add more proxies here...
]
# Randomly pick a proxy
return random.choice(proxies)
# Pick a random proxy URL
proxy = get_random_proxy()
# Selenium Wire configuration with the proxy
seleniumwire_options = {
"proxy": {
"http": proxy,
"https": proxy
}
}
# Browser configuration
chrome_options = Options()
chrome_options.add_argument("--headless") # Run the browser in headless mode
# Initialize a browser instance with the given configurations
driver = webdriver.Chrome(service=Service(), options=chrome_options, seleniumwire_options=seleniumwire_options)
try:
# Visit the target page
driver.get("https://httpbin.io/ip")
# Extract the page output
body = driver.find_element(By.TAG_NAME, "body").text
print(body)
except Exception as e:
# Handle any errors that occur with the browser or the proxy
print(f"Error with proxy {proxy}: {e}")
finally:
# Close the browser
driver.quit()
huairuwoShi Xing suruniha, Ci woQi Dong shimasu.
Shi Xing gotoni, Chu Li haCi noyouninarimasu.
"origin": "PROXY_1:XXXX"
}
matahaCi noyouninarimasu.
"origin": "PROXY_2:YYYY"
}
nado...
sukuriputowoFu Shu Hui Shi Xing suruto, Mei Hui Yi naruIPadoresugaBiao Shi sareruhazudesu.
yoriLiang ipurokishiro-te-shiyonnoapuro-chi: Bright Data Proxies
Selenium WiredeShou Dong nopurokishiro-te-shiyonwoXing uniha, Duo kunoboira-pure-toko-dogaBi Yao de, You Xiao napurokishiURLnorisutowoWei Chi suruBi Yao gaarimasu. Dai warini, IPadoresunoBian Geng woZi Dong De niChu Li suruBright Datanoro-te-teingupurokishiwoShi Yong dekimasu. Shi Yong Fang Fa haCi notooridesu.
sudeniakauntowooChi chinoChang He haBright Dataniroguinshitekudasai. soudenaiChang He ha, Wu Liao deakauntowoZuo Cheng shitekudasai. Ci noyu-za-datsushiyubo-doniakusesudekiruyouninarimasu.
[View proxy products] botanwokuritsukushimasu.
Ci no[Proxies & Scraping Infrastructure] pe-ziniridairekutosaremasu.
Xia nisukuro-rushite[Residential Proxies] ka-dowoJian tsuke, [Get started] botanwokuritsukushimasu.
rezidenshiyarupurokishinoShe Ding datsushiyubo-doniYi Dong shimasu. gaidoFu kiuiza-doniCong i, ni-zuniHe wasetepurokishisa-bisuwoShe Ding shitekudasai.
[Access parameters] tabuniYi Dong shi, Ci noyounipurokishinohost, port, username, passwordwoQu De shimasu.
[Host] hui-rudonihasudeniportgaHan mareteiruDian niZhu Yi shitekudasai.
korede, purokishiURLwoGou Zhu shiteSelenium WireniShe Ding surutameniBi Yao namonohasubeteJian imashita. subetenoQing Bao woShou Ji shi, Ci noGou Wen deURLwoGou Zhu shimasu.
:@
tatoeba, konoke-sudehaCi noyouninarimasu.
brd-customer-hl_4hgu8dwd-zone-residential:[email protected]:XXXXX
[Active proxy] woQie riTi e, Zui Hou noZhi Shi niCong ebaZhun Bei Wan Liao desu.
Yi Xia ha, Bright DatawoTong He surutamenoSelenium Wirepurokishisunipetsutodesu.
proxy = "brd-customer-hl_4hgu8dwd-zone-residential:[email protected]:XXXXX"
# Set up Selenium Wire options
options = {
"proxy": {
"http": proxy,
"https": proxy
}
}
# Initialize the WebDriver with Selenium Wire
driver = webdriver.Chrome(seleniumwire_options=options)
WebsukureipinguniokeruSeleniumtoSelenium WirenoBi Jiao
Yao Yue suruto, SeleniumtoSelenium WirenoBi Jiao haCi notooridesu.
| Selenium | Selenium Wire | |
|---|---|---|
| Purpose | UItesutooyobiWebCao Zuo woShi Xing surutameniWebburauzawoZi Dong Hua shimasu | SeleniumwoKuo Zhang shi, HTTP/HTTPSrikuesutotoresuponsunoJian Cha oyobiBian Geng notamenoZhui Jia Ji Neng woTi Gong shimasu |
| HTTP/HTTPS request handling | HTTP/HTTPSrikuesutoyaresuponsuheZhi Jie akusesusuruJi Neng haTi Gong shimasen | HTTP/HTTPSrikuesutotoresuponsunoJian Cha *Bian Geng *kiyapuchiyagaKe Neng desu |
| Proxy support | purokishisapo-togaXian Ding De desu(Shou Dong She Ding gaBi Yao ) | Dong De She Ding wosapo-tosuruGao Du napurokishiGuan Li |
| Performance | Qing Liang deGao Su desu | netsutowa-kutorahuitsukunokiyapuchiyatoChu Li notame, yayaDi Su desu |
| Use cases | Zhu niWebapurike-shiyonnoJi Neng tesutoniShi Yong sare, Ji Ben De naWebsukureipingunimoBian Li desu | APInotesuto, netsutowa-kutorahuitsukunodebatsugu, WebsukureipinguniYou Yong desu |
Jie Lun
Selenium WirehaWebsukureipinguniXiao Lu De niShi Yong dekimasuga, mentenansusareteinaisohutoueadeari, Mo Neng nasoriyu-shiyondehaarimasen.
Dai warini, Su noSeleniumto, Bright DatanoScraping Browser noyounaZhuan Yong nosukureipinguburauzanoLi Yong woJian Tao shitekudasai. korehaPlaywright, Puppeteer, SeleniumnadotoLian Xi surusuke-raburunakuraudoburauzadesu. Ge rikuesutogotoniTui Chu IPwoshi-muresuniro-te-shiyonshitsutsu, burauzahuinga-purinto, ritorai, CAPTCHAJie Jue nadomoGuan Li shimasu. burotsukingunoWen Ti woJie Xiao shi, sukureipingunowa-kuhuro-woZui Shi Hua surutamenioShi shikudasai.