NewsCrawler
Duo Ping Tai Xin Wen & Nei Rong Pa Chong Ji He
Zhi Chi Wei Xin Gong Zhong Hao , Jin Ri Tou Tiao , Wang Yi Xin Wen , Sou Hu , Teng Xun , Naver, Detik, Quora, BBC, CNN, Twitter/X Deng 12 Ge Zhu Liu Ping Tai
Ti Gong Ming Ling Xing Diao Yong , Ke Shi Hua Web UI, Tong Yi JSON Shu Chu , Zhi Chi MCP Xie Yi , Claude Code Skills
English * Zhong Wen
Kai Xiang Ji Yong De Web UI - Zi Dong Shi Bie Ping Tai , Shi Shi Ti Qu Jin Du , JSON/Markdown Shuang Ge Shi Dao Chu
Wei Shi Yao Xuan Ze NewsCrawler?
| Duo Ping Tai Zhi Chi | Shuang Mo Shi Shi Yong | Biao Zhun Hua Shu Chu | Kuai Su Bu Shu | MCP Zhi Chi | Skills Zhi Chi |
|---|---|---|---|---|---|
| 12 Ge Zhu Liu Ping Tai Fu Gai Zhong Ying Han Yin Ni |
Python API + Web UI |
Tong Yi JSON Ge Shi Yi Yu Ji Cheng |
uv Bao Guan Li Qi Ji Su An Zhuang |
Ji Cheng Ge Lei AIZong Jie Wen Zhang | Claude Code Ke Qian Yi Ji Neng |
He Xin Te Xing :
- Quan Ping Tai Fu Gai - Zhi Chi Wei Xin Gong Zhong Hao , Jin Ri Tou Tiao , Wang Yi , Sou Hu , Teng Xun , Lenny's Newsletter, Naver Blog, Detik News, Quora, BBC News, CNN News, Twitter/X
- Zhi Neng Ti Qu - Zi Dong Shi Bie Ping Tai Lei Xing ,Ti Qu Biao Ti , Zheng Wen , Tu Pian , Shi Pin Deng Duo Mei Ti Nei Rong
- Tong Yi Shu Chu - Suo You Ping Tai Shu Chu Biao Zhun Hua JSON Ge Shi ,Wan Mei Gua Pei Shu Ju Fen Xi , Ru Ku , Xia You Chu Li
- Ling Huo Shi Yong - Zhi Chi Python API(Zi Dong Hua ) + Web UI(Ke Shi Hua ) + MCP Server(AI Agent) + Claude Code Skills
- Yi Jian Bu Shu - Docker Compose Bian Pai Suo You Fu Wu (Hou Duan + Qian Duan + MCP)
- AI Zhi Neng Ti Ji Cheng - Zhi Chi MCP Xie Yi ,Ke Jie Ru Claude Desktop Deng AI Gong Ju
- Mo Kuai Hua She Ji - Ge Ping Tai Pa Chong Jie Ou ,Yi Yu Kuo Zhan Xin Ping Tai Huo You Hua Xian You Shi Xian
- Qing Liang Gao Xiao - Shi Yong uv Guan Li Yi Lai ,An Zhuang Kuai Su ,Yun Xing Wen Ding
Kuai Su Kai Shi
Fang Shi Yi :Docker Compose ( Tui Jian - Yi Jian Bu Shu )
# Fang Wen : https://docs.docker.com/get-docker/
# 2. Ke Long Xiang Mu
git clone https://github.com/NanmiCoder/NewsCrawler.git
cd NewsCrawler
# 3. Yi Jian Qi Dong Suo You Fu Wu (Hou Duan + Qian Duan + MCP)
docker compose up -d
# 4. Fang Wen Fu Wu
# - Qian Duan Jie Mian : http://localhost:3021
# - Hou Duan API: http://localhost:8000/docs
# - MCP Fu Wu : http://localhost:8765/mcp
Bao Han Fu Wu :
- Backend Fu Wu (FastAPI) - Xin Wen Ti Qu API
- Frontend Fu Wu (Vue 3 + Nginx) - Web UI Jie Mian
- MCP Fu Wu - AI Agent Gong Ju (Zhi Chi Claude Desktop)
- Zi Dong Jian Kang Jian Cha - Que Bao Suo You Fu Wu Zheng Chang Yun Xing
- Shu Ju Chi Jiu Hua - Ti Qu De Xin Wen Bao Cun Zai
./data/Mu Lu
Docker Guan Li Ming Ling :
docker compose logs -f
# Ting Zhi Fu Wu
docker compose down
# Dai Ma Geng Xin Hou Zhong Xin Gou Jian
docker compose up -d --build
Wan Zheng Wen Dang : DOCKER_DEPLOYMENT.md
Fang Shi Er :Web UI (Shou Dong Bu Shu )
curl -LsSf https://astral.sh/uv/install.sh | sh # macOS/Linux
# Huo : pip install uv
# 2. Ke Long Xiang Mu
git clone https://github.com/NanmiCoder/NewsCrawler.git
cd NewsCrawler
# 3. An Zhuang Suo You Yi Lai (uv workspace Mo Shi )
uv sync
# 4. Qi Dong Hou Duan (Zai Xiang Mu Gen Mu Lu )
uv run news-extractor-backend --host 0.0.0.0 --port 8000
# 5. Qi Dong Qian Duan (Xin Zhong Duan )
cd news-extractor-ui/frontend
npm install && npm run dev
# 6. Fang Wen http://localhost:3000
Web UI Gong Neng :
- Nian Tie URL,Zi Dong Shi Bie Ping Tai Lei Xing
- Shi Shi Xian Shi Ti Qu Jin Du
- Zhi Chi JSON / Markdown Shuang Ge Shi Dao Chu
- Nei Rong Yu Lan Yu Yi Jian Xia Zai
Fang Shi San :Python API (Gua He Zi Dong Hua Ji Cheng )
from news_crawler.toutiao_news import ToutiaoNewsCrawler
# Wei Xin Gong Zhong Hao
wechat_url = "https://mp.weixin.qq.com/s/xxxxxx"
crawler = WeChatNewsCrawler(wechat_url)
result = crawler.run() # Zi Dong Bao Cun Dao data/ Mu Lu
# Jin Ri Tou Tiao
toutiao_url = "https://www.toutiao.com/article/xxxxxx"
crawler = ToutiaoNewsCrawler(toutiao_url)
result = crawler.run()
print(result) # Fan Hui JSON Ge Shi Shu Ju
Yun Xing Shi Li :
Fang Shi Si :MCP Server (AI Zhi Neng Ti Ji Cheng )
Shi Yao Shi MCP? Model Context Protocol (MCP) Shi Yi Ge Lian Jie AI Zhu Shou (Ru Claude Desktop)Yu Wai Bu Gong Ju He Shu Ju Yuan De Biao Zhun Xie Yi .
Shi Yong Chang Jing :
- Rang Claude, Cursor, ChatGPTDeng Gong Ju Tong Guo Dui Hua Zhi Jie Ti Qu Xin Wen Nei Rong
- Tong Guo AI Zhi Ling Pi Liang Chu Li Duo Ge URL
- AI Qu Dong De Nei Rong Fen Xi Gong Zuo Liu
- Gou Jian Ju You Xin Wen Ti Qu Neng Li De Zi Ding Yi AI Zhi Neng Ti
Kuai Su Pei Zhi :
docker compose up -d mcp
# 2. Huo Shou Dong Qi Dong (Zai Xiang Mu Gen Mu Lu )
# Shou Xian An Zhuang Yi Lai
uv sync
# Qi Dong MCP Fu Wu Qi
uv run news-extractor-mcp --host 0.0.0.0 --port 8765
# 3. MCP Fu Wu Yun Xing Zai : http://localhost:8765/mcp
AI Gong Ju Pei Zhi (Streamable HTTP Fang Shi ):
Cursor (Dian Ji Zhan Kai )
Pei Zhi Wen Jian Wei Zhi : ~/.cursor/mcp.json (Quan Ju ) Huo .cursor/mcp.json (Xiang Mu Ji Bie )
"mcpServers": {
"newscrawler": {
"url": "http://127.0.0.1:8765/mcp"
}
}
}
Windsurf (Dian Ji Zhan Kai )
Pei Zhi Wen Jian Wei Zhi : ~/.codeium/windsurf/mcp_server_config.json
"mcpServers": {
"newscrawler": {
"url": "http://127.0.0.1:8765/mcp"
}
}
}
Trae (Dian Ji Zhan Kai )
She Zhi - Gong Ju - MCP Fu Wu Qi - Tian Jia Fu Wu Qi
"name": "newscrawler",
"url": "http://127.0.0.1:8765/mcp"
}
Claude Desktop (Dian Ji Zhan Kai )
Pei Zhi Wen Jian Wei Zhi :
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%/Claude/claude_desktop_config.json
"mcpServers": {
"newscrawler": {
"url": "http://127.0.0.1:8765/mcp"
}
}
}
Qi Ta Zhi Chi MCP De Gong Ju (Dian Ji Zhan Kai )
Suo You Zhi Chi Streamable HTTP Chuan Shu De MCP Ke Hu Duan Du Ke Yi Shi Yong Yi Xia Pei Zhi :
"mcpServers": {
"newscrawler": {
"url": "http://127.0.0.1:8765/mcp"
}
}
}
Zhu Yi : Ru Guo Shi Yong Docker Qie AI Gong Ju Yun Xing Zai Docker Wai ,Qing Jiang 127.0.0.1 Ti Huan Wei Su Zhu Ji IP Huo host.docker.internal
Ke Yong MCP Gong Ju :
extract_news- Ti Qu Dan Pian Xin Wen (JSON Huo Markdown Ge Shi )batch_extract_news- Pi Liang Ti Qu Duo Ge URLdetect_news_platform- Cong URL Shi Bie Ping Tai Lei Xinglist_supported_platforms- Xian Shi Suo You Zhi Chi De Ping Tai
Wan Zheng MCP Wen Dang : news_extractor_mcp/README.md
Fang Shi Wu :Claude Code Skills (AI Bian Cheng Zhu Shou Ji Cheng )
Shi Yao Shi Claude Code Skills?
Claude Code Shi Anthropic Tui Chu De AI Bian Cheng Zhu Shou . Skills Shi Claude Code De Ke Qian Yi Ji Neng Mo Kuai ,Ke Yi Fu Zhi Dao Ren He Xiang Mu Zhong Shi Yong ,Rang AI Zhu Shou Zi Dong Huo De Xin Wen Ti Qu Neng Li .
Yu MCP De Qu Bie :
- MCP Server - Xu Yao Qi Dong Du Li Fu Wu ,Gua He Chang Qi Yun Xing De AI Gong Zuo Liu
- Claude Code Skills - Fu Zhi Dao Xiang Mu Ji Ke Shi Yong ,Wu Xu Qi Dong Fu Wu ,Gua He Kai Fa Zhe Zai Bian Ma Shi Kuai Su Ti Qu Xin Wen
Shi Yong Chang Jing :
- Zai Ren He Xiang Mu Zhong Rang Claude Code Ju Bei Xin Wen Ti Qu Neng Li
- Zi Bao Han , Wu Xu Wai Bu Fu Wu ,Fu Zhi Ji Yong
- Kai Fa Guo Cheng Zhong Kuai Su Ti Qu Xin Wen Nei Rong Yong Yu Ce Shi Huo Fen Xi
An Zhuang Fang Shi :
Jiang Ben Xiang Mu De .claude/skills/news-extractor/ Mu Lu Fu Zhi Dao Ni De Mu Biao Xiang Mu ,Ran Hou An Zhuang Yi Lai Ji Ke :
cp -r NewsCrawler/.claude/skills/news-extractor <Ni De Xiang Mu >/.claude/skills/news-extractor
# 2. An Zhuang Yi Lai
cd <Ni De Xiang Mu >/.claude/skills/news-extractor
uv sync
# 3. Zai Claude Code Zhong Zhi Jie Shi Yong
# Claude Code Hui Zi Dong Du Qu SKILL.md Bing Huo De Xin Wen Ti Qu Neng Li
Zhi Chi 12 Ge Ping Tai : Wei Xin Gong Zhong Hao , Jin Ri Tou Tiao , Wang Yi Xin Wen , Sou Hu Xin Wen , Teng Xun Xin Wen , BBC News, CNN News, Twitter/X, Lenny's Newsletter, Naver Blog, Detik News, Quora
Wan Zheng An Zhuang Zhi Nan : INSTALL_SKILL.md
Zhi Chi De Ping Tai
Xin Wen / Nei Rong Ping Tai
| Ping Tai | URL Shi Li | Yu Yan | Te Xing |
|---|---|---|---|
| Wei Xin Gong Zhong Hao | mp.weixin.qq.com |
Zhong Wen | Zhi Chi Tu Wen Ti Qu |
| Jin Ri Tou Tiao | toutiao.com |
Zhong Wen | Fu Mei Ti Nei Rong |
| Wang Yi Xin Wen | 163.com |
Zhong Wen | Tu Pian Hua Lang Zhi Chi |
| Sou Hu Xin Wen | sohu.com |
Zhong Wen | Duo Mei Ti Nei Rong |
| Teng Xun Xin Wen | news.qq.com |
Zhong Wen | Xin Wen Zhi Chi |
| Lenny's Newsletter | lennysnewsletter.com |
Ying Wen | Chang Wen Nei Rong |
| Naver Blog | blog.naver.com |
Han Yu | Bo Ke Ping Tai |
| Detik News | detik.com |
Yin Ni Yu | Dong Nan Ya Xin Wen |
| Quora | quora.com |
Ying Wen | Wen Da Nei Rong |
| Twitter/X | x.com twitter.com |
Duo Yu Yan | Tui Wen Ti Qu |
Shi Pin Su Cai Ping Tai
Pexels * Pixabay * Coverr * Mixkit - Gao Zhi Liang Mian Fei Shi Pin Su Cai Xia Zai
Shi Yong Chang Jing
Duo Yuan Xin Wen Ju He Ping Tai / Yu Qing Jian Kong Xi Tong
Mei Ti Nei Rong Fen Xi , Shu Ju Wa Jue , Tui Jian Xi Tong
Xue Zhu Yan Jiu / Shu Ju Ke Xue - Kua Ping Tai Nei Rong Zhua Qu
Jiao Xue Xiang Mu / Ge Ren Xue Xi - Pa Chong Kuang Jia Mo Ban
AI Xun Lian Shu Ju Cai Ji / Nei Rong Zhi Liang Fen Xi
Shu Ju Shu Chu Ge Shi
Suo You Pa Chong Shu Chu Tong Yi De JSON Ge Shi ,Bao Cun Zai data/ Mu Lu :
"title": "Wen Zhang Biao Ti ",
"news_url": "Yuan Wen Lian Jie ",
"news_id": "Wen Zhang ID",
"meta_info": {
"author_name": "Zuo Zhe Ming Cheng ",
"author_url": "Zuo Zhe Zhu Ye ",
"publish_time": "2024-10-15 10:30:00"
},
"contents": [
{"type": "text", "content": "Duan Luo Wen Ben Nei Rong ", "desc": ""},
{"type": "image", "content": "https://example.com/image.jpg", "desc": "Tu Pian Miao Shu "},
{"type": "video", "content": "https://example.com/video.mp4", "desc": "Shi Pin Miao Shu "}
],
"texts": ["Duan Luo 1Wen Ben ", "Duan Luo 2Wen Ben "],
"images": ["Tu Pian URL1", "Tu Pian URL2"],
"videos": ["Shi Pin URL1"]
}
Zi Duan Shuo Ming :
contents- Jie Gou Hua Nei Rong ,Bao Liu Shun Xu He Lei Xing (Wen Ben /Tu Pian /Shi Pin )texts/images/videos- Bian Ping Hua Lie Biao ,Bian Yu Kuai Su Fang Wen Te Ding Lei Xing Nei Rongmeta_info- Wen Zhang Yuan Xin Xi (Zuo Zhe , Fa Bu Shi Jian Deng )
Ji Zhu Jia Gou
Hou Duan Ji Zhu
Python 3.8+ * FastAPI * Pydantic * curl_cffi * parsel * tenacity
Qian Duan Ji Zhu
Vue 3 * TypeScript * Vite * Axios
Kai Fa Gong Ju
uv (Bao Guan Li Qi ) * Playwright (Liu Lan Qi Zi Dong Hua ,Ke Xuan )
Xiang Mu Jie Gou
NewsCrawler/
+-- news_crawler/ # He Xin Pa Chong Mo Kuai
| +-- wechat_news/ # Wei Xin Gong Zhong Hao
| +-- toutiao_news/ # Jin Ri Tou Tiao
| +-- netease_news/ # Wang Yi Xin Wen
| +-- sohu_news/ # Sou Hu Xin Wen
| +-- tencent_news/ # Teng Xun Xin Wen
| +-- ... # Qi Ta Ping Tai
|
+-- news_extractor_core/ # Gong Xiang He Xin Ku (uv workspace Cheng Yuan )
| +-- adapters/ # Ping Tai Gua Pei Qi
| +-- services/ # Ye Wu Luo Ji
| +-- models/ # Shu Ju Mo Xing
|
+-- news_extractor_backend/ # FastAPI Hou Duan Fu Wu (uv workspace Cheng Yuan )
| +-- api/ # API Lu You
| +-- main.py # Ying Yong Ru Kou
| +-- cli.py # Ming Ling Xing Ru Kou
|
+-- news_extractor_mcp/ # MCP Fu Wu Qi (uv workspace Cheng Yuan )
| +-- server.py # MCP Shi Xian
| +-- README.md # MCP Wen Dang
|
+-- news-extractor-ui/ # Web UI Ying Yong
| +-- frontend/ # Vue 3 Qian Duan
|
+-- video_crawler/ # Shi Pin Su Cai Xia Zai Qi
+-- libs/ # Gong Ju Ku
+-- data/ # Shu Chu Shu Ju Mu Lu
|
+-- pyproject.toml # uv workspace Gen Pei Zhi
+-- uv.lock # Yi Lai Suo Wen Jian
+-- Dockerfile # Duo Jie Duan Docker Gou Jian
+-- docker-compose.yml # Fu Wu Bian Pai Pei Zhi
+-- DOCKER_DEPLOYMENT.md # Docker Bu Shu Zhi Nan
+-- MANUAL_DEPLOYMENT.md # Shou Dong Bu Shu Zhi Nan
Ben Xiang Mu Jin Gong Xue Xi He Yan Jiu Shi Yong ,Jin Zhi Yong Yu Shang Ye Yong Tu
Shi Yong Xu Zhi :
- Jin Yong Yu Ge Ren Xue Xi , Yan Jiu , Jiao Xue Mu De
- Zun Shou Mu Biao Wang Zhan De robots.txt He Fu Wu Tiao Kuan
- Kong Zhi Qing Qiu Pin Lu ,Bi Mian Gei Fu Wu Qi Zao Cheng Ya Li
- Bu De Yong Yu Fei Fa Yong Tu Huo Qin Fan Ta Ren Quan Yi
- Bu De Jin Xing Da Gui Mo Shang Ye Hua Pa Qu
Ji Zhu Shuo Ming :
- Bu Fen Ping Tai Ke Neng You Fan Pa Ji Zhi ,Xu Gua Dang Diao Zheng Ce Lue
- Mo Ren Headers Ke Neng Guo Qi ,Ke Shi Yong Playwright Zi Dong Huo Qu Zui Xin Cookie
- Wang Ye Jie Gou Bian Hua Ke Neng Dao Zhi Jie Xi Shi Bai ,Huan Ying Ti Jiao Issue
Gong Xian Zhi Nan
Huan Ying Ti Jiao Issue He Pull Request!
Gong Xian Fang Xiang :
- Xiu Fu Bug
- Tian Jia Xin Ping Tai Zhi Chi
- Gai Jin Wen Dang
- You Hua UI/UX
- Xing Neng You Hua
Ti Jiao Liu Cheng :
- Fork Ben Cang Ku
- Chuang Jian Te Xing Fen Zhi (
git checkout -b feature/AmazingFeature) - Ti Jiao Geng Gai (
git commit -m 'Add some AmazingFeature') - Tui Song Dao Fen Zhi (
git push origin feature/AmazingFeature) - Kai Qi Pull Request
Xu Ke Zheng
Ben Xiang Mu Jin Gong Xue Xi He Yan Jiu Shi Yong . Shi Yong Ben Xiang Mu Ji Biao Shi Nin Tong Yi :
- Bu Jiang Qi Yong Yu Shang Ye Mu De
- Bu Jin Xing Da Gui Mo Pa Qu
- Zun Shou Xiang Guan Fa Lu Fa Gui He Mu Biao Wang Zhan De Shi Yong Tiao Kuan
Dui Yu Yin Shi Yong Ben Xiang Mu Nei Rong Er Yin Qi De Ren He Fa Lu Ze Ren ,Ben Xiang Mu Bu Cheng Dan Ze Ren .
Xiang Guan Zi Yuan
Star History
Ru Guo Zhe Ge Xiang Mu Dui Ni You Bang Zhu ,Qing Gei Ge Star Zhi Chi Yi Xia !
Made with by NanmiCoder