Intro
Goose was originally an article extractor written in Java that has most recently (Aug2011) been converted to a scala project.
This is a complete rewrite in Python. The aim of the software is to take any news article or article-type web page and not only extract what is the main body of the article but also all meta data and most probable image candidate.
Goose will try to extract the following information:
- Main text of an article
- Main image of article
- Any YouTube/Vimeo movies embedded in article
- Meta Description
- Meta tags
The Python version was rewritten by:
- Xavier Grangier
Licensing
If you find Goose useful or have issues please drop me a line. I'd love to hear how you're using it or what features should be improved.
Goose is licensed by Gravity.com under the Apache 2.0 license; see the LICENSE file for more details.
Setup
git clone https://github.com/grangier/python-goose.git
cd python-goose
pip install -r requirements.txt
python setup.py install
Take it for a spin
>>> url = 'http://edition.cnn.com/2012/02/22/world/europe/uk-occupy-london/index.html?hpt=ieu_c2'
>>> g = Goose()
>>> article = g.extract(url=url)
>>> article.title
u'Occupy London loses eviction fight'
>>> article.meta_description
"Occupy London protesters who have been camped outside the landmark St. Paul's Cathedral for the past four months lost their court bid to avoid eviction Wednesday in a decision made by London's Court of Appeal."
>>> article.cleaned_text[:150]
(CNN) -- Occupy London protesters who have been camped outside the landmark St. Paul's Cathedral for the past four months lost their court bid to avoi
>>> article.top_image.src
http://i2.cdn.turner.com/cnn/dam/assets/111017024308-occupy-london-st-paul-s-cathedral-story-top.jpg
Configuration
There are two ways to pass configuration to goose. The first one is to pass goose a Configuration() object. The second one is to pass a configuration dict.
For instance, if you want to change the userAgent used by Goose just pass:
>>> g = Goose({'browser_user_agent': 'Mozilla'})Switching parsers : Goose can now be used with lxml html parser or lxml soup parser. By default the html parser is used. If you want to use the soup parser pass it in the configuration dict :
>>> g = Goose({'browser_user_agent': 'Mozilla', 'parser_class':'soup'})Goose is now language aware
For example, scraping a Spanish content page with correct meta language tags:
>>> from goose import Goose>>> url = 'http://sociedad.elpais.com/sociedad/2012/10/27/actualidad/1351332873_157836.html'
>>> g = Goose()
>>> article = g.extract(url=url)
>>> article.title
u'Las listas de espera se agravan'
>>> article.cleaned_text[:150]
u'Los recortes pasan factura a los pacientes. De diciembre de 2010 a junio de 2012 las listas de espera para operarse aumentaron un 125%. Hay m\xe1s ciudad'
Some pages don't have correct meta language tags, you can force it using configuration :
>>> from goose import Goose>>> url = 'http://www.elmundo.es/elmundo/2012/10/28/espana/1351388909.html'
>>> g = Goose({'use_meta_language': False, 'target_language':'es'})
>>> article = g.extract(url=url)
>>> article.cleaned_text[:150]
u'Importante golpe a la banda terrorista ETA en Francia. La Guardia Civil ha detenido en un hotel de Macon, a 70 kil\xf3metros de Lyon, a Izaskun Lesaka y '
Passing {'use_meta_language': False, 'target_language':'es'} will forcibly select Spanish.
Video extraction
>>> url = 'http://www.liberation.fr/politiques/2013/08/12/journee-de-jeux-pour-ayrault-dans-les-jardins-de-matignon_924350'
>>> g = goose.Goose({'target_language':'fr'})
>>> article = g.extract(url=url)
>>> article.movies
[
>>> article.movies[0].src
'http://sa.kewego.com/embed/vp/?language_code=fr&playerKey=1764a824c13c&configKey=dcc707ec373f&suffix=&sig=9bc77afb496s&autostart=false'
>>> article.movies[0].embed_code
''
>>> article.movies[0].embed_type
'iframe'
>>> article.movies[0].width
'476'
>>> article.movies[0].height
'357'
Goose in Chinese
Some users want to use Goose for Chinese content. Chinese word segmentation is way more difficult to deal with than occidental languages. Chinese needs a dedicated StopWord analyser that need to be passed to the config object.
>>> from goose import Goose>>> from goose.text import StopWordsChinese
>>> url = 'http://www.bbc.co.uk/zhongwen/simp/chinese_news/2012/12/121210_hongkong_politics.shtml'
>>> g = Goose({'stopwords_class': StopWordsChinese})
>>> article = g.extract(url=url)
>>> print article.cleaned_text[:150]
Xiang Gang Xing Zheng Chang Guan Liang Zhen Ying Zai Ge Fang Ya Li Xia Jiu Qi Da Zhai De Wei Zhang Jian Zhu (Jian Jian )Wen Ti Dao Li Fa Hui Jie Shou Zhi Xun ,Bing Xiang Xiang Gang Min Zhong Dao Qian .
Liang Zhen Ying Zai Xing Qi Er (12Yue 10Ri )De Da Wen Da Hui Kai Shi Zhi Ji Zai Qi Yan Shuo Zhong Dao Qian ,Dan Qiang Diao Ta Zai Wei Zhang Jian Zhu Wen Ti Shang Mei You Yin Man De Yi Tu He Dong Ji .
Yi Xie Qin Bei Jing Zhen Ying Yi Yuan Huan Ying Liang Zhen Ying Dao Qian ,Qie Ren Wei Ying Neng Huo De Xiang Gang Min Zhong Jie Shou ,Dan Zhe Xie Yi Yuan Ye Zhi Wen Liang Zhen Ying You
Goose in Arabic
In order to use Goose in Arabic you have to use the StopWordsArabic class.
>>> from goose import Goose>>> from goose.text import StopWordsArabic
>>> url = 'http://arabic.cnn.com/2013/middle_east/8/3/syria.clashes/index.html'
>>> g = Goose({'stopwords_class': StopWordsArabic})
>>> article = g.extract(url=url)
>>> print article.cleaned_text[:150]
dmshq, swry (CNN) -- 'kdt jht swry@ m`rD@ 'n fSy'l mslH@ m`rD@ lnZm lry'ys bshr l'sd w`l~ Sl@ b"ljysh lHr" tmknt mn lsyTr@ `l~ mstwd`t ll'sl
Goose in Korean
In order to use Goose in Korean you have to use the StopWordsKorean class.
>>> from goose import Goose>>> from goose.text import StopWordsKorean
>>> url='http://news.donga.com/3/all/20131023/58406128/1'
>>> g = Goose({'stopwords_class':StopWordsKorean})
>>> article = g.extract(url=url)
>>> print article.cleaned_text[:150]
gyeonggido yongine jari jabeun mingan siheominjeung jeonmungieob (ju)dijiteoliemssi(www.digitalemc.com).
14nyeonjjae segye gaggugyi tongsin*anjeon*jeonpa gyugyeog siheomgwa injeung han umulman pago issneun i hoesa bagcaegyu daepyoga mannagiro han juingongida.
geuneun jeongijeonja*museontongsin*jadongca jeonjangpum bunyae
Known issues
There are some issues with unicode URLs.
Cookie handling : Some websites need cookie handling. At the moment the only work around is to use the raw_html extraction. For instance:
>>> import urllib2
>>> import goose
>>> url = "http://www.nytimes.com/2013/08/18/world/middleeast/pressure-by-us-failed-to-sway-egypts-leaders.html?hp"
>>> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
>>> response = opener.open(url)
>>> raw_html = response.read()
>>> g = goose.Goose()
>>> a = g.extract(raw_html=raw_html)
>>> a.cleaned_text
u'CAIRO \u2014 For a moment, at least, American and European diplomats trying to defuse the volatile standoff in Egypt thought they had a breakthrough.\n\nAs t'
TODO
- Video html5 tag extraction