scrapy-extract links
from scrapy.spiders import CrawlSpider
class SuperSpider(CrawlSpider):
name = 'extractor'
allowed_domains = ['en.wikipedia.org']
start_urls = ['https://en.wikipedia.org/wiki/Python_(programming_language)']
base_url = 'https://en.wikipedia.org'
def parse(self, response):
for link in response.xpath('//div/p/a'):
yield {
"link": self.base_url + link.xpath('.//@href').get()
}
Are there any code examples left?
New code examples in category Python
-
Python 2023-04-11 03:04:20
-
Python 2022-03-27 22:40:04 pycharm no module named
-
Python 2022-03-27 22:25:05 assign multiple variablesin one line
-
Python 2022-03-27 22:20:02 levenshtein distance
-
Python 2022-03-27 21:35:09 get text from url python last slash
-
Python 2022-03-27 21:30:30 df concatenate df
-
Python 2022-03-27 21:25:09 python odd or even
-
Python 2022-03-27 21:15:32 python include function from another file
-
Python 2022-03-27 21:10:01 color module python
-
Python 2022-03-27 21:00:27 python tkinter cursor types