python - Scrapy Spider not Following Links -

-

i'm writing scrapy spider crawl today's nyt articles homepage, reason doesn't follow links. when instantiate link extractor in scrapy shell http://www.nytimes.com, extracts list of article urls le.extract_links(response), can't crawl command (scrapy crawl nyt -o out.json) scrape homepage. i'm sort of @ wit's end. because homepage not yield article parse function? appreciated. 

from datetime import date                                                         import scrapy                                                                    scrapy.contrib.spiders import rule                                          scrapy.contrib.linkextractors import linkextractor                            ..items import newsarticle                                                   open('urls/debug/nyt.txt') debug_urls:                                       debug_urls = debug_urls.readlines()                                           open('urls/release/nyt.txt') release_urls:                                   release_urls = release_urls.readlines() # ["http://www.nytimes.com"]                                   today = date.today().strftime('%y/%m/%d')                                        print today                                                                        class nytspider(scrapy.spider):                                                      name = "nyt"                                                                     allowed_domains = ["nytimes.com"]                                                start_urls = release_urls                                                           rules = (                                                                                   rule(linkextractor(allow=(r'/%s/[a-z]+/.*\.html' % today, )),                            callback='parse', follow=true),                                        )                                                                                    def parse(self, response):                                                              article = newsarticle()                                                                                  story in response.xpath('//article[@id="story"]'):                                  article['url'] = response.url                                                       article['title'] = story.xpath(                                                             '//h1[@id="story-heading"]/text()').extract()                               article['author'] = story.xpath(                                                            '//span[@class="byline-author"]/@data-byline-name'                          ).extract()                                                                      article['published'] = story.xpath(                                                      '//time[@class="dateline"]/@datetime').extract()                         article['content'] = story.xpath(                                                        '//div[@id="story-body"]/p//text()').extract()                           yield article

i have found solution problem. doing 2 things wrong:

  1. i needed subclass crawlspider rather spider if wanted automatically crawl sublinks.
  2. when using crawlspider, needed use callback function rather overriding parse. per docs, overriding parse breaks crawlspider functionality.

Comments

Post a Comment

Popular posts from this blog

PHP DOM loadHTML() method unusual warning -

-
i had following code works fine on localhost: $document = new domdocument(); $document->loadhtml($p_result); $form = $document->getelementsbytagname('form')->item(0); // code continues using $form variable after same code updated on outside server, loadhtml() failed , issued warning. warning: domdocument::loadhtml() expects parameter 1 valid path, string given in path/name/to/script.php returned null instead of object code pretty gets fatal error. note contents of $p_result absolutely same on outside server , on localhost. but why displays kind of warning , why not work? doesn't loadhtml() expects argument 1 string in first place? why method expects parameter 1 valid path ? just make clear i'm not calling loadhtmlfile() , i'm calling loadhtml() . thanks. you're affected one of php bugs . issue present in php 5.6.8 , 5.6.9. have affected php version on server, , bug-free version on localhost. the bug forbids null
Read more

python - How to create jsonb index using GIN on SQLAlchemy? -

-
here's current code creating index jsonb. index("mytable_data_idx_id_key", mytable.data['id'].astext, postgresql_using='gin') but got error. sqlalchemy.exc.programmingerror: (psycopg2.programmingerror) data type text has no default operator class access method "gin" hint: must specify operator class index or define default operator class data type. [sql: "create index event_data_idx_id_key on event using gin ((data ->> 'id'))"] is there way create index on sqlalchemy? the postgresql specific sqlalchemy docs @ http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#operator-classes mention postgresql_ops dictionary provide "operator class" used postgresql, , provide example illustrating use: index('my_index', my_table.c.id, my_table.c.data, postgresql_ops={ 'data': 'text_pattern_ops',
Read more

c# - TransactionScope not rolling back although no complete() is called -

-
i'm using transactionscope rollback transaction fail bool errorreported = false; action<importerrorlog> newerrorcallback = e => { errorreported = true; errorcallback(e); }; using (var transaction = new transactionscope()) { foreach (importtaskdefinition task in taskdefinition) { loader.load(streamfile, newerrorcallback, task.destinationtable, processingtaskid); } if (!errorreported) transaction.complete(); } i'm sure there no transactionscope started ahead or after code. i'm using entity framework insert in db. regardless state of errorreported transaction never rolled in case of error. what missing ? transactionscope sets transaction.current . that's does. wants transacted must @ property. i believe ef each time connection opened reason. probably, connection open when scope installed. open connection inside of scope or enlist manually. ef has nasty "design decision": default opens
Read more