Search results

Switch to <code>Addressable::URI</code> for URI parsing

feature

Status: Open.

#86 In postmodern/spidr;

· postmodern opened on Dec 22, 2022

Switch to using async-http

http

improvement

Status: Open.

#76 In postmodern/spidr;

· postmodern opened on Jan 29, 2022

Add Logging

feature

Status: Open.

#72 In postmodern/spidr;

· postmodern opened on Oct 25, 2019

 ·  

0.8.0

Add low-level HTTP request methods

feature

Status: Open.

#62 In postmodern/spidr;

· postmodern opened on May 25, 2017

Add ignore_paths and ignore_paths_like

feature

Status: Open.

#61 In postmodern/spidr;

· postmodern opened on May 25, 2017

 ·  

0.8.0

Following redirects

Status: Open.

#56 In postmodern/spidr;

· ZackMattor opened on Nov 29, 2016

Infinite path loop

needs info

Status: Open.

#31 In postmodern/spidr;

· ethicalhack3r opened on Feb 1, 2012

SSL session reuse may fail

Status: Open.

#30 In postmodern/spidr;

· nirvdrum opened on Jan 19, 2012

Multithreading

feature

Status: Open.

#26 In postmodern/spidr;

· ethicalhack3r opened on Jun 6, 2011

Automatically detect and parse sitemap.xml

feature

Status: Open.

#19 In postmodern/spidr;

· postmodern opened on Oct 16, 2010

Store history queue in Hash of host:port and paths.

feature

Status: Open.

#18 In postmodern/spidr;

· postmodern opened on Oct 16, 2010

Redirect fails silently - need debug messages

feature

Status: Open.

#13 In postmodern/spidr;

· perplexes opened on Aug 19, 2010