Hello!

First amazing script, lately got more familiar with it, and now use almost in every security assesment! It helps me to get very clear insight of web application and makes job more easy! Have some ideas and functions what maybe could be added! When ready gonna let you know you!

Second thing i noticed that on large scale crawl, I,m getting this error: Database karta.db initialized, crawl started with 10 threads
[==== ] 966 of 7928 pages processed in 67 minutes Exception in thread Thread-9:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/share/htcap/core/crawl/crawler_thread.py", line 62, in run
self.crawl()
File "/usr/share/htcap/core/crawl/crawler_thread.py", line 212, in crawl
probe = self.send_probe(request, errors)
File "/usr/share/htcap/core/crawl/crawler_thread.py", line 161, in send_probe
probeArray = self.load_probe_json(jsn)
File "/usr/share/htcap/core/crawl/crawler_thread.py", line 99, in load_probe_json
return json.loads(jsn)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 367, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 4 column 1 - line 4 column 249 (char 58 - 306)
Maybe its just my system warning! I,m Using Kali Linux but its highly modified and build up for me and maybe because of this!

It seems that there is no impact on work flow, it continues crawling and after when working on database there is no error!

Anyways good script and good luck developing it!