Bug fix 🐛

Http req @timeout annotation not working

The @timeout annotation for HTTP requests is not working in the latest release. This issue is likely a side effect of recent bug fixes, including those for goroutine leaks and nuclei hang/stuck issues. The problem seems to stem from the context in httpclientpool being always overwritten. Adding selective overwrite has resolved the issue

• closes timeout request annotation not working in templates #5137

Max Host Error causes input to be skipped

Nuclei was skipping active domains or IPs due to errors returned while executing a template on closed or filtered ports. These errors were being matched across the maxhosterrors set of errors. This issue was partly addressed by updating the definitions of errors that should be skipped in maxhosterrors. In addition, the hosterrorcache has been refactored to consider the address as host:port instead of host. This change avoids counting legitimate port closed errors against the host.

• closes Nuclei skips the active ip from scanning problem #5177

Performance Improvements 🚀

For complete context, refer projectdiscovery/fastdialer#283. In summary, errors were being stacked when a domain, such as [hackerone.com](http://hackerone.com), resolved to multiple IPs. This issue occurred during attempts to connect to closed ports, significantly reducing the scan speed, particularly in the middle or end of the scan.

------------------------------
Command: nuclei -u https://hackerone.com -stats -elog latest.log -mhe 5000
Max RSS: 465 MB
Sys Time: 660.651µs
User Time: 719.72µs
Actual Time: 7m21.824803042s
Voluntary Context Switch (nvcsw): 10464

[0:07:12] | Templates: 7972 | Hosts: 1 | RPS: 25 | Matched: 24 | Errors: 938 | Requests: 11197/12689 (88%)

After several iterations in fastdialer and changes in retryablehttp, we have addressed this issue. We implemented a singleflight parallel dial on all IPs during the first dial of a given address. This approach effectively prevents the significant speed drop in the middle or end of the scan. As a result, we have observed remarkable speed improvements of 40-70%, depending on the target and concurrency levels.

------------------------------
Command: ./nuclei -u https://hackerone.com -stats -elog latest.log -mhe 5000
Max RSS: 474 MB
Sys Time: 438.782µs
User Time: 236.765µs
Actual Time: 2m56.388615959s
Voluntary Context Switch (nvcsw): 10825

[0:02:46] | Templates: 7972 | Hosts: 1 | RPS: 67 | Matched: 25 | Errors: 930 | Requests: 11186/12689 (88%)

• closes Closed Port connections have very high timeout #5153
• closes Add port status caching to reduce redundant and heavy calls #5152
• closes incorrect port in http variables + context deadline on closed ports #5150

Enhancements 🧨

Improving Error Ecosystem in Nuclei

This PR, implementing the errkit package from projectdiscovery/errkit, enhances error accuracy and user-friendliness. For more details, refer to errkit.

In the context of nuclei, we have reduced vague errors like context deadline exceeded and improved the -elog output format to include address , kind and attrs (optional)

the overall result of this change can be verified using -elog output , sorted , uniq errors

Before

152 context deadline exceeded
  23 i/o timeout
  18 [:RUNTIME] ztls fallback failed <- dial tcp [2606:4700:90d1:5cc5:4243:0:a7a8:efc4]:21: i/o timeout
  15 ---
  12 could not connect to any address found for host
  10 [:RUNTIME] ztls fallback failed <- dial tcp [2606:4700:90d1:5cc5:4243:0:a7a8:efc4]:25: i/o timeout
   9 [:RUNTIME] ztls fallback failed <- dial tcp [2606:4700:90d1:5cc5:4243:0:a7a8:efc4]:22: i/o timeout
...

refer https://gist.github.com/tarunKoyalwar/8ddb1fc5a9e21dbd90ac26d2330c705b#file-old_unique_errors-txt for complete output

After

 288 port closed or filtered
   9 could not connect to host
   5 i/o timeout
   4 context deadline exceeded
   2 EOF
   1 tls: handshake failure
   1 context cancelled before establishing connection

• In addition to the actual error, the newly introduced error kind can be used to summarize the error output and identify

cat latest.log| jq -r '.kind' | sort | uniq -c                
 288 network-permanent-error
   8 network-temporary-error
   2 null
  12 unknown-error

• and an understanding of errors split by address

cat latest.log | jq -r '"\(.address)|\(.kind)"' | sort | uniq -c | sort -r
  75 hackerone.com:5432|network-permanent-error
  68 hackerone.com:3306|network-permanent-error
  25 hackerone.com:21|network-permanent-error
  11 hackerone.com:445|network-permanent-error
  11 hackerone.com:443|unknown-error
  11 hackerone.com:25|network-permanent-error
  10 hackerone.com:22|network-permanent-error
  ...

• closes Use WithTimeoutCause instead of WithTimeout #5104

Other Improvements

• Register wrapper driver to use fastdialer in nuclei/postgres js module
• Fix unexpected resource closure in ThreadSafeNucleiEngine (closes 使用sdk包创建的并发安全的引擎，因共享资源的释放无法正常执行完全部任务 #5202)
• Introduce Go Style debug args for experimental or debug features

 NUCLEI_ARGS="host-error-stats" cmdutil ./nuclei -u https://hackerone.com -stats -elog newupdate2.log -c 200

• Set Js Engine Timeout to 1.5 x -timeout
• Implement early exit on unresponsive hosts for network and javascript protocols
• Set Http ResponseHeaderTimeout to 10 Sec