Multiple NucleiEngine instances cause HTTP scan failures

[Teruya-Higashi]

Nuclei version:

v3.2.6

Current Behavior:

• Initializing multiple instances of NucleiEngine and executing HTTP scans.
• After executing the scan, the instance is closed.
• From the second instance initialization onwards, the following log is output during HTTP requests, causing the scan to fail.

  [WRN] [test_status_200] Could not execute request for http://127.0.0.1:54961: [:RUNTIME] got err while executing http://127.0.0.1:54961 <- GET http://127.0.0.1:54961 giving up after 2 attempts: Get "http://127.0.0.1:54961": leveldb: closed

Expected Behavior:

When initializing multiple instances of NucleiEngine and executing HTTP scans, the scans are performed without any errors occurring in any of the HTTP requests.

Steps To Reproduce:

1. Execute the following test code:

   func TestThreeTimesNewNucleiEngine(t *testing.T) {
       sv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
           w.WriteHeader(http.StatusOK)
       }))
       defer sv.Close()
   
       wd, _ := os.Getwd()
       templatePath := filepath.Join(wd, "test_status_200.yaml")
   
       var mu sync.Mutex
       events := make([]*output.ResultEvent, 0, 3)
       fn := func(event *output.ResultEvent) {
           mu.Lock()
           defer mu.Unlock()
           events = append(events, event)
       }
   
       for i := 0; i < 3; i++ {
           ne, err := nuclei.NewNucleiEngine(
               nuclei.WithTemplatesOrWorkflows(nuclei.TemplateSources{Templates: []string{templatePath}}),
               nuclei.WithVerbosity(nuclei.VerbosityOptions{Debug: true}),
           )
           require.Nil(t, err)
           ne.LoadTargets([]string{sv.URL}, false)
           err = ne.ExecuteWithCallback(fn)
           require.Nil(t, err)
           ne.Close()
       }
   
       require.Len(t, events, 3)
   }

   • The contents of test_status_200.yaml are as follows:

     id: test_status_200
     info:
       name: test_status_200
       author: test
       severity: Low
     http:
       - raw:
         - |+
           GET / HTTP/1.1
           Host: {{Hostname}}
         payloads: {}
         matchers:
           - type: status
             status:
               - 200

2. The following log is output, causing the test to fail:

   [INF] [test_status_200] Dumped HTTP request for http://127.0.0.1:55719
   
   GET / HTTP/1.1
   Host: 127.0.0.1:55719
   User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.1983.75
   Connection: close
   Accept-Encoding: gzip
   
   [DBG] [test_status_200] Dumped HTTP response http://127.0.0.1:55719
   
   HTTP/1.1 200 OK
   Connection: close
   Date: Fri, 10 May 2024 09:43:52 GMT
   Content-Length: 0
   
   [INF] [test_status_200] Dumped HTTP request for http://127.0.0.1:55719
   
   GET / HTTP/1.1
   Host: 127.0.0.1:55719
   User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36
   Connection: close
   Accept-Encoding: gzip
   
   [WRN] [test_status_200] Could not execute request for http://127.0.0.1:55719: [:RUNTIME] got err while executing http://127.0.0.1:55719 <- GET http://127.0.0.1:55719 giving up after 2 attempts: Get "http://127.0.0.1:55719": leveldb: closed
   [INF] [test_status_200] Dumped HTTP request for http://127.0.0.1:55719
   
   GET / HTTP/1.1
   Host: 127.0.0.1:55719
   User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/601.6.14 (KHTML, like Gecko) Version/9.1.2 Safari/601.6.14
   Connection: close
   Accept-Encoding: gzip
   
   [WRN] [test_status_200] Could not execute request for http://127.0.0.1:55719: [:RUNTIME] got err while executing http://127.0.0.1:55719 <- GET http://127.0.0.1:55719 giving up after 2 attempts: Get "http://127.0.0.1:55719": leveldb: closed
   --- FAIL: TestThreeTimesNewNucleiEngine (4.85s)
       sample_test.go:59: 
           	Error Trace:	/Users/teruyahigashi/projects/nuclei/lib/tests/sample_test.go:59
           	Error:      	"[0x14000bffb88]" should have 3 item(s), but has 1
           	Test:       	TestThreeTimesNewNucleiEngine
   FAIL

Anything else:

By removing the following from the Close method of NucleiEngine, the issue no longer occurs. However, removing this code will cause a leak.

nuclei/lib/sdk.go

Line 210 in 5957381

protocolstate.Close()