Valgrind Reporting Memory Leaks in Flashlight Libraries.

[vchagari]

Bug Description

Running Inference with Valgrind tool reports memory leaks in Flashlight and ArrayFire libraires.

==209== 35,580,804 (472 direct, 35,580,332 indirect) bytes in 1 blocks are definitely lost in loss record 4,810 of 4,817
==209== at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==209== by 0x932E53: std::_Hashtable<int, std::pair<int const, std::vector<fl::lib::text::LexiconDecoderState, std::allocatorfl::lib::text::LexiconDecoderState > >, std::allocator<std::pair<int const, std::vector<fl::lib::text::LexiconDecoderState, std::allocatorfl::lib::text::LexiconDecoderState > > >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) )
==209== by 0x932FA7: std::_Hashtable<int, std::pair<int const, std::vector<fl::lib::text::LexiconDecoderState, std::allocatorfl::lib::text::LexiconDecoderState > >, std::allocator<std::pair<int const, std::vector<fl::lib::text::LexiconDecoderState, std::allocatorfl::lib::text::LexiconDecoderState > > >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<int const, std::vector<fl::lib::text::LexiconDecoderState, std::allocatorfl::lib::text::LexiconDecoderState > >, false>, unsigned long) )
==209== by 0x93018D: fl::lib::text::LexiconDecoder::decodeStep(float const, int, int) )
==209== by 0x5BFE8C: fl::lib::text::Decoder::decode(float const*, int, int) (Decoder.h:54)
==209== by 0xE837608: start_thread (pthread_create.c:477)
==209== by 0x1B495132: clone (clone.S:95)

==4513== 269,528 (752 direct, 268,776 indirect) bytes in 2 blocks are definitely lost in loss record 4,446 of 4,519
==4513== at 0x4C3217F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4513== by 0x17E8E4F59: cudnnCreate (in /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5)
==4513== by 0x7F5A32: fl::getCudnnHandle()
==4513== by 0x86D85A: fl::batchnorm(fl::Variable const&, fl::Variable const&, fl::Variable const&, fl::Variable&, fl::Variable&, std::vector<int, std::allocator > const&, bool, double, double) )
==4513== by 0x81FF31: fl::LayerNorm::forward(fl::Variable const&) )
==4513== by 0x8278BD: fl::UnaryModule::forward(std::vector<fl::Variable, std::allocatorfl::Variable > const&) )
==4513== by 0x858796: fl::ext::forwardSequentialModuleWithPadMask(fl::Variable const&, std::shared_ptrfl::Module, af::array const&) )
==4513== by 0xEF866DA: start_thread (pthread_create.c:463)
==4513== by 0x1D57961E: clone (clone.S:95)

==209== 25,952 (416 direct, 25,536 indirect) bytes in 4 blocks are definitely lost in loss record 4,642 of 4,817
==209== at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==209== by 0x935C73: std::_Hashtable<int, std::pair<int const, std::shared_ptrfl::lib::text::TrieNode >, std::allocator<std::pair<int const, std::shared_ptrfl::lib::text::TrieNode > >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) )
==209== by 0x935E6A: std::__detail::_Map_base<int, std::pair<int const, std::shared_ptrfl::lib::text::TrieNode >, std::allocator<std::pair<int const, std::shared_ptrfl::lib::text::TrieNode > >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](int const&) )
==209== by 0x9356B9: fl::lib::text::Trie::insert(std::vector<int, std::allocator > const&, int, float) )
==209== by 0x91835A: fl::app::asr::buildTrie(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, std::shared_ptrfl::lib::text::LM, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, fl::lib::text::Dictionary const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > > > > > const&, fl::lib::text::Dictionary const&, int, int) )
==209== by 0xE837608: start_thread (pthread_create.c:477)
==209== by 0x1B495132: clone (clone.S:95)

==4513== 13,880 (1,680 direct, 12,200 indirect) bytes in 42 blocks are definitely lost in loss record 4,317 of 4,519
==4513== at 0x4C3217F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4513== by 0x807896: fl::CachingMemoryManager::alloc(bool, unsigned int, long long*, unsigned int) )
==4513== by 0x80278C: fl::MemoryManagerInstaller::MemoryManagerInstaller(std::shared_ptrfl::MemoryManagerAdapter)::{lambda(void*, void**, int, unsigned int, long long*, unsigned int)#3}::_FUN(void*, void**, int, unsigned int, long long*, unsigned int) )
==4513== by 0xB19AAA22: MemoryManagerFunctionWrapper::alloc(bool, unsigned int, long long*, unsigned int) (in /opt/arrayfire/lib/libafcuda.so.3.7.3)
==4513== by 0xB1229295: std::unique_ptr<float [], std::function<void (float*)> > cuda::memAlloc(unsigned long const&) (in /opt/arrayfire/lib/libafcuda.so.3.7.3)
==4513== by 0xB0CD3F6C: cuda::Array::Array(af::dim4 const&) (in /opt/arrayfire/lib/libafcuda.so.3.7.3)
==4513== by 0xB0CD4038: cuda::Array cuda::createEmptyArray(af::dim4 const&) (in /opt/arrayfire/lib/libafcuda.so.3.7.3)
==4513== by 0xB17DD2F0: createHandle(af::dim4 const&, af_dtype) (in /opt/arrayfire/lib/libafcuda.so.3.7.3)
==4513== by 0xB17E0D1D: af_create_handle (in /opt/arrayfire/lib/libafcuda.so.3.7.3)
==4513== by 0x7A49B5B: af_create_handle (in /opt/arrayfire/lib/libaf.so.3.7.3)
==4513== by 0x7B600BD: (anonymous namespace)::initEmptyArray(af_dtype, long long, long long, long long, long long) (in /opt/arrayfire/lib/libaf.so.3.7.3)
==4513== by 0x7B61424: af::array::array(af::dim4 const&, af_dtype) (in /opt/arrayfire/lib/libaf.so.3.7.3

Reproduction Steps

Use Valgrind tool with the inference application.
Example:
valgrind --log-file=/tmp/valgrind_$( date +"%Y_%m_%d-%H_%M_%S" ) --leak-check=full ./bin/inference.exe

Platform and Hardware

Intel Processor
Nvidia GPU
Ubuntu 18.04 LTS

Additional Context

Flashlight Commit: Version 0.31 Tag

[jacobkahn]

@vchagari — can you share more info about your setup? What is inference.exe?