Are ENGINESDIR and MODULESDIR incorrect in openssl@3 ?

[avoidik]

Output of brew config

brew config
HOMEBREW_VERSION: 4.0.27
ORIGIN: https://github.com/Homebrew/brew
HEAD: 32ff40f688a49e02428eb3f20232aac69ffeeea0
Last commit: 2 days ago
Core tap JSON: 06 Jul 20:57 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 10
HOMEBREW_NO_ANALYTICS: set
HOMEBREW_NO_ENV_HINTS: set
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.3 build 1403
Git: 2.41.0 => /opt/homebrew/bin/git
Curl: 7.88.1 => /usr/bin/curl
macOS: 13.4.1-arm64
CLT: 14.3.1.0.1.1683849156
Xcode: N/A
Rosetta 2: false

Output of brew doctor

brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed manually.
You should find replacements for the following formulae:
  gem-cfn-nag

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause formulae that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  opensc

Description of issue

hello all,

looking at the output below, shouldn't the ENGINESDIR point to some other directory like /opt/homebrew/lib/engines-3/?

$ openssl version -a
OpenSSL 3.1.1 30 May 2023 (Library: OpenSSL 3.1.1 30 May 2023)
built on: Tue May 30 12:13:24 2023 UTC
platform: darwin64-arm64-cc
options:  bn(64,64)
compiler: clang -fPIC -arch arm64 -O3 -Wall -DL_ENDIAN -DOPENSSL_PIC -D_REENTRANT -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
OPENSSLDIR: "/opt/homebrew/etc/openssl@3"
ENGINESDIR: "/opt/homebrew/Cellar/openssl@3/3.1.1_1/lib/engines-3"
MODULESDIR: "/opt/homebrew/Cellar/openssl@3/3.1.1_1/lib/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_armcap=0x187d

to this one

$ ls -la /opt/homebrew/lib/engines-3/
ls -la /opt/homebrew/lib/engines-3/
total 0
drwxr-xr-x    7 blah  admin   224 Jul  6 21:43 .
drwxrwxr-x  272 blah  admin  8704 Jul  6 21:43 ..
lrwxr-xr-x    1 blah  admin    55 Jul  6 21:43 capi.dylib -> ../../Cellar/openssl@3/3.1.1_1/lib/engines-3/capi.dylib
lrwxr-xr-x    1 blah  admin    56 Jul  6 21:43 libpkcs11.dylib -> ../../Cellar/libp11/0.4.12/lib/engines-3/libpkcs11.dylib
lrwxr-xr-x    1 blah  admin    63 Jul  6 21:43 loader_attic.dylib -> ../../Cellar/openssl@3/3.1.1_1/lib/engines-3/loader_attic.dylib
lrwxr-xr-x    1 blah  admin    58 Jul  6 21:43 padlock.dylib -> ../../Cellar/openssl@3/3.1.1_1/lib/engines-3/padlock.dylib
lrwxr-xr-x    1 blah  admin    53 Jul  6 21:43 pkcs11.dylib -> ../../Cellar/libp11/0.4.12/lib/engines-3/pkcs11.dylib

instead of that one

$ ls -la /opt/homebrew/Cellar/openssl@3/3.1.1_1/lib/engines-3
total 344
drwxr-xr-x   5 blah  admin    160 Jun 29 09:43 .
drwxr-xr-x  11 blah  admin    352 Jun 29 09:43 ..
-r--r--r--   1 blah  admin  34960 Jun 29 09:43 capi.dylib
-r--r--r--   1 blah  admin  98864 Jun 29 09:43 loader_attic.dylib
-r--r--r--   1 blah  admin  34976 Jun 29 09:43 padlock.dylib

probably the same issue is valid for MODULESDIR too, /opt/homebrew/lib/ossl-modules vs /opt/homebrew/Cellar/openssl@3/3.1.1_1/lib/ossl-modules

I guess both could be fixed by providing --enginesdir and --modulesdir at compilation-time

however in the same time

$ pkg-config --variable=enginesdir libcrypto
/opt/homebrew/Cellar/openssl@3/3.1.1_1/lib/engines-3
$ pkg-config --variable=modulesdir libcrypto
/opt/homebrew/Cellar/openssl@3/3.1.1_1/lib/ossl-modules
$ cat /opt/homebrew/opt/openssl@3/lib/pkgconfig/libcrypto.pc
prefix=/opt/homebrew/Cellar/openssl@3/3.1.1_1
exec_prefix=${prefix}
libdir=/opt/homebrew/Cellar/openssl@3/3.1.1_1/lib
includedir=${prefix}/include
enginesdir=${libdir}/engines-3
modulesdir=${libdir}/ossl-modules

Name: OpenSSL-libcrypto
Description: OpenSSL cryptography library
Version: 3.1.1
Libs: -L${libdir} -lcrypto
Libs.private:
Cflags: -I${includedir}

[SMillerDev]

As you can see one does symlink to the other. The only problem would be that anything in that directory is not persistent between updates but I don't know if it should be either.

[avoidik]

Yes, it does have symlinks, but by default openssl would use another directory without symlinks in it. This makes it impossible to use external engines (libp11 in this case) in openssl without overriding configuration or relinking symlinks.

[cho-m]

In Homebrew, we only started linking OpenSSL recently, so HOMEBREW_PREFIX/lib path wasn't even an option. The current paths are at least functional for openssl@3 but may not be optimal behavior for multiple formulae.

The tricky thing is that Homebrew uses a symlink directory structure which is sometimes not supported by build scripts (i.e. separate directories for installing files and for reading/loading files from). For OpenSSL, this would require modifying generated Makefile:

• install/load path: https://github.com/openssl/openssl/blob/openssl-3.1.1/Configurations/unix-Makefile.tmpl#L307-L308

  ENGINESDIR=$(libdir)/engines-{- $sover_dirname -}
  MODULESDIR=$(libdir)/ossl-modules
  

• pkg-config path: https://github.com/openssl/openssl/blob/openssl-3.1.1/Configurations/unix-Makefile.tmpl#L1403-L1404

              echo 'enginesdir=$${libdir}/engines-{- $sover_dirname -}'; \
              echo 'modulesdir=$${libdir}/ossl-modules'; \
  

---

One option would be to modify Makefile, maybe like:

    inreplace "Makefile" do |s|
      enginesdir = s.get_make_var("ENGINESDIR").sub("$(libdir)", "#{HOMEBREW_PREFIX}/lib")
      modulesdir = s.get_make_var("MODULESDIR").sub("$(libdir)", "#{HOMEBREW_PREFIX}/lib")
      s.gsub! '-DENGINESDIR="\"$(ENGINESDIR)', "-DENGINESDIR=\"\\\"#{enginesdir}"
      s.gsub! '-DMODULESDIR="\"$(MODULESDIR)', "-DMODULESDIR=\"\\\"#{modulesdir}"
      s.gsub!(/(echo '(?:enginesdir|modulesdir)=)\$\$\{libdir\}/, "\\1#{HOMEBREW_PREFIX}/lib")
    end

I guess another option would be to run build step with different paths (make ENGINESDIR=... MODULESDIR=...) and the install step with the default paths. Would still need to inreplace pkgconfig file as those aren't impacted by Makefile variables since they use pkgconfig variable ${libdir}. Something like following (though hardcoding subdir may be less ideal than reading it and not sure if soversion numbering may change from version number):

    system "make", "ENGINESDIR=#{HOMEBREW_PREFIX}/lib/engines-#{version.major}",
                   "MODULESDIR=#{HOMEBREW_PREFIX}/lib/ossl-modules"
    system "make", "install", "MANDIR=#{man}", "MANSUFFIX=ssl"
    inreplace  lib/"pkgconfig/libcrypto.pc", /^((?:enginesdir|modulesdir)=)\$\{libdir\}/, "\\1#{HOMEBREW_PREFIX}/lib"
    system "make", "test"