Build dynamic libraries (#357)

This commit re-adds support for building a dynamically linked ksh with libast, libdll, libcmd, and libshell available as dynamic libraries for other applications to use. Previously, this required nmake; as of this commit, we use a helper script dylink.sh instead, which does all the necessary platform-specific things and can be easily extended to add support for new platforms. Currently tested and supported platforms are Linux, Android, macOS, FreeBSD, NetBSD, OpenBSD, and illumos/Solaris. Summary of important changes: src/cmd/ksh93/SHOPT.sh: - Enable SHOPT_DYNAMIC by default. This enables the '-f' option to the 'builtin' command, making it possible to load builtins from dynamic/shared libraries. This mostly makes sense for a dynamically linked ksh, as that is needed to allow ksh and builtins to share the same libraries such as libast. src/cmd/INIT/dylink.sh: - Added. This is a dynamic linking tool to invoke from Mamfiles. - Disable dynamic libraries if AST_NO_DYLIB is exported. - Disable dynamic libraries on untested systems unless AST_DYLIB_TEST is exported. bin/package: - Add support for $INSTALLROOT/dyn subdirectory. - Set LD_LIBRARY_PATH, etc. to point to dyn/lib. - Prefix dyn/bin to $PATH. - do_install(): - Prefer dynamically linked versions for installing, if they have been built. - If dyn/lib exists, install not only the dynamic libraries but also the dev stuff: AST headers and section 3 manual pages. - Remove generation of .paths file. It causes regression test failures because of how built-ins are bound to different paths when found via a .paths file. This functionality is deprecated and may be removed from ksh 93u+m/1.1 and up. bin/shtests: - Relaunch self via 'bin/package use' to pull in the necessary environment stuff to make the pre-installed dynamically linked binaries work. src/cmd/INIT/make.probe: - Add probe for the --as-needed and --no-as-needed linker options. To make dynamic libraries work on Linux arm64, we need to pass --no-as-needed to the linker on Linux to avoid glibc interfering with libast in identically named functions like AST regcomp(3). Linux systems vary as to whether --as-needed is on by default. Thanks to Lukáš Zaoral at Red Hat for finding the key to the fix. src/cmd/INIT/mamprobe.sh: - Update version number. This change causes mamake to redo the probe (it checks for a change in mamprobe but not in make.probe; I should probably fix that at some point). We need mamake to redo the probe for the --no-as-needed addition to take effect. **/Mamfile: - Use the --no-as-needed probe result (either -Wl,-no-as-needed or empty) from ${mam_cc_LD_NOASNEEDED} whenever linking anything; add it to LDFLAGS before invoking dylink. - ksh and shcomp also need the results of the existing --export-dynamic probe; this option is needed on Linux as without it ksh fails to export symbols to loadable builtins. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821806 Thanks to Johnothan King. src/lib/libast/features/map.c, src/lib/libast/regex/regexec.c, src/lib/libast/regex/regnexec.c, src/lib/libast/regex/regrexec.c, src/lib/libast/regex/regsubexec.c: - Map regcomp and all the other AST regex functions via macros so they get an _ast_ prefix. This allows a functional dynamically linked executable to exist under ASan because the regcomp function gets overriden by ASan, despite linking with --no-as-needed. Thanks to Johnothan King. src/lib/libast/comp/errno.c: - Make one trivial change (copyright year) to force libast to relink -- which then triggers a relink of everything else. Co-authored-by: Johnothan King <[email protected]> Co-authored-by: Lukáš Zaoral <[email protected]>
ksh93 · Mar 24, 2024 · f2bc1f4 · f2bc1f4
1 parent a2fbf54
commit f2bc1f4
Show file tree

Hide file tree

Showing 27 changed files with 415 additions and 265 deletions.
diff --git a/NEWS b/NEWS
@@ -2,6 +2,16 @@ This documents significant changes in the dev branch of ksh 93u+m.
 For full details, see the git log at: https://github.com/ksh93/ksh
 Uppercase BUG_* IDs are shell bug IDs as used by the Modernish shell library.
 
+2024-03-24:
+
+- We now support building a dynamically linked ksh 93u+m with libast,
+ libdll, libcmd, and libshell available as dynamic libraries for other
+ applications to use. The dynamically linked version is built in a 'dyn'
+ subdirectory; there are no changes to the statically linked version. See
+ README.md for more information. Dynamic linking is tested and supported
+ on Linux, Android, macOS, FreeBSD, NetBSD, OpenBSD, illumos and Solaris.
+ Please test this on other platforms and send us your reports and patches.
+
 2024-03-22:
 
 - Fixed a crash in the getconf built-in that could occur after the 'builtin'

diff --git a/README.md b/README.md
@@ -78,6 +78,11 @@ The compiled binaries are stored in the `arch` directory, in a subdirectory
 that corresponds to your architecture. The command `bin/package host type`
 outputs the name of this subdirectory.
 
+Dynamically linked binaries, if supported for your system, are stored in
+`dyn/bin` and `dyn/lib` subdirectories of your architecture directory.
+If built, they are built in addition to the statically linked versions.
+Export `AST_NO_DYLIB` to deactivate building dyanmically linked versions.
+
 If you have trouble or want to tune the binaries, you may pass additional
 compiler and linker flags. It is usually best to export these as environment
 variables *before* running `bin/package` as they could change the name of
@@ -141,6 +146,13 @@ available, is installed in `share/man`.
 Destination directories with whitespace or shell pattern characters in their
 pathnames are not yet supported.
 
+If a dynamically linked version of ksh and associated commands has been
+built, then the `install` subcommand will prefer that: commands, dynamic
+libraries and associated header files will be installed then. To install the
+statically linked version instead (and skip the header files), either delete
+the `dyn` subdirectory, or export `AST_NO_DYLIB=y` before building to prevent
+it from being created in the first place.
+
 ## What is ksh93?
 
 The following is the official AT&T description from 1993 that came with the

diff --git a/bin/package b/bin/package
@@ -115,7 +115,7 @@ command=${0##*/}
 case $(getopts '[-][123:xyz]' opt --xyz 2>/dev/null; echo 0$opt) in
 0123) USAGE=$'
 [-?
-@(#)$Id: '$command$' (ksh 93u+m) 2024-03-20 $
+@(#)$Id: '$command$' (ksh 93u+m) 2024-03-23 $
 ]
 [-author?Glenn Fowler <[email protected]>]
 [-author?Contributors to https://github.com/ksh93/ksh]
@@ -205,6 +205,9 @@ case $(getopts '[-][123:xyz]' opt --xyz 2>/dev/null; echo 0$opt) in
  to install the \acommand\as directly,
  or a temporary directory like \a/tmp/pkgtree/usr\a
  to prepare for packaging with operating system-specific tools.
+ Each \acommand\a is first searched in \b$INSTALLROOT/dyn/bin\b
+ (dynamically linked version)
+ and then in \b$INSTALLROOT/bin\b.
  If no \acommand\a is specified,
  then \aksh\a and \ashcomp\a are assumed.]
  [+make\b [ \apackage\a ]] [ \aoption\a ... ]] [ \atarget\a ... ]]?Build
@@ -464,8 +467,10 @@ DESCRIPTION
  any necessary subdirectories are created. dest_dir can be a directory
  like /usr/local to install the commands directly, or a temporary
  directory like /tmp/pkgtree/usr to prepare for packaging with
- operating system-specific tools. If no command is specified, then ksh
- and shcomp are assumed.
+ operating system-specific tools. Each command is first searched in
+ $INSTALLROOT/dyn/bin (dynamically linked version) and then in
+ $INSTALLROOT/bin. If no command is specified, then ksh and shcomp
+ are assumed.
  make [ package ] [ option ... ] [ target ... ]
  Build and install. The default target is install, which makes and
  installs package. If the standard output is a terminal then the
@@ -543,7 +548,7 @@ SEE ALSO
  pkgadd(1), pkgmk(1), rpm(1), sh(1), tar(1), optget(3)
 
 IMPLEMENTATION
- version package (ksh 93u+m) 2024-03-20
+ version package (ksh 93u+m) 2024-03-23
  author Glenn Fowler <[email protected]>
  author Contributors to https://github.com/ksh93/ksh
  copyright (c) 1994-2012 AT&T Intellectual Property
@@ -1611,6 +1616,14 @@ trace()
  "$@"
 )
 
+trace()
+(
+ PS4="${action}: executing: "
+ exec 2>&1 # trace to standard output
+ set -o xtrace
+ "$@"
+)
+
 # cc checks
 #
 # CC: compiler base name name
@@ -2049,14 +2062,14 @@ case $x in
  do eval \$a
  eval \"
  case \\\$LD_LIBRARY\${v}_PATH: in
- \\\$d/lib:*)
+ \\\$d/dyn/lib:*)
  ;;
  *) x=\\\$LD_LIBRARY\${v}_PATH
  case \\\$x in
  ''|:*) ;;
  *) x=:\\\$x ;;
  esac
- LD_LIBRARY\${v}_PATH=\$d/lib\\\$x
+ LD_LIBRARY\${v}_PATH=\$d/dyn/lib\\\$x
  export LD_LIBRARY\${v}_PATH
  p=1
  ;;
@@ -2091,30 +2104,30 @@ case $x in
  ;;
  esac
  case $LIBPATH: in
- $INSTALLROOT/bin:$INSTALLROOT/lib:*)
+ $INSTALLROOT/dyn/bin:$INSTALLROOT/dyn/lib:*)
  ;;
  *) case $LIBPATH in
  '') LIBPATH=/usr/lib:/lib ;;
  esac
- LIBPATH=$INSTALLROOT/bin:$INSTALLROOT/lib:$LIBPATH
+ LIBPATH=$INSTALLROOT/dyn/bin:$INSTALLROOT/dyn/lib:$LIBPATH
  $show LIBPATH=$LIBPATH
  $show export LIBPATH
  export LIBPATH
  ;;
  esac
  case $SHLIB_PATH: in
- $INSTALLROOT/lib:*)
+ $INSTALLROOT/dyn/lib:*)
  ;;
- *) SHLIB_PATH=$INSTALLROOT/lib${SHLIB_PATH:+:$SHLIB_PATH}
+ *) SHLIB_PATH=$INSTALLROOT/dyn/lib${SHLIB_PATH:+:$SHLIB_PATH}
  $show SHLIB_PATH=$SHLIB_PATH
  $show export SHLIB_PATH
  export SHLIB_PATH
  ;;
  esac
  case $DYLD_LIBRARY_PATH: in
- $INSTALLROOT/lib:*)
+ $INSTALLROOT/dyn/lib:*)
  ;;
- *) DYLD_LIBRARY_PATH=$INSTALLROOT/lib${DYLD_LIBRARY_PATH:+:$DYLD_LIBRARY_PATH}
+ *) DYLD_LIBRARY_PATH=$INSTALLROOT/dyn/lib${DYLD_LIBRARY_PATH:+:$DYLD_LIBRARY_PATH}
  $show DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
  $show export DYLD_LIBRARY_PATH
  export DYLD_LIBRARY_PATH
@@ -2147,6 +2160,12 @@ case $x in
  test -n "${MANPATH+s}" && MANPATH=$INSTALLROOT/man:$MANPATH
  ;;
  esac
+ case $PATH: in
+ $INSTALLROOT/dyn/bin:*)
+ ;;
+ *) PATH=$INSTALLROOT/dyn/bin:$PATH
+ ;;
+ esac
  case $FPATH: in
  $INSTALLROOT/fun:*)
  ;;
@@ -2636,14 +2655,25 @@ do_install() # dir [ command ... ]
  done
  # set install directories
  bindir=$dd/bin
- man1dir=${dd:-/usr}/share/man/man1
  fundir=${dd:-/usr}/share/fun
+ mandir=${dd:-/usr}/share/man
+ man1dir=$mandir/man1
+ man3dir=$mandir/man3
+ libdir=$dd/lib
+ includedir=$dd/include
  # and off we go
  trace mkdir -p "$bindir" "$man1dir" || exit
  for f
- do # install executable
- trace cp "bin/$f" "$bindir/" || exit
- # install manual and autoloadable functions
+ do # macOS throws code signature error if 'cp' overwrites Mach-O binary; must remove first
+ if test -e "$bindir/$f"
+ then rm -f "$bindir/$f" || exit
+ fi
+ # install executable
+ if test -f "dyn/bin/$f"
+ then trace cp "dyn/bin/$f" "$bindir/" || exit
+ else trace cp "bin/$f" "$bindir/" || exit
+ fi
+ # install manual pages and autoloadable functions
  case $f in
  ksh) trace cp "$PACKAGEROOT/src/cmd/ksh93/sh.1" "$man1dir/ksh.1" || exit
  trace mkdir -p "$fundir" || exit
@@ -2662,6 +2692,34 @@ do_install() # dir [ command ... ]
  ;;
  esac
  done
+ if test -d "dyn/lib"
+ then trace mkdir -p "$libdir" "$man3dir" "$includedir"
+ # install libraries
+ # note: to copy symlinks with BSD cp, we need to specify -R as well as -P, contra POSIX
+ set +o noglob
+ for f in dyn/lib/*
+ do set -o noglob
+ # macOS throws code signature error if 'cp' overwrites Mach-O binary; must remove first
+ if test -e "$libdir/$f"
+ then rm -f "$libdir/$f" || exit
+ fi
+ # to copy symlinks with BSD cp, we need to specify -R as well as -P, contra POSIX
+ trace cp -PR "$f" "$libdir"/
+ done
+ # install developer stuff
+ test -d "$includedir/ast" && trace rm -rf -- "$includedir/ast"
+ trace cp -R "include/ast" "$includedir"/ || exit
+ printf "install: writing into %s:" "$man3dir"
+ (
+ set +o noglob
+ for f in man/man3/*.3
+ do printf " %sast" "${f##*/}"
+ # give .3 pages an ast suffix to avoid conflicts
+ sed '/^\.TH .* 3/ s/ 3/ 3ast/' "$f" > "$man3dir/${f##*/}ast"
+ done
+ )
+ printf '\n'
+ fi
 }
 
 error_status=0
@@ -3067,81 +3125,9 @@ make|view)
  ;;
  esac
 
- # mamprobe data should have been generated by this point
+ # transition - TODO: remove when dust settles
 
- case $exec in
- '') if test ! -f "$INSTALLROOT/bin/.paths" || test -w "$INSTALLROOT/bin/.paths"
- then N='
-'
- b= f= h= n= p= u= B= L=
- if test -f $INSTALLROOT/bin/.paths
- then exec < $INSTALLROOT/bin/.paths
- while read x
- do case $x in
- '#'?*) case $h in
- '') h=$x ;;
- esac
- ;;
- *BUILTIN_LIB=*) b=$x
- ;;
- *FPATH=*) f=$x
- ;;
- *PLUGIN_LIB=*) p=$x
- ;;
- *) case $u in
- ?*) u=$u$N ;;
- esac
- u=$u$x
- ;;
- esac
- done
- fi
- ifs=$IFS
- m=
- case $p in
- ?*) b=
- ;;
- esac
- case $b in
- ?*) IFS='='
- set $b
- IFS=$ifs
- shift
- p="PLUGIN_LIB=$*"
- case $b in
- [Nn][Oo]*) p=no$p ;;
- esac
- m=1
- ;;
- esac
- case $f in
- '') f="FPATH=../fun"
- m=1
- ;;
- esac
- case $h in
- '') h='# use { no NO } prefix to permanently disable #' ;;
- esac
- case $p in
- '') p="PLUGIN_LIB=cmd"
- if ( set +o noglob
- grep '^setv mam_cc_DIALECT .* EXPORT=[AD]LL' "$INSTALLROOT"/lib/probe/C/mam/*
- ) >/dev/null 2>&1
- then p=no$p
- fi
- m=1
- ;;
- esac
- case $m in
- 1) case $u in
- ?*) u=$N$u ;;
- esac
- echo "$h$N$p$N$f$N$u" > $INSTALLROOT/bin/.paths
- ;;
- esac
- fi
- ;;
- esac
+ rm -f "$INSTALLROOT/bin/.paths" &
 
  # run from separate copies since ksh may be rebuilt
 
@@ -3405,6 +3391,13 @@ use) # finalize the environment
  export PACKAGE_USE
  unset LC_ALL # respect the user's locale again; avoids multibyte corruption
 
+ for s in "$INSTALLROOT/dyn/bin/ksh" "$INSTALLROOT/$OK/bin/ksh" "$INSTALLROOT/bin/ksh"
+ do if "$s" -c ':' 2>/dev/null
+ then export SHELL=$s
+ break
+ fi
+ done
+
  # run the command
 
  case $run in

diff --git a/bin/shtests b/bin/shtests
@@ -1,5 +1,5 @@
 # Wrapper script to run the ksh93 regression tests directly.
-# By Martijn Dekker <[email protected]> 2020-05-14, 2023-03-17
+# By Martijn Dekker <[email protected]> 2020-05-14, 2023-03-17, 2024-03-05
 # Public domain. https://creativecommons.org/publicdomain/zero/1.0/
 #
 # The manual: bin/shtests --man
@@ -26,29 +26,30 @@ else "$SHELL" -c "$min_posix" 2>/dev/null && exec "$SHELL" -- "$0" ${1+"$@"}
  exit 128
 fi
 
-# The test suite uses $SHELL to indicate the shell to test. But we cannot
-# use a $SHELL environment value inherited by this script, as that is
-# already used for the user's default login shell on most systems. So
-# allow inheriting $KSH instead, or passing SHELL= or KSH= as arguments.
+# bin/package will have set $SHELL to our ksh.
+# Allow override by passing SHELL= or KSH= as arguments.
 for arg do
  case $arg in
  ( SHELL=* | KSH=* )
- KSH=${arg#*=} ;;
+ export KSH=${arg#*=} ;;
  ( * ) set -- "$@" "$1" ;;
  esac
  shift
 done
 
-# Find root dir of ksh source
-mydir=$(dirname "$0") \
-&& mydir=$(CDPATH='' cd -P -- "$mydir/.." && printf '%sX' "$PWD") \
-&& mydir=${mydir%X} \
-|| exit
-myarch=$("$mydir/bin/package" host type) || exit
+# Relaunch with necessary environment stuff from bin/package
+case ${HOSTTYPE+h}${INSTALLROOT+i}${PACKAGEROOT+p}${LD_LIBRARY_PATH+l} in
+hipl) ;;
+*) mydir=$(dirname "$0") \
+ && mydir=$(CDPATH='' cd -P -- "$mydir/.." && printf '%sX' "$PWD") \
+ && mydir=${mydir%X} \
+ || exit
+ exec "$mydir/bin/package" use "$mydir" "$0" "$@" ;;
+esac
 
 # Check if there is a ksh to test.
 case ${KSH+set} in
-( '' ) KSH=$mydir/arch/$myarch/bin/ksh ;;
+( '' ) KSH=$SHELL ;;
 esac
 if ! test -x "$KSH" || ! test -f "$KSH"; then
  printf '%s: shell not found: %s\n' "${0##*/}" "$KSH" >&2
@@ -62,10 +63,8 @@ KSH=$(CDPATH='' cd -P -- "$(dirname "$KSH")" \
 && KSH=${KSH%X}
 
 # Run the test suite
-CDPATH='' cd -P -- "$mydir/src/cmd/ksh93/tests" || exit
+CDPATH='' cd -P -- "$PACKAGEROOT/src/cmd/ksh93/tests" || exit
 SHELL=$KSH
-INSTALLROOT=${INSTALLROOT:-$mydir/arch/$myarch}
-export SHELL INSTALLROOT
 unset -v KSH
 printf '#### Regression-testing %s ####\n' "$SHELL"
 exec "$SHELL" shtests "$@"