macOS build

[marcastel]

Apple's official build of ksh on macOS is maintained on their Darwin site. We should merge their patches and ensure our build works from here-on on macOS.

That done we should encourage update of Homebrew's formula. Likewise for MacPorts.

[marcastel]

A quick and dirty script to get a local copy of the Darwin site: download-apple-ksh.gz

[marcastel]

diff -ruN apple-ksh27 ksh-community

where ksh-community is commit e7f2542 outputs apple-build.diff.gz

To be honest was expecting a much bigger diff :-)

[marcastel]

Build fails. Wasn't expecting this (build.log.gz). I hope it is just my build environment which needs refreshing.

[jelmd]

Just tried on the WS of my colleague - works. Little recipe to reproduce and being able to play with compiler flags:

rm -rf /tmp/xxx && mkdir /tmp/xxx && cd /tmp/xxx
curl -O http://iks.cs.ovgu.de/~elkner/tmp/ksh93/apfel/Makefile
make src
make bin

To try different compiler flags, ditch all cached stuff with make clean first.

[jghub]

@jelmd: I just gave this a very quick try (copy+paste of the last 3 commands above into my (differentlhy named, pristine) tmpdir):

the build does not succeed on my machine (OSX 10.15.3) but aborts with:

...
+ test '' = fun/pushd
+ /usr/bin/cmp -s /Users/vdh/kshmac/ksh/src/cmd/ksh93/fun/pushd /Users/vdh/kshmac/ksh/arch/darwin.i386/fun/pushd
+ /bin/mv /Users/vdh/kshmac/ksh/arch/darwin.i386/fun/pushd /Users/vdh/kshmac/ksh/arch/darwin.i386/fun/pushd.old
+ true
+ /bin/cp /Users/vdh/kshmac/ksh/src/cmd/ksh93/fun/pushd /Users/vdh/kshmac/ksh/arch/darwin.i386/fun/pushd
+ chmod ugo+x /Users/vdh/kshmac/ksh/arch/darwin.i386/fun/pushd
package: make done  at Tue Mar  3 19:58:53 CET 2020 in /Users/vdh/kshmac/ksh/arch/darwin.i386
cd ksh && ./bin/package results failed
==> /Users/vdh/kshmac/ksh/arch/darwin.i386/lib/package/gen/make.out <==

this is only a first feedback. I have not yet tried to find out why it fails. (if you see the problem or know, where I should start looking, please let me know, of course).

[jelmd]

Sure? Looks like success. ./bin/package results failed should show 1+ lines with failed stuff right after ==> .... <==. Wondering what ls -al ksh/arch/darwin.i386/bin/ shows (I build it on 10.13.6 with kernel 17.7.0).

[jghub]

oops, you're right. there it is :|. I misinterpreted the message. sorry for the noise. and I thought I had scanned the whole dir for ksh, but obviously not so. so you right. very good :). I now can confirm that there is a functional binary present. compiler flags will need tweaking as a first q&d run of the benchmark shows. but this is encouraging. maybe we can offer a way for macports/homebrew to provide a ksh93u+ rather soon :)

[jelmd]

Ahhh, cool! :) All the important stuff lands in ksh/arch/*/*/.

Compiler flags: Can you try (possibly a -O2 is sufficient) and bench? I've had only ~ half an hour to check+setup+compile, before he went home (switched off the box)...

[jghub]

@jelmd

I just ran shbench -l5 /bin/ksh kshcomO2 (where the latter is "our" binary when compiled with -O2):

-----------------------------------
name             /bin/ksh  kshcomO2
-----------------------------------
braces.ksh       1.81      1.68
charsplit.ksh    3.41      2.91
extglob.ksh      1.83      1.56
fibonacci.ksh    1.81      1.73
gsub.ksh         4.39      4.29
numfor.ksh       3.18      3.19
numloop.ksh      2.58      2.54
parens.ksh       3.63      3.24
piln2.ksh        5.90      6.03
plus-equals.ksh  3.47      3.39
printf.ksh       4.14      3.74
qsort.ksh        4.84      4.65
rand.ksh         3.83      3.66
rand2.ksh        2.96      2.89
sample.ksh       4.91      4.82
shuffle.ksh      3.54      3.41
-----------------------------------

so that is cumulative run time (sec) when executing each benchmark 5 times. you were right -O2 alone does the trick regarding performance. /bin/ksh is the system's binary provided by apple. from looking at the source in the Darwin repo (or, rather, a diff against our starting point), apple uses a somewhat older/less patched variant of 93u+ (assorted older copyright comments, also different instances of changes to the code where, apparently some additions/corrections have been made relative to the Darwin repo).

so this might explain the residual run time differences, e.g. in charsplit and parens (which are in favour of the new variant).

so: looks very good indeed I'd say: it's quite some years that I have seen ksh93 compile under OSX :)

[marcastel]

Great work gents. Unfortunately my macOS build environment is out-of-service until I rebuilt it, so I can't yell hurrah with you just yet.

Since we have a macOS build would it not be a good idea to publicise this... to show the community their is actual work in progress ?

If so:

• What should our rule be for handling the patches?
• How to we communicate back with Apple?
• Is it time to look into the Travis macOS environment ?

[jghub]

macOS build/publicise:
I presume it is too rough around the edges so far, no? at least a install target is missing and and analogue to configure --prefix (or a README/INSTALL what to do to customize the install).

patches: I presume the patches used by @jelmd where those found in the Darwin repo? so that would be Apples own patches I guess? and nothing to communicate back to Apple AFAICS for now.

it also is not clear to me, yet, what the best strategy for patch review and integration will be. I guess there will be substantial overlap between the patch sets used by different distros. I wonder whether those should be (after thorough review/testing....) not be integrated one-by-one so that further patching during building/installation might not be necessary? this sure is now an option, given that ATT/AST has definitely stated that they are not going beyond 93u+ themselves.

[marcastel]

patches: I presume the patches used by @jelmd where those found in the Darwin repo? so that would be Apples own patches I guess? and nothing to communicate back to Apple AFAICS for now.

We have on Apples site the patches to backward support all versions of macOS since OS X. So should we merge or otherwise handle Apple patches we need to be clear on what responsibility we take over.

macOS may be departing from POSIX compliance with some security focused changes to the operating system (e.g. sip). This would mean that patches will become specific not to a BSD variant but to macOS specifically. Should that end up in the code ? Ideally for me this would be preprocessor directives.

More generally IMHO clean code would be one where we integrate all patches as preprocessing directives controlled though the build system.

[marcastel]

I presume it is too rough around the edges so far, no? at least a install target is missing and and analogue to configure --prefix (or a README/INSTALL what to do to customize the install).

MacOS is not like a Linux distribution. We don't have access to their system build. So we could probably deliver:

a) a ready-made binary tarball
b) a custom HomeBrew formula
c) and the likes for MacPorts which I know much less about

And all these could be GitHub releases so that extra integration work can be done by others. As the process matures we can reorganise, dispatch, automate as required.

[jghub]

macports does have "portfiles" (Tcl) but that will be done by the package maintainer (or rather: it exists but did no longer suceed with the build from the ATT repo). so I presume it suffices to provide the necessary ingredients like @jelmd did yesterday but as part of the ksh repo. then like other distros, macports could just start to use (if they are willing to -- remains to be seen) the ksh-community repo as upstream. we don't need to act as 'package maintainer' ourselves, I'd guess ;)

[jelmd]

Ehmm, the provided Makefile should be seen as a proof of concept, i.e. check whether vendor patches work as expected, build process works, performance is about the same or better as with the OS shipped ksh. If so, it is worth to start to unify/integrate the patches into the repo (otherwise sort out later). Not more!

And as @jghub said, we definitely do not want to provide OS packages. OS vendors/package maintainers can do this better and we have quite a bit of other tasks, which require their own time, too.