`base` changes for exception backtrace proposal

[bgamari]

In ghc-proposals/ghc-proposals#330 we propose a significant number of interface additions to base, as well as changes to a few central abstractions (namely, the Exception class and SomeException type). These are articulated in the "Proposed Change Specification" section of the proposal.

I have broken the GHC Proposal into five subproposals:

• Use displayException instead of show in top-level exception handler #198 proposes changing the top-level exception handler to use displayException
• Exception backtrace proposal: Part 1: Annotations #200 describes the general-purpose exception annotation mechanism
• Exception backtrace proposal: Part 2: Backtraces #199 describes how this mechanism is used to capture backtraces
• Exception backtrace proposal: Part 3: HasCallStack Improvements #201 proposes changes to ensure that HasCallStack backtraces are propagated such that they can be attached as context
• Exception backtrace proposal: Part 4: Rethrowing #202 describes how exception context is preserved during rethrowing

The operative sections of the GHC proposal are reflected in the following CLC proposals:

Section	Description	Proposal
2.1	Annotations	part 1
2.2	Representing backtraces	part 2
2.3	Representing ExceptionContext	part 1
2.4	Attaching context to exceptions	part 1
2.5	Providing context to handlers	part 1
2.6	Rethrowing	part 4
2.7	Capturing backtraces	part 2
2.8	HasCallStack improvements	part 3
2.9	Asynchronous exceptions	part 2
2.10	Modifying top-level handler	#198
N/A	Collecting nested exceptions and annotations	part 5

[Bodigrim]

@parsonsmatt since the proposal quotes annotated-exception as prior art, could you please comment?

[bgamari]

@haskell/core-language-committee, I would also appreciate guidance on how we can address the issue brought up in #145.

To summarize: in order to provide stack trace decoding for the Haskell stack, we will need to make some or all of ghc-heap (which is exposes internal implementation with a high likelihood of changing even across minor GHC versions) available to base. Naturally, this is complicated by the fact that ghc-heap rather unavoidably depends upon base. Given the constraints of our packaging system, I see a few ways of accomplishing this:

1. moving the relevant modules into base, advertising them loudly as internal and not subject to the PVP, and reexporting them from ghc-heap (which will continue to be their canonical home)
2. initiating a split such as suggested in Buy-in for Technical Proposal #47 #145, moving the relevant modules and reexporting them from ghc-heap (which will continue to be their canonical home). Naturally, this would allow us to continue to version these declarations under the PVP.
3. option (1) but retire ghc-heap
4. option (2) but retire ghc-heap

Thoughts?

While we could in principle proceed with ghc-proposals/ghc-proposals#330 without addressing this, I worry that the utility of the proposal would be severely compromised. Afterall, Haskell stack decoding (that is, IPEBacktrace) is very likely to be the most widely-used of the four backtrace mechanisms introduced in the proposal.

[parsonsmatt]

I helped a small amount with the proposal and am in favor. While there are some design quibbles re exactly what information gets retained and how to handle duplication, they aren't going to alter my approval of it - if I don't get my favorite design in base, then I'll continue to maintain annotated-exception with the behavior that I want.

[hasufell]

moving the relevant modules into base, advertising them loudly as internal and not subject to the PVP

I think I've said it too many times by now, but -1 to any PVP exclusion. Please provide evidence that it has been a problem in the past and proper motivation why we can't achieve compliance. "Makes our workflow easier" is not sufficient motivation.

[bgamari]

I think I've said it too many times by now, but -1 to any PVP exclusion. Please provide evidence that it has been a problem in the past and proper motivation why we can't achieve compliance. "Makes our workflow easier" is not sufficient motivation.

GHC as a compiler explicitly provides no ABI compatibility guarantees, even across minor releases. The reason that the ghc-heap package exists is so that it can truthfully reflect the ABI implemented by GHC (in particular, heap structure) while being versioned independently. Making the major version of base depend upon something that we explicitly claim to be an unstable implementation detail which the user cannot otherwise observe is simply not tenable.

This now puts us in a difficult situation: in order to implement a standard library feature we now find that we need access to ghc-heap. The natural way to accomplish this would be to introduce a package --- call it, for instance, ghc-base --- which we can freely bump across GHC versions without affecting users of base. This package would contain both ghc-heap as well as the bits of base that require ghc-heap. This is option (2), which we are discussing in #145, and is my preferred option.

However, as there appears to be some resistance to #145, I feel obligated to suggest alternatives. The most plausible alternative I can think of is option (3), which sadly requires a targeted PVP exemption. In practice, I don't believe this would be problematic as these definitions are only exposed to be reexported by ghc-heap; users should not be using them directly.. While our package system doesn't give us the tools to prevent users from using them entirely, I think a {-# HADDOCK hide #-} pragma would be a sufficient deterrent.

If we nevertheless think that this is unacceptable, then we are left only with options (2) and (4). I simply do not see a way around this.

[Bodigrim]

GHC as a compiler explicitly provides no ABI compatibility guarantees, even across minor releases. The reason that the ghc-heap package exists is so that it can truthfully reflect the ABI implemented by GHC (in particular, heap structure) while being versioned independently.

Could you possibly elaborate a bit?

ghc-heap releases have versions 9.0.1, 9.2.1, 9.2.2, 9.4.1, 9.6.1, which means that major version bumps of ghc-heap perfectly match major version bumps of base. Is ghc-heap itself PVP-compliant?

AFAICT the only difference between ghc-heap-9.2.1 and ghc-heap-9.2.2 is adding

#if __GLASGOW_HASKELL__ >= 811 && __GLASGOW_HASKELL__ < 902
  | BlockedOnIOCompletion
#endif

which seems to be fixing an outright bug and not reflecting an ABI difference between GHC 9.2.1 and 9.2.2 (in which case I would expect to see __GLASGOW_HASKELL_PATCHLEVEL1__ checked).

[bgamari]

@Bodigrim, sure.

Currently we version ghc-heap according to the GHC version number for simplicity. Moreover, several minor GHC releases have required no changes to ghc-heap at all (hence there being . However, when we find we need to make a change requiring a major version bump we will revisit this. The important thing here is that we have the ability to make versioning decisions independently of base.

[hasufell]

Currently we version ghc-heap according to the GHC version number for simplicity.

So there's no particular reason to not be strictly PVP compliant and deviate from the GHC version scheme? Boot packages already correspond to a particular GHC version. I don't see a problem doing the same here.

Is it more work? Maybe.

[Bodigrim]

@hasufell AFAIU the GHC ABI wrt heap structure (accidentally) happened to remain stable within each major version of GHC so far, but @bgamari is reluctant to guarantee that it is to remain so in future.

I'm not really sure what I think about it: if the past behaviour is any predictor of the future, there does not seem to be too much pressure to change heap structure in patch versions of GHC. Maybe it's time to commit to keep at least this part of ABI stable within each single major release? Especially if we expect people to write and operate heap-analysing tools: it would be unfortunate if they stop working after a minor GHC update.

[hasufell]

@Bodigrim I'm a bit confused here.

I don't see a particular good reason to have a library versioned exactly like GHC (even if it contains internals) other than the fact you don't have to think about the versioning at all.

Do people expect that there are no major PVP bumps of boot libraries in a minor GHC bump? I think that doesn't have to be guaranteed when it's about internals. We can be more aggressive there.

[bgamari]

I'm not really sure what I think about it: if the past behaviour is any predictor of the future, there does not seem to be too much pressure to change heap structure in patch versions of GHC. Maybe it's time to commit to keep at least this part of ABI stable within each single major release?

This is not an outcome that we would be willing to entertain. Ultimately, heap structure is the very definition of "implementation detail" and one that we need to be able to iterate on freely. Of course, as always, we will attempt to keep breakage to a minimum in minor releases. However, we cannot commit to a situation which forecloses such changes entirely.

[Bodigrim]

Do people expect that there are no major PVP bumps of boot libraries in a minor GHC bump?

Yes, this is the expectation. Currently Stackage can accomodate a minor GHC bump almost immediately, within the same LTS series. If a minor GHC bump incurs major bumps of boot libraries (be it base, or ghc-heap, or ghc-base), it would require a new Stackage LTS series.

That said, I'd rather sacrifice the speed of Stackage adoption than PVP compliance. After all, the very point of Stackage LTS evolution is that no one sneaks breaking changes underneath.

[parsonsmatt]

FWIW, I'm totally fine with PVP exclusions, provided the modules are named in a manner that suggests that and documented as excluded. While I do prefer to see packages extracted instead, I can't deny that base and other boot packages have a much harder time with that. We have proposals in place to mitigate these issues in the future. I'd rather not hold up good new stuff for ideological purity on a version scheme which essentially no one in the Haskell ecosystem actually follows, anyway (lookin at you, open imports without minor version bump limits).