Skip to content

Conversation

@ablaom
Copy link
Member

@ablaom ablaom commented Nov 9, 2025

This PR bumps the stated compatibility requirements (to close the substantial #1172) but also pares down the number of packages included in MLJ integration tests. Integration tests for the following packages are being permanently removed:

| BetaML
│ CatBoost
│ MLJXGBoostInterface
│ MLJScikitLearnInterface
│ OutlierDetectionNeighbors
│ SIRUS
│ MLJTSVDInterface
│ PartitionedLS
│ Maxnet
│ ParallelKMeans
│ MLJNaiveBayesInterface
│ Imbalance
│ EvoLinear
│ LightGBM
│ LaplaceRedux
│ MLJText

I am deeming this necessary because:

  • Posting issues and follows ups for packages in non-compliance has become an unsustainable maintenance burden for the core MLJ team (mostly me)
  • Packages providing MLJ model implementations can easily enough add integration tests to their own Continuous Integration workflows using MLJIntegrationTests.jl, which is now mature enough for this purpose
  • As currently structured, including all the model-providing packages as test dependencies in the Project.toml is creating development bottlenecks when packages are slow to make compatibility updates. The current CategoricalArrays update is a case-in-point. If we do not remove half a dozen packages from testing, in may be months before we can complete this roll out. Testing using package-specific test environments is an alternative, but it's not trivial to set up, with dubious benefits, to be honest. It's basically just trying to do what the packages should be taking on in their own CI.
  • To test "integration" of the various MLJ components, it is excessive to test every single implementation of the interface. The main reason they were introduced was to scope out potentially disruptive changes upstream (MLJBase, MLJModelInterface, etc). Based on our experiences up-to-date, testing a few core packages is enough. For critical changes we still need to run individual component CI anyhow.

@ablaom ablaom marked this pull request as draft November 9, 2025 20:06
Packages that are to be permanently removed from integration tests:

| BetaML
│ CatBoost
│ MLJXGBoostInterface
│ MLJScikitLearnInterface
│ OutlierDetectionNeighbors
│ SIRUS
│ MLJTSVDInterface
│ PartitionedLS
│ Maxnet
│ ParallelKMeans
│ MLJNaiveBayesInterface
│ Imbalance
│ EvoLinear
│ LightGBM
│ LaplaceRedux
│ MLJText
@ablaom ablaom changed the title Bump CategoricalArrays = "1", StatisticalMeasures = "0.3" Bump CategoricalArrays, etc and major testing change 😱 Nov 20, 2025
@ablaom ablaom marked this pull request as ready for review November 20, 2025 06:56
@ablaom ablaom requested a review from DilumAluthge November 20, 2025 06:56
@DilumAluthge
Copy link
Member

For comparison, can you post the list of packages for which we'll continue running the integration tests?

@ablaom
Copy link
Member Author

ablaom commented Nov 20, 2025

cc @OkonSamuel

@ablaom
Copy link
Member Author

ablaom commented Nov 20, 2025

Sure, they are:

EvoTrees = "f6006082-12f8-11e9-0c9c-0d5d367ab1e5"
MLJClusteringInterface = "d354fa79-ed1c-40d4-88ef-b8c7bd1568af"
MLJDecisionTreeInterface = "c6f25543-311c-4c74-83dc-3ea6d1015661"
MLJFlux = "094fc8d1-fd35-5302-93ea-dabda2abf845"
MLJGLMInterface = "caf8df21-4939-456d-ac9c-5fefbfb04c0c"
MLJLinearModels = "6ee0df7b-362f-4a72-a706-9e79364fb692"
MLJMultivariateStatsInterface = "1b6a4a23-ba22-4f51-9698-8599985d3728"
NearestNeighborModels = "636a865e-7cf4-491e-846c-de09b730eb36"

Basically everything in [extras]

@ablaom
Copy link
Member Author

ablaom commented Nov 20, 2025

These are all packages we maintain ourselves anyway, except EvoTrees, which is very popular, very-well maintained over a long period, with responsive lead developer. Even this list may be longer than really necessary, I reckon.

@ablaom
Copy link
Member Author

ablaom commented Nov 20, 2025

BTW, While integration tests are begin skipped in this PR (because base of merge is not master) I have confirmed all tests pass locally,

Copy link
Member

@DilumAluthge DilumAluthge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable to me, especially if it will free up some of your time.

@ablaom ablaom merged commit 2542fb6 into dev Nov 20, 2025
3 checks passed
@ablaom ablaom mentioned this pull request Nov 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants