My partner sent me a paper on making machine learning models transparent and I expect those of you who tinker with training the ML models will also find it good food for thought:

http://vanderschaar-lab.com/papers/NIPS2019_DBM.pdf

Jump ahead to section 5.2 "Predicting prognosis for breast cancer" to get an idea about what they're doing.

This could be a very useful tool for combining machine learning with public policy.

Pragmatically, it might allow Signalen to publish a transparent and re-usable model without having to publish any sensitive data required for generating a model.

At a glance, it seems there's more work on building models that model models -- what a sentence! -- and "Explainable AI" seems to be a good keyword to look up ... but I am very novice in this subject.

Cheers,

 -Eric