Definitely the most frustrating thing as working as a Data Scientist. You run an experiment, find that building a model greatly increase metric X at almost no cost, has safe model metrics, present it to stakeholders, everybody agrees with proceeding to deploying and utilizing the model in production, and yet every time the model takes a wrong decision, we get questioned about it. Why did the model say this?
Man when did I ever say the model obtained a 100% accuracy in the validation phase? Why is it so hard for stakeholders to understand that the best models humankind ever created are expected to make wrong calls once in a while?
Stakeholder did a buisness major and got cash from parents. They couldn't calculate where a function reaches 0, don't expect them to understand ANYTHING about ML. Also it's AI and chatgpt is sentient, it just told me.
I will never forget hearing the boss with a MBA from Harvard asking for Venn diagrams to be removed because they were confusing.
Some wrong decisions are more important than others - are these mistakes that are largely inconsequential, or are they leading to further downstream problems?
Errors are bad, of course, but again, what data scientist has ever promised a 0% error rate in any model?
That's the real kicker. Most people don't care about error rate in the abstract, they care about the one time it screws something up on their desk. A 2% miss rate sounds great until your model denies a loan to the CEO's nephew or whatever.
Exactly. This indicates that either something was overpromised in the planning stages, or a major requirement is missing. Every model makes mistakes, but not all mistakes are equal. If I build a trading algorithm on top of some data streams that I have access to and I predict the movement of an option with 99% accuracy, I still want to be cautious in how much I put down and not go crazy, otherwise that 1% could represent a loss from which I cannot recover. In that example I would control my overal liabilities. In a client situation I might ask what the costs of each class of errors are up-front. It's an ongoing process, munging data and then sitting in a high tower with it is untenable
Sounds like you need to work a bit on explainability. Get some SHAP plots or other very simple visual guide to what affects the model and set up a wik/confluence/whatever you use. We got tons of "why is this customer being denied!?!?" support complaints and when checking logs the users had so many warning flags it would have been denied 8 times over by a human analyst as well, it's very neat to be able to point other stakeholders to a simple quick guide on which params have a big impact on the outcome.
if they can't understand the simple concept of a model not getting 100% accuracy, SHAP will look like arcane runes to them
Conditional Feature Contributions my beloved, you never answer any questions but you do point to something vaguely answer shaped Usually
The issue here doesn't seem to be model explanability. He stated that the models performs at no cost. I would say the issue is either 1. There is a cost but he does not understand the cost (like in your example where denied customers cost the company) and/or 2. Communication: he did not get across the point that the cost is not substantial vs the gains.
Because they think everything is deterministic
This is the correct answer, but a little thin. We're fighting against forty years of educating people that computers do *precisely* what you tell them to, the same way every time. LLMs do the opposite. It's hardly any wonder that most people outside the field haven't made that mental adjustment yet.
This kind of stakeholder question is absolutely normal. Don't you ask the exact same question to yourself as you build and interrogate the model? It is a huge trust building opportunity to be able to say something like "yes, this is expected because of XYZ aspect of the model/data/whatever. It's still a huge improvement over the previous approach because (it also would have gotten this wrong)/(it got this right but at the cost of getting a bunch of other stuff wrong)/etc.
This is a good opportunity to work on your soft skills and do some education and expectations setting.
In my experience, sometimes it depends on whether the data scientist set honest expectations after deploying the model. I always recommend to provide error ranges for every model-based decision you provide to stakeholders and get their alignment BEFORE implementation. As long as the error is within the range you provided, they shouldn’t reach out for explanations. I’ve seen the scenario you described play out in cases where the scientists sold a “too good to be true” outcome to quickly get alignment from stakeholders.
I don’t believe they think that, but did you aside of model accuracy metrics provide a business case on how the model errors impact bottom line? If you did you could pull it up every time such a question arises. If you didn’t then their inquiries are on you.
The “All models are wrong, some are useful.” phrase has helped me in the past communicate this idea. Ultimately some stakeholders will be receptive, some will never understand. Life is fun.
Thank you, people are making it seem like OP has a communication issue. Some stakeholders get it with one explanation and some don't get it no matter how many times you explain.
In my experience, it is because the data science team doesn't really communicate properly about what the model does and how it achieves what it does. I have to justify performance to my stakeholders in ordinary people language. Around 40% of my job is making sure that data science autism translates nicely into actual human language. It's a soft skill that once you acquire can 2-3x your salary.
There's always the balance between accuracy and interpretability. A foundational DS lesson in the workplace is that if you can't actually explain what your model does and why it does it, even if it's highly accurate, you're going to have a hard time to get people buying in.
This is largely how it is for any technical role. I sometimes think people are paid not only for their specific skills but also how well they can dumb it down to people higher up without making them feel stupid.
Bingo. I don't think there is anything to add.
Know where to obtain starting materials to begin to work on this skill?
Sounds like a good YouTube channel to start. Data science for dumb executives
you can only be so clear about the limitations of predictive models to people without training in theory wrt epistemic difference between inferential vs descriptive stats, sometimes explaining things "properly" will connect sometimes it wont
Agree, most of the business user expect data science = black magic.
Could you please explain with an example?
100% agree. That is super annoying, especially when you are surrounded by many very young collegues and bad bosses, who then claim you'd be a pessimist or spoilsport... But it's maths not magic 🤷
I would say self driving cars is like this. To guarantee 100% it would take hundreds of engineers to reduce the error. Over the years its still not perfect so... 100% isnt possible even eith the resources
There is an important distinction in how you respond to this type of situation. If you are responding to these questions with “well models get it wrong sometimes” then frankly you’re the one in the wrong here. You should be capable enough and understand the model well enough to be able to answer this question with “here is how the model came to the decision that it came to” and also be able to work with the stakeholder to determine if that error is acceptable. Not all errors are created equal and most stakeholders do not know enough about ML to be able to look at a model and just understand how it works. Building resources for your stakeholders to help them understand the work you do is part of the job.
Because they’re not data scientists. That’s why they have you. I agree it’s the trickiest part of the job. Before even agreeing to a project, you have to have this conversation, and have it again and again throughout development. Just part of the job! Focus on communicating what it does for them, why does it matter, the value.
The same reason they want customer sentiment scores to have a decimal place. Seriously, they asked for decimal precision on a parameter that ranges from "unhappy" to "meh" to "happy"
It's your job to explain this to them.
What everyone else has said, plus; Computers make decisions very fast.. so errors get big (in volume terms) quickly.. much scarier than a nice squishy human. DS projects are often expensive in time or £.. so value for momey is v important
This is partly an interface problem rather than just a stakeholder expectation problem.
A model finishing inference isn't the same thing as its output being allowed to steer operations.
The highest-value production systems increasingly separate generation from governance. High-confidence, low-risk cases can flow straight through. Borderline, novel, or contradictory cases should escalate to a human, a rules engine, or additional verification.
If stakeholders keep asking "Why did the model make this mistake?", that often means they're treating the model as the final decision-maker instead of one component in a decision pipeline.
Nobody expects accountants, pilots, or physicians to have a 0% error rate. They expect systems that detect, contain, and recover from inevitable errors before they become expensive. If every pilot mistake immediately became a plane crash, we'd all agree the problem wasn't "pilots make mistakes." It would be that the system had no safeguards. ML should be engineered the same way.
Because most stakeholders don't know what non-deterministic means. Walk them through that concept and see if they can connect the dots. If not, make the connection for them.
All models are wrong, but sometimes they can be useful
Stakeholders do not understand statistics … like not even a little bit
First, this should be foundational training for any stakeholder in an ML project. What is ML, what does it do, why is perfection impossible, what questions to ask to be sure you understand the benefit and what to expect after its in production. Second, I like to remind them that humans have an error rate too.
Because they’re fucking stupid
Usually, it's because someone failed to explain it in a manner / language that the stakeholders can understand and slot into their own loved experience, sometimes because they were fed BS by someone else - or misunderstood something, often because it was never explicitly stated what the model should actually achieve. Then there's maybe 1% who are just unwilling to learn.
My experience was reverted. We had a classification model yielding a 0.99 Gini/AUC. It was that good and the problem is deterministic. Yet we had to under-report the accuracy to avoid immediate "overfitting" eyebrow raises.
Imagine, when we work with telemetry data and are so happy about an R2 of 0.47
You always hear more during the times you’re wrong vs. the times you’re right. It is frustrating
So fat I never saw stakeholder’s expect 0 error. They do however expect a lift which is what you should anchor on. There is always a baseline cost and what you do is slightly better. Therefore we take it.
If they have zero error rate expectation, it sounds like you didn't communicate properly, which is one of the main responsibilities of a data scientist. Tbh, it's a very easy expectation to fix.
How are they to know anyone about how machine learning models work? In their world, if something is ready, it’s usually error free. You need to set their expectations about what’s possible with data science from the get go.
Because you haven’t done a good enough job communicating that models are always wrong, it is a fluke when they are right.
You may want to explain how a casino works. If you take a small enough time chunk, you may see the players gaining an advantage, but over time the house always wins.
yes, but i also think part of it is that people don't experience model accuracy as a percentage, they experience it one prediction at a time. if the model gets 99 decisions right and the 100th affects their customer or team, that one mistake becomes their entire impression of the system. i've found it's often easier to frame discussions around the expected number and type of errors, plus what happens when the model is uncertain, rather than just quoting an accuracy metric.
I had a PhD neuroscientist arguing with me that good models should have 98% or more accuracy. I was like bitch we did something wrong if our accuracy is that high. (Not literally I was polite ) I didn’t last long on that team.
What would you say is an appropriate accuracy? Can you explain why it should be below 98%? I would tend to think higher is better so what am I missing?
Said someone who never passed statistics 101
Can I ask what the accuracy was that your stakeholders were dissatisfied with? Was it 70%? or 99,5%?
It's because we are hardwired to feel a loss more than a win.
you got to evaluate the cost of each element of the confusion matrix, if error means nuclear meltdown, may be you need 100% accuracy.
Sounds like a presentation / company intelligence problem. Never come across that mindset in insurance where pricing teams expect a model to predict everyone who will ever have a claim
Because you didn't do a good enough job explaining to them probability space. You didn't do a good enough job explaining application of a model to the ROI.
Because they are literally business, not tech colleagues. Their understanding of the computer goes as far as turning the computer on and off. Lol. Honestly.
Because they are not technical, but business. You know, if they could, they would want everything done yesterday. As an SWE, it's a never ending battle, also, its your PM who are supposed to defend your interests.
Stupidity
Believe it or not, this is on you and your team. Setting and managing expectations is the single most important thing you can do in a project. If you neglect it, then you run into this situation.
Next time:
Describe the problem
Tell them in jargon-free language the data you will show the model
Tell them the metric you'll use to evaluate the model, and why that metric is important
Build the model
Show them the current state metric (BAU) vs model's metric
Let them decide for themselves that it's worth their time
When they ask why it's not perfect, simply say that there is always some error when predicting the future. Nobody gets things perfectly. After the A/B test you can apply any lessons learned to explore future improvements.
Boom. You're done.
Because getting to 0% error rate was a big goal for a corporate computing for a decades. Back when the main job of the main frames was data processing, computer errors were seen as deterministic program flaws that could be eliminated by better process and quality control. Management was taught that for the most part the computer doesn’t make mistakes, but mistakes can happen if the programmers tell the computer to do the wrong thing or if the data is bad in the first place. You need to explain how this stuff is different and how the benefits still exist and outweigh the “problems”.
Because stakeholders are the nontechnical execs who has no idea what goes behind the scene 😂
Tell the stakeholders you also demand a 0% error rate from them as well
This is common but slowly gets better... I think the gap usually comes down to two things:
Statistically, there is always irreducible error. (Bayes Error Rate if you feel confident enough to share that with your stakeholders) A slightly hammy way, I have used to break the 100% expectation was to show a large decision tree during an initial review. I pointed out that even the leaf nodes are impure, i.e. we had mixed outcomes in the same super specific bucket. That made clear how limited our predictability was with the available information.
To manage this going forward:
All models are wrong. So all models make wrong calls all the time, not once in awhile. Modeling is statistical modeling - whether probabilistic (“traditional” models) or algorithmic (ML) - and therefore relies on imperfect data and imperfect models. And that’s OK. I suggest then, to set expectations accordingly. Hopefully, your stakeholders don’t think models are magic and predict like something out of the movies. They are not and they do not. They are human inventions - very clever for sure - but nevertheless, human.
Accuracy is the wrong contract for many deployments. Before launch, define the cost of each error class, the acceptable range by slice, and what action follows an outlier. Then monitor calibration, drift, abstention or escalation rates, and downstream cost in a canary, with a rollback path. A model can have stable aggregate metrics and still be unsafe if a low-frequency slice carries most of the loss, so the useful postmortem is usually which error class and control failed rather than why was this row wrong?
The classic "model error = scandal, human error = just one of those things" double standard. The frustrating part is that the model's error rate was presented, agreed upon, and signed off before deployment. The moment it makes a wrong call in production, that context disappears and it becomes "why did the AI do this." I've started framing it differently in stakeholder presentations: "this model will be wrong X times per 1000 decisions — here's what wrong looks like." Making the failure mode concrete and visible upfront seems to reduce the shock when it actually happens. Doesn't fully solve it, but it sets expectations better than accuracy/F1 scores alone.
Bad communication and poor understanding what a model is. “But why male models?” Treat them like Derek Zoolander. Sometimes you will have to explain it again. I use weather forecasting models as examples as models we use every day to make decisions about our lives and they are rarely exactly right. If the model says 92F high and it gets only to 91F, no one cares. We don’t change our behavior between 91/92. But if it says 92F and it snows - everyone loses their mind. But less extreme if you are planning on an outdoor party and the forecast says 10% rain you have one backup plan. If it’s 50/50 rain, you make sure you have a full backup plan. Same with business - what does the model say and how much is it known to be off? Adjust to the range. That communicated well keeps the expectations in check.
Most just don't understand what they're asking for or how to interpret a confusion matrix.