I inherited the same detection problem twice
Twice I took over security for an organization paying a managed detection provider that never learned its network. What I tried first, the options I gave leadership, and what I check now.
The moment
Some of the alerts were about attacks on services we didn’t even run.
When I took over security for one organization, I inherited a managed detection provider along with everything else. We were paying $48 per endpoint, every month, for a service that was supposed to watch the network and tell us when something mattered. We’d log in and find alerts for services and attacks that weren’t even in use in our environment. That was most of what it sent us. The price was the same either way.
It wasn’t the only time. At another organization, with a different provider, I found the same pattern.
The situation
In both places, the provider had never been tuned to the organization, so it treated everything the same and most of what it flagged was noise.
At the second organization the numbers were easy to see: about 60 alerts a month, and about 55 of them were false positives.
Every one of those false alarms took time from a small team that was already stretched. And when the alerts that matter arrive in the same queue as the ones that don’t, the real ones are easier to miss.
The decision
The obvious move was to fire the provider and find a new one. I didn’t start there.
I tried to fix the relationship first. We held meetings with the account representatives. At the second organization I asked for a standing meeting so the provider could learn the business and tune its detections. At the first, I offered to help them learn the environment and tailor the tools myself. We held them to the service level agreement they had signed.
It didn’t work. When we held them to the SLA, the account team made it clear they didn’t want to be bothered. There was no initiative, no resolution, and no sign they wanted to do the work. At $48 per endpoint per month, that gap between cost and value was hard to justify.
So I brought leadership options. At the first organization there were three:
- Keep the provider and have our own team build the tuning. On paper this fixed the noise. In practice it wasn’t possible, because the team didn’t have the staff to handle basic IT, let alone detection engineering.
- Escalate to the contract attorneys and cancel. This ended the spend, but it didn’t replace the coverage.
- Move to a different platform. I had evaluated two alternatives, so leadership could weigh them against the cost of staying put.
The organization owned the risk and the budget, so the decision belonged to its leadership. My job was to frame the choice honestly, including the options I didn’t recommend, and make sure they could see the tradeoffs.
What happened
At the first organization, leadership chose to replace the provider. It wasn’t the last change. Later we moved again, to a provider whose tickets, detections, and remediation were better still.
At the second organization, the replacement handled alerts with a person in the loop and remediated threats directly. Monthly alerts went from about 60 to about 6, and all six were worth acting on. Each one arrived with notes a person had written about what happened and what to do next, so the team stopped spending its time on false positives.
The lesson
Know the difference between what you’re paying for and what marketing sold you, and hold every provider to the first one.
What I’d tell a security leader facing this
- In your first weeks in a new role, inventory every vendor, partner, and provider that touches security, and what each one is supposed to deliver. For each, write down what it covers, what the SLA promises, when the contract renews, and who on your side owns the relationship.
- Review each SLA, contract, and statement of work, and know the line between what you’re paying for and what marketing sold.
- Give a struggling provider a real chance: tell them specifically what’s wrong, help them learn your environment, and hold them to the SLA they signed.
- Bring leadership options with real costs, including the ones you don’t recommend. It keeps the decision with the people who own the risk.
- Measure a detection service by how many of its alerts are worth acting on. Sixty alerts that nobody believes are worth less than six that everyone acts on.
Closing
Managed providers keep a lot of small security teams covered. They only help when someone on your side reads the contract, watches what actually comes through, and is willing to have the hard conversation when the two don’t match.