Master IT Change Management to Avoid Major Security Breaches

Most security breaches don’t start with a hooded figure typing furiously in a dark room. They start with a change nobody documented.

A patch that got scheduled but never applied. A firewall rule someone opened “temporarily” three years ago. A configuration file edited on a Friday afternoon with the best of intentions and no peer review. A cloud permission set that was supposed to be scoped down after a migration and wasn’t.

If you’ve spent any time in IT operations, you already know the feeling. The alert fires at 2 a.m. The engineer on call pulls up the logs and finds something that shouldn’t be there. Then comes the question that always makes a room go quiet: when did that change?

The answer, more often than not, is that nobody knows.

This is why IT change management sits at the center of any serious security program. Not because auditors like paperwork, and not because process is inherently virtuous, but because unmanaged change is the single most reliable way to turn a well-configured system into a compromised one. The flip side is just as true: an organization with disciplined change control tends to spot intrusions faster, recover from incidents quicker, and pass audits with far less drama.

Scott Alldridge and the IT Process Institute built the VisibleOps Cybersecurity framework around exactly this idea. Their argument, sharpened over decades of IT operations research and several bestselling handbooks, is that security and operations aren’t two separate problems. They’re the same problem viewed from different angles. Let’s dig into why that matters, and what it looks like in practice.

Why Change Management Is a Security Control, Not Just an Ops Ritual

Here’s a useful way to think about it. Every security control you own is a configuration. Your firewall rules are configuration. Your identity provider’s conditional access policies are configuration. Your endpoint detection agent’s exclusions list is configuration. Your Kubernetes admission controllers, your database encryption settings, your backup schedules, your MFA enrollment policies—all configuration.

Now ask yourself: what changes configuration?

Change. That’s it. Nothing else. New software, new patches, new cloud resources, new vendor integrations, new hires, new priorities. Every one of those arrives as a change.

So when change happens without a record, a review, or a test, your security posture shifts without anyone noticing. Sometimes it shifts in a good direction—a critical patch lands, a vulnerability closes. Often it shifts sideways, quietly weakening something that used to be strong.

The 80% figure you’ve probably heard

You’ll find a version of this claim in a lot of IT operations literature: roughly 80% of unplanned downtime traces back to some kind of change. The exact number varies by study and by how you define “change,” and honestly, I’d treat any single percentage with suspicion. But the directional finding holds up across decades of incident postmortems. When you roll back the timeline of an outage, you usually find a human action near the start of it.

Security incidents follow the same pattern, just with a longer fuse. An attacker doesn’t need to break in if a door was left open during a migration. They don’t need to exploit a zero-day if a patch window was missed. The breach becomes possible months or years before it becomes visible.

The cost side of the ledger

Gartner’s long-cited estimate of $5,600 per minute of downtime gets thrown around a lot, and it’s clearly a rough average across wildly different industries. A hospital system and a boutique design agency don’t feel an outage the same way. Still, the math is instructive. A two-hour outage at that rate runs about $672,000. Add regulatory notification, forensics, legal, customer churn, and the engineering time burned on firefighting, and you’re well past seven figures for a mid-size company.

And that’s just downtime. A breach where data leaves the building has a different cost structure entirely—one that includes fines, credit monitoring, and the slow erosion of customer trust that never shows up cleanly on a balance sheet.

Security teams and ops teams are usually solving the same puzzle separately

Walk into most organizations and you’ll find two groups with overlapping mandates and separate tooling. The operations team owns uptime, capacity, and the change calendar. The security team owns risk, compliance, and incident response. They meet during incidents, argue about priorities, and then go back to their corners.

The problem isn’t competence. It’s that they’re tracking different objects. Ops tracks systems and services. Security tracks vulnerabilities and controls. Neither group owns the change itself in a way that gives both sides what they need.

This is the gap VisibleOps Cybersecurity was built to close. Its core premise is disarmingly simple: if you get change management, configuration management, release management, and incident management working together, you get operational excellence and security assurance from the same set of practices. You don’t need two programs. You need one program with two audiences.

Four Practices That Do the Heavy Lifting

The VisibleOps methodology organizes around four disciplines. They’re not sequential steps so much as interlocking systems, and weakening any one of them degrades the other three.

Change management

Every change gets recorded, classified by risk, reviewed at a level appropriate to that risk, tested where feasible, and scheduled with a rollback plan. The goal isn’t to slow things down. It’s to make change visible and reversible.

Configuration management

You can’t tell what changed if you never knew what you had. Configuration management means maintaining an accurate picture of your systems, their owners, their dependencies, and their baseline settings—and then comparing reality against that baseline on a regular cadence.

Release management

This is where change meets the pipeline. Releases bundle multiple changes into a testable, deployable unit. Good release management gives you staging environments, automated tests, progressive rollouts, and a clean way to undo what you just did. It’s also where security gates fit most naturally, because a release pipeline can enforce checks that a human reviewer would eventually get tired of enforcing.

Incident management

When something breaks—whether it’s an outage or a breach—incident management determines how fast you detect it, how you respond, and what you learn afterward. Crucially, it feeds back into change management. Every incident review should produce a change record, whether that’s a fix, a monitoring adjustment, or a process update.

How the four reinforce each other

|—|—|—|—|

Read that table left to right and you’ll notice something. The security benefit isn’t a separate column bolted on. It falls out of doing operations well.

What Unmanaged Change Actually Looks Like in the Wild

Abstract arguments about process are easy to nod along with and easy to ignore. Concrete stories stick. Here are several where change control—or its absence—was the hinge the whole event turned on.

Equifax, 2017

Apache Struts had a known remote code execution vulnerability, CVE-2017-5638, with a patch available in March 2017. Equifax’s dispute portal ran an unpatched version. Attackers got in, moved laterally, and exfiltrated personal data on roughly 147 million people before the intrusion was discovered in late July.

You can frame that as a patching failure, which it was. But patching is a change management problem. There was a change that needed to happen, an owner who needed to be accountable for it, a verification step that would have caught the gap, and a deadline. All four were missing or broken.

The AWS S3 outage, February 2017

An engineer debugging a billing system ran a command intended to remove a small number of servers. A typo in the command removed far more capacity than intended, and the S3 index subsystem went down across a large chunk of the eastern US. The command was legitimate work. The absence of a guardrail—a dry run, a confirmation step, a change review—turned a routine maintenance task into a multi-hour outage affecting thousands of companies.

Nothing malicious happened here. That’s the point. Change management exists to protect you from your own competent people operating under pressure.

GitLab, January 2017

During a spam incident, an engineer tried to delete a directory on a database server—and deleted it on the primary instead of a replica. Five different backup and replication mechanisms failed to restore the data, and the company ended up restoring from a snapshot that was about six hours old. GitLab did something unusual: they published a live-stream style incident report with raw detail.

Reading it is a masterclass in how a small operational change, executed without a clear runbook or confirmation step, can cascade. It’s also a masterclass in honesty. The company’s willingness to show the failure did more for the industry than a hundred polished case studies.

Code Spaces, 2014

A small hosted code repository company suffered a coordinated attack that ended with an intruder deleting the company’s AWS control plane configuration. With no viable recovery path for the environment, the business shut down. Code Spaces was, in effect, killed by a change—an unauthorized one—that nobody had designed a recovery plan for.

If your disaster recovery plan only covers hardware failure and not deliberate configuration destruction, it has a hole in it.

SolarWinds, 2020

Attackers compromised the build pipeline itself, injecting malicious code into a signed software update that then went out to roughly 18,000 customers. Defense against this class of attack is largely a release management problem: who can commit to the build, what review is required, how are artifacts signed and verified, and can you prove the pipeline wasn’t touched?

Most organizations had never asked those questions. Many still haven’t.

Change Healthcare, 2024

A portal without multi-factor authentication became the entry point for a ransomware operation that disrupted healthcare payments across the United States for weeks. Again, no exotic exploit. A configuration that should have been different, and no process that reliably catches those gaps.

The pattern across all six: no nation-state magic, no zero-day wizardry. Just changes that happened without the guardrails that would have caught them.

Building a Change Management Process People Actually Follow

Here’s the hard part. Almost everyone who has worked under a bloated change process has also learned to route around it. If your process takes three weeks and a committee meeting to approve a DNS record update, engineers will find a shortcut. Then you’ll have a process on paper and no process in practice, which is worse than having none at all, because you’ll believe you’re covered.

So the design goal is not maximum control. It’s maximum compliance with the process at the lowest reasonable cost in friction.

Classify changes by risk, then treat them differently

A three-tier model works for most organizations.

Standard changes are pre-approved, low-risk, and repeatable. Password resets. Adding a user to a standard group. Deploying a routine application update that’s passed automated tests. These should flow through without a human approval step, because forcing approval on them teaches people to hate the process.

Normal changes need review. Adding a firewall rule, changing a database schema, onboarding a new SaaS vendor, modifying an IAM policy. Risk-appropriate review, which for most of these means a peer who understands the system plus an automated check or two.

Emergency changes need to happen now, with review afterward. This category is where honesty matters. If your process has no legitimate emergency path, every change becomes an “emergency,” and your classification system collapses.

|—|—|—|—|—|

Make the change record short enough to actually fill out

A change record needs, at minimum: what’s changing, who’s doing it, when, which systems are affected, what the rollback is, and how you’ll know it worked. Six fields. Some organizations turn that into a forty-field form and then wonder why the data is garbage.

If a field doesn’t inform a decision someone will actually make, cut it.

Kill the change advisory board as you know it

The traditional CAB—a weekly meeting where a dozen people review every change—is a bottleneck that breeds resentment and workarounds. It made sense when systems were fewer and changes were rare. In a world of continuous deployment, it’s a speed bump with no steering wheel.

What works better:

Automate approval for standard changes. Let the pipeline be the approver.
Route normal changes to the specific reviewers who understand the affected system. Not a standing committee. Specific people.
Reserve the group meeting for genuinely cross-cutting changes—the ones that touch multiple teams or carry real blast radius.
Use a lightweight emergency channel with post-hoc review, and track how often it’s used. If emergency changes are more than about 10% of your total, your normal process is too slow.

Build rollback into the change, not the incident

Every change record should answer one question before deployment: how do we undo this, and how long will that take? For a code deploy, it’s a revert and a redeploy. For a database migration, it might be a restore from a snapshot taken thirty minutes prior. For a firewall rule, it’s removing the rule.

If the answer is “we’d have to figure it out,” the change isn’t ready.

Verify after deployment, not just before

The most commonly skipped step. You deployed the change. Did it land? Did it land everywhere, or did one node in the cluster miss it? Did the configuration drift back within a week because a management tool overwrote it?

A post-change verification step catches the gap between “we did the change” and “the change is in effect.” That gap is where breaches live.

Where Zero Trust and Change Management Intersect

Zero Trust gets discussed as though it were a product category. It isn’t. It’s an architectural philosophy: verify explicitly, assume breach, grant least privilege, and never trust a request just because it came from inside the network.

Which is fine as philosophy. The trouble starts at implementation, because Zero Trust requires an enormous number of configuration changes. Identity providers, device posture checks, micro-segmentation policies, conditional access rules, service-to-service authentication, logging pipelines. Each of those is a change. Each one can be misconfigured. Each one can drift.

So the question is: how do you deploy a Zero Trust architecture without creating a sprawling, undocumented mess that you can’t audit six months later?

You use change management as the delivery mechanism. Every policy gets a record. Every policy gets a test. Every policy gets an owner. And every policy gets re-verified on a schedule, because policies that aren’t tested rot.

Micro-segmentation without change control is a self-inflicted outage

Micro-segmentation is one of the highest-value Zero Trust practices and one of the highest-risk changes you can make. You’re adding deny rules between workloads that used to talk freely. Miss a dependency and you’ve broken a production service.

The organizations that pull this off do it incrementally, in monitor-only mode first, with a documented list of every allowed flow and a rollback that takes seconds. The ones that don’t end up rolling the whole thing back after an outage and losing executive support for the next attempt.

Identity changes deserve extra scrutiny

Identity is the new perimeter, which means identity configuration is the new firewall. A single misconfigured conditional access policy can lock out an entire region—or, worse, silently stop enforcing MFA for a subset of users.

Treat identity changes with the same rigor you’d apply to a perimeter firewall change. Peer review. Staged rollout. Monitoring for authentication failures and unexpected success patterns.

Compliance Frameworks Are Basically Change Management Frameworks

If you’ve ever read PCI DSS, HIPAA’s Security Rule, or the IT general controls section of a Sarbanes-Oxley audit, you’ve read change management requirements wearing different clothes. They just don’t always say so directly.

PCI DSS

Requirement 6 covers secure development and change control for systems. Requirement 10 covers logging of all access to system components. Requirement 12 requires a documented change control process with roles and responsibilities. In practice, an assessor will ask to see a sample of changes and trace each one from request through approval, testing, deployment, and verification.

If your only evidence is a screenshot of a Slack message saying “looks good to me,” that won’t pass.

HIPAA

The Security Rule’s administrative safeguards include a requirement to implement policies governing how systems are modified. Addressable, not optional-and-ignorable. Auditors want to see that you can demonstrate what changed in your environment and who authorized it.

Sarbanes-Oxley

SOX auditors care about financial reporting systems, which means they care intensely about who can change the applications and databases that produce financial statements. Segregation of duties matters here. The person who writes the code shouldn’t be the person who deploys it to production, and the person who deploys it shouldn’t be the person who approves it.

The good news: a well-run change management process generates this evidence as a byproduct. You’re not doing extra work for the audit. You’re doing the work once and exporting a report.

The bad news: retrofitting change control onto a chaotic environment in the six weeks before an audit is miserable and usually produces theater rather than actual control. Start earlier. It’s the same amount of work, spread over a period where it can actually reduce risk.

A Worked Example: The Firewall Rule That Wasn’t Temporary

Let’s walk through a realistic scenario, because the abstract version is too easy to dismiss.

Month 0. A vendor needs to connect to an internal API for a proof of concept. An engineer adds a firewall rule scoping the vendor’s IP range to port 443 on one host. The change is logged, approved by a peer, and given a ticket with a review date ninety days out.

Month 1. The proof of concept succeeds. The vendor gets a contract. Someone intends to widen the rule to production hosts and does so, but through a different ticketing system because the work got absorbed into a project. Two change records now exist for the same rule, neither aware of the other.

Month 4. The review date on the first ticket passes. Nobody looks, because the ticket is assigned to an engineer who left the company.

Month 9. The vendor’s IP range is reassigned by their hosting provider. It now belongs to a completely unrelated company. Your firewall still trusts it. Nothing alerts, because the rule is functioning exactly as configured.

Month 14. A penetration test finds an open path from an external IP into an internal API. The finding is rated high. Remediation takes two weeks of investigation because nobody can explain why the rule exists.

Now consider the same scenario under a working process.

The rule is recorded with a business justification, an owner, and an expiry date. An automated job sweeps for changes with expired justifications and opens a review task. The owner’s departure triggers a reassignment because the asset inventory links the rule to a team, not just a person. When the vendor’s IP range changes, the monitoring that watches for configuration drift against the baseline flags it. Total time to resolution: hours instead of weeks.

Same rule. Same risk in principle. Radically different outcome, and the difference is entirely process.

Common Mistakes That Undermine Change Management

Even teams that buy into the concept tend to trip on the same handful of rocks.

Treating the ticketing system as the process. A ticket is a container. The process is the review, the test, the rollback plan, and the verification. Filling out fields doesn’t create control.

Making the process scale-blind. A startup with fifteen engineers doesn’t need the same process as a bank with three thousand. Copying a large organization’s CAB structure into a small company is a reliable way to kill velocity without improving security.

Ignoring emergency changes in the metrics. If your change records show 400 normal changes and 900 emergency changes, the numbers are telling you the normal path is unusable. Track that ratio.

Skipping post-incident review. Every incident—outage or breach—should produce at least one change record. If your incident reports never result in changes, you’re treating symptoms.

Letting the configuration baseline go stale. A baseline from last quarter is a historical document, not a control. Automation is the only way to keep it current.

No owner for shared infrastructure. “Everyone’s responsible” means nobody runs the review when the person who set it up moves teams.

Punishing people for reporting emergency changes. If an engineer fesses up to a shortcut and gets a lecture, the next shortcut goes unreported. Psychologically safe reporting is a security control.

Ignoring non-production environments. Attackers don’t care that the misconfigured S3 bucket was “just a dev environment.” Dev environments often hold real data, real credentials, and real network paths.

Metrics That Tell You Whether This Is Working

You can’t manage what you don’t measure, and change management has a handful of metrics that reveal a lot.

Change success rate. Percentage of changes that complete without causing an incident or requiring rollback. Teams with mature processes often land in the mid-to-high 90s. If yours is below 80, look at testing and review quality.
Change lead time. How long from approved to deployed. Long lead times push people toward shortcuts.
Emergency change percentage. Target under 10%. Higher means the normal path is too expensive.
Mean time to restore (MTTR). The single biggest operational lever. High-performing teams measure this in minutes or hours; struggling teams measure it in days.
Unauthorized change rate. Changes detected by configuration comparison that have no matching record. This is your drift metric, and it’s the one that most directly correlates to security exposure.
Percent of changes with verified rollback. If nobody tested the rollback, you don’t have one.
Incidents traced to change. Some number is inevitable. A rising trend is a warning.

Plot these monthly. The trend matters more than the absolute number, and the trend will tell you where to invest next.

Making the Business Case to Executives

Here’s where a lot of security leaders stall. They walk into a budget conversation with technical arguments and get politely deferred.

The VisibleOps Cybersecurity: Executive Companion Handbook exists because that conversation keeps failing. It’s written for CEOs, COOs, CFOs, board members, and business owners—people who don’t want to know what a firewall rule is, but very much want to know what a breach would cost and how likely it is.

For a non-technical audience, the argument that works goes something like this:

Framing it as risk, not technology. “We have a 30% chance of a material security incident this year based on our current control gaps. The average cost for a company our size is $X. Investing in change discipline reduces both the likelihood and the recovery time.”

Tying it to metrics the board already tracks. Downtime costs money. Audit findings cost money. Customer churn after an incident costs money. Change management improves all three, and it’s measurable.

Showing the compliance angle. If you operate in a regulated industry, change management isn’t optional. Framing the investment as “avoiding a compliance finding” is easier than framing it as “improving security hygiene.”

Making it a business-alignment conversation. The strongest version is that change management lets the business move faster with confidence. You can ship more, take more risk deliberately, and stop guessing about what’s actually running in production.

Scott Alldridge’s materials include ROI frameworks and benchmark data specifically aimed at these conversations, which is a large part of why they get used in boardrooms rather than just data centers.

A 90-Day Plan to Get Started

If you’re starting from near-zero, don’t try to fix everything at once. Here’s a sequence that works.

Days 1–30: See what you have

Inventory your systems, owners, and dependencies. It will be incomplete. Ship it anyway.
Capture the current baseline configuration for your highest-value systems.
Start logging changes even if you can’t yet control them. Awareness first.
Interview the people who do the work. Ask them where they route around the process and why.

Days 31–60: Reduce the obvious risk

Define standard, normal, and emergency change categories with concrete examples.
Automate approval for standard changes so people stop hating the process.
Add a lightweight change record template. Six fields, no more.
Set up automated baseline comparison on your top ten systems and review the drift weekly.
Establish a 48-hour post-incident review habit.

Days 61–90: Close the loop

Add a post-change verification step to your deployment pipeline.
Introduce rollback requirements for any change touching production data or identity.
Start tracking the metrics above and publish a monthly dashboard.
Run one tabletop exercise for a breach scenario that involves a configuration change.
Pick one compliance framework and map your change records to its requirements.

By day 90 you won’t have a perfect process. You’ll have a functioning one, real data about where the gaps are, and evidence you can show leadership. That’s a better position than most organizations reach in a year.

Where Scott Alldridge Fits In

If this all sounds reasonable but you’re staring at a specific environment with specific constraints, the frameworks exist so you don’t have to invent the approach from scratch.

Scott Alldridge holds an MBA in Cybersecurity, is a Certified Chief Information Security Officer, a CISSP, and Harvard-certified in Privacy and Technology, with more than 30 years in IT management and cybersecurity. The VisibleOps Cybersecurity handbooks have sold over 400,000 copies and are used across industries, from small businesses to regulated enterprises.

The offerings split along audience lines. The main VisibleOps Cybersecurity Handbook covers integrating the VisibleOps framework with Zero Trust, including micro-segmentation, identity management, real-time monitoring, and continuous visibility. The Executive Companion Handbook strips the jargon for non-technical leaders—CEOs, COOs, CFOs, and board members—and translates the same concepts into business language with ROI graphs, benchmarks, and leadership takeaways.

There’s also a newer addition: VisibleOps AI: Governance, Risk, and Leadership in the Age of Intelligent Systems, which extends the same thinking to AI governance. That’s a conversation more organizations need to be having, and it’s not one that change management frameworks from 2005 were designed to answer.

On top of the books, Scott runs personalized training and coaching, publishes through the Forbes Technology Council, and delivers hands-on managed IT and cybersecurity services through IP Services. So the path runs from “read the book and implement it yourself” all the way to “have someone do it with you.”

Frequently Asked Questions

Isn’t change management just bureaucracy that slows down developers?

Only if you build it that way. The version that works pre-approves low-risk changes, automates the checks that can be automated, and reserves human review for changes that actually carry risk. Teams with strong change discipline often deploy faster than teams without it, because they spend less time firefighting and have better rollback options when something goes wrong.

How does change management relate to vulnerability management?

Vulnerability management identifies changes that need to happen. Change management is how they get scheduled, tested, deployed, and verified. A vulnerability program without a change process produces a long list of open findings and slow remediation. Equifax is the cautionary tale.

What’s the smallest viable change management process?

A shared log with six fields—what, who, when, affected systems, rollback, verification—plus a peer review step for anything touching production, plus a weekly review of the log. That’s it. You can run it in a spreadsheet. Tools help, but they’re not the starting point.

Do we need a formal Change Advisory Board?

Probably not in the traditional sense. Standing weekly CAB meetings are a bottleneck for most organizations. Route approvals to the specific reviewers who understand the affected systems, automate standard changes, and reserve group review for cross-cutting changes with real blast radius.

How does configuration management differ from change management?

Change management governs the act of modifying something. Configuration management maintains an accurate picture of what exists and what state it’s in. They’re complementary: configuration management tells you what changed, and change management makes sure changes go through review. Without configuration management, you can’t detect unauthorized changes at all.

How do we handle changes made by cloud providers or SaaS vendors?

You can’t control their changes, but you can monitor for impact. Track your dependency on their services, subscribe to their status and deprecation notices, and build monitoring that catches behavior changes on your side. Some vendor changes will require you to adapt—those adaptations are changes in your own environment and should be recorded and reviewed like any other.

What about infrastructure as code? Doesn’t that solve this?

Partly. Infrastructure as code makes changes explicit, versioned, and reviewable, which is a huge improvement over clicking through a console. It doesn’t automatically give you drift detection for resources changed outside the pipeline, post-deployment verification, or rollback for stateful resources. You still need the surrounding process.

How often should we re-verify firewall rules and access policies?

At minimum, every time access is granted with an expiry date, plus a quarterly sweep of all rules with business justifications attached. The goal is to catch rules that outlived their purpose, which is one of the most common findings in penetration tests.

What to Do Next

Start with one thing. Pick your ten most important systems and write down what you know about each one: who owns it, what it depends on, and what its baseline configuration is. That’s a week of work, and it will immediately surface gaps you didn’t know you had.

Then add a change log. Six fields. No approval workflow, no committee, just visibility. You’ll be surprised how much the simple act of writing changes down changes behavior.

From there, layer in the rest: risk classification, automated approval for standard changes, post-change verification, drift detection, and a metric dashboard you review monthly. It’s not glamorous work. It’s the work that keeps the breach headline from having your company’s name in it.

If you’d rather not build the whole thing from first principles, the VisibleOps Cybersecurity handbooks give you the framework, the benchmarks, and the executive language in one place. Start with the main handbook if you’re hands-on, or the Executive Companion if you’re the person signing the checks and asking the hard questions. Either way, the point is the same: change is going to happen. The only question is whether you’ll know about it before someone else does.