Lessons Learned From Years of Red Teaming in Cybersecurity
Red teaming can mean a lot of things to a lot of people. In its truest sense, and how I will define it here for the purpose of this blog, red teaming is the practice of adopting the adversarial mindset - thinking like the bad guy or how a situation can go wrong. Understanding and viewing how things can go wrong and why it’s extremely important to have that mindset in everyday life. With random acts of violence and terror occurring seemingly every day, it’s important to understand how to think critically and apply it to your life. For red teams, this is paramount, and over time becomes second-nature. It is paramount as this type of applied critical thinking is a make it or a break it skill set.
After almost 20 years of being a part of and running red team exercises within the U.S. government, I often get asked what lessons I’ve learned, some pertinent (and sometimes difficult) lessons, so I started putting together a list of the the lessons learned and that are critical to keep in mind as you’re running your own red teams. While many of these feel like concepts, vice lessons learned, I hope the reader finds them thought provoking as they formulate and execute red teams of their own. As always, feedback and comments are welcome. Here are a few key lessons to share with your team:
🔹Don’t Overthink It and Don’t Fight Your Gut
Sometimes you just need to go with your initial instinct, or gut feeling. If you start to think something might be amiss or just plain wrong for the environment you’re in, chances are, it is. There are times where you need to embrace that feeling of uncertainty and either get the hell out of the area or ensure you approach the situation in front of you with multiple strategies in mind. If you are too reactive to what’s around you, you will be one step behind. A primary goal of the red team is to dictate and develop the situation, not to let others do it for you. The less control over a situation you have, the more opportunities for failure can present themselves. Good proactive actions will always win the day over bad reactions or freezing in place. One caveat to this is sometimes inaction is in fact an action itself; if you make the decision to not do anything, you are still dictating the tempo of events. If you’re conducting a red team operation, drill and practice the plan over and over. Approach the problem sets from multiple angles to better understand what could go wrong and what success actually looks like.
The adage that the Jedi want to bring balance to the force is a farce. Adversaries, competitors, and other actors/entities never seek balance. They seek asymmetry. Over time, I’ve come to recognize that order does not equate with balance, and the scales are never equally weighed regardless of whether we are talking international relations, economics, or societal frameworks like civil liberties versus security.
Asymmetry is a key objective of the red teamer. This point was highlighted for me with Michael Moore’s presentation at the Boyd & Beyond conference when he advocated that the concept of Yin and Yang is a lie. We always seek advantage, or at least less disadvantage, and that needs to be the guiding ethos of your red team. Out smart, out play, and be driven to actually win something. You can drive asymmetry through overwhelming force, technology, or tactical surprise, attacks of disproportionality or long-term strategy.
🔹There is No Spoon
I often relate my red teaming approach to a scene in the movie the Matrix. In the movie a young child bends a spoon using only his mind. When our hero Neo attempts the same feat, the child prodigy notes that the secret is to remember that there is no spoon. “Do not try and bend the spoon, that’s impossible. Instead, only try to realize the truth…there is no spoon. Then you will see it is not the spoon that bends, it is only yourself.”
A methodology is an artificial constraint on the red team. To truly red team, you need to unleash the creativity and ingenuity of the experts on the team. The key is not to think outside the box, but to think without the box.
🔹Accept That You Don’t Know Everything. Because You Don’t
A good red team is composed of unique members from all walks of life with their own specialties, abilities, and perceptions. To build a good team, you need to think about what skills and experience you’ll need to get the job done. In that same vein, you cannot settle for less. A team may be successful on one hand for a particular situation, but if the playing field were to change, you should consider these modifiers and adjust team composition as necessary. The other benefit of having a variety of talent, skills, and ability on the team is that you are less likely to fall into groupthink, which can lead to blind spots.
Good team members can adapt given enough time; however, if you have timelines that can’t be moved, it’s better to swap in teammates that can multiply your chances of success or call the operation off. The worst-case scenario would be to proceed with an inadequate team where you don’t even achieve mission success or someone gets hurt, or worse.
To prove an example of a diverse team, below is the notional structure that was leveraged on a real red team operation in the past:
🔹Red Team Lead (RTL) - responsible for clarifying goals and objectives and ensuring success through strategy and tactics; also responsible for translating real-world risks into business impacts and communicating them to the client.
🔹Project Manager/Analyst/Technical Writer - responsible for ensuring that the red team is executing against the operational order (OPORD) and tracking risks to the operation, and documenting results.
🔹Business Analyst - partnered with the client to understand business impacts, similar to the RTL, but also provide feedback loop to the operators for situational context during the operation which provides strategic opportunities to exploit processes, not just tangible assets.
🔹Technical Analysts (Networking, Exploit Development) - point for assessing the target for technical vulnerabilities, identifying ingress opportunities, and establishing command control through covert exploitation.
🔹Non-Technical Analyst (Social) - performs open source intelligence (OSINT), analyzes relationships and identified opportunities for social engineering. Also aids in analyzing risk and impact of weaknesses identified.
🔹Physical Security Specialist - performs on-site reconnaissance, wireless network sniffing (cross-trained), security control review and testing of deterrence and preventative physical countermeasures, and identifies opportunities for compromise.
🔹Context is King
A successful red team can articulate their results in a way that brings context to the red team’s sponsor and supports their decision making process. Your red team briefing should have a valid threat/competitor model, an attack narrative with contextual outcomes, and the value proposition for the attacker and the defender. I’ve especially found this to be true when briefing red team results to executives and boards of directors.
🔹You Can’t Script an Adversary
When an IT security organization tells us that they red team by running Nessus or Metasploit, we’d often ask “what nation state does Nessus represent?” Tools are intent agnostic. An adversary is not. Tools treat all systems as equal. An adversary does not. There is great value in pro-actively probing your network with available tools, but they are not a replacement for a real human-led red team. By that same manner, there is great value to exercises conducted with structured injects, but a real red team takes place in real-time and is unscripted.
🔹The Attacker is Not a One or Zero
My best mentor first articulated this concept years ago to focus our attention away from the noise on the wire and back on the living, breathing, human adversary on the other side. You can’t think of your adversary only in the context of the technical attack that manifests itself, but rather in the context of their human behavior. A good red team will re-enforce this fact for the blue team.
🔹The 10th Man Rule
Previous efforts to articulate this concept were flawed until the movie World War Z did it so simply and brilliantly with the 10th man rule:
“If nine intelligence analysts came to the same conclusion, it was the duty of the tenth to disagree. No matter how unlikely or far-fetched a possibility might be, one must always dig deeper.”
As humans we are fundamentally flawed towards consensus, hive mind, and an inherent desire to believe the lie. A good red teamer has to break outside the chains of conception and imagine the unimaginable and see whether the unimaginable can manifest itself as red action. Here the 10th man rule has great value, not in requiring the 10th man to automatically dissent, but also as a mental exercise to expand the potential of the red team. As a red teamer your job is to help prevent failures of imagination.
🔹OODA Loop Compression
Operating within a faster OODA Loop than your adversary is the core precept of the OODA Loop concept itself. The fighter pilot able to complete the OODA Loop will have the advantage as Boyd did with his 40 second wins. However, we need to look beyond just completing the OODA Loop quicker and acknowledge that in some situations the OODA Loop might be compressed to just Observe - Act. This is especially true when red teaming in the cyber domain. What can you do to force your adversary to compress or truncate their OODA Loop and if they do, how you can you take advantage of it? This goes beyond surprise or deception, but can also be achieved through exploitation of procedures or other constructs narrowing your adversaries response options or their ability to respond in the first place. The red team’s agility and ability to operate within compressed OODA Loops can be a tactical enabler of the red team’s success.
🔹Encourage Disagreement and Alternative Planning
Until planning has been completed and you’re in an operational state, encourage feedback and dissenting opinions during the plan design phase. This is where it matters most. During the operation, ensure your operators are continually providing feedback on what is working or not and why. If you have a red team leader (RTL) dictating every step of the plan and operation while ignoring the team, then you have a bad RTL. This doesn’t happen too often; however, just like the above point where you don’t know everything, the same is true for the RTL. This is why it’s a good idea to shift RTLs based on the engagement specifications. If the predominant activity will involve physical entry, then the RTL should be someone with a solid understanding of this area and critical thinking skills, not a computer expert, or someone that has no experience in that area.
🔹No Artificial Constraints
I learned early on working with some of my best mentors not to accept artificial constraints. The minute you constrain the red team, you’ve officially moved outside the realm of red teaming an into the realm of experimentation. If the blue forces try to impose artificial constraints ask them if they are also able to impose those constraints on the real-world adversary. Real attackers don’t steer away from operational systems or restrict their activity to business off-hours. Want to use mylar balloons and aluminum foil to make your red team jeeps look like tanks? Does the adversary have access to mylar balloons and foil? Fair game.
🔹A Good Red Team is a Thinking, Adapting Team
It might be obvious to most, but if you’re not adapting to what I like to call the atmospherics around you, things can go very wrong, very fast. When creating your operational plan, it’s important to have a backup plan (which you should also dry run). I like to follow the PACE prinicple of planning. Having a primary, alternative, contingency, and emergency plan based on the original operations order (OPORD) is key. Just because your primary plan goes to shit, doesn’t mean you can’t still be successful - but you have to plan for it to ensure success the next time around.
🔹Know When You’ve Reached the Point of No Return or Failure
I don’t know how many times I’ve been faced with this, bit the inside of my lip, and thought “oh fuck.” Often I’ve seen red teams continue to hammer away even though their chance of success is nil. You must be honest with yourself and your teammates when you’ve reached that point. We all make mistakes, and mistakes are part of learning and what makes you grow. Continuing to execute can bring further harm to your operation, team, or the target itself. Lastly, understand what the point of no return is within each stage of your plan. This is vital in case you need to shift or adapt your tactics or go to plan B. Otherwise, it may be too late, and you’ll find yourself staring at failure, feeling like a failure.
🔹Speak Truth to Power
I once spent 45 minutes backing two of my red teamers who were engaged in a heated discussion with a customer over a request to change a single word in a red team report. We never backed down as the word carried true value and context for the decision maker. A red team should never compromise the integrity of their results to satisfy the red team sponsor. True value comes from speaking truth to power which often means articulating unpopular findings or diverging from the status quo or common conceptions.
In you red team efforts, you need to apply critical thinking to all aspects of planning and executing to make sure you’re not wasting time and effort, but instead, achieving results. The most important thing you can do is reflect on the performance of the team as a whole and on the individual level; this includes self-reflection as well. Conducting an after-action report (AAR) and walking through how the operation was planned and executed will help you to better understand how each component was leveraged or mis-leveraged based on expectations and the achieved results or failures. These are especially useful for improving tactics and strategy as a whole for the team, but also opportunities for improvement at the individual level.
Hopefully, the insights from my experiences, key perceptions and mind-think will help you to further achieve success the next time you’re in an engagement. Just say to yourself, “remember what mindhack diva said,” internalize it, live with it and grow with it and you’ll find yourself more mature in a better position to execute for success the next time around.