Table of Contents
The goal of this project is to position attacks on ML systems in an ATT&CK-style framework so that security analysts can orient themselves in this new and upcoming threats.
If you are new to how ML systems can be attacked, we suggest starting at this no-frills Adversarial ML 101 aimed at security analysts.
Or if you want to dive right in, head to Adversarial ML Threat Matrix
Why Adversarial ML Threat Matrix?
- In the last three years, major companies such as Google, Amazon, Microsoft, and Tesla, have had their ML systems tricked, evaded, or misled.
- This trend is only set to rise: According to Gartner report. 30% of cyberattacks by 2022 will involve data poisoning, model theft or adversarial examples.
- However, industry is underprepared. In a survey of 28 organizations spanning small as well as large organizations, 25 organizations did not know how to secure their ML systems.
Unlike traditional cybersecurity vulnerabilities that are tied to specific software and hardware systems, adversarial ML vulnerabilities are enabled by inherent limitations underlying ML algorithms. As a result, data can now be weaponized in new ways requiring that we extend the way we model cyber adversary behavior, reflecting emerging threat vectors and the rapidly evolving adversarial machine learning attack lifecycle.
This threat matrix came out of partnership with 12 industry and academic research groups with the goal of empowering security analysts to orient themselves in this new and upcoming threats. We are seeding this framework with a curated set of vulnerabilities and adversary behaviors that Microsoft and MITRE have vetted to be effective against production ML systems Since the primary audience is security analysts, we used ATT&CK as template to position attacks on ML systems given its popularity and wide adoption in the industry.
We recommend digging into Adversarial ML Threat Matrix next
Contributors
| Organization | Contributors |
|---|---|
| Microsoft | Ram Shankar Siva Kumar, Hyrum Anderson, Will Pearce, Suzy Shapperle, Blake Strom, Madeline Carmichael, Matt Swann, Mark Russinovich, Nick Beede, Kathy Vu, Andi Comissioneru, Sharon Xia, Mario Goertzel, Jeffrey Snover, Derek Adam, Deepak Manohar, Bhairav Mehta, Peter Waxman, Abhishek Gupta, Ann Johnson, Andrew Paverd |
| MITRE | Mikel D. Rodriguez, Christina E Liaghati, Keith R. Manville, Michael R Krumdick |
| Bosch | Manojkumar Parmar |
| IBM | Pin-Yu Chen |
| NVIDIA | David Reber Jr., Keith Kozo, Christopher Cottrell, Daniel Rohrer |
| Airbus | Adam Wedgbury |
| Deep Instinct | Nadav Maman |
| TwoSix | David Slater |
| University of Toronto | Adelin Travers, Jonas Guan, Nicolas Papernot |
| Cardiff University | Pete Burnap |
| Software Engineering Institute/Carnegie Mellon University | Nathan M. VanHoudnos |
| Berryville Institute of Machine Learning | Gary McGraw, Harold Figueroa, Victor Shepardson, Richie Bonett |
Feedback and Getting Involved
The Adversarial ML Threat Matrix is a first-cut attempt at collating a knowledge base of how ML systems can be attacked. We need your help to make it holistic and fill in the missing gaps!
Corrections and Improvement
- For immediate corrections, please submit a Pull Request with suggested changes! We are excited to make this system better with you!
- For a more hands on feedback session, we are partnering with Defcon's AI Village to open up the framework to all community members to get feedback and make it better. Current thinking is to have this event circa Jan/Feb 2021.Please register here for the workshop for more hands on feedback session
Join our Adversarial ML Threat Matrix Google Group
- For discussions around Adversarial ML Threat Matrix, we invite everyone to join our Google Group here
- If you want to access this forum using your corporate email (as opposed to your gmail)
- Open your browser in Incognito mode.
- Once you sign up with your corporate, and complete captcha, you may
- Get an error, ignore it!
- Also note, emails from Google Forums generally go to "Other"/"Spam" folder. So, you may want to create a rule to go into your inbox instead
Contact Us
For corrections and improvement, see Feedback
For questions/comments/discussion:
- Our public email group is advmlthreatmatrix@googlegroups.com. This emails all the members of the distribution group.
- For private questions/comments, please email: Ram.Shankar@microsoft.com and Mikel@mitre.org