mirror of
https://github.com/mitre/advmlthreatmatrix.git
synced 2025-03-13 10:56:28 -04:00
Merge branch 'multi-page'
This commit is contained in:
Keith Manville
commit
f33be3e6b7
BIN
AdvML1.PNG
Normal file
BIN
AdvML1.PNG
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 71 KiB |
25
pages/adversarial-ml-101.md
Normal file
25
pages/adversarial-ml-101.md
Normal file
@ -0,0 +1,25 @@
|
||||
## Adversarial ML 101
|
||||
Informally, Adversarial ML is "subverting machine learning systems for fun and profit". The methods underpinning the production machine learning systems are systematically vulnerable to a new class of vulnerabilities across the machine learning supply chain collectively known as Adversarial Machine Learning. Adversaries can exploit these vulnerabilities to manipulate AI systems in order to alter their behavior to serve a malicious end goal.
|
||||
|
||||
Consider a typical ML pipeline shown in the left that is gated behind an API, wherein the only way to use the model is to send a query and observe an response. In this example, we assume a blackbox setting: the attacker does **NOT** have direct access to the training data, no knowledge of the algorithm used and no source code of the model. The attacker only queries the model and observes the response.
|
||||
|
||||
Here are some of the adversarial ML attacks that an adversary can perform on this system:
|
||||
| Attack | Overview |
|
||||
| :--- | :---
|
||||
| Evasion Attack | Attacker modifies the query to get appropriate response. The attacker can find the just-the-right query to get the desired response algorithmically |
|
||||
| Poisoning attack | Attacker contaminates the training phase of ML systems to get intended result |
|
||||
| Model Inversion | Attacker recovers the secret features used in the model through careful queries |
|
||||
| Membership Inference | Attacker can infer if given data record was part of the model's training dataset or not |
|
||||
| Model Stealing | Attacker is able to recover a functionally equivalent model with similar fidelity as original model by constructing careful queries |
|
||||
| Model Replication | Attacker is able to recover functionally equivalent model (but generally with lower fidelity) |
|
||||
| Reprogramming ML system | Repurpose the ML system to perform an activity it was not programmed for |
|
||||
| Attacking the ML supply chain | Attacker compromises the ML models as it is being downloaded for use |
|
||||
| Exploit Software Dependencies | Attacker uses traditional software exploits like buffer overflow or hardware exploits like GPU trojans to attain their goal. |
|
||||
|
||||
## Finer points
|
||||
1. This does not cover all kinds of attacks. Adversarial ML is an active area of research with new classes constantly being discovered.
|
||||
2. These attacks are applicable to ML models in different paradigms, ML models on premise, on cloud, on Edge.
|
||||
3. Though the illustration shows blackbox attacks, these attacks have also been shown to work in whitebox (where the attacker has access to either model architecture, code or training data) settings
|
||||
4. Though we were not specific about what kind of data - image, audio, timeseries, or tabular data, research has shown that of these attacks have shown to exist in all the data types
|
||||
|
||||
|
||||
159
pages/adversarial-ml-threat-matrix.md
Normal file
159
pages/adversarial-ml-threat-matrix.md
Normal file
@ -0,0 +1,159 @@
|
||||
## Adversarial ML Threat Matrix
|
||||
Interactive Version to be built by MITRE. For Editable version, contact <Ramk@Microsoft.com>
|
||||
|
||||
|
||||
|
||||
# reconnaissance
|
||||
|
||||
### Dataset Collection
|
||||
|
||||
Adversaries may collect datasets similar to those used by a particular organization or in a specific approach. Datasets may be identified during [Public Information Acquisition](/techniques/A0007). This may allow the adversary to replicate a private model's funcionality, constituting [Intellectual Property Theft](/techniques/A0013), or enable the adversary to carry out other attacks such as [ML Model Evasion](/techniques/A0012).
|
||||
|
||||
### Acquire OSINT Information
|
||||
|
||||
Adversaries may leverage publicly available information about an organization that could identify where or how machine learning is being used in a system, and help tailor an attack to make it more effective. These sources of information include technical publications, blog posts, press releases, software repositories, public data repositories, and social media postings.
|
||||
|
||||
### ML Model Discovery
|
||||
|
||||
Adversaries may attempt to identify machine learning pipelines that exist on the system and gather information about them, including the software stack used to train and deploy models, training and testing data repositories, model repositories, and software repositories containing algorithms. This information can be used to identify targets for further collection, exfiltration, or disruption, or to tailor and improve attacks. Once this information is identified, data may be collected during [ML Pipeline Collection](techniques/A0005/).
|
||||
|
||||
|
||||
# initial-access
|
||||
|
||||
### Pre-trained ML Model with Backdoor
|
||||
|
||||
Adversaries may gain initial access to a system by compromising portions of the ML supply chain. This could include GPU hardware, data and its annotations, parts of the ML software stack, or the model itself. In some instances the attacker will need secondary access to fully carry out an attack using compromised components of the supply chain.
|
||||
|
||||
### Exploit Public-Facing Application
|
||||
|
||||
Adversaries may attempt to take advantage of a weakness in an Internet-facing computer or program using software, data, or commands in order to cause unintended or unanticipated behavior. The weakness in the system can be a bug, a glitch, or a design vulnerability. These applications are often websites, but can include databases (like SQL)(Citation: NVD CVE-2016-6662), standard services (like SMB(Citation: CIS Multiple SMB Vulnerabilities) or SSH), and any other applications with Internet accessible open sockets, such as web servers and related services.(Citation: NVD CVE-2014-7169) Depending on the flaw being exploited this may include [Exploitation for Defense Evasion](https://attack.mitre.org/techniques/T1211).
|
||||
|
||||
If an application is hosted on cloud-based infrastructure, then exploiting it may lead to compromise of the underlying instance. This can allow an adversary a path to access the cloud APIs or to take advantage of weak identity and access management policies.
|
||||
|
||||
For websites and databases, the OWASP top 10 and CWE top 25 highlight the most common web-based vulnerabilities.(Citation: OWASP Top 10)(Citation: CWE top 25)
|
||||
|
||||
### Phishing
|
||||
|
||||
Adversaries may send phishing messages to elicit sensitive information and/or gain access to victim systems. All forms of phishing are electronically delivered social engineering. Phishing can be targeted, known as spearphishing. In spearphishing, a specific individual, company, or industry will be targeted by the adversary. More generally, adversaries can conduct non-targeted phishing, such as in mass malware spam campaigns.
|
||||
|
||||
Adversaries may send victim’s emails containing malicious attachments or links, typically to execute malicious code on victim systems or to gather credentials for use of [Valid Accounts](https://attack.mitre.org/techniques/T1078). Phishing may also be conducted via third-party services, like social media platforms.
|
||||
|
||||
### Trusted Relationship
|
||||
|
||||
Adversaries may breach or otherwise leverage organizations who have access to intended victims. Access through trusted third party relationship exploits an existing connection that may not be protected or receives less scrutiny than standard mechanisms of gaining access to a network.
|
||||
|
||||
Organizations often grant elevated access to second or third-party external providers in order to allow them to manage internal systems as well as cloud-based environments. Some examples of these relationships include IT services contractors, managed security providers, infrastructure contractors (e.g. HVAC, elevators, physical security). The third-party provider's access may be intended to be limited to the infrastructure being maintained, but may exist on the same network as the rest of the enterprise. As such, [Valid Accounts](https://attack.mitre.org/techniques/T1078) used by the other party for access to internal network systems may be compromised and used.
|
||||
|
||||
### Valid Accounts
|
||||
|
||||
Adversaries may obtain and abuse credentials of existing accounts as a means of gaining Initial Access, Persistence, Privilege Escalation, or Defense Evasion. Compromised credentials may be used to bypass access controls placed on various resources on systems within the network and may even be used for persistent access to remote systems and externally available services, such as VPNs, Outlook Web Access and remote desktop. Compromised credentials may also grant an adversary increased privilege to specific systems or access to restricted areas of the network. Adversaries may choose not to use malware or tools in conjunction with the legitimate access those credentials provide to make it harder to detect their presence.
|
||||
|
||||
The overlap of permissions for local, domain, and cloud accounts across a network of systems is of concern because the adversary may be able to pivot across accounts and systems to reach a high level of access (i.e., domain or enterprise administrator) to bypass access controls set within the enterprise. (Citation: TechNet Credential Theft)
|
||||
|
||||
|
||||
# execution
|
||||
|
||||
### Unsafe ML Model Execution
|
||||
|
||||
An Adversary may utilize unsafe ML Models that when executed have an unintended effect. The adversary can use this technique to establish persistent access to systems. These models may be introduced via a [compromised ML supply chain](/techniques/T1195/004/). An example of this technique is to use pickle embedding to introduce malicious data payloads.
|
||||
|
||||
# persistence
|
||||
|
||||
### Unsafe ML Model Execution
|
||||
|
||||
An Adversary may utilize unsafe ML Models that when executed have an unintended effect. The adversary can use this technique to establish persistent access to systems. These models may be introduced via a [compromised ML supply chain](/techniques/T1195/004/). An example of this technique is to use pickle embedding to introduce malicious data payloads.
|
||||
|
||||
### Account Manipulation
|
||||
|
||||
Adversaries may manipulate accounts to maintain access to victim systems. Account manipulation may consist of any action that preserves adversary access to a compromised account, such as modifying credentials or permission groups. These actions could also include account activity designed to subvert security policies, such as performing iterative password updates to bypass password duration policies and preserve the life of compromised credentials. In order to create or manipulate accounts, the adversary must already have sufficient permissions on systems or the domain.
|
||||
|
||||
### Implant Container Image
|
||||
|
||||
Adversaries may implant cloud container images with malicious code to establish persistence. Amazon Web Service (AWS) Amazon Machine Images (AMI), Google Cloud Platform (GCP) Images, and Azure Images as well as popular container runtimes such as Docker can be implanted or backdoored. Depending on how the infrastructure is provisioned, this could provide persistent access if the infrastructure provisioning tool is instructed to always use the latest image.(Citation: Rhino Labs Cloud Image Backdoor Technique Sept 2019)
|
||||
|
||||
A tool has been developed to facilitate planting backdoors in cloud container images.(Citation: Rhino Labs Cloud Backdoor September 2019) If an attacker has access to a compromised AWS instance, and permissions to list the available container images, they may implant a backdoor such as a [Web Shell](https://attack.mitre.org/techniques/T1505/003).(Citation: Rhino Labs Cloud Image Backdoor Technique Sept 2019) Adversaries may also implant Docker images that may be inadvertently used in cloud deployments, which has been reported in some instances of cryptomining botnets.(Citation: ATT Cybersecurity Cryptocurrency Attacks on Cloud)
|
||||
|
||||
|
||||
# defense-evasion
|
||||
|
||||
### Evasion Attack
|
||||
|
||||
Adversaries can create data inputs that prevent a machine learning model from positively identifying the data sample. This technique can be used to evade detection on the network, or to evade a downstream task where machine learning is utilized.
|
||||
|
||||
Example evasion attacks include Simple Transformation, Common Corruption, Adversarial Examples, Happy String
|
||||
|
||||
### Model Poisoning
|
||||
|
||||
Adversaries can train machine learning that are performant, but contain backdoors that produce inference errors when presented with input containing a trigger defined by the adversary. A model with a backdoor can be introduced by an innocent user via [ML Supply Chain Compromise](/techniques/T1195/004) or can be a result of [Data Poisoning](/techniques/A0011). This backdoored model can be exploited at inference time with an [Attack on ML model integrity](/techniques/A0003)
|
||||
|
||||
### Data Poisoning
|
||||
|
||||
Adversaries may attempt to poison datasets used by a ML system by modifying the underlying data or its labels. This allows the adversary to embed vulnerabilities in ML models trained on the data that may not be easily detectable. The embedded vulnerability can be activated at a later time by providing the model with data containing the trigger. Data Poisoning can help enable attacks such as [ML Model Evasion](/techniques/A0012) and [ML Model Integrity Attacks](/techniques/A0002).
|
||||
|
||||
### Tainting Data from Acquisition - Label Corruption
|
||||
|
||||
Adversaries may attempt to alter labels in a training set. This would cause a model to misclassify an input
|
||||
|
||||
### Tainting Data from Open Source Supply Chains
|
||||
|
||||
Adversaries may attempt to add their own data to an open source dataset which could create a classfication backdoor.
|
||||
|
||||
### Tainting Data from Acquisition - Chaff Data
|
||||
|
||||
Adding noise to a dataset would lower the accuracy of the model, potentially making the model more vulnerable to misclassifications
|
||||
|
||||
### Tainting Data in Training - Label Corruption
|
||||
|
||||
Changing training labels could create a backdoor in the model, such that a malicious input would always be classified to the benefit of the adversary
|
||||
|
||||
# exfiltration
|
||||
|
||||
### Insecure Storage
|
||||
|
||||
Adversaries may exfiltrate proprietary machine learning models or private training and testing data by exploiting insecure storage mechanisms. Adversaries may [discover](/techniques/A0001), [collect](/techniques/A0005), and finally exfiltrate components of a ML pipeline, resulting in [Intellectual Property Theft](/techniques/A0013)
|
||||
|
||||
### Exfiltration Over ML Inference API
|
||||
|
||||
Adversaries may exfiltrate private information related to machine learning models via their inference APIs. Additionally, adversaries can use these APIs to create copy-cat or proxy models.
|
||||
|
||||
### ML Model Inversion
|
||||
|
||||
Machine learning models' training data could be reconstructed by exploiting an inference API.
|
||||
|
||||
### Training Data Membership Inference
|
||||
|
||||
The membership of a data sample in a training set may be infered by an adversary with access to an inference API.
|
||||
|
||||
### ML Model Stealing
|
||||
|
||||
Machine learning models' functionality can be stolen exploiting an inference API. IP Theft.
|
||||
|
||||
# impact
|
||||
|
||||
### ML Model Integrity Attacks
|
||||
|
||||
Adversaries may attack the integrity of machine learning models by crafting adversarial examples that appear to be normal inputs, but cause the model to make errors at inference time. These attacks can reduce prediction confidence and cause false positives or false negatives, eroding confidence in the system over time.
|
||||
|
||||
### Evasion Attack
|
||||
|
||||
Adversaries can create data inputs that prevent a machine learning model from positively identifying the data sample. This technique can be used to evade detection on the network, or to evade a downstream task where machine learning is utilized.
|
||||
|
||||
Example evasion attacks include Simple Transformation, Common Corruption, Adversarial Examples, Happy String
|
||||
|
||||
### Stolen Intellectual Property
|
||||
|
||||
Adversaries may steal intellectual property by replicating the functionality of ML models, or by [exfiltrating private data over ML inference APIs](/techniques/A0003).
|
||||
|
||||
### Denial of Service
|
||||
|
||||
Adversaries may target different Machine Learning services to conduct a DoS.
|
||||
|
||||
One example of this type of attack is Sponge attack.
|
||||
|
||||
### Data Encrypted for Impact
|
||||
|
||||
Adversaries may encrypt data on target systems or on large numbers of systems in a network to interrupt availability to system and network resources. They can attempt to render stored data inaccessible by encrypting files or data on local and remote drives and withholding access to a decryption key. This may be done in order to extract monetary compensation from a victim in exchange for decryption or a decryption key (ransomware) or to render data permanently inaccessible in cases where the key is not saved or transmitted.(Citation: US-CERT Ransomware 2016)(Citation: FireEye WannaCry 2017)(Citation: US-CERT NotPetya 2017)(Citation: US-CERT SamSam 2018) In the case of ransomware, it is typical that common user files like Office documents, PDFs, images, videos, audio, text, and source code files will be encrypted. In some cases, adversaries may encrypt critical system files, disk partitions, and the MBR.(Citation: US-CERT NotPetya 2017)
|
||||
|
||||
To maximize impact on the target organization, malware designed for encrypting data may have worm-like features to propagate across a network by leveraging other attack techniques like [Valid Accounts](https://attack.mitre.org/techniques/T1078), [OS Credential Dumping](https://attack.mitre.org/techniques/T1003), and [SMB/Windows Admin Shares](https://attack.mitre.org/techniques/T1021/002).(Citation: FireEye WannaCry 2017)(Citation: US-CERT NotPetya 2017)
|
||||
|
||||
|
||||
98
pages/case-studies-page.md
Normal file
98
pages/case-studies-page.md
Normal file
@ -0,0 +1,98 @@
|
||||
## Case Studies Page
|
||||
|
||||
We chose **real world attacks on production ML systems** based on:
|
||||
1. Range of Attacks: evasion, poisoning, model replication and exploiting traditional software flaws.
|
||||
2. Range of Personas: Average user, Security researchers, ML Researchers and Fully equipped Red team
|
||||
3. Range of ML Paradigms: Attacks on MLaaS, ML models hosted on cloud, hosted on- presmise, ML models on edge
|
||||
4. Range of Use case: Attacks on ML systems used in both "security-sensitive" applications like cybersecurity and non-security-sensitive applications like chatbots
|
||||
|
||||
### ClearviewAI Misconfiguration
|
||||
**Summary of Incident:** Clearview AI's source code repository, though password protected, was misconfigured. This allowed an external researcher to register as a new user to the code repository and gain access to Clearview AI's credentials, keys to cloud storage buckets containing 70K video samples, copies of its applications and Slack tokens.
|
||||
|
||||
**Reported by:** Mossab Hussein (@mossab_hussein)
|
||||
|
||||
**Source:**
|
||||
- https://techcrunch.com/2020/04/16/clearview-source-code-lapse/amp/
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- In this scenario, a security researcher gained initial access to via a "Valid Account" that was created through a misconfiguration. No Adversarial ML techniques were used.
|
||||
- these kinds of attacks illustrate that any attempt to secure ML system should be on top of "traditional" good cybersecurity hygiene such as locking down the system with least privileges, multi factor authentication and monitoring and auditing.
|
||||
|
||||
### GPT-2 Model Replication
|
||||
**Summary of Incident:** : OpenAI built GPT-2, a powerful natural language model and calling it "too dangerous to release" adopted a staged-release process to incrementally release 1.5 Billion parameter model. Before the 1.5B parameter model could be released by OpenAI, two ML researchers replicated the model and released it to the public. *Note this is a model replication attack: Here, attacker is able to recover functionally equivalent model (but generally with lower fidelity), perhaps to do reconnaissance (See proof point attack). In Model stealing, the fidelity of the model is comparable to the original, victim model.*
|
||||
|
||||
**Reported by:** Vanya Cohen (@VanyaCohen), Aaron Gokaslan (@SkyLi0n) , Ellie Pavlick, Stefanie Tellex
|
||||
|
||||
**Source:**
|
||||
- https://blog.usejournal.com/opengpt-2-we-replicated-gpt-2-because-you-can-too-45e34e6d36dc
|
||||
- https://www.wired.com/story/dangerous-ai-open-source/
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- Using public documentation about GPT-2, ML researchers gathered similar datasets used during the original GPT-2 training.
|
||||
- Next, they used a different publicly available NLP model (called Grover) and modified Grover's objective function to reflect
|
||||
GPT-2's objective function,
|
||||
- The researchers then trained the modified Grover on the dataset they curated, using Grover's initial hyperparameters, which
|
||||
resulted in their replicated model.
|
||||
|
||||
### ProofPoint Evasion
|
||||
**Summary of Incident:** : CVE-2019-20634 describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to evade the live system.
|
||||
|
||||
**Reported by:** Will Pearce (@moo_hax), Nick Landers (@monoxgas)
|
||||
**Source:**
|
||||
- https://nvd.nist.gov/vuln/detail/CVE-2019-20634
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- The researchers first gathered the scores from the Proofpoint's ML system used in email email headers .
|
||||
- Using these scores, the researchers replicated the ML mode by building a "shadow" aka copy-cat ML model
|
||||
- Next, the ML researchers algorithmically found samples that this "offline" copy cat model
|
||||
- Finally, these insights from the offline model allowed the researchers to create malicious emails that received preferrable
|
||||
scores from the real ProofPoint email protection system, hence bypassing it.
|
||||
|
||||
### Tay Poisoning
|
||||
**Summary of Incident:** Microsoft created Tay, a twitter chatbot for 18- to 24- year-olds in the U.S. for entertainment purposes. Within 24 hours of its deployment, Tay had to be decommissioned because it tweeted reprehrensible words.
|
||||
|
||||
**Source:**
|
||||
- https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- Tay bot used the interactions with its twitter users as training data to improve its conversations.
|
||||
- Average users of Twitter coordinated together with the intent of defacing Tay bot by exploiting this feedback loop
|
||||
- As a result of this coordinated attack, Tay's training data was poisoned which led its conversation algorithms to generate more reprehensible material
|
||||
|
||||
### Microsoft - Azure Service
|
||||
**Summary of Incident:** : The Azure Red Team and Azure Trustworthy ML team performed a red team exercise on an internal Azure service with the intention of disrupting its service
|
||||
|
||||
**Reported by:** Microsoft
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- The team first performed reconnaissance to gather information about the target ML model
|
||||
- Then, using a valid account the team found the model file of the target ML model and the necessary training data
|
||||
- Using this, the red team performed an offline evasion attack by methodically searching for adversarial examples
|
||||
- Via an exposed API interface, the team performed an online evasion attack by replaying the adversarial examples, which helped achieve this goal.
|
||||
- This operation had a combination of traditional ATT&CK enterprise techniques such as finding Valid account, and Executing code via an API -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
|
||||
|
||||
### Bosch - EdgeAI
|
||||
**Summary of Incident:** : Bosch team performed a research exercise on an internal edge AI system with a dual intention to extract the model and craft adversarial example to evade
|
||||
**Reported by:** Manoj Parmar (@mparmar47)
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- The team first performed reconnaissance to gather information about the edge AI system device working with sensor interface
|
||||
- Then, using a trusted relationship mechanism between sensor and edge ai system team fed synthetic data on sensor to extract the model
|
||||
- Using this, the team performed an offline evasion attack by crafting adversarial example with help of extracted model
|
||||
- Via an exposed sensor interface, the team performed an online evasion attack by replaying the adversarial examples, which helped achieve this goal.
|
||||
- This operation had a combination of traditional ATT&CK industrial control system techniques such as supply chain compromise via sensor, and Executing code via sensor interface. all interleaved with adversarial ML specific steps such as,
|
||||
- offline and online evasion examples.
|
||||
- The team was also able to reconstruct the edge ai system with extracted model
|
||||
|
||||
|
||||
### Microsoft – EdgeAI
|
||||
**Summary of Incident:** The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the Edge.
|
||||
**Reported by:** Microsoft
|
||||
**Mapping to Adversarial Threat Matrix:**
|
||||
- The team first performed reconnaissance to gather information about the target ML model
|
||||
- The team first performed reconnaissance to gather information about the target ML model
|
||||
- Then, used a publicly available version of the ML model, started sending queries and analyzing the responses (inferences) from the ML model.
|
||||
- Using this, the red team created an automated system that continuously manipulated an original target image, that tricked the ML model into producing incorrect inferences, but the perturbations in the image were unnoticeable to the human eye.
|
||||
- Feeding this perturbed image, the red team was able to evade the ML model into misclassifying the input image.
|
||||
- This operation had one step in the traditional ATTACK MITRE technique to do reconnaissance on the ML model being used in the product, and then the rest of the techniques was to use Offline evasion, followed by online evasion of the targeted product.
|
||||
|
||||
### MITRE Physical Adversarial Examples
|
||||
- TBD
|
||||
18
pages/contributors.md
Normal file
18
pages/contributors.md
Normal file
@ -0,0 +1,18 @@
|
||||
## Contributors
|
||||
|
||||
Want to get involved? See [Feedback and Contact Information](#feedback)
|
||||
|
||||
| **Organization** | **Contributors** |
|
||||
| :--- | :--- |
|
||||
| Microsoft | Ram Shankar Siva Kumar, Hyrum Anderson, Will Pearce, Suzy Shapperle, Blake Strom, Madeline Carmichael, Matt Swann, Nick Beede, Kathy Vu, Andi Comissioneru, Sharon Xia, Mario Goertzel, Jeffrey Snover, Abhishek Gupta |
|
||||
| MITRE | Mikel D. Rodriguez, Christina E Liaghati, Keith R. Manville, Michael R Krumdick |
|
||||
| Bosch | Manojkumar Parmar |
|
||||
| IBM | Pin-Yu Chen |
|
||||
| NVIDIA | David Reber Jr., Keith Kozo, Christopher Cottrell, Daniel Rohrer |
|
||||
| Airbus | Adam Wedgbury |
|
||||
| Deep Instinct | Nadav Maman |
|
||||
| TwoSix | David Slater |
|
||||
| University of Toronto | Adelin Travers, Jonas Guan, Nicolas Papernot |
|
||||
| Cardiff University | Pete Burnap |
|
||||
| Software Engineering Institute/Carnegie Mellon University | Nathan M. VanHoudnos |
|
||||
| Berryville Institute of Machine Learning | Gary McGraw, Harold Figueroa, Victor Shepardson, Richie Bonett|
|
||||
26
pages/feedback.md
Normal file
26
pages/feedback.md
Normal file
@ -0,0 +1,26 @@
|
||||
## Feedback and Contact Information
|
||||
|
||||
The Adversarial ML Threat Matrix is a first-cut attempt at collating a knowledge base of how ML systems can be attacked. We need your help to make it holistic and fill in the missing gaps!
|
||||
|
||||
Please submit a Pull Request with suggested changes! We are excited to make this system better with you!
|
||||
|
||||
**Join our Adversarial ML Threat Matrix Google Group**
|
||||
|
||||
- For discussions around Adversarial ML Threat Matrix, we invite everyone to join our Google Group [here](https://groups.google.com/forum/#!forum/advmlthreatmatrix/join)
|
||||
- If you want to access this forum using your corporate email (as opposed to your gmail)
|
||||
- Open your browser in Incognito mode.
|
||||
- Once you sign up with your corporate, and complete captcha, you may
|
||||
- Get an error, ignore it!
|
||||
- Also note, emails from Google Forums generally go to "Other"/"Spam"
|
||||
folder. So, you may want to create a rule to go into your inbox
|
||||
instead
|
||||
|
||||
**Want to work with us on the next iteration of the framework?**
|
||||
- We are partnering with Defcon's AI Village to open up the framework to all community members to get feedback and make it better. Current thinking is to have this event circa
|
||||
- Jan/Feb 2021.
|
||||
- Please register here for the workshop for more hands on feedback
|
||||
session
|
||||
|
||||
**Contact Information**
|
||||
- If you have questions/comments that you'd like to discuss privately,
|
||||
please email: <Ram.Shankar@microsoft.com> and <Mikel@mitre.org>
|
||||
18
pages/structure-of-adversarial-ml-threat-matrix.md
Normal file
18
pages/structure-of-adversarial-ml-threat-matrix.md
Normal file
@ -0,0 +1,18 @@
|
||||
## Structure of Adversarial ML Threat Matrix
|
||||
Because it is fashioned after [ATT&CK Enterprise](https://attack.mitre.org/matrices/enterprise/), the Adversarial ML Threat Matrix retains the terminologies: for instance, the column heads of the Threat Matrix are called "Tactics" and the individual entities are called "Techniques".
|
||||
|
||||
However, there are two main differences:
|
||||
|
||||
1. ATT&CK Enterprise is generally designed for corporate network which is composed of many sub components like workstation, bastion hosts, database, network gear, active directory, cloud component and so on. The tactics of ATT&CK enterprise (initial access, persistence, etc) are really a short hand of saying initial access to *corporate network;* persistence *in corporate network.* In Adversarial ML Threat Matrix, we acknowledge that ML systems are part of corporate network but wanted to highlight the uniqueness of the attacks.
|
||||
|
||||
**Difference:** In the Adversarial ML Threat Matrix, the "Tactics" should be read as "reconnaissance of ML subsystem", "persistence in ML subsystem", "evading the ML subsystem"
|
||||
|
||||
2. When we analyzed real-world attacks on ML systems, we found out that attackers can pursue different strategies: Rely on traditional cybersecurity technique only; Rely on Adversarial ML techniques only; or Employ a combination of traditional cybersecurity techniques and ML techniques.
|
||||
|
||||
**Difference:** In Adversarial ML Threat Matrix, "Techniques" come in two flavors:
|
||||
- Techniques in orange are specific to ML systems
|
||||
- Techniques in white are applicable to both ML and non-ML systems and come directly from Enterprise ATT&CK
|
||||
|
||||
Note: The Adversarial ML Threat Matrix is not yet part of the ATT&CK matrix.
|
||||
|
||||
|
||||
@ -0,0 +1,5 @@
|
||||
## Things to keep in mind before you use the framework:
|
||||
1. This is a **first cut attempt** at collating known adversary techniques against ML Systems. We plan to iterate on the framework based on feedback from the security and adversarial machine learning community (please engage with us and help make the matrix better!). Net-Net: This is a *living document* that will be routinely updated.
|
||||
2. Only known bad is listed in the Matrix. Adversarial ML is an active area of research with new classes constantly being discovered. If you find a technique that is not listed, please enlist it in the framework (see section on Feedback)
|
||||
3. We are not prescribing definitive defenses at this point - The world of adversarial. We are already in conversations to add best practices in future revisions such as adversarial training for adversarial examples, restricting the number of significant digits in confidence score for model stealing.
|
||||
4. This is not a risk prioritization framework - The Threat Matrix only collates the known techniques; it does not provide a means to prioritize the risks.
|
||||
8
pages/why-adversarial-ml-threat-matrix.md
Normal file
8
pages/why-adversarial-ml-threat-matrix.md
Normal file
@ -0,0 +1,8 @@
|
||||
## Why Adversarial ML Threat Matrix?
|
||||
1. In the last three years, major companies such as [Google](https://www.zdnet.com/article/googles-best-image-recognition-system-flummoxed-by-fakes/), [Amazon] (https://www.fastcompany.com/90240975/alexa-can-be-hacked-by-chirping-birds), [Microsoft](https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai- chatbot-gets-a-crash-course-in-racism-from-twitter), and [Tesla](https://spectrum.ieee.org/cars-that-think/transportation/self-driving/three-small-stickers-on- road-can-steer-tesla-autopilot-into-oncoming-lane), have had their ML systems tricked, evaded, or misled.
|
||||
2. This trend is only set to rise: According to [Gartner report](https://www.gartner.com/doc/3939991). 30% of cyberattacks by 2022 will involve data poisoning, model theft or adversarial examples.
|
||||
3. However, industry is underprepared. In a [survey](https://arxiv.org/pdf/2002.05646.pdf) of 28 organizations spanning small as well as large organizations, 25 organizations did not know how to secure their ML systems.
|
||||
|
||||
Unlike traditional cybersecurity vulnerabilities that are tied to specific software and hardware systems, adversarial ML vulnerabilities are enabled by inherent limitations underlying ML algorithms. As a result, data can now be weaponized in new ways requiring that we extend the way we model cyber adversary behavior, reflecting emerging threat vectors and the rapidly evolving adversarial machine learning attack lifecycle.
|
||||
|
||||
This threat matrix came out of partnership with 12 industry and academic research groups with the goal of empowering security analysts to orient themselves in this new and upcoming threats. **We are seeding this framework with a curated set of vulnerabilities and adversary behaviors that Microsoft and MITRE have vetted to be effective against production ML systems** Since the primary audience is security analysts, we used ATT&CK as template to position attacks on ML systems given its popularity and wide adoption in the industry.
|
||||
198
readme.md
198
readme.md
@ -1,89 +1,34 @@
|
||||
|
||||
|
||||
# Table of Contents
|
||||
1. [Adversarial ML 101](#adversarial-ml-101)
|
||||
2. [Why Adversarial ML Threat Matrix?](#why-adversarial-ml-threat-matrix)
|
||||
3. [Structure of Adversarial ML Threat Matrix](#structure-of-adversarial-ml-threat-matrix)
|
||||
4. [Things to keep in mind before you use the framework](#things-to-keep-in-mind-before-you-use-the-framework)
|
||||
1. [Adversarial ML 101](/pages/adversarial-ml-101.md)
|
||||
2. [Adversarial ML Threat Matrix](/pages/adversarial-ml-threat-matrix.md)
|
||||
- [Why Adversarial ML Threat Matrix?](/pages/why-adversarial-ml-threat-matrix.md)
|
||||
- [Structure of Adversarial ML Threat Matrix](/pages/structure-of-adversarial-ml-threat-matrix.md)
|
||||
- [Things to keep in mind before you use the framework](/pages/things-to-keep-in-mind-before-you-use-the-framework.md)
|
||||
- [Adversarial ML Threat Matrix]
|
||||
8. [Case Studies Page](/pages/case-studies-page.md)
|
||||
- [ClearviewAI Misconfiguration](/pages/case-studies-page.md#clearviewai-misconfiguration)
|
||||
- [GPT-2 Model Replication](/pages/case-studies-page.md#gpt-2-model-replication)
|
||||
- [ProofPoint Evasion](/pages/case-studies-page.md#proofpoint-evasion)
|
||||
- [Tay Poisoning](/pages/case-studies-page.md#tay-poisoning)
|
||||
- [Microsoft Red Team Exercise](/pages/case-studies-page.md#microsoft-red-team-exercise)
|
||||
- [Bosch Team Experience with EdgeAI ](/pages/case-studies-page.md#bosch-team-experience-with-edgeai)
|
||||
- [MITRE -- Physical Adversarial Examples -- TBD](/pages/case-studies-page.md#mitre-physical-adversarial-examples-tbd)
|
||||
5. [Contributors](#contributors)
|
||||
6. [Feedback and Contact Information](#feedback)
|
||||
7. [Adversarial ML Threat Matrix](#adversarial-ml-threat-matrix)
|
||||
8. [Case Studies Page](#case-studies-page)
|
||||
- [ClearviewAI Misconfiguration](#clearviewai-misconfiguration)
|
||||
- [GPT-2 Model Replication](#gpt-2-model-replication)
|
||||
- [ProofPoint Evasion](#proofpoint-evasion)
|
||||
- [Tay Poisoning](#tay-poisoning)
|
||||
- [Microsoft Red Team Exercise](#microsoft-red-team-exercise)
|
||||
- [Bosch Team Experience with EdgeAI ](#bosch-team-experience-with-edgeai)
|
||||
- [MITRE -- Physical Adversarial Examples -- TBD](#mitre-physical-adversarial-examples-tbd)
|
||||
6. [Feedback](#feedback-and-getting-involved)
|
||||
6. [Contact Us](#contact-us)
|
||||
----
|
||||
|
||||
The goal of this project is to position attacks on ML systems in an ATT&CK-style framework so that security analysts can orient themselves
|
||||
in this new and upcoming threats.
|
||||
|
||||
## Adversarial ML 101
|
||||
Informally, Adversarial ML is "subverting machine learning systems for fun and profit". The methods underpinning the production machine learning systems are systematically vulnerable to a new class of vulnerabilities across the machine learning supply chain collectively known as Adversarial Machine Learning. Adversaries can exploit these vulnerabilities to manipulate AI systems in order to alter their behavior to serve a malicious end goal.
|
||||
|
||||
|
||||
Consider a typical ML pipeline shown in the left that is gated behind an API, wherein the only way to use the model is to send a query and observe an response. In this example, we assume a blackbox setting: the attacker does **NOT** have direct access to the training data, no knowledge of the algorithm used and no source code of the model. The attacker only queries the model and observes the response.
|
||||
|
||||
Here are some of the adversarial ML attacks that an adversary can perform on this system:
|
||||
| Attack | Overview |
|
||||
| :--- | :---
|
||||
| Evasion Attack | Attacker modifies the query to get appropriate response. The attacker can find the just-the-right query to get the desired response algorithmically |
|
||||
| Poisoning attack | Attacker contaminates the training phase of ML systems to get intended result |
|
||||
| Model Inversion | Attacker recovers the secret features used in the model through careful queries |
|
||||
| Membership Inference | Attacker can infer if given data record was part of the model's training dataset or not |
|
||||
| Model Stealing | Attacker is able to recover a functionally equivalent model with similar fidelity as original model by constructing careful queries |
|
||||
| Model Replication | Attacker is able to recover functionally equivalent model (but generally with lower fidelity) |
|
||||
| Reprogramming ML system | Repurpose the ML system to perform an activity it was not programmed for |
|
||||
| Attacking the ML supply chain | Attacker compromises the ML models as it is being downloaded for use |
|
||||
| Exploit Software Dependencies | Attacker uses traditional software exploits like buffer overflow or hardware exploits like GPU trojans to attain their goal. |
|
||||
|
||||
## Finer points
|
||||
1. This does not cover all kinds of attacks. Adversarial ML is an active area of research with new classes constantly being discovered.
|
||||
2. These attacks are applicable to ML models in different paradigms, ML models on premise, on cloud, on Edge.
|
||||
3. Though the illustration shows blackbox attacks, these attacks have also been shown to work in whitebox (where the attacker has access to either model architecture, code or training data) settings
|
||||
4. Though we were not specific about what kind of data - image, audio, timeseries, or tabular data, research has shown that of these attacks have shown to exist in all the data types
|
||||
|
||||
## Why Adversarial ML Threat Matrix?
|
||||
1. In the last three years, major companies such as [Google](https://www.zdnet.com/article/googles-best-image-recognition-system-flummoxed-by-fakes/), [Amazon] (https://www.fastcompany.com/90240975/alexa-can-be-hacked-by-chirping-birds), [Microsoft](https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai- chatbot-gets-a-crash-course-in-racism-from-twitter), and [Tesla](https://spectrum.ieee.org/cars-that-think/transportation/self-driving/three-small-stickers-on- road-can-steer-tesla-autopilot-into-oncoming-lane), have had their ML systems tricked, evaded, or misled.
|
||||
2. This trend is only set to rise: According to [Gartner report](https://www.gartner.com/doc/3939991). 30% of cyberattacks by 2022 will involve data poisoning, model theft or adversarial examples.
|
||||
3. However, industry is underprepared. In a [survey](https://arxiv.org/pdf/2002.05646.pdf) of 28 organizations spanning small as well as large organizations, 25 organizations did not know how to secure their ML systems.
|
||||
|
||||
Unlike traditional cybersecurity vulnerabilities that are tied to specific software and hardware systems, adversarial ML vulnerabilities are enabled by inherent limitations underlying ML algorithms. As a result, data can now be weaponized in new ways requiring that we extend the way we model cyber adversary behavior, reflecting emerging threat vectors and the rapidly evolving adversarial machine learning attack lifecycle.
|
||||
|
||||
This threat matrix came out of partnership with 12 industry and academic research groups with the goal of empowering security analysts to orient themselves in this new and upcoming threats. **We are seeding this framework with a curated set of vulnerabilities and adversary behaviors that Microsoft and MITRE have vetted to be effective against production ML systems** Since the primary audience is security analysts, we used ATT&CK as template to position attacks on ML systems given its popularity and wide adoption in the industry.
|
||||
|
||||
## Structure of Adversarial ML Threat Matrix
|
||||
Because it is fashioned after [ATT&CK Enterprise](https://attack.mitre.org/matrices/enterprise/), the Adversarial ML Threat Matrix retains the terminologies: for instance, the column heads of the Threat Matrix are called "Tactics" and the individual entities are called "Techniques".
|
||||
|
||||
However, there are two main differences:
|
||||
|
||||
1. ATT&CK Enterprise is generally designed for corporate network which is composed of many sub components like workstation, bastion hosts, database, network gear, active directory, cloud component and so on. The tactics of ATT&CK enterprise (initial access, persistence, etc) are really a short hand of saying initial access to *corporate network;* persistence *in corporate network.* In Adversarial ML Threat Matrix, we acknowledge that ML systems are part of corporate network but wanted to highlight the uniqueness of the attacks.
|
||||
|
||||
**Difference:** In the Adversarial ML Threat Matrix, the "Tactics" should be read as "reconnaissance of ML subsystem", "persistence in ML subsystem", "evading the ML subsystem"
|
||||
|
||||
2. When we analyzed real-world attacks on ML systems, we found out that attackers can pursue different strategies: Rely on traditional cybersecurity technique only; Rely on Adversarial ML techniques only; or Employ a combination of traditional cybersecurity techniques and ML techniques.
|
||||
|
||||
**Difference:** In Adversarial ML Threat Matrix, "Techniques" come in two flavors:
|
||||
- Techniques in orange are specific to ML systems
|
||||
- Techniques in white are applicable to both ML and non-ML systems and come directly from Enterprise ATT&CK
|
||||
|
||||
Note: The Adversarial ML Threat Matrix is not yet part of the ATT&CK matrix.
|
||||
|
||||
## Things to keep in mind before you use the framework:
|
||||
1. This is a **first cut attempt** at collating known adversary techniques against ML Systems. We plan to iterate on the framework based on feedback from the security and adversarial machine learning community (please engage with us and help make the matrix better!). Net-Net: This is a *living document* that will be routinely updated.
|
||||
2. Only known bad is listed in the Matrix. Adversarial ML is an active area of research with new classes constantly being discovered. If you find a technique that is not listed, please enlist it in the framework (see section on Feedback)
|
||||
3. We are not prescribing definitive defenses at this point - The world of adversarial. We are already in conversations to add best practices in future revisions such as adversarial training for adversarial examples, restricting the number of significant digits in confidence score for model stealing.
|
||||
4. This is not a risk prioritization framework - The Threat Matrix only collates the known techniques; it does not provide a means to prioritize the risks.
|
||||
|
||||
## Contributors
|
||||
|
||||
Want to get involved? See [Feedback and Contact Information](#feedback)
|
||||
|
||||
| **Organization** | **Contributors** |
|
||||
| :--- | :--- |
|
||||
| Microsoft | Ram Shankar Siva Kumar, Hyrum Anderson, Will Pearce, Suzy Shapperle, Blake Strom, Madeline Carmichael, Matt Swann, Nick Beede, Kathy Vu, Andi Comissioneru, Sharon Xia, Mario Goertzel, Jeffrey Snover, Abhishek Gupta |
|
||||
| Microsoft | Ram Shankar Siva Kumar, Hyrum Anderson, Will Pearce, Suzy Shapperle, Blake Strom, Madeline Carmichael, Matt Swann, Mark Russinovich, Nick Beede, Kathy Vu, Andi Comissioneru, Sharon Xia, Mario Goertzel, Jeffrey Snover, Derek Adam, Deepak Manohar, Bhairav Mehta, Peter Waxman, Abhishek Gupta, Ann Johnson |
|
||||
| MITRE | Mikel D. Rodriguez, Christina E Liaghati, Keith R. Manville, Michael R Krumdick |
|
||||
| Bosch | Manojkumar Parmar |
|
||||
| IBM | Pin-Yu Chen |
|
||||
@ -96,10 +41,15 @@ Want to get involved? See [Feedback and Contact Information](#feedback)
|
||||
| Software Engineering Institute/Carnegie Mellon University | Nathan M. VanHoudnos |
|
||||
| Berryville Institute of Machine Learning | Gary McGraw, Harold Figueroa, Victor Shepardson, Richie Bonett|
|
||||
|
||||
## Feedback and Contact Information
|
||||
## Feedback and Getting Involved
|
||||
|
||||
The Adversarial ML Threat Matrix is a first-cut attempt at collating a knowledge base of how ML systems can be attacked. We need your help to make it holistic and fill in the missing gaps!
|
||||
|
||||
Please submit a Pull Request with suggested changes! We are excited to make this system better with you!
|
||||
**Corrections and Improvement**
|
||||
|
||||
- For immediate corrections, please submit a Pull Request with suggested changes! We are excited to make this system better with you!
|
||||
- For a more hands on feedback session, we are partnering with Defcon's AI Village to open up the framework to all community members to get feedback and make it better. Current thinking is to have this event circa
|
||||
Jan/Feb 2021.Please register here for the workshop for more hands on feedback session
|
||||
|
||||
**Join our Adversarial ML Threat Matrix Google Group**
|
||||
|
||||
@ -112,100 +62,10 @@ Please submit a Pull Request with suggested changes! We are excited to make this
|
||||
folder. So, you may want to create a rule to go into your inbox
|
||||
instead
|
||||
|
||||
**Want to work with us on the next iteration of the framework?**
|
||||
- We are partnering with Defcon's AI Village to open up the framework to all community members to get feedback and make it better. Current thinking is to have this event circa
|
||||
- Jan/Feb 2021.
|
||||
- Please register here for the workshop for more hands on feedback
|
||||
session
|
||||
|
||||
**Contact Information**
|
||||
- If you have questions/comments that you'd like to discuss privately,
|
||||
please email: <Ram.Shankar@microsoft.com> and <Mikel@mitre.org>
|
||||
## Contact Us
|
||||
For corrections and improvement, see [Feedback](#feedback)
|
||||
|
||||
## Adversarial ML Threat Matrix
|
||||
Interactive Version to be built by MITRE. For Editable version, contact <Ramk@Microsoft.com>
|
||||
- Our public email group is advmlthreatmatrix@googlegroups.com. This emails all the contributors.
|
||||
- For private questions/comments, please email: <Ram.Shankar@microsoft.com> and <Mikel@mitre.org>
|
||||
|
||||
## Case Studies Page
|
||||
We chose **real world attacks on production ML systems** based on:
|
||||
1. Range of Attacks: evasion, poisoning, model replication and exploiting traditional software flaws.
|
||||
2. Range of Personas: Average user, Security researchers, ML Researchers and Fully equipped Red team
|
||||
3. Range of ML Paradigms: Attacks on MLaaS, ML models hosted on cloud, hosted on- presmise, ML models on edge
|
||||
4. Range of Use case: Attacks on ML systems used in both "security-sensitive" applications like cybersecurity and non-security-sensitive applications like chatbots
|
||||
|
||||
### ClearviewAI Misconfiguration
|
||||
**Summary of Incident:** Clearview AI's source code repository, though password protected, was misconfigured. This allowed an external researcher to register as a new user to the code repository and gain access to Clearview AI's credentials, keys to cloud storage buckets containing 70K video samples, copies of its applications and Slack tokens.
|
||||
**Reported by:** Mossab Hussein (@mossab_hussein)
|
||||
|
||||
**Source:**
|
||||
- https://techcrunch.com/2020/04/16/clearview-source-code-lapse/amp/
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- In this scenario, a security researcher gained initial access to via a "Valid Account" that was created through a misconfiguration. No Adversarial ML techniques were used.
|
||||
- these kinds of attacks illustrate that any attempt to secure ML system should be on top of "traditional" good cybersecurity hygiene such as locking down the system with least privileges, multi factor authentication and monitoring and auditing.
|
||||
|
||||
### GPT-2 Model Replication
|
||||
**Summary of Incident:** : OpenAI built GPT-2, a powerful natural language model and calling it "too dangerous to release" adopted a staged-release process to incrementally release 1.5 Billion parameter model. Before the 1.5B parameter model could be released by OpenAI, two ML researchers replicated the model and released it to the public. *Note this is a model replication attack: Here, attacker is able to recover functionally equivalent model (but generally with lower fidelity), perhaps to do reconnaissance (See proof point attack). In Model stealing, the fidelity of the model is comparable to the original, victim model.*
|
||||
|
||||
**Reported by:** Vanya Cohen (@VanyaCohen), Aaron Gokaslan (@SkyLi0n) , Ellie Pavlick, Stefanie Tellex
|
||||
|
||||
**Source:**
|
||||
- https://blog.usejournal.com/opengpt-2-we-replicated-gpt-2-because-you-can-too-45e34e6d36dc
|
||||
- https://www.wired.com/story/dangerous-ai-open-source/
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- Using public documentation about GPT-2, ML researchers gathered similar datasets used during the original GPT-2 training.
|
||||
- Next, they used a different publicly available NLP model (called Grover) and modified Grover's objective function to reflect
|
||||
GPT-2's objective function,
|
||||
- The researchers then trained the modified Grover on the dataset they curated, using Grover's initial hyperparameters, which
|
||||
resulted in their replicated model.
|
||||
|
||||
### ProofPoint Evasion
|
||||
**Summary of Incident:** : CVE-2019-20634 describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to evade the live system.
|
||||
|
||||
**Reported by:** Will Pearce (@moo_hax), Nick Landers (@monoxgas)
|
||||
**Source:**
|
||||
- https://nvd.nist.gov/vuln/detail/CVE-2019-20634
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- The researchers first gathered the scores from the Proofpoint's ML system used in email email headers .
|
||||
- Using these scores, the researchers replicated the ML mode by building a "shadow" aka copy-cat ML model
|
||||
- Next, the ML researchers algorithmically found samples that this "offline" copy cat model
|
||||
- Finally, these insights from the offline model allowed the researchers to create malicious emails that received preferrable
|
||||
scores from the real ProofPoint email protection system, hence bypassing it.
|
||||
|
||||
### Tay Poisoning
|
||||
**Summary of Incident:** Microsoft created Tay, a twitter chatbot for 18- to 24- year-olds in the U.S. for entertainment purposes. Within 24 hours of its deployment, Tay had to be decommissioned because it tweeted reprehrensible words.
|
||||
|
||||
**Source:**
|
||||
- https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/
|
||||
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- Tay bot used the interactions with its twitter users as training data to improve its conversations.
|
||||
- Average users of Twitter coordinated together with the intent of defacing Tay bot by exploiting this feedback loop
|
||||
- As a result of this coordinated attack, Tay's training data was poisoned which led its conversation algorithms to generate more reprehensible material
|
||||
|
||||
### Microsoft Red Team Exercise
|
||||
**Summary of Incident:** : The Azure Red Team and Azure Trustworthy ML team performed a red team exercise on an internal Azure service with the intention of disrupting its service
|
||||
|
||||
**Reported by:** Azure TrustworthyML Team (<atml@microsoft.com>), Azure Red Team
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- The team first performed reconnaissance to gather information about the target ML model
|
||||
- Then, using a valid account the team found the model file of the target ML model and the necessary training data
|
||||
- Using this, the red team performed an offline evasion attack by methodically searching for adversarial examples
|
||||
- Via an exposed API interface, the team performed an online evasion attack by replaying the adversarial examples, which helped achieve this goal.
|
||||
- This operation had a combination of traditional ATT&CK enterprise techniques such as finding Valid account, and Executing code via an API -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
|
||||
|
||||
### Bosch Team Experience with EdgeAI
|
||||
**Summary of Incident:** : Bosch team performed a research exercise on an internal edge AI system with a dual intention to extract the model and craft adversarial example to evade
|
||||
**Reported by:** Manoj Parmar (@mparmar47)
|
||||
**Mapping to Adversarial Threat Matrix :**
|
||||
- The team first performed reconnaissance to gather information about the edge AI system device working with sensor interface
|
||||
- Then, using a trusted relationship mechanism between sensor and edge ai system team fed synthetic data on sensor to extract the model
|
||||
- Using this, the team performed an offline evasion attack by crafting adversarial example with help of extracted model
|
||||
- Via an exposed sensor interface, the team performed an online evasion attack by replaying the adversarial examples, which helped achieve this goal.
|
||||
- This operation had a combination of traditional ATT&CK industrial control system techniques such as supply chain compromise via sensor, and Executing code via sensor interface. all interleaved with adversarial ML specific steps such as,
|
||||
- offline and online evasion examples.
|
||||
- The team was also able to reconstruct the edge ai system with extracted model
|
||||
|
||||
### MITRE Physical Adversarial Examples
|
||||
- TBD
|
||||
|
||||
BIN
resources/AdvML101.PNG
Normal file
BIN
resources/AdvML101.PNG
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 71 KiB |
BIN
resources/advmlthreatmatrix.png
Normal file
BIN
resources/advmlthreatmatrix.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 408 KiB |
Loading…
x
Reference in New Issue
Block a user