advmlthreatmatrix initial release

pages/adversarial-ml-101.md

@@ -0,0 +1,55 @@
+# Adversarial Machine Learning 101 
+Informally, Adversarial ML is “subverting machine learning systems for fun and profit”. The methods underpinning the production machine learning systems are systematically vulnerable to a new class of vulnerabilities across the machine learning supply chain collectively known as Adversarial Machine Learning. Adversaries can exploit these vulnerabilities to manipulate AI systems in order to alter their behavior to serve a malicious end goal.
+
+Consider a typical ML pipeline shown below that is gated behind an API, wherein the only way to use the model is to send a query and observe a response.  In this example, we assume a blackbox setting: the attacker does NOT have direct access to the training data, no knowledge of the algorithm used and no source code of the model. The attacker only queries the model and observes the response. We will look at two broad categories of attacks: 
+
+**Train time vs Inference time:** 
+
+Training refers to the process by which data is modeled. This process includes collecting and processing data, training a model, validating the model works, and then finally deploying the model. An attack that happens at "train time" is an attack that happens while the model is learning prior its deployment. After a model is deployed, consumers of the model can submit queries and receive outputs (inferences).  
+An attack that happens at "inference time" is an attack where the learned state of the model does not change and the model is just providing outputs. In practice, a model could be re-trained after every new query providing an attacker with some interesting scenarios by which they could use an inference endpoint to perform a "train-time" attack. In any case, the delineation is useful to describe how an attacker could be interacting with a target model.
+
+With this in mind, we can jump into the attacks on ML systems. 
+
+![Adversarial ML 101](/images/AdvML101.PNG)
+
+
+# Machine Learning Attacks
+Attacks on machine learning systems can be categorized as follows.
+
+| Attack 		        | Overview	| Type |
+| :---			        | :---      | :---:|
+| Model Evasion         | Attacker modifies a query in order to get a desired outcome. These attacks are performed by iteratively querying a model and observing the output. | Inference |
+| Functional Extraction | Attacker is able to recover a functionally equivalent model by iteratively querying the model. This allows an attacker to examine the offline copy of the model before further attacking the online model. | Inference |
+| Model Poisoning 	    | Attacker contaminates the training data of an ML system in order to get a desired outcome at inference time. With influence over training data an attacker can create "backdoors" where an arbitrary input will result in a particular output. The model could be "reprogrammed" to perform a new undesired task. Further, access to training data would allow the attacker to create an offline model and create a Model Evasion. Access to training data could also result in the compromise of private data.  | Train |
+| Model Inversion 	    | Attacker recovers the features used to train the model. A successful attack would result in an attacker being able to launch a Membership inference attack. This attack could result in compromise of private data. | Inference | 
+| Traditional Attacks   | Attacker uses well established TTPs to attain their goal. | Both |
+
+
+
+# Attack Scenarios
+## Attack Scenario #1: Inference Attack 
+Consider the most common deployment scenario where a model is deployed as an API endpoint. In this blackbox setting an attacker can only query the model and observe the response. The attacker controls the input to the model, but the attacker does not know how it is processed.
+
+![Adversarial ML 101](/images/AdvML101_Inference.PNG)
+
+---- 
+## Attack Scenario #2: Training Time Attack
+Consider that an attacker has control over training data. This flavor of attack is shown in [Tay Poisoning Case Study](/pages/case-studies-page.md#tay-poisoning) where the attacker was able to compromise the training data via the feedback mechanism.
+
+
+![Adversarial ML 101](/images/AdvML101_Traintime.PNG)
+
+
+## Attack Scenario #3: Attack on Edge/Client
+Consider that a model exists on a client (like a phone) or on the edge (such as IoT) . An attacker might have access to model code through reversing the service on the client. This flavor of attack is shown in [Bosch Case Study with EdgeAI](/pages/case-studies-page.md#bosch---edge-ai).
+
+![Adversarial ML 101](/images/AdvML101_Client.PNG)
+
+
+# Important Notes
+1.  This does not cover all kinds of attacks -- adversarial ML is an active area of research with new classes of attacks constantly being discovered.
+2.  Though the illustration shows blackbox attacks, these attacks have also been shown to work in whitebox (where the attacker has access to either model architecture, code or training data)  settings.
+3.  Though we were not specific about what kind of data – image, audio, time series, or tabular data - research has shown that of these attacks are data agnostic.
+
+# Next Recommended Reading 
+Head over to [Adversarial ML Threat Matrix](/pages/adversarial-ml-threat-matrix.md#adversarial-ml-threat-matrix) page to see a compendium of attacks in ATT&CK style. 

pages/adversarial-ml-threat-matrix.md

@@ -0,0 +1,276 @@
+# Adversarial ML Threat Matrix
+
+## Things to keep in mind before you use the framework:
+1.  This is a **first cut attempt** at collating known adversary techniques against ML Systems. We plan to iterate on the framework based on feedback from the security     and adversarial machine learning community (please engage with us and help make the matrix better!). Note: This is a *living document* that will be routinely updated.
+   -  Have feedback or improvements? We want it! See [Feedback](/readme.md#feedback-and-getting-involved).
+2.  Only known bad is listed in the Matrix. Adversarial ML is an active area of research with new classes constantly being discovered. If you find a technique that is     not listed, please enlist it in the framework (see section on [Feedback](/readme.md#feedback-and-getting-involved)).
+3.  We are not prescribing definitive defenses at this point since the field has not reached consensus. We are already in conversations with industry members to add best practices in future revisions such as adversarial training for adversarial examples, restricting the number of significant digits in confidence score for model stealing.
+4.  This is not a risk prioritization framework - The Threat Matrix only collates the known techniques; it does not provide a means to prioritize the risks.
+
+## Structure of Adversarial ML Threat Matrix
+Because the Adversarial ML Threat Matrix is fashioned after [ATT&CK Enterprise](https://attack.mitre.org/matrices/enterprise/), it retains the terminologies: for instance, the column heads of the Threat Matrix are called "Tactics" and the individual entities are called "Techniques".
+
+However, there are two main differences:
+
+1.  ATT&CK Enterprise is generally designed for corporate network which is composed of many sub components like workstation, bastion hosts, database, network gear,         active directory, cloud component and so on. The tactics of ATT&CK enterprise (initial access, persistence, etc) are really a short hand of saying initial access       to *corporate network;* persistence *in corporate network.* In Adversarial ML Threat Matrix, we acknowledge that ML systems are part of corporate network but           wanted to highlight the uniqueness of the attacks.
+
+    **Difference:** In the Adversarial ML Threat Matrix, the "Tactics" should be read as "reconnaissance of ML subsystem", "persistence in ML subsystem", "evading the      ML subsystem".
+
+2.  When we analyzed real-world attacks on ML systems, we found out that attackers can pursue different strategies: Rely on traditional cybersecurity technique only;       Rely on Adversarial ML techniques only; or Employ a combination of traditional cybersecurity techniques and ML techniques.
+
+    **Difference:** In Adversarial ML Threat Matrix, "Techniques" come in two flavors:
+     -  Techniques in orange are specific to ML systems
+     -  Techniques in white are applicable to both ML and non-ML systems and come directly from Enterprise ATT&CK
+
+        Note: The Adversarial ML Threat Matrix is not yet part of the ATT&CK matrix.
+
+
+![Adversarial ML Threat Matrix](/images/AdvMLThreatMatrix.jpg)
+
+|Legend                            | Description                                                                             |
+|:---:                             | :---                                                                                    |
+|![Cyber](/images/color_cyber.png) | Techniques that are applicable to both ML and non-ML systems and come directly from Enterprise ATT&CK                    |
+|![AdvML](/images/color_advml.png) |  Techniques that are specific to ML systems                               |
+
+## Adversarial ML Matrix - Description
+
+### Reconnaissance
+
+
+#### ![AdvML](/images/color_advml.png)Acquire OSINT Information
+
+Adversaries may leverage publicly available information, or Open Source Intelligence (OSINT), about an organization that could identify where or how machine learning is being used in a system, and help tailor an attack to make it more effective. These sources of information include technical publications, blog posts, press releases, software repositories, public data repositories, and social media postings.
+
+#### ![AdvML](/images/color_advml.png)ML Model Discovery
+
+Adversaries may attempt to identify machine learning pipelines that exist on the system and gather information about them, including the software stack used to train and deploy models, training and testing data repositories, model repositories, and software repositories containing algorithms. This information can be used to identify targets for further collection, exfiltration, or disruption, or to tailor and improve attacks. 
+
+> ##### ![AdvML](/images/color_advml.png)Reveal ML Ontology
+> By ML ML Ontology, we are referring to specific components of the ML system such as dataset (image, audio, tabular, NLP), features (handcrafted or learned), model / learning algorithm (gradient based or non-gradient based), parameters / weights. Depending on how much information is known, it can be a greybox or whitebox level of attacker knowledge. 
+> 
+>
+> ##### ![AdvML](/images/color_advml.png)Reveal ML Model Family
+> Here the specifics of ML Models are not known and can generally be thought of as blackbox attacks. The attacker is able to only glean the model task, model input and model output. But because of the nature of the blog posts or papers that are published, some mention of the algorithms such as "Deep Learning" may squarely indicate that the underlying algorithm is gradient based.  
+
+
+#### ![AdvML](/images/color_advml.png)Gathering Datasets
+
+Adversaries may collect datasets similar to those used by a particular organization or in a specific approach. Datasets may be identified when [Acquiring OSINT Information](#Acquire-OSINT-Information). This may allow the adversary to replicate a private model's functionality, constituting Intellectual Property Theft, or enable the adversary to carry out other attacks such as an [Evasion Attack](#Evasion-Attack).
+
+#### ![AdvML](/images/color_advml.png)Exploit Physical Environment
+
+In addition to the attacks that take place purely in the digital domain, adversaries may also exploit the physical environment for their attacks. Recent work has show successful false positive and evasion attacks using physically printed patterns that are placed into scenes to disrupt and attack machine learning models. MITRE has recently created a dataset based on these [physically printed patterns](https://apricot.mitre.org/) to help researchers and practitioners better understand these attacks.
+
+#### ![AdvML](/images/color_advml.png)Model Replication
+
+An adversary may replicate a model's functionality by training a shadow model by exploiting its API, or by leveraging pre-trained weights.
+
+> ##### ![AdvML](/images/color_advml.png)Exploit API - Shadow Model 
+>
+> An adversary may replicate a machine learning model's functionality by exploiting its inference API. In this case of model replication, the attacker repeatedly queries the victim's inference API and uses it as an oracle to collect combination of data and label. From the combination of (data,label), the attacker builds a shadow model, that effectively functions as the victim model -- but with lower fidelity. This is generally the first step in model evasion. 
+>
+> ##### ![AdvML](/images/color_advml.png)Alter Publicly Available, Pre-Trained Weights
+>
+> An adversary uses pre-trained weights of one model to replicate a related model's functionality. For instance, researchers wanted to replicated GPT-2, a large language model. So, the researchers used the pre-trained weights of Grover, another NLP model, and modified it using GPT-2's objective function and training data, which effectively resulted in a shadow GPT-2 model (though with lower fidelity).
+
+#### ![AdvML](/images/color_advml.png)Model Stealing
+
+Machine learning models' functionality can be stolen exploiting an inference API. There is a difference between Model Extraction and Model Replication: in model extraction attacks, the attacker is able to build a shadow model whose fidelity matches that of the victim model and hence, model stealing/extraction attacks lead to Stolen Intellectual Property. In [Model Replication](#Model-Replication) attacks, the shadow model does not have the same fidelity as that of the victim model. 
+
+### Initial Access
+
+#### ![AdvML](/images/color_advml.png)Pre-Trained ML Model with Backdoor
+
+Adversaries may gain initial access to a system by compromising portions of the ML supply chain. This could include GPU hardware, data and its annotations, parts of the ML software stack, or the model itself. In some instances the attacker will need secondary access to fully carry out an attack using compromised components of the supply chain.
+
+### ![Cyber](/images/color_cyber.png) Included ATT&CK Techniques
+<details>
+<summary>Exploit Public-Facing Application</summary>
+
+Adversaries may attempt to take advantage of a weakness in an Internet-facing computer or program using software, data, or commands in order to cause unintended or unanticipated behavior. The weakness in the system can be a bug, a glitch, or a design vulnerability. These applications are often websites, but can include databases (like SQL)(Citation: NVD CVE-2016-6662), standard services (like SMB(Citation: CIS Multiple SMB Vulnerabilities) or SSH), and any other applications with Internet accessible open sockets, such as web servers and related services.(Citation: NVD CVE-2014-7169) Depending on the flaw being exploited this may include [Exploitation for Defense Evasion](https://attack.mitre.org/techniques/T1211).
+
+If an application is hosted on cloud-based infrastructure, then exploiting it may lead to compromise of the underlying instance. This can allow an adversary a path to access the cloud APIs or to take advantage of weak identity and access management policies.
+
+For websites and databases, the OWASP top 10 and CWE top 25 highlight the most common web-based vulnerabilities.(Citation: OWASP Top 10)(Citation: CWE top 25)
+</details>
+
+<details>
+<summary>Valid Accounts</summary>
+
+Adversaries may obtain and abuse credentials of existing accounts as a means of gaining Initial Access, Persistence, Privilege Escalation, or Defense Evasion. Compromised credentials may be used to bypass access controls placed on various resources on systems within the network and may even be used for persistent access to remote systems and externally available services, such as VPNs, Outlook Web Access and remote desktop. Compromised credentials may also grant an adversary increased privilege to specific systems or access to restricted areas of the network. Adversaries may choose not to use malware or tools in conjunction with the legitimate access those credentials provide to make it harder to detect their presence.
+
+The overlap of permissions for local, domain, and cloud accounts across a network of systems is of concern because the adversary may be able to pivot across accounts and systems to reach a high level of access (i.e., domain or enterprise administrator) to bypass access controls set within the enterprise. (Citation: TechNet Credential Theft)
+</details>
+
+<details>
+<summary>Phishing</summary>
+
+Adversaries may send phishing messages to elicit sensitive information and/or gain access to victim systems. All forms of phishing are electronically delivered social engineering. Phishing can be targeted, known as spear phishing. In spear phishing, a specific individual, company, or industry will be targeted by the adversary. More generally, adversaries can conduct non-targeted phishing, such as in mass malware spam campaigns.
+
+Adversaries may send victim’s emails containing malicious attachments or links, typically to execute malicious code on victim systems or to gather credentials for use of [Valid Accounts](https://attack.mitre.org/techniques/T1078). Phishing may also be conducted via third-party services, like social media platforms.
+</details>
+
+<details>
+<summary>External Remote Services</summary>
+
+Adversaries may leverage external-facing remote services to initially access and/or persist within a network. Remote services such as VPNs, Citrix, and other access mechanisms allow users to connect to internal enterprise network resources from external locations. There are often remote service gateways that manage connections and credential authentication for these services. Services such as [Windows Remote Management](https://attack.mitre.org/techniques/T1021/006/) can also be used externally.
+
+Access to [Valid Accounts](https://attack.mitre.org/techniques/T1078) to use the service is often a requirement, which could be obtained through credential pharming or by obtaining the credentials from users after compromising the enterprise network. Access to remote services may be used as a redundant or persistent access mechanism during an operation.
+</details>
+
+<details>
+<summary>Trusted Relationship</summary>
+
+Adversaries may breach or otherwise leverage organizations who have access to intended victims. Access through trusted third party relationship exploits an existing connection that may not be protected or receives less scrutiny than standard mechanisms of gaining access to a network.
+
+Organizations often grant elevated access to second or third-party external providers in order to allow them to manage internal systems as well as cloud-based environments. Some examples of these relationships include IT services contractors, managed security providers, infrastructure contractors (e.g. HVAC, elevators, physical security). The third-party provider's access may be intended to be limited to the infrastructure being maintained, but may exist on the same network as the rest of the enterprise. As such, [Valid Accounts](https://attack.mitre.org/techniques/T1078) used by the other party for access to internal network systems may be compromised and used.
+</details>
+
+### Execution
+#### ![AdvML](/images/color_advml.png)Execute Unsafe ML Models
+
+An Adversary may utilize unsafe ML Models that when executed have an unintended effect. The adversary can use this technique to establish persistent access to systems. These models may be introduced via a [Pre-Trained Model with Backdoor](#Pre-Trained-ML-Model-with-Backdoor).
+
+> ##### ![AdvML](/images/color_advml.png)ML Models from Compromised Sources
+>
+> TIn Model Zoo such as "Caffe Model Zoo" or "ONNX Model Zoo" a collection of state of the art ML, pre-trained ML models are available so that ML engineers do not have to spend resources training ML models from scratch (hence "pre-trained"). An adversary may be able to compromise the model by checking in malicious code into the repository or perform a Man-in-the-Middle attack as the models are downloaded.  
+>
+> ##### ![AdvML](/images/color_advml.png)Pickle Embedding
+>
+> Python is one of the most commonly used ML language. Python pickles are used in serializing and de-serializing a Python object structures. ML models are sometimes stored as pickles and shared. An adversary may use pickle embedding to introduce malicious data payloads which may result in remote code execution.
+
+### ![Cyber](/images/color_cyber.png) Included ATT&CK Techniques
+<details>
+<summary>Execution via API</summary>
+For most Machine Learning as a Service (MLaaS), the primary interaction point is via an API. So, attackers interact with the API in three ways: To build an offline copy of the model through Model Stealing or Model Replication; or to do online attacks like Model Inversion, Online Evasion, Membership inference. Execution via API is also possible for causative attacks if the adversary can taint the training data of the model via feedback loop, 
+</details>
+
+<details>
+<summary>Traditional Software Attacks</summary>
+All ML models exist in code, and thus vulnerable to "traditional software attacks" if the underlying system is not secured appropriately. For instance, [researchers](https://arxiv.org/abs/1711.11008) found that a number of popular ML dev packages like Tensorflow, Caffe, OpenCV had open CVEs against them, making them vulnerable to traditional heap overflow and buffer overflow attacks. 
+   
+   
+</details>
+
+### Persistence
+#### ![AdvML](/images/color_advml.png)Execute unsafe ML Model Execution
+
+An Adversary may utilize unsafe ML Models that when executed have an unintended effect. The adversary can use this technique to establish persistent access to systems. These models may be introduced via a [Pre-trained Model with Backdoor](#Pre-Trained-ML-Model-with-Backdoor). An example of this technique is to use pickle embedding to introduce malicious data payloads.
+
+### ![Cyber](/images/color_cyber.png) Included ATT&CK Techniques
+<details> 
+<summary>Account Manipulation</summary>
+Adversaries may manipulate accounts to maintain access to victim systems. Account manipulation may consist of any action that preserves adversary access to a compromised account, such as modifying credentials or permission groups. These actions could also include account activity designed to subvert security policies, such as performing iterative password updates to bypass password duration policies and preserve the life of compromised credentials. In order to create or manipulate accounts, the adversary must already have sufficient permissions on systems or the domain.
+</details>
+
+<details>
+<summary>Implant Container Image</summary>
+Adversaries may implant cloud container images with malicious code to establish persistence. Amazon Web Service (AWS) Amazon Machine Images (AMI), Google Cloud Platform (GCP) Images, and Azure Images as well as popular container runtimes such as Docker can be implanted or backdoored. Depending on how the infrastructure is provisioned, this could provide persistent access if the infrastructure provisioning tool is instructed to always use the latest image.(Citation: Rhino Labs Cloud Image Backdoor Technique Sept 2019)
+
+A tool has been developed to facilitate planting backdoors in cloud container images.(Citation: Rhino Labs Cloud Backdoor September 2019) If an attacker has access to a compromised AWS instance, and permissions to list the available container images, they may implant a backdoor such as a [Web Shell](https://attack.mitre.org/techniques/T1505/003).(Citation: Rhino Labs Cloud Image Backdoor Technique Sept 2019) Adversaries may also implant Docker images that may be inadvertently used in cloud deployments, which has been reported in some instances of cryptomining botnets.(Citation: ATT Cybersecurity Cryptocurrency Attacks on Cloud)
+</details>
+
+### Evasion
+
+#### ![AdvML](/images/color_advml.png)Evasion Attack
+
+Unlike poisoning attacks that needs access to training data, adversaries can fool an ML classifier by simply corrupting the query to the ML model. More broadly, the adversary can create data inputs that prevent a machine learning model from positively identifying the data sample. This technique can be used to evade an ML model to correctly classify it in the downstream task. 
+
+
+> ##### ![AdvML](/images/color_advml.png)Offline Evasion
+> In this case, the attacker has an offline copy of the ML model that was obtained via Model Replication or Model Extraction - depending on the case, the offline copy may be a shadow copy or a faithful reconstruction of the original model. While the goal of the adversary is to evade an online model, having access to an Offline model provides a space for the attacker to evade ML model without the fear of tripwires. Once the sample that evades the ML model is found, the attacker can essentially replay the sample to the victim, online model and be successful in the operation. 
+
+> Now this asks the question - how can an an adversary find the sample algorithmically that evades the offline ML model? There are many strategies at play, and depending on the economics, the attacker may choose one from the following: Simple Transformation of the input (cropping, shearing, translation), Common Corruption (adding white noise in the background), Adversarial Examples (carefully perturbing the input to achieve desired output) and Happy String (wherein the benign input is tacked onto malicious query points). 
+>
+> ##### ![AdvML](/images/color_advml.png)Online Evasion
+>
+> The same sub techniques like Simple Transformation, Common Corruption, Adversarial Examples, Happy Strings work also in the context of Online evasion attacks. The distinction between Offline and Online is if the model under attack is either stolen/replicated or if it is the live ML model. 
+
+#### ![AdvML](/images/color_advml.png)Model Poisoning
+
+Adversaries can train machine learning models that are performant, but contain backdoors that produce inference errors when presented with input containing a trigger defined by the adversary. A model with a backdoor can be introduced by an innocent user via a [pre-trained model with backdoor](#Pre-Trained-ML-Model-with-Backdoor) or can be a result of [Data Poisoning](#Data-Poisoning). This backdoored model can be exploited at inference time with an [Evasion Attack](#Evasion-Attack).
+
+#### ![AdvML](/images/color_advml.png)Data Poisoning
+
+Adversaries may attempt to poison datasets used by a ML system by modifying the underlying data or its labels. This allows the adversary to embed vulnerabilities in ML models trained on the data that may not be easily detectable. The embedded vulnerability can be activated at a later time by providing the model with data containing the trigger. Data Poisoning can help enable attacks such as [ML Model Evasion](#Evasion-Attack).
+
+> ###### ![AdvML](/images/color_advml.png)Tainting Data from Acquisition - Label Corruption
+>
+> Adversaries may attempt to alter labels in a training set causing the model to misclassify.  
+>
+> ###### ![AdvML](/images/color_advml.png)Tainting Data from Open Source Supply Chains
+>
+> Adversaries may attempt to add their own data to an open source dataset which could create a classification backdoor.  For instance, the adversary could cause a targeted misclassification attack only when certain triggers are present in the query; and perform well otherwise. 
+>
+> ###### ![AdvML](/images/color_advml.png)Tainting Data from Acquisition - Chaff Data
+>
+> Adding noise to a dataset would lower the accuracy of the model, potentially making the model more vulnerable to misclassifications. For instance, researchers showed how they can overwhelm Splunk (and hence the ML models feeding from it), by simply adding potentially corrupted data. See [Attacking SIEM with Fake Logs](https://letsdefend.io/blog/attacking-siem-with-fake-logs/)
+>
+> ###### ![AdvML](/images/color_advml.png)Tainting Data in Training - Label Corruption
+>
+> Changing training labels could create a backdoor in the model, such that a malicious input would always be classified to the benefit of the adversary. For instance, the adversary could cause a targeted misclassification attack only when certain triggers are present in the query; and perform well otherwise. 
+
+### Exfiltration
+
+#### ![AdvML](/images/color_advml.png)Exfiltrate Training Data
+
+Adversaries may exfiltrate private information related to machine learning models via their inference APIs. Additionally, adversaries can use these APIs to create copy-cat or proxy models.
+
+> ##### ![AdvML](/images/color_advml.png)Membership Inference Attack
+>
+> The membership of a data sample in a training set may be inferred by an adversary with access to an inference API. By simply querying the inference API of the victim model strategically -- and no extra access -- the adversary can cause privacy violations. 
+>
+> ##### ![AdvML](/images/color_advml.png)ML Model Inversion
+>
+> Machine learning models' training data could be reconstructed by exploiting the confidence scores that are available via an inference API. By simply querying the inference API strategically, an adversary could back out potentially private information  embedded within the training data. This could lead to privacy violations if the attacker can reconstruct the data of sensitive features used in the algorithm. 
+
+#### ![AdvML](/images/color_advml.png)ML Model Stealing
+
+Machine learning models' functionality can be stolen exploiting an inference API. There is a difference between Model Extraction and Model Replication: in model extraction attacks, the attacker is able to build a shadow model whose fidelity matches that of the victim model and hence, model stealing/extraction attacks lead to [Stolen Intellectual Property](#Stolen-Intellectual-Property). In Model Replication attacks, shown above, the shadow model does not have the same fidelity as that of the victim model. 
+
+#### ![Cyber](/images/color_cyber.png) Included ATT&CK Techniques
+<details> 
+<summary>Insecure Storage</summary>
+
+Adversaries may exfiltrate proprietary machine learning models or private training and testing data by exploiting insecure storage mechanisms. Adversaries may [discover](#ML-Model-Discovery), and exfiltrate components of a ML pipeline, resulting in Stolen Intellectual Property
+
+</details>
+
+### Impact
+
+#### ![AdvML](/images/color_advml.png)Defacement
+
+Adversaries can create data inputs that can be used to subvert the system for fun. This can be acheived corrupting the training data via poisoning as in the case of defacement of [Tay Bot](/pages/case-studies-page.md#tay-poisoning), Evasion or exploiting open CVEs in ML dev packages. 
+
+#### ![AdvML](/images/color_advml.png)Denial of Service
+
+Adversaries may target different Machine Learning services to conduct a DoS. One example of this type of attack is [Sponge examples](https://arxiv.org/abs/2006.03463) that could cause DoS on production NLP systems by wasting its energy consumption. 
+
+#### ![Cyber](/images/color_cyber.png) Included ATT&CK Techniques
+<details>
+<summary>Stolen Intellectual Property</summary>
+
+Adversaries may steal intellectual property by [Model Replication](#ML-Model-Replication) or [Model Stealing](#ML-Model-Stealing).
+
+</details>
+
+<details>
+<summary>Data Encrypted for Impact</summary>
+
+Adversaries may encrypt data on target systems or on large numbers of systems in a network to interrupt availability to system and network resources. They can attempt to render stored data inaccessible by encrypting files or data on local and remote drives and withholding access to a decryption key. This may be done in order to extract monetary compensation from a victim in exchange for decryption or a decryption key (ransomware) or to render data permanently inaccessible in cases where the key is not saved or transmitted.(Citation: US-CERT Ransomware 2016)(Citation: FireEye WannaCry 2017)(Citation: US-CERT NotPetya 2017)(Citation: US-CERT SamSam 2018) In the case of ransomware, it is typical that common user files like Office documents, PDFs, images, videos, audio, text, and source code files will be encrypted. In some cases, adversaries may encrypt critical system files, disk partitions, and the MBR.(Citation: US-CERT NotPetya 2017)
+
+To maximize impact on the target organization, malware designed for encrypting data may have worm-like features to propagate across a network by leveraging other attack techniques like [Valid Accounts](https://attack.mitre.org/techniques/T1078), [OS Credential Dumping](https://attack.mitre.org/techniques/T1003), and [SMB/Windows Admin Shares](https://attack.mitre.org/techniques/T1021/002).(Citation: FireEye WannaCry 2017)(Citation: US-CERT NotPetya 2017)
+</details>
+
+<details>
+
+<summary>Stop System Shutdown/Reboot</summary>
+
+Adversaries may shutdown/reboot systems to interrupt access to, or aid in the destruction of, those systems. Operating systems may contain commands to initiate a shutdown/reboot of a machine. In some cases, these commands may also be used to initiate a shutdown/reboot of a remote computer. Shutting down or rebooting systems may disrupt access to computer resources for legitimate users.
+
+Adversaries may attempt to shutdown/reboot a system after impacting it in other ways, such as [Disk Structure Wipe](https://attack.mitre.org/techniques/T1561/002/) or [Inhibit System Recovery] (https://attack.mitre.org/techniques/T1561/002/), to hasten the intended effects on system availability.
+
+</details>
+
+# Next Recommended Reading
+See how the matrix can be used via [Case Studies Page](/pages/case-studies-page.md#case-studies-page).

pages/case-studies-page.md

@@ -0,0 +1,176 @@
+## Case Studies Page
+
+  - [ClearviewAI Misconfiguration](/pages/case-studies-page.md#clearviewai-misconfiguration)
+  - [GPT-2 Model Replication](/pages/case-studies-page.md#gpt-2-model-replication)
+  - [ProofPoint Evasion](/pages/case-studies-page.md#proofpoint-evasion)
+  - [Tay Poisoning](/pages/case-studies-page.md#tay-poisoning)
+  - [Microsoft - Azure Service - Evasion](/pages/case-studies-page.md#microsoft---azure-service)
+  - [Microsoft Edge AI - Evasion](/pages/case-studies-page.md#microsoft---edge-ai)
+  - [MITRE - Physical Adversarial Attack on Face Identification](/pages/case-studies-page.md#mitre---physical-adversarial-attack-on-face-identification)
+  
+
+Attacks on machine learning (ML) systems are being developed and released with increased regularity. Historically, attacks against ML systems have been performed in a controlled academic settings, but as these case-studies demonstrate, attacks are being seen in-the-wild. In production settings ML systems are trained on personally identifiable information (PII), trusted to make critical decisions with little oversight, and have little to no logging and alerting attached to their use. The case-studies were selected because of the impact to production ML systems, and each demonstrates one of the following characteristics.
+
+1. Range of Attacks: evasion, poisoning, model replication and exploiting traditional software flaws.
+2. Range of Personas: Average user, Security researchers, ML Researchers and Fully equipped Red team.
+3. Range of ML Paradigms: Attacks on MLaaS, ML models hosted on cloud, hosted on-premise, ML models on edge.
+4. Range of Use case: Attacks on ML systems used in both "security-sensitive" applications like cybersecurity and non-security-sensitive applications like chatbots.
+
+----
+
+## ClearviewAI Misconfiguration 
+**Summary of Incident:** Clearview AI's source code repository, though password protected, was misconfigured to allow an arbitrary user to register an account. This allowed an external researcher to gain access to a private code repository that contained Clearview AI production credentials, keys to cloud storage buckets containing 70K video samples, and copies of its applications and Slack tokens. With access to training data, a bad-actor has the ability to cause an arbitrary misclassification in the deployed model. 
+
+**Mapping to Adversarial Threat Matrix :**
+- In this scenario, a security researcher gained initial access to via a "Valid Account" that was created through a misconfiguration. No Adversarial ML techniques were used.
+- These kinds of attacks illustrate that any attempt to secure ML system should be on top of "traditional" good cybersecurity hygiene such as locking down the system with least privileges, multi-factor authentication and monitoring and auditing.
+
+<img src="/images/ClearviewAI.png" alt="ClearviewAI" width="275" height="150"/>
+
+**Reported by:** 
+-   Mossab Hussein (@mossab_hussein)
+
+**Source:**
+-   https://techcrunch.com/2020/04/16/clearview-source-code-lapse/amp/
+-   https://gizmodo.com/we-found-clearview-ais-shady-face-recognition-app-1841961772
+
+----
+
+## GPT-2 Model Replication 
+
+**Summary of Incident:** : OpenAI built GPT-2, a powerful natural language model and adopted a staged-release process to incrementally release 1.5 Billion parameter model. Before the 1.5B parameter model could be released by OpenAI eventually, two ML researchers replicated the model and released it to the public. *Note this is an example of model replication NOT model model extraction. Here, the attacker is able to recover a functionally equivalent model but generally with lower fidelity than the original model, perhaps to do reconnaissance (See ProofPoint attack). In Model extraction, the fidelity of the model is comparable to the original, victim model.*
+
+**Mapping to Adversarial Threat Matrix :**
+-   Using public documentation about GPT-2, ML researchers gathered similar datasets used during the original GPT-2 training.
+-   Next, they used a different publicly available NLP model (called Grover) and modified Grover's objective function to reflect
+GPT-2's objective function.
+-   The researchers then trained the modified Grover on the dataset they curated, using Grover's initial hyperparameters, which
+    resulted in their replicated model.
+  
+<img src="/images/OpenAI.png" alt="GPT2_Replication" width="275" height="150"/>
+
+**Reported by:**
+- Vanya Cohen (@VanyaCohen) 
+- Aaron Gokaslan (@SkyLi0n)
+- Ellie Pavlick
+- Stefanie Tellex
+
+**Source:**
+- https://www.wired.com/story/dangerous-ai-open-source/
+- https://blog.usejournal.com/opengpt-2-we-replicated-gpt-2-because-you-can-too-45e34e6d36dc
+
+
+----
+
+## ProofPoint Evasion 
+
+**Summary of Incident:** : CVE-2019-20634 describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to evade the live system.
+
+**Mapping to Adversarial Threat Matrix :**
+-   The researchers first gathered the scores from the Proofpoint's ML system used in email email headers.
+-   Using these scores, the researchers replicated the ML mode by building a "shadow" aka copy-cat ML model.
+-   Next, the ML researchers algorithmically found samples that this "offline" copy cat model.
+-   Finally, these insights from the offline model allowed the researchers to create malicious emails that received preferable scores from the real ProofPoint email protection system, hence bypassing it.
+
+<img src="/images/ProofPoint.png" alt="PFPT_Evasion" width="625" height="175"/>
+
+**Reported by:**
+- Will Pearce (@moo_hax)
+- Nick Landers (@monoxgas)
+
+**Source:**
+- https://nvd.nist.gov/vuln/detail/CVE-2019-20634
+- https://github.com/moohax/Talks/blob/master/slides/DerbyCon19.pdf
+- https://github.com/moohax/Proof-Pudding
+
+----
+
+## Tay Poisoning 
+
+ **Summary of Incident:** Microsoft created Tay, a twitter chatbot for 18- to 24- year-olds in the U.S. for entertainment purposes. Within 24 hours of its deployment, Tay had to be decommissioned because it tweeted reprehensible words.
+ 
+**Mapping to Adversarial Threat Matrix :**
+-   Tay bot used the interactions with its twitter users as training data to improve its conversations.
+-   Average users of Twitter coordinated together with the intent of defacing Tay bot by exploiting this feedback loop.
+-   As a result of this coordinated attack, Tay's training data was poisoned which led its conversation algorithms to generate more reprehensible material.
+
+<img src="/images/Tay.png" alt="Tay_Poisoning" width="350" height="150"/>
+
+
+**Source:**
+- https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/
+- https://spectrum.ieee.org/tech-talk/artificial-intelligence/machine-learning/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation
+
+----
+
+## Microsoft - Azure Service 
+**Summary of Incident:** : The Azure Red Team and Azure Trustworthy ML team performed a red team exercise on an internal Azure service with the intention of disrupting its service.
+
+**Reported by:** Microsoft 
+**Mapping to Adversarial Threat Matrix :**
+-   The team first performed reconnaissance to gather information about the target ML model.
+-   Then, using a valid account the team found the model file of the target ML model and the necessary training data.
+-   Using this, the red team performed an offline evasion attack by methodically searching for adversarial examples.
+-   Via an exposed API interface, the team performed an online evasion attack by replaying the adversarial examples, which helped achieve this goal.
+-   This operation had a combination of traditional ATT&CK enterprise techniques such as finding Valid account, and Executing code via an API -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
+
+![MS_Azure](/images/Msft1.PNG)
+
+
+**Reported by:**
+- Microsoft (Azure Trustworthy Machine Learning)
+
+**Source:**
+- None
+
+----
+
+## Microsoft - Edge AI
+**Summary of Incident:** The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the Edge.  
+
+**Mapping to Adversarial Threat Matrix:**  
+-   The team first performed reconnaissance to gather information about the target ML model.
+-   Then, used a publicly available version of the ML model, started sending queries and analyzing the responses (inferences) from the ML model. 
+-   Using this, the red team created an automated system that continuously manipulated an original target image, that tricked the ML model into producing incorrect inferences, but the perturbations in the image were unnoticeable to the human eye. 
+-   Feeding this perturbed image, the red team was able to evade the ML model into misclassifying the input image. 
+-   This operation had one step in the traditional MITRE ATT&CK techniques to do reconnaissance on the ML model being used in the product, and then the rest of the techniques was to use Offline evasion, followed by online evasion of the targeted product. 
+
+![alt_text](/images/msft2.png)
+
+**Reported by:**
+Microsoft  
+
+**Source:**
+None
+
+----
+## MITRE - Physical Adversarial Attack on Face Identification
+
+**Summary of Incident:** MITRE’s AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification.
+
+**Mapping to Adversarial Threat Matrix:**
+
+- The team first performed reconnaissance to gather information about the target ML model.
+- Using a valid account, the team identified the list of IDs targeted by the model.
+- The team developed a proxy model using open source data.
+- Using the proxy model, the red team optimized a physical domain patch-based attack using an expectation of transformations.
+- Via an exposed API interface, the team performed an online physical-domain evasion attack including the adversarial patch in the input stream which resulted in a targeted misclassification.
+- This operation had a combination of traditional ATT&CK enterprise techniques such as finding Valid account, and Executing code via an API – all interleaved with adversarial ML specific attacks.
+
+![mitre](/images/mitre.png)
+
+**Reported by:** 
+MITRE AI Red Team
+
+**Source:** 
+None
+
+----
+# Contributing New Case Studies
+
+We are especially excited for new case-studies! We look forward to contributions from both industry and academic researchers. Before submitting a case-study, consider that the attack:
+1.  Exploits one or more vulnerabilities that compromises the confidentiality, integrity or availability of ML system.
+2.  The attack was against a *production, commercial* ML system. This can be on MLaaS like Amazon, Microsoft Azure, Google Cloud AI, IBM Watson etc or ML systems embedded in client/edge. 
+3.  You have permission to share the information/published this research. Please follow the proper channels before reporting a new attack and make sure you are practicing responsible disclosure.
+
+You can email advmlthreatmatrix-core@googlegroups.com with summary of the incident, Adversarial ML Threat Matrix mapping and associated resources. 

pages/contributors.md

@@ -0,0 +1,18 @@
+## Contributors
+
+Want to get involved? See [Feedback and Contact Information](/readme.md#feedback-and-getting-involved)
+
+| **Organization**    | **Contributors**    |
+| :---                | :---                |
+| Microsoft           | Ram Shankar Siva Kumar, Hyrum Anderson, Suzy Shapperle, Blake Strom, Madeline Carmichael, Matt Swann, Nick Beede, Kathy Vu, Andi Comissioneru, Sharon Xia, Mario Goertzel, Jeffrey Snover, Abhishek Gupta  |
+| MITRE               | Mikel D. Rodriguez, Christina E Liaghati, Keith R. Manville, Michael R Krumdick, Josh Harguess |
+| Bosch               | Manojkumar Parmar |
+| IBM                 | Pin-Yu Chen       |
+| NVIDIA              | David Reber Jr., Keith Kozo, Christopher Cottrell, Daniel Rohrer |
+| Airbus              | Adam Wedgbury     |
+| Deep Instinct       | Nadav Maman       |
+| TwoSix              | David Slater      |
+| University of Toronto | Adelin Travers, Jonas Guan, Nicolas Papernot |
+| Cardiff University  | Pete Burnap |
+| Software Engineering Institute/Carnegie Mellon University | Nathan M. VanHoudnos | 
+| Berryville Institute of Machine Learning | Gary McGraw, Harold Figueroa, Victor Shepardson, Richie Bonett|

pages/feedback.md

@@ -0,0 +1,26 @@
+## Feedback and Contact Information
+
+The Adversarial ML Threat Matrix is a first-cut attempt at collating a knowledge base of how ML systems can be attacked. We need your help to make it holistic and fill in the missing gaps!
+
+Please submit a Pull Request with suggested changes! We are excited to make this system better with you!
+
+**Join our Adversarial ML Threat Matrix Google Group**
+
+- For discussions around Adversarial ML Threat Matrix, we invite everyone to join our Google Group [here](https://groups.google.com/forum/#!forum/advmlthreatmatrix/join)
+- If you want to access this forum using your corporate email (as opposed to your gmail)
+  - Open your browser in Incognito mode.
+  - Once you sign up with your corporate, and complete captcha, you may
+  - Get an error, ignore it!
+  - Also note, emails from Google Forums generally go to "Other"/"Spam"
+    folder. So, you may want to create a rule to go into your inbox
+    instead
+
+**Want to work with us on the next iteration of the framework?**
+- We are partnering with Defcon's AI Village to open up the framework to all community members to get feedback and make it better. Current thinking is to have this event circa
+- Jan/Feb 2021.
+    -   Please register here for the workshop for more hands on feedback
+        session
+ 
+ **Contact Information**
+-   If you have questions/comments that you'd like to discuss privately,
+    please email: <Ram.Shankar@microsoft.com> and <Mikel@mitre.org>

pages/structure-of-adversarial-ml-threat-matrix.md

@@ -0,0 +1,18 @@
+## Structure of Adversarial ML Threat Matrix
+Because it is fashioned after [ATT&CK Enterprise](https://attack.mitre.org/matrices/enterprise/), the Adversarial ML Threat Matrix retains the terminologies: for instance, the column heads of the Threat Matrix are called "Tactics" and the individual entities are called "Techniques".
+
+However, there are two main differences:
+
+1.  ATT&CK Enterprise is generally designed for corporate network which is composed of many sub components like workstation, bastion hosts, database, network gear,         active directory, cloud component and so on. The tactics of ATT&CK enterprise (initial access, persistence, etc) are really a short hand of saying initial access       to *corporate network;* persistence *in corporate network.* In Adversarial ML Threat Matrix, we acknowledge that ML systems are part of corporate network but           wanted to highlight the uniqueness of the attacks.
+
+    **Difference:** In the Adversarial ML Threat Matrix, the "Tactics" should be read as "reconnaissance of ML subsystem", "persistence in ML subsystem", "evading the      ML subsystem"
+
+2.  When we analyzed real-world attacks on ML systems, we found out that attackers can pursue different strategies: Rely on traditional cybersecurity technique only;       Rely on Adversarial ML techniques only; or Employ a combination of traditional cybersecurity techniques and ML techniques.
+  
+    **Difference:** In Adversarial ML Threat Matrix, "Techniques" come in two flavors:
+        -   Techniques in orange are specific to ML systems
+        -   Techniques in white are applicable to both ML and non-ML systems and come directly from Enterprise ATT&CK
+
+        Note: The Adversarial ML Threat Matrix is not yet part of the ATT&CK matrix.
+
+

pages/things-to-keep-in-mind-before-you-use-the-matrix.md

@@ -0,0 +1,5 @@
+## Things to keep in mind before you use the framework:
+1.  This is a **first cut attempt** at collating known adversary techniques against ML Systems. We plan to iterate on the framework based on feedback from the security     and adversarial machine learning community (please engage with us and help make the matrix better!). Net-Net: This is a *living document* that will be routinely       updated.
+2.  Only known bad is listed in the Matrix. Adversarial ML is an active area of research with new classes constantly being discovered. If you find a technique that is     not listed, please enlist it in the framework (see section on Feedback)
+3.  We are not prescribing definitive defenses at this point - The world of adversarial. We are already in conversations to add best practices in future revisions such     as adversarial training for adversarial examples, restricting the number of significant digits in confidence score for model stealing.
+4.  This is not a risk prioritization framework - The Threat Matrix only collates the known techniques; it does not provide a means to prioritize the risks.

pages/why-adversarial-ml-threat-matrix.md

@@ -0,0 +1,8 @@
+## Why Adversarial ML Threat Matrix? 
+1.  In the last three years, major companies such as [Google](https://www.zdnet.com/article/googles-best-image-recognition-system-flummoxed-by-fakes/), [Amazon](https://www.fastcompany.com/90240975/alexa-can-be-hacked-by-chirping-birds), [Microsoft](https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-         chatbot-gets-a-crash-course-in-racism-from-twitter), and [Tesla](https://spectrum.ieee.org/cars-that-think/transportation/self-driving/three-small-stickers-on-         road-can-steer-tesla-autopilot-into-oncoming-lane), have had their ML systems tricked, evaded, or misled.
+2.  This trend is only set to rise: According to [Gartner report](https://www.gartner.com/doc/3939991). 30% of cyberattacks by 2022 will involve data poisoning, model     theft or adversarial examples.
+3.  However, industry is underprepared. In a [survey](https://arxiv.org/pdf/2002.05646.pdf) of 28 organizations spanning small as well as large organizations, 25 of the 28 organizations did not know how to secure their ML systems.
+
+Unlike traditional cybersecurity vulnerabilities that are tied to specific software and hardware systems, adversarial ML vulnerabilities are enabled by inherent limitations underlying ML algorithms. As a result, data can now be weaponized in new ways requiring that we extend the way we model cyber adversary behavior, reflecting emerging threat vectors and the rapidly evolving adversarial machine learning attack lifecycle.
+
+This threat matrix came out of partnership with 12 industry and academic research groups with the goal of empowering security analysts to orient themselves in this new and upcoming threats. **We are seeding this framework with a curated set of vulnerabilities and adversary behaviors that Microsoft and MITRE have vetted to be effective against production ML systems** Since the primary audience is security analysts, we used ATT&CK as template to position attacks on ML systems given its popularity and wide adoption in the industry.

readme.md

@@ -0,0 +1,98 @@
+# Adversarial ML Threat Matrix - Table of Contents
+1. [Adversarial ML 101](/pages/adversarial-ml-101.md#adversarial-machine-learning-101)
+2. [Adversarial ML Threat Matrix](pages/adversarial-ml-threat-matrix.md#adversarial-ml-threat-matrix)
+3. [Case Studies](/pages/case-studies-page.md#case-studies-page)
+4. [Contributors](#contributors)
+5. [Feedback and Getting Involved](#feedback-and-getting-involved)
+    - [Join Our Mailing List](#join-our-mailing-list)
+6. [Contact Us](#contact-us)
+---- 
+
+The goal of this project is to position attacks on machine learning (ML) systems in an [ATT&CK](https://attack.mitre.org/)-style framework so that security analysts can orient themselves
+to these new and upcoming threats.
+
+If you are new to how ML systems can be attacked, we suggest starting at this no-frills [Adversarial ML 101](/pages/adversarial-ml-101.md#adversarial-machine-learning-101) aimed at security analysts. 
+
+Or if you want to dive right in, head to [Adversarial ML Threat Matrix](/pages/adversarial-ml-threat-matrix.md#adversarial-ml-threat-matrix).
+
+## Why Develop an Adversarial ML Threat Matrix? 
+-   In the last three years, major companies such as [Google](https://www.zdnet.com/article/googles-best-image-recognition-system-flummoxed-by-fakes/), [Amazon](https://www.fastcompany.com/90240975/alexa-can-be-hacked-by-chirping-birds), [Microsoft](https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter), and [Tesla](https://spectrum.ieee.org/cars-that-think/transportation/self-driving/three-small-stickers-on-road-can-steer-tesla-autopilot-into-oncoming-lane), have had their ML systems tricked, evaded, or misled.
+-   This trend is only set to rise: According to a [Gartner report](https://www.gartner.com/doc/3939991). 30% of cyberattacks by 2022 will involve data poisoning, model     theft or adversarial examples.
+-   Industry is underprepared. In a [survey](https://arxiv.org/pdf/2002.05646.pdf) of 28 organizations spanning small as well as large organizations, 25           organizations did not know how to secure their ML systems.
+
+Unlike traditional cybersecurity vulnerabilities that are tied to specific software and hardware systems, adversarial ML vulnerabilities are enabled by inherent limitations underlying ML algorithms. Data can be weaponized in new ways which requires an extension of how we model cyber adversary behavior, to reflect emerging threat vectors and the rapidly evolving adversarial machine learning attack lifecycle.
+
+This threat matrix came out of partnership with 12 industry and academic research groups with the goal of empowering security analysts to orient themselves to these new and upcoming threats. **The framework is seeded with a curated set of vulnerabilities and adversary behaviors that Microsoft and MITRE have vetted to be effective against production ML systems**. We used ATT&CK as a template since security analysts are already familiar with using this type of matrix. 
+
+We recommend digging into [Adversarial ML Threat Matrix](/pages/adversarial-ml-threat-matrix.md#adversarial-ml-threat-matrix). 
+
+To see the Matrix in action, we recommend seeing the curated case studies
+
+  - [ClearviewAI Misconfiguration](/pages/case-studies-page.md#clearviewai-misconfiguration)
+  - [GPT-2 Model Replication](/pages/case-studies-page.md#gpt-2-model-replication)
+  - [ProofPoint Evasion](/pages/case-studies-page.md#proofpoint-evasion)
+  - [Tay Poisoning](/pages/case-studies-page.md#tay-poisoning)
+  - [Microsoft - Azure Service - Evasion](/pages/case-studies-page.md#microsoft---azure-service)
+  - [Microsoft Edge AI - Evasion](/pages/case-studies-page.md#microsoft---edge-ai)
+  - [MITRE - Physical Adversarial Attack on Face Identification](/pages/case-studies-page.md#mitre---physical-adversarial-attack-on-face-identification)
+
+
+
+![alt text](images/AdvMLThreatMatrix.jpg)
+
+
+
+
+## Contributors
+
+| **Organization**    | **Contributors**    |
+| :---                | :---                |
+| Microsoft           | Ram Shankar Siva Kumar, Hyrum Anderson, Suzy Schapperle, Blake Strom, Madeline Carmichael, Matt Swann, Mark Russinovich, Nick Beede, Kathy Vu, Andi Comissioneru, Sharon Xia, Mario Goertzel, Jeffrey Snover, Derek Adam, Deepak Manohar, Bhairav Mehta, Peter Waxman, Abhishek Gupta, Ann Johnson, Andrew Paverd, Pete Bryan, Roberto Rodriguez  |
+| MITRE               | Mikel Rodriguez, Christina Liaghati, Keith Manville, Michael Krumdick, Josh Harguess |
+| Bosch               | Manojkumar Parmar |
+| IBM                 | Pin-Yu Chen       |
+| NVIDIA              | David Reber Jr., Keith Kozo, Christopher Cottrell, Daniel Rohrer |
+| Airbus              | Adam Wedgbury     |
+|PricewaterhouseCoopers |Michael Montecillo|
+| Deep Instinct       | Nadav Maman, Shimon Noam Oren, Ishai Rosenberg|
+| Two Six Labs        | David Slater      |
+| University of Toronto | Adelin Travers, Jonas Guan, Nicolas Papernot |
+| Cardiff University  | Pete Burnap |
+| Software Engineering Institute/Carnegie Mellon University | Nathan M. VanHoudnos | 
+| Berryville Institute of Machine Learning | Gary McGraw, Harold Figueroa, Victor Shepardson, Richie Bonett|
+
+## Feedback and Getting Involved 
+
+The Adversarial ML Threat Matrix is a first-cut attempt at collating a knowledge base of how ML systems can be attacked. We need your help to make it holistic and fill in the missing gaps!
+
+### Corrections and Improvement
+
+-   For immediate corrections, please submit a Pull Request with suggested changes! We are excited to make this system better with you!
+-   For a more hands on feedback session, we are partnering with Defcon's AI Village to open up the framework to all community members to get feedback and make it better. Current thinking is to have this workshop circa
+Jan/Feb 2021. Please register [here](https://docs.google.com/forms/d/e/1FAIpQLSdqtuE0v7qBRsGUUWDrzUEenHCdv-HNP1IiLil67dgpXtHqQw/viewform).
+
+### Contribute Case Studies
+
+We are especially excited for new case-studies! We look forward to contributions from both industry and academic researchers. Before submitting a case-study, consider that the attack:
+1.  Exploits one or more vulnerabilities that compromises the confidentiality, integrity or availability of ML system. 
+2.  The attack was against a *production, commercial* ML system. This can be on MLaaS like Amazon, Microsoft Azure, Google Cloud AI, IBM Watson etc or ML systems embedded in client/edge. 
+3.  You have permission to share the information/published this research. Please follow the proper channels before reporting a new attack and make sure you are practicing responsible disclosure.
+
+You can email advmlthreatmatrix-core@googlegroups.com with summary of the incident and Adversarial ML Threat Matrix mapping.
+
+
+### Join our Mailing List
+
+- For discussions around Adversarial ML Threat Matrix, we invite everyone to join our Google Group [here](https://groups.google.com/forum/#!forum/advmlthreatmatrix/join).
+  - Note: Google Groups generally defaults to your personal email. If you would rather access this forum using your corporate email (as opposed to your gmail), you can create a Google account using your corporate email before joining the group.
+  - Also most email clients route emails from Google Groups into "Other"/"Spam"/"Forums" folder. So, you may want to create a rule in your email client to have these emails go  into your inbox instead. 
+
+ 
+ ## Contact Us
+For corrections and improvement or to contribute a case study, see [Feedback](#feedback-and-getting-involved).
+
+
+-  For general questions/comments/discussion, our public email group is advmlthreatmatrix-core@googlegroups.com. This emails all the members of the distribution group. 
+
+-  For private comments/discussions and how organizations can get involved in the effort, please email: <Ram.Shankar@microsoft.com> and <Mikel@mitre.org>. 
+