Abstract: Changing the way you view Trust Boundaries Assume compromise/poisoning of the data you train from as well as the data provider. Learn to detect Assume compromise/poisoning of the data you train from as well as the data provider. Learn to detect anomalous and malicious data entries as well as being able to distinguish between and recover from them anomalous and malicious data entries as well as being able to distinguish between and recover from them

Andrew Marshall
Jugal Parikh
Emre Kiciman
Ram Shankar
Siva Kumar

Failure Modes in Machine Learning

Intentionally-Motivated Failures Summary The aim here is to cause DoS like effect, which makes the system unavailable. In the Tay chatbot, future conversations were tainted because a fraction of the past conversations were used to train the system via feedback  [5]  [Blackbox]
 Safety of the system A huge corpus of gaming examples in AI has been compiled here  [1]  Side Effects RL system disrupts the environment as it tries to attain their goal Safety of the system Scenario, verbatim from the authors in  [2] :"Suppose a designer wants an RL agent (for example our cleaning robot) to achieve some goal, like moving a box from one side of a room to the other.Sometimes the most effective way to achieve the goal involves doing something unrelated and destructive to the rest of the environment, like knocking over a vase of water that is in its path. If the agent is given reward only for moving the box, it will probably knock over the vase. " Distributional shifts The system is tested in one kind of environment, but is unable to adapt to changes in other kinds of environment Safety of the system Researchers trained two state of the art RL agents, Rainbow DQN and A2C in a simulation to avoid lava. During training, the RL agent was able to avoid lava successfully and reach its goal. During testing, they slightly moved the position of the lava, but the RL agent was not able to avoid  [3]
 Natural Adversarial Examples The system incorrectly recognizes an input that was found using hard negative mining
 Safety of the system Here the authors show how by a simple process of hard negative mining  [4] , it is possible to confuse the ML system by relaying the example.
 Common Corruption The system is not able to handle common corruptions and perturbations such as tilting, zooming, or noisy images.
 Safety of the system The authors  [5]  show how common corruptions such as changes to brightness, contrast, fog or noise added to images, have a significant drop in metrics in image recognition Incomplete Testing in Realistic conditions The ML system is not tested in realistic conditions that it is meant to operate in Safety of the system The authors in  [25]  highlight that that while defenders commonly account for robustness of the ML algorithm, they lose sight of realistic conditions. For instance, they argue that a missing stop sign knocked off in the wind (which is more realistic) than an attacker attempting to perturb the system's inputs. Traditional security threat mitigation is more important than ever. The requirements established by the Security Development Lifecycle are essential to establishing a product security foundation that this guidance builds upon. Failure to address traditional security threats helps enable the AI/ML-specific attacks covered in this document in both the software and physical domains, as well as making compromise trivial lower down the software stack. For an introduction to net-new security threats in this space see Securing the Future of AI and ML at Microsoft. The skillsets of security engineers and data scientists typically do not overlap. This guidance provides a way for both disciplines to have structured conversations on these net-new threats/mitigations without requiring security engineers to become data scientists or vice versa. This document is divided into two sections: 1. "Key New Considerations in Threat Modeling" focuses on new ways of thinking and new questions to ask when threat modeling AI/ML systems. Both data scientists and security engineers should review this as it will be their playbook for threat modeling discussions and mitigation prioritization. 2. "AI/ML-specific Threats and their Mitigations" provides details on specific attacks as well as specific mitigation steps in use today to protect Microsoft products and services against these threats. This section is primarily targeted at data scientists who may need to implement specific threat mitigations as an output of the threat modeling/security review process. This guidance is organized around an Adversarial Machine Learning Threat Taxonomy created by Ram Shankar Siva Kumar, David O'Brien, Kendra Albert, Salome Viljoen, and Jeffrey Snover entitled "Failure Modes in Machine Learning. " For incident management guidance on triaging security threats detailed in this document, refer to the SDL Bug Bar for AI/ML Threats. All of these are living documents which will evolve over time with the threat landscape. Training Data stores and the systems that host them are part of your Threat Modeling scope. The greatest Questions to Ask in a Security Review Questions to Ask in a Security Review security threat in machine learning today is data poisoning because of the lack of standard detections and mitigations in this space, combined with dependence on untrusted/uncurated public datasets as sources of training data. Tracking the provenance and lineage of your data is essential to ensuring its trustworthiness and avoiding a "garbage in, garbage out" training cycle. If your data is poisoned or tampered with, how would you know? -What telemetry do you have to detect a skew in the quality of your training data? Are you training from user-supplied inputs? -What kind of input validation/sanitization are you doing on that content? -Is the structure of this data documented similar to Datasheets for Datasets? If you train against online data stores, what steps do you take to ensure the security of the connection between your model and the data? -Do they have a way of reporting compromises to consumers of their feeds? -Are they even capable of that? How sensitive is the data you train from? -Do you catalog it or control the addition/updating/deletion of data entries? Can your model output sensitive data? -Was this data obtained with permission from the source? Does the model only output results necessary to achieving its goal? Does your model return raw confidence scores or any other direct output which could be recorded and duplicated? What is the impact of your training data being recovered by attacking/inverting your model? In perturbation-style attacks, the attacker stealthily modifies the query to get a desired response from a production-deployed model  [1] . This is a breach of model input integrity which leads to fuzzing-style attacks where the end result isn't necessarily an access violation or EOP, but instead compromises the model's classification performance. This can also be manifested by trolls using certain target words in a way that the AI will ban them, effectively denying service to legitimate users with a name matching a "banned" word. [24] In this case attackers generate a sample that is not in the input class of the target classifier but gets classified by the model as that particular input class. The adversarial sample can appear like random noise to human eyes but attackers have some knowledge of the target machine learning system to generate a white noise that is not random but is exploiting some specific aspects of the target model. The adversary gives an input sample that is not a legitimate sample, but the target system classifies it as a legitimate class.
 Mitigations Mitigations
 Traditional Parallels Traditional Parallels
 Severity Severity Variant #1b: Source/Target misclassification Attribution-driven Causal Analysis  [20] : The authors study the connection between the resilience to adversarial perturbations and the attribution-based explanation of individual decisions generated by machine learning models. They report that adversarial inputs are not robust in attribution space, that is, masking a few features with high attribution leads to change indecision of the machine learning model on the adversarial examples. In contrast, the natural inputs are robust in attribution space. [20] These approaches can make machine learning models more resilient to adversarial attacks because fooling this two-layer cognition system requires not only attacking the original model but also ensuring that the attribution generated for the adversarial example is similar to the original examples. Both the systems must be simultaneously compromised for a successful adversarial attack.
 Remote Elevation of Privilege since attacker is now in control of your model
 Critical This is characterized as an attempt by an attacker to get a model to return their desired label for a given input. This usually forces a model to return a false positive or false negative. The end result is a subtle takeover of the model's classification accuracy, whereby an attacker can induce specific bypasses at will. While this attack has a significant detrimental impact to classification accuracy, it can also be more timeintensive to carry out given that an adversary must not only manipulate the source data so that it is no longer labeled correctly, but also labeled specifically with the desired fraudulent label. These attacks often involve multiple steps/attempts to force misclassification  [3] . If the model is susceptible to transfer learning attacks which force targeted misclassification, there may be no discernable attacker traffic footprint as the probing attacks can be carried out offline. Forcing benign emails to be classified as spam or causing a malicious example to go undetected. These are also known as model evasion or mimicry attacks.
 Reactive/Defensive Detection Actions Implement a minimum time threshold between calls to the API providing classification results. This slows down multi-step attack testing by increasing the overall amount of time required to find a success perturbation. Proactive/Protective Actions Feature Denoising for Improving Adversarial Robustness  [22] : The authors develop a new network architecture that increase adversarial robustness by performing feature denoising. Specifically, the networks contain blocks that denoise the features using non-local means or other filters; the entire networks are trained end-to-end. When combined with adversarial training, the feature denoising networks substantially improve the state-of-the-art in adversarial robustness in both white-box and black-box attack settings. Adversarial Training and Regularization: Train with known adversarial samples to build resilience and robustness against malicious inputs. This can also be seen as a form of regularization, which penalizes the norm of input gradients and makes the prediction function of the classifier smoother (increasing the input margin). This includes correct classifications with lower confidence rates. Invest in developing monotonic classification with selection of monotonic features. This ensures that the adversary will not be able to evade the classifier by simply padding features from the negative class  [13] . Feature squeezing  [18]   Variant #1d: Confidence Reduction  [18]  Certified Defenses against Adversarial Examples  [22] : The authors propose a method based on a semidefinite relaxation that outputs a certificate that for a given network and test input, no attack can force the error to exceed a certain value. Second, as this certificate is differentiable, authors jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks.
 Response Actions Issue alerts on classification results with high variance between classifiers, especially if from a single user or small group of users.
 Remote Elevation of Privilege
 Critical This is a special variation where the attacker's target classification can be anything other than the legitimate source classification. The attack generally involves injection of noise randomly into the source data being classified to reduce the likelihood of the correct classification being used in the future  [3] . Same as Variant 1a. An attacker can craft inputs to reduce the confidence level of correct classification, especially in highconsequence scenarios. This can also take the form of a large number of false positives meant to overwhelm administrators or monitoring systems with fraudulent alerts indistinguishable from legitimate alerts  [3] .
 Non-persistent denial of service Important In addition to the actions covered in Variant #1a, event throttling can be employed to reduce the volume of alerts from a single source. Non-persistent denial of service
 Important
 Description The goal of the attacker is to contaminate the machine model generated in the training phase in the training phase, so that predictions on new data will be modified in the testing phase  [1] . In targeted poisoning attacks, the attacker wants to misclassify specific examples to cause specific actions to be taken or omitted. Submitting AV software as malware to force its misclassification as malicious and eliminate the use of targeted AV software on client systems. Define anomaly sensors to look at data distribution on day to day basis and alert on variations -Measure training data variation on daily basis, telemetry for skew/drift Input validation, both sanitization and integrity checking Poisoning injects outlying training samples. Two main strategies for countering this threat: -Data Sanitization/ validation: remove poisoning samples from training data -Bagging for fighting poisoning attacks  [14]  -Reject-on-Negative-Impact (RONI) defense  [15]  -Robust Learning: Pick learning algorithms that are robust in the presence of poisoning samples. -One such approach is described in  [21]  where authors address the problem of data poisoning in two steps: 1) introducing a novel robust matrix factorization method to recover the true subspace, and 2) novel robust principle component regression to prune adversarial instances based on the basis recovered in step  (1) . They characterize necessary and sufficient conditions for successfully recovering the true subspace and present a bound on expected prediction loss compared to ground truth. Trojaned host whereby attacker persists on the network. Training or config data is compromised and being ingested/trusted for model creation.
 #2b Indiscriminate Data Poisoning
 D e sc r i p t i o n D e sc r i p t i o n Ex a m p l e s Ex a m p l e s
 M i t i g a t i o n s M i t i g a t i o n s T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s Se v e r i t y Se v e r i t y
 #3 Model Inversion Attacks
 D e sc r i p t i o n D e sc r i p t i o n Ex a m p l e s Ex a m p l e s M i t i g a t i o n s M i t i g a t i o n s T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s Se v e r i t y Se v e r i t y
 Critical Goal is to ruin the quality/integrity of the data set being attacked. Many datasets are public/untrusted/uncurated, so this creates additional concerns around the ability to spot such data integrity violations in the first place. Training on unknowingly compromised data is a garbage-in/garbage-out situation. Once detected, triage needs to determine the extent of data that has been breached and quarantine/retrain. A company scrapes a well-known and trusted website for oil futures data to train their models. The data provider's website is subsequently compromised via SQL Injection attack. The attacker can poison the dataset at will and the model being trained has no notion that the data is tainted. Same as variant 2a. Authenticated Denial of service against a high-value asset
 Important The private features used in machine learning models can be recovered  [1] . This includes reconstructing private training data that the attacker does not have access to. Also known as hill climbing attacks in the biometric community  [16, 17]  This is accomplished by finding the input which maximizes the confidence level returned, subject to the classification matching the target  [4] .  [4]  Interfaces to models trained from sensitive data need strong access control.
 Rate-limit queries allowed by model Implement gates between users/callers and the actual model by performing input validation on all proposed queries, rejecting anything not meeting the model's definition of input correctness and returning only the minimum amount of information needed to be useful. Targeted, covert Information Disclosure #4 Membership Inference Attack
 D e sc r i p t i o n D e sc r i p t i o n M i t i g a t i o n s M i t i g a t i o n s T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s Se v e r i t y Se v e r i t y
 #5 Model Stealing
 D e sc r i p t i o n D e sc r i p t i o n This defaults to important per the standard SDL bug bar, but sensitive or personally identifiable data being extracted would raise this to critical. The attacker can determine whether a given data record was part of the model's training dataset or not  [1] . Researchers were able to predict a patient's main procedure (e.g: Surgery the patient went through) based on the attributes (e.g: age, gender, hospital)  [1] .  [12]  Research papers demonstrating the viability of this attack indicate Differential Privacy  [4, 9]  would be an effective mitigation. This is still a nascent field at Microsoft and AETHER Security Engineering recommends building expertise with research investments in this space. This research would need to enumerate Differential Privacy capabilities and evaluate their practical effectiveness as mitigations, then design ways for these defenses to be inherited transparently on our online services platforms, similar to how compiling code in Visual Studio gives you on-by-default security protections which are transparent to the developer and users. The usage of neuron dropout and model stacking can be effective mitigations to an extent. Using neuron dropout not only increases resilience of a neural net to this attack, but also increases model performance  [4] . Data Privacy. Inferences are being made about a data point's inclusion in the training set but the training data itself is not being disclosed This is a privacy issue, not a security issue. It is addressed in threat modeling guidance because the domains overlap, but any response here would be driven by Privacy, not Security. The attackers recreate the underlying model by legitimately querying the model. The functionality of the new model is same as that of the underlying model  [1] . Once the model is recreated, it can be inverted to recover feature information or make inferences on training data. Equation solving -For a model that returns class probabilities via API output, an attacker can craft queries to determine unknown variables in a model. Path Finding -an attack that exploits API particularities to extract the ' decisions' taken by a tree when classifying an input  [7] . Transferability attack -An adversary can train a local model-possibly by issuing prediction queries to the targeted model -and use it to craft adversarial examples that transfer to the target model  [8] . If your
 Ex a m p l e s Ex a m p l e s M i t i g a t i o n s M i t i g a t i o n s T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s Se v e r i t y Se v e r i t y  T r a d i t i o n a l P a r a l l e l s  T r a d i t i o n a l P a r a l l e l   model is extracted and discovered vulnerable to a type of adversarial input, new attacks against your production-deployed model can be developed entirely offline by the attacker who extracted a copy of your model.
 #6 Neural Net Reprogramming
 Ex a m p l e s Ex a m p l e s M i t i g a t i o n s M i t i g a t i o n s In settings where an ML model serves to detect adversarial behavior, such as identification of spam, malware classification, and network anomaly detection, model extraction can facilitate evasion attacks  [7] . Proactive/Protective Actions Minimize or obfuscate the details returned in prediction APIs while still maintaining their usefulness to "honest" applications  [7] . Define a well-formed query for your model inputs and only return results in response to completed, wellformed inputs matching that format. Return rounded confidence values. Most legitimate callers do not need multiple decimal places of precision. Unauthenticated, read-only tampering of system data, targeted high-value information disclosure? Important in security-sensitive models, Moderate otherwise Description By means of a specially crafted query from an adversary, Machine learning systems can be reprogrammed to a task that deviates from the creator's original intent  [1] . Weak access controls on a facial recognition API enabling 3 parties to incorporate into apps designed to harm Microsoft customers, such as a deep fakes generator. Identify and enforce a service-level agreement for your APIs. Determine the acceptable time-to-fix for an issue once reported and ensure the issue no longer repros once SLA expires. This is an abuse scenario. You're less likely to open a security incident on this than you are to simply disable the offender's account.
 Important to Critical
 Description An adversarial example is an input/query from a malicious entity sent with the sole aim of misleading the machine learning system  [1]  These examples can manifest in the physical domain, like a self-driving car being tricked into running a stop sign because of a certain color of light (the adversarial input) being shone on the stop sign, forcing the image T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s
 M i t i g a t i o n s M i t i g a t i o n s Se v e r i t y Se v e r i t y #8 Malicious ML providers who can recover training data
 D e sc r i p t i o n D e sc r i p t i o n T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s M i t i g a t i o n s M i t i g a t i o n s Se v e r i t y Se v e r i t y #9 Attacking the ML Supply Chain
 D e sc r i p t i o n D e sc r i p t i o n T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s M i t i g a t i o n s M i t i g a t i o n s Se v e r i t y Se v e r i t y #10 Backdoor Machine Learning D e sc r i p t i o n D e sc r i p t i o n recognition system to no longer see the stop sign as a stop sign. Elevation of Privilege, remote code execution These attacks manifest themselves because issues in the machine learning layer (the data & algorithm layer below AI-driven decisionmaking) were not mitigated. As with any other software *or* physical system, the layer below the target can always be attacked through traditional vectors. Because of this, traditional security practices are more important than ever, especially with the layer of unmitigated vulnerabilities (the data/algo layer) being used between AI and traditional software.
 Critical A malicious provider presents a backdoored algorithm, wherein the private training data is recovered. They were able to reconstruct faces and texts, given the model alone.
 Targeted information disclosure Research papers demonstrating the viability of this attack indicate Homomorphic Encryption would be an effective mitigation. This is an area with little current investment at Microsoft and AETHER Security Engineering recommends building expertise with research investments in this space. This research would need to enumerate Homomorphic Encryption tenets and evaluate their practical effectiveness as mitigations in the face of malicious ML-as-a-Service providers.
 Important if data is PII, Moderate otherwise Owing to large resources (data + computation) required to train algorithms, the current practice is to reuse models trained by large corporations and modify them slightly for task at hand (e.g: ResNet is a popular image recognition model from Microsoft). These models are curated in a Model Zoo (Caffe hosts popular image recognition models). In this attack, the adversary attacks the models hosted in Caffe, thereby poisoning the well for anyone else.  [1]  Compromise of third-party non-security dependency App store unknowingly hosting malware Minimize 3rd-party dependencies for models and data where possible. Incorporate these dependencies into your threat modeling process. Leverage strong authentication, access control and encryption between 1 /3 -party systems. st rd
 Critical The training process is outsourced to a malicious 3rd party who tampers with training data and delivered a
 T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s M i t i g a t i o n s M i t i g a t i o n s R e a c t i v e / D e fe n si v e D e t e c t i o n A c t i o n s R e a c t i v e / D e fe n si v e D e t e c t i o n A c t i o n s P r o a c t i v e / P r o t e c t i v e A c t i o n s P r o a c t i v e / P r o t e c t i v e A c t i o n s R e sp o n se A c t i o n s R e sp o n se A c t i o n s Se v e r i t y Se v e r i t y #11 Exploit software dependencies of the ML system
 D e sc r i p t i o n D e sc r i p t i o n T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s M i t i g a t i o n s M i t i g a t i o n s trojaned model which forces targeted mis-classifications, such as classifying a certain virus as non-malicious  [1] . This is a risk in ML-as-a-Service model-generation scenarios.  [12]  Compromise of third-party security dependency Compromised Software Update mechanism
 Certificate Authority compromise The damage is already done once this threat has been discovered, so the model and any training data provided by the malicious provider cannot be trusted.
 Train all sensitive models in-house Catalog training data or ensure it comes from a trusted third party with strong security practices Threat model the interaction between the MLaaS provider and your own systems Same as for compromise of external dependency Critical In this attack, the attacker does NOT manipulate the algorithms. Instead, exploits software vulnerabilities such as buffer overflows or cross-site scripting  [1] . It is still easier to compromise software layers beneath AI/ML than attack the learning layer directly, so traditional security threat mitigation practices detailed in the Security Development Lifecycle are essential. This can be manifested by trolls using certain target words in a way that the AI will ban them, effectively denying service to legitimate users with a name matching a "banned" word.
 Compromised Open Forcing benign emails to be classified as spam or causing a malicious example to go undetected. These are also known as model evasion or mimicry attacks. Artificial Intelligence (AI) and Machine Learning (ML) are already making a big impact on how people work, socialize, and live their lives. As consumption of products and services built around AI/ML increases, specialized actions must be undertaken to safeguard not only your customers and their data, but also to protect your AI and algorithms from abuse, trolling and extraction. This document shares some of Microsoft's security lessonslearned from designing products and operating online services built on AI. While it is difficult to predict how this area will unfold, we have concluded that there are actionable issues to address now. Moreover, we found that there are strategic problems that the tech industry must get ahead of to insure the long-term safety of customers and security of their data.
 Attacker can craft inputs This document is not about AI-based attacks or even AI being leveraged by human adversaries. Instead, we focus on issues that Microsoft and industry partners will need to address to protect AI-based products and services from highly sophisticated, creative and malicious attacks, whether carried out by individual trolls or entire wolfpacks. This document focuses entirely on security engineering issues unique to the AI/ML space, but due to the expansive nature of the InfoSec domain it is understood that issues and findings discussed here will overlap to a degree with the domains of privacy and ethics. As this document highlights challenges of strategic importance to the tech industry, the target audience for this document is security engineering leadership industry-wide. Our early findings suggest that: AI/ML-specific pivots to existing security practices are required to mitigate the types of security issues discussed in this document. Machine Learning models are largely unable to discern between malicious input and benign anomalous data. A significant source of training data is derived from un-curated, unmoderated, public datasets which are open to 3 -party contributions. Attackers don't need to compromise datasets when they are free to contribute to them. Over time, low-confidence malicious data becomes high-confidence trusted data, provided that the data structure/formatting remains correct. rd Given the great number of layers of hidden classifiers/neurons which can be leveraged in a deep learning model, too much trust is placed on the output of AI/ML decision-making processes and algorithms without a critical understanding of how these decisions were reached. This obfuscation creates an inability to "show your work" and makes it difficult to provably defend AI/ML findings when called into question. AI/ML is increasingly used in support of high-value decision-making processes in medicine and other industries where the wrong decision may result in serious injury or death. A lack of forensics reporting capabilities in AI/ML prevent these high-value conclusions from being defensible in both the court of law and court of public opinion. The goals of this document are to (1) highlight security engineering issues which are unique to the AI/ML space, New Security Engineering Challenges AI requires new pivots to traditional secure design/secure operations models: the introduction of Resilience and Discretion (2) surface some initial thoughts and observations on emerging threats and (  3 ) share early thoughts on potential remediation. Some of the challenges in this document are problems that the industry needs to get ahead of in the next two years, others are issues that we're already being forced to address today. Without deeper investigation into the areas covered in this document, we risk future AI becoming a black box to its through our inability to trust or understand (and modify if necessary) AI decision-making processes at a mathematical level  [7] . From a security perspective, this effectively means loss of control and a departure from Microsoft's guiding principles on Artificial Intelligence  [4, 8] . Traditional software attack vectors are still critical to address, but they do not provide sufficient coverage in the AI/ML threat landscape. The tech industry must avoid fighting next-gen issues with last-gen solutions by building new frameworks and adopting new approaches which address gaps in the design and operation of AI/ML-based services: 1. As discussed below, secure development and operations foundations must incorporate the concepts of Resilience and Discretion when protecting AI and the data under its control. AI-specific pivots are required in the areas of Authentication, Separation of Duty, Input Validation and Denial of Service mitigation. Without investments in these areas, AI/ML services will continue to fight an uphill battle against adversaries of all skill levels. 2. AI must be able to recognize bias in others, without being biased in its own interactions with humans. Accomplishing this requires a collective and evolving understanding of biases, stereotypes, vernacular, and other cultural constructs. Such an understanding will help protect AI from social engineering and dataset tampering attacks. A properly implemented system will actually become stronger from such attacks and be able to share its expanded understanding with other AIs. 3. Machine Learning algorithms must be capable of discerning maliciously-introduced data from benign "Black Swan" events  [1]  by rejecting training data with negative impact on results. Otherwise, learning models will always be susceptible to gaming by attackers and trolls. 4. AI must have built-in forensic capabilities. This enables enterprises to provide customers with transparency and accountability of their AI, ensuring its actions are not only verifiably correct but also legally defensible. These capabilities also function as an early form of "AI intrusion detection", allowing engineers to determine the exact point in time that a decision was made by a classifier, what data influenced it, and whether or not that data was trustworthy. Data visualization capabilities in this area are rapidly advancing and show promise to help engineers identify and resolve root causes for these complex issues  [11] . 5. AI must recognize and safeguard sensitive information, even if humans don't recognize it as such. Rich user experiences in AI require vast amounts of raw data to train on, so "over-sharing" by customers must be planned for. Each of these areas, including threats and potential mitigations, is discussed in detail below. AI designers will always need to ensure the confidentiality, integrity and availability of sensitive data, that the AI system is free of known vulnerabilities, and provide controls for the protection, detection and response to malicious behavior against the system or the user's data. The traditional ways of defending against malicious attacks do not provide the same coverage in this new paradigm, where voice/video/image-based attacks can circumvent current filters and defenses. New threat modeling aspects must be explored in order to prevent new abuses from exploiting our AI. This goes far beyond AI must be able to recognize bias in others without being biased on its own identifying traditional attack surface via fuzzing or input manipulation (those attacks have their own AI-specific pivots too). It requires incorporating scenarios unique to the AI/ML space. Key among these are AI user experiences such as voice, video and gestures. The threats associated with these experiences have not been traditionally modeled. For example, video content is now being tailored to induce physical effects. Additionally, research has demonstrated that audio-based attack commands can be crafted  [10] . The unpredictability, creativity, and maliciousness of criminals, determined adversaries, and trolls requires us to instill our AIs with the values of Resilience Resilience and Discretion Discretion: Resilience: Resilience: The system should be able to identify abnormal behaviors and prevent manipulation or coercion outside of normal boundaries of acceptable behavior in relation to the AI system and the specific task. These are new types of attacks specific to the AI/ML space. Systems should be designed to resist inputs that would otherwise conflict with local laws, ethics and values held by the community and its creators. This means providing AI with the capability to determine when an interaction is going "off script. " This could be achieved with the following methods: 1. Pinpoint individual users who deviate from norms set by the various large clusters of similar users e.g. users who seem to type to fast, respond too fast, don't sleep, or trigger parts of the system other users do not. 2. Identify patterns of behavior that are known to be indicators of malicious intent probing attacks and the start of the Network Intrusion Kill Chain. 3. Recognize any time when multiple users act in a coordinated fashion; e.g., multiple users all issuing the same unexplainable yet deliberately crafted query, sudden spikes in the number of users or sudden spikes in activation of specific parts of an AI system. Attacks of this type should be considered on par with Denial of Service attacks since the AI may require bugfixes and retraining in order not to fall for the same tricks again. Of critical importance is the ability to identify malicious intent in the presence of countermeasures such as those used to defeat sentiment analysis APIs  [5] .
 Discretion Discretion: AI should be a responsible and trustworthy custodian of any information it has access to. As humans, we will undoubtedly assign a certain level of trust in our AI relationships. At some point, these agents will talk to other agents or other humans on our behalf. We must be able to trust that an AI system has enough discretion to only share in a restricted form what it needs to share about us so other agents can complete tasks on its behalf. Furthermore, multiple agents interacting with personal data on our behalf should not each need global access to it. Any data access scenarios involving multiple AIs or bot agents should limit the lifetime of access to the minimum extent required. Users should also be able to deny data and reject the authentication of agents from specific corporations or locales just as web browsers allow site blacklisting today. Solving this problem requires new thinking on inter-agent authentication and data access privileges like the cloud-based user authentication investments made in the early years of cloud computing. While AI should be fair and inclusive without discriminating against any particular group of individuals or valid outcomes, it needs to have an innate understanding of bias to achieve this. Without being trained to recognize bias, trolling, or sarcasm, AI will be duped by those seeking cheap laughs at best, or cause harm to customers at worst. Achieving this level of awareness calls for "good people teaching AI bad things" as it effectively requires a comprehensive and evolving understanding of cultural biases. AI should be able to recognize a user it has had negative interactions with in the past and exercise appropriate caution, similar to how parents teach their children to be wary of strangers. The best way to approach this is by carefully exposing the AI to trolls in a controlled/moderated/limited fashion. This way AI can learn the difference between a benign user "kicking the Machine Learning Algorithms must be capable of discerning maliciously-introduced data from benign "Black Swan" events AI must have built-in forensics and security logging to provide transparency and accountability tires" and actual maliciousness/trolling. Trolls provide a valuable stream of training data for AI, making it more resilient against future attacks. AI should also be able to recognize bias in datasets it trains on. This could be cultural or regional, containing the vernacular in use by a particular group of people, or topics/viewpoints of specific interest to one group. As with maliciously-introduced training data, AI must be resilient to the effects of this data on its own inferences and deductions. At its core, this is a sophisticated input validation issue with similarities to bounds checking. Instead of dealing with buffer lengths and offsets, the buffer and bounds checks are red-flagged words from a broad range of sources. The conversation history and context in which words are used are also key. Just as defense-indepth practices are used to layer protections on top of a traditional Web Service API frontend, multiple layers of protection should be leveraged in bias recognition and avoidance techniques. Numerous whitepapers have been published on the theoretical potential of ML model/classifier tampering and extraction/theft from services where attackers have access to both the training data set and an informed understanding of the model in use  [2, 3, 6, 7] . The over-arching issue here is that all ML classifiers can be tricked by an attacker who has control over training set data. Attackers don't even need the ability to modify existing training set data, they just need to be able to add to it and have their inputs become "trusted" over time through the ML classifier's inability to discern malicious data from genuine anomalous data. This training data supply chain issue introduces us to the concept of "Decision Integrity" -the ability to identify and reject maliciously introduced training data or user input before it has a negative impact on classifier behavior. The rationale here is that trustworthy training data has a higher probability of generating trustworthy outcomes/decisions. While it is still crucial to train on and be resilient to untrusted data, the malicious nature of that data should be analyzed prior to it becoming part of a high-confidence body of training data. Without such measures, AI could be coerced into overreacting to trolling and deny service to legitimate users. This is of particular concern where unsupervised learning algorithms are training on uncurated or untrusted datasets. This means that attackers can introduce any data they want provided the format is valid and the algorithm is trained on it, effectively trusting that data point equally with the rest of the training set. With enough crafted inputs from the attacker, the training algorithm loses the ability to discern noise and anomalies from high-confidence data. As an example of this threat, imagine a database of stop signs throughout the world, in every language. That would be extremely challenging to curate because of the number of images and languages involved. Malicious contribution to that dataset would go largely unnoticed until self-driving cars no longer recognize stop signs. Data resilience and decision integrity mitigations will have to work hand in hand here to identify and eliminate the training damage done by malicious data to prevent it from becoming a core part of the learning model. AI will eventually be capable of acting in a professional capacity as an agent on our behalf, assisting us with high-impact decision-making. An example of this could be an AI that assists in the processing of financial transactions. If the AI were exploited, and transactions manipulated in some way, the consequences could range from the individual to the systemic. In high-value scenarios, AI will need appropriate forensic and security logging to provide integrity, transparency, accountability, and in some instances, evidence where civil or criminal liability may arise. Essential AI services will need auditing/event-tracing facilities at the algorithm level whereby developers can examine the recorded state of specific classifiers which may have led to an inaccurate decision. This capability is needed industry-wide in order to prove the correctness and transparency of AI-generated decisions whenever AI must safeguard sensitive information, even if humans don't Early Observations on Addressing AI Security Issues called into question. Event tracing facilities could start with the correlation of basic decision-making information such as: 1. The timeframe in which the last training event occurred 2. The timestamp of the most recent dataset entry trained upon 3. Weights and confidence levels of key classifiers used to arrive at high-impact decisions 4. The classifiers or components involved in the decision 5. The final high-value decision reached by the algorithm Such tracing is overkill for the majority of algorithm-assisted decision making. However, having the ability to identify the data points and algorithm metadata leading to specific results will be of great benefit in high-value decision making. Such capabilities will not only demonstrate trustworthiness and integrity through the algorithm's ability to "show its work", but this data could also be used for fine-tuning as well. Another forensic capability needed in AI/ML is tamper detection. Just as we need our AIs to recognize bias and not be susceptible to it, we should have forensic capabilities available to aid our engineers in detecting and responding to such attacks. Such forensic capabilities will be of tremendous value when paired with data visualization techniques  [11]  allowing the auditing, debugging and tuning of algorithms for more effective results. Rich experiences require rich data. Humans already volunteer vast amounts of data for ML to train against. This ranges from the mundane video streaming queue contents to trends in credit card purchases/transaction histories used to detect fraud. AI should have an ingrained sense of discretion when it comes to handling user data, always acting to protect it even when it is volunteered freely by an over-sharing public. As an AI can have an authenticated group of "peers" it talks to in order to accomplish complex tasks, it must also recognize the need to restrict the data it shares with those peers. Despite the nascent state of this project, we believe the evidence compiled to date shows deeper investigation into each of the areas below will be key in moving our industry towards more trustworthy and secure AI/ML products/services. The following are our early observations and thoughts on what we' d like to see done in this space. 1. AI/ML-focused penetration testing and security review bodies could be established to ensure that our future AI shares our values and aligns to the Asilomar AI Principles. a. Such a group could also develop tools and frameworks that could be consumed industry-wide in support of securing their AI/ML-based services. b. Over time, this expertise will build up within engineering groups organically, as it did with traditional security expertise over the past 10 years. 2. Training could be developed which empowers enterprises to deliver on goals such as democratizing AI while mitigating the challenges discussed in this document. a. AI-specific security training ensures that engineers are aware of the risks posed to to their AI and the resources at their disposal. This material needs to be delivered in conjunction with current training on protecting customer data. b. This could be accomplished without requiring every data scientist to become a security expertinstead, the focus is placed on educating developers on Resilience and Discretion as applied to their AI Conclusion Bibliography use cases. c. Developers will need to understand the secure "building blocks" of AI services that will be reused across their enterprise. There will need to be an emphasis on fault-tolerant design with subsystems which can be easily turned off (e.g. image processors, text parsers). 3. ML Classifiers and their underlying algorithms could be hardened and capable of detecting malicious training data without it contaminating valid training data currently in use or skewing the results. a. Techniques such as Reject on Negative Input  [6]  need researcher cycles to investigate. b. This work involves mathematical verification, proof-of-concept in code, and testing against both malicious and benign anomalous data. c. Human spot-checking/moderation may be beneficial here, particularly where statistical anomalies are present. d. "Overseer classifiers" could be built to have a more universal understanding of threats across multiple AIs. This vastly improves the security of the system because the attacker can no longer exfiltrate any one particular model. e. AIs could be linked together to identify threats in each other's systems 4. A centralized ML auditing/forensics library could be built that establishes a standard for transparency and trustworthiness of AI. a. Query capabilities could also be built for auditing and reconstruction of high business impact decisions by AI. 5. The vernacular in use by adversaries across different cultural groups and social media could be continuously inventoried and analyzed by AI in order to detect and respond to trolling, sarcasm, etc. a. AIs need to be resilient in the face of all kinds of vernacular, whether technical, regional, or forumspecific. b. This body of knowledge could also be leveraged in content filtering/labeling/blocking automation to address moderator scalability issues. c. This global database of terms could be hosted in development libraries or even exposed via cloud service APIs for reuse by different AIs, ensuring new AIs benefit from the combined wisdom of older ones. 6. A "Machine Learning Fuzzing Framework" could be created which provides engineers with the ability to inject various types of attacks into test training sets for AI to evaluate. a. This could focus on not only text vernacular, but image, voice and gesture data as well as permutations of those data types. The Asilomar AI Principles illustrate the complexity of delivering on AI in a fashion that consistently benefits humanity. Future AIs will need to interact with other AIs to deliver rich, compelling user experiences. That means it simply is not good enough for Microsoft to "get AI right" from a security perspective -the world has to. We need industry alignment and collaboration with greater visibility brought to the issues in this document in a fashion similar to our worldwide push for a Digital Geneva Convention  [9] . By addressing the issues presented here, we can begin guiding our customers and industry partners down a path where AI is truly democratized and augments the intelligence of all humanity. Identifying Security Bug Reports Based Solely on Report Titles and Noisy Data Index Terms -Machine Learning, Mislabeling, Noise, Security Bug Report, Bug Repositories Identifying security related issues among reported bugs is a pressing need among software development teams as such issues call for more expedited fixes in order to meet compliance requirements and ensure the integrity of the software and customer data. Machine learning and artificial intelligence tools promise to make the software development faster, agile and correct. Several researchers have applied machine learning to the problem of identifying security bugs  [2] ,  [7] ,  [8] ,  [18] .Previous published studies have assumed that the entire bug report is available for training and scoring a machine learning model. This is not necessarily the case. There are situations where the entire bug report cannot be made available. For example, the bug report might contain passwords, personally identifying information (PII) or other kinds of sensitive data -a case we are currently facing at Microsoft. It is therefore important to establish how well security bug identification can be performed using less information, such as when only the title of the bug report is available. Additionally, bug repositories often contain mislabeled entries  [7] : non-security bug reports classified as security related and vice-versa. There are several reasons for the occurrence of mislabeling, ranging from the development team's lack of expertise in security, to the fuzziness of certain problems, e.g. it is possible for nonsecurity bugs to be exploited in an indirect way as to cause a security implication. This is a serious problem since the mislabeling of SBRs results in security experts having to manually review bug database in an expensive and time-consuming effort. Understanding how noise affects different classifiers and how robust (or fragile) different machine learning techniques are in the presence of data sets contaminated with different kinds of noise is a problem that must be addressed for to bring automatic classification to the practice of software engineering. Preliminary work argues that bug repositories are intrinsically noisy and that the noise might have an adverse effect on the performance machine learning classifiers  [7] . There lacks, however, any systematic and quantitative study of how different levels and types of noise affect the performance of different supervised machine learning algorithms for the problem of identifying security bug reports (SRBs). In this study, we show that the classification of bug reports can be performed even when solely the title is available for training and scoring. To the best of our knowledge this is the very first work to do so. Additionally, we provide the first systematic study of the effect of noise in bug report classification. We make a comparative OUR RESEARCH CONTRIBUTIONS: OUR RESEARCH CONTRIBUTIONS:
 II. PREVIOUS WORKS
 MACHINE LEARNING APPLICATIONS TO BUG REPOSITORIES. MACHINE LEARNING APPLICATIONS TO BUG REPOSITORIES. study of robustness of three machine learning techniques (logistic regression, naive Bayes and AdaBoost) against class-independent noise. While there are some analytical models that capture the general influence of noise for a few simple classifiers  [5] ,  [6] , these results do not provide tight bounds on the effect of the noise on precision and are valid only for a particular machine learning technique. An accurate analysis of the effect of noise in machine learning models is usually performed by running computational experiments. Such analyses have been done for several scenarios ranging from software measurement data  [4] , to satellite image classification  [13]  and medical data  [12] . Yet these results cannot be translated to our specific problem, due to its high dependency on the nature of the data sets and underlying classification problem. To the best of our knowledge, there are no published results on the problem of the effect of noisy data sets on security bug report classification in particular. We train classifiers for the identification of security bug reports (SBRs) based solely on the title of the reports. To the best of our knowledge this is the first work to do so. Previous works either used the complete bug report or enhanced the bug report with additional complementary features. Classifying bugs based solely on the tile is particularly relevant when the complete bug reports cannot be made available due to privacy concerns. For example, it is notorious the case of bug reports that contain passwords and other sensitive data. We also provide the first systematic study of the label noise-tolerance of different machine learning models and techniques used for the automatic classification of SBRs. We make a comparative study of robustness of three distinct machine learning techniques (logistic regression, naive Bayes and AdaBoost) against class-dependent and class-independent noise. The remaining of the paper is presented as follows: In section II we present some of the previous works in the literature. In section III we describe the data set and how data is pre-processed. The methodology is described in section IV and the results of our experiments analyzed in section V. Finally, our conclusions and future works are presented in VI. There exists an extensive literature in applying text mining, natural language processing and machine learning on bug repositories in an attempt to automate laborious tasks such as security bug detection  [2] ,  [7] ,  [8] ,  [18] , bug duplicate identification  [3] , bug triage  [1] ,  [11] , to name a few applications. Ideally, the marriage of machine learning (ML) and natural language processing potentially reduces the manual work required to curate bug databases, shorten the required time for accomplishing these tasks and can increase the reliability of the results. In  [7]  the authors propose a natural language model to automate the classification of SBRs based on the description of the bug. The authors extract a vocabulary from all bug description in the training data set and manually curate it into three lists of words: relevant words, stop words (common words that seem irrelevant for classification), and synonyms. They compare the performance of security bug classifier trained on data that is all evaluated by security engineers, and a classifier trained on data that was labeled by bug reporters in general. Although their model is clearly more effective when trained on data reviewed by security engineers, the proposed model is based on a manually derived vocabulary, which makes it dependent on human curation. Moreover, there is no analysis of how different levels of noise affects their model, how different classifiers respond to noise, and if noise in either class affects performance differently. Zou et. al  [18]  make use of multiple types of information contained in a bug report which involve the nontextual fields of a bug report (meta features, e.g., time, severity, and priority) and the textual content of a bug report (textual features, i.e., the text in summary fields). Based on these features, they build a model to automatically identify the SBRs via natural language processing and machine learning techniques. In  [8]  the authors perform a similar analysis, but additionally they compare the performance of supervised and In  [2]  the authors also explore different machine learning techniques to classify bugs as SBRs or NSBRs (Non-Security Bug Report) based on their descriptions. They propose a pipeline for data processing and model training based on TFIDF. They compare the proposed pipeline with a model based on bag-of-words and naive Bayes. Wijayasekara et al.  [16]  also used text mining techniques to generate the feature vector of each bug report based on frequent words to identify Hidden Impact Bugs (HIBs). Yang et al.  [17]  claimed to identify high impact bug reports (e.g., SBRs) with the help of Term Frequency (TF) and naive Bayes. In  [9]  the authors propose a model to predict the severity of a bug. The problem of dealing with data sets with label noise has been extensively studied. Frenay and Verleysen propose a label noise taxonomy in  [6] , in order to distinguish different types of noisy label. The authors propose three different types of noise: label noise which occurs independently of the true class and of the values of the instance features; label noise that depends only on the true label; and label noise where the mislabelling probability also depends on the feature values. In our work we study the first two types of noise. From a theoretical perspective, label noise usually decreases a model's performance  [10] , except in some specific cases  [14] . In general, robust methods rely on overfitting avoidance to handle label noise  [15] . The study of noise effects in classification has been done before in many areas such as satellite image classification  [13] , software quality classification  [4]  and medical domain classification  [12] . To the best of our knowledge, there are no published works studying the precise quantification of the effects of noisy labels in the problem of SBRs classification. In this scenario, the precise relationship among noise levels, noise types and performance degradation has not been established. Moreover, it is worthwhile to understand how different classifiers behave in the presence of noise. More generally, we are unaware of any work that systematically studies the effect of noisy data sets on the performance of different machine learning algorithms in the context of software bug reports. Our data set consists of 1,073,149 bug titles, 552,073 of which correspond to SBRs and 521,076 to NSBRs. The data was collected from various teams across Microsoft in the years 2015, 2016, 2017 and 2018. All the labels were obtained by signature-based bug verification systems or human labeled. Bug titles in our data set are very short texts, containing around 10 words, with an overview of the problem. A. Data Pre-processing We parse each bug title by its blank spaces, resulting in a list of tokens. We process each list of tokens as follows: Remove all tokens that are file paths Split tokens where the following symbols are present: { , (, ), -, }, {, [, ], } Remove stop words , tokens that are composed by numeric characters only and tokens that appear fewer than 5 times in the entire corpus. The process of training our machine learning models consists of two main steps: encoding the data into feature vectors and training supervised machine learning classifiers. The first part involves encoding data into feature vectors using the term frequencyinverse document frequency algorithm (TF-IDF), as used in  [2] . TF-IDF is an information retrieval technique that weighs a terms frequency (TF) and its inverse document frequency (IDF). Each word or term has its respective TF and IDF score. The TF-IDF algorithm assigns the importance to that word based on the number of times it appears in the document, and more importantly, it checks how relevant the keyword is throughout the collection of titles in the data set. We
 B. Types of Noise B. Types of Noise
 V. EXPERIMENTAL RESULTS trained and compared three classification techniques: naive Bayes (NB), boosted decision trees (AdaBoost) and logistic regression (LR). We have chosen these techniques because they have been shown to performance well for the related task of identifying security bug reports based on the entire report in the literature. These results were confirmed in a preliminary analysis where these three classifiers outperformed support vector machines and random forests. In our experiments we utilize the scikit-learn library for encoding and model training. The noise studied in this work refers to noise in the class label in the training data. In the presence of such noise, as a consequence, the learning process and resulting model are impaired by mislabelled examples. We analyze the impact of different noise levels applied to the class information. Types of label noise have been discussed previously in the literature using various terminologies. In our work, we analyze the effects of two different label noise in our classifiers: class-independent label noise, which is introduced by picking instances at random and flipping their label; and class-dependent noise, where classes have different likelihood to be noisy. a) Class-independent noise: Class-independent noise refers to the noise that occurs independently of the true class of the instances. In this type of noise, the probability of mislabeling p is the same for all instances in the data set. We introduce class-independent noise in our data sets by flipping each label in our data set at random with probability p . We add noise to the training and validation data sets for different levels of p , p and p . We do not make any modifications to the test data set. The different noise levels used are P = {0.05 × i|0 < i < 10}. Tune models using validation data set (with noise); Test models using test data set (noiseless). In this section analyze the results of experiments conducted according to the methodology described in section IV. a) Model performance without noise in the training data set: One of the contributions of this paper is the proposal of a machine learning model to identify security bugs by using only the title of the bug as data for decision making. This enables the training of machine learning models even when development teams do not wish to share bug reports in full due to presence of sensitive data. We compare the performance of three machine learning models when trained using only bug titles. The logistic regression model is the best performing classifier. It is the classifier with the highest AUC value, of 0.9826, recall of 0.9353 for a FPR value of 0.0735. The naive Bayes Classifier presents slightly lower performance than the logistic regression Classifier, with an AUC of 0.9779 and a recall of 0.9189 for a FPR of 0.0769. The AdaBoost classifier has an inferior performance in comparison to the two previously mentioned classifiers. It achieves an AUC of 0.9143, and a recall of 0.7018 for a 0.0774 FPR. The area under the ROC curve (AUC) is a good metric for comparing performance of several models, as it summarizes in a single value the TPR vs. FPR relation. In the subsequent analysis we will restrict our comparative analysis to AUC values.
 A. Class Noise: single-class One can imagine a scenario where all bugs are assigned to class NSBR by default, and a bug will only be assigned to class SBR if there is a security expert reviewing the bug repository. This scenario is represented in the single-class experimental setting, where we assume that p = 0 and 0 < p < 0.5. nsbr sbr From table II we observe a very small impact in the AUC for all three classifiers. The AUC-ROC from a model trained on p = 0 compared to a AUC-ROC of model where p = 0.25 differs by 0.003 for logistic regression, 0.006 for naive Bayes, and 0.006 for AdaBoost. In the case of p = 0.50, the AUC measured for each of the models differ from the model trained with p = 0 by 0.007 for logistic regression, 0.011 for naive Bayes, and 0.010 for AdaBoost. logistic regression classifier trained in the presence of single-class noise presents the smallest variation in its AUC metric, i.e. a more robust behavior, when compared to our naive Bayes and AdaBoost classifiers. In Table  III  we observe a decrease in the AUC-ROC for every noise increment in the experiment. The AUC-ROC measured from a model trained on noiseless data compared to a AUC-ROC of model trained with classindependent noise with p = 0.25 differs by 0.011 for logistic regression, 0.008 for naive Bayes, and 0.0038 for AdaBoost. We observe that label noise does not impact the AUC of naive Bayes and AdaBoost classifiers significantly when noise levels are lower than 40%. On the other hand, logistic regression Classifier experiences an impact in AUC measure for label noise levels above 30%. Tables IV, V, VI show the variation of AUC as noise is increased in different levels in each class for logistic regression in Table  IV , for naive Bayes in Table V and for AdaBoost in Table  VI . For all classifiers, we notice an impact in AUC metric when both classes contains noise level above 30%. naive Bayes behaves mot robustly. The impact on AUC is very small even when the 50% of the label in the positive class are flipped, provided that the negative class contains 30% of noisy labels or less. In this case, the drop in AUC is of 0.03. AdaBoost presented the most robust behavior of all three classifiers. A significant change in AUC will only happen for noise levels greater than 45% in both classes. In that case, we start observing an AUC decay greater than 0.02.
 D. On The Presence of Residual Noise in the Original Data Set Our data set was labeled by signature-based automated systems and by human experts. Moreover, all bugs reports have been further reviewed and closed by human experts. While we expect that the amount of noise in our data set is minimal and not statistically significant, the presence of residual noise does not invalidate our
 VI. CONCLUSIONS AND FUTURE WORKS
 FUTURE WORKS FUTURE WORKS
 REFERENCES REFERENCES conclusions. Indeed, for the sake of illustration, assume that the original data set is corrupted by a classindependent noise equal to 0 < p < 1/2 independent and identically distributed (i.i.d) for every entry. If we, on top of the original noise, add a class-independent noise with probability p i.i.d, the resulting noise per entry will be p * = p(1 − p )+(1 − p)p . For 0 < p,p < 1/2, we have that the actual noise per label p * is strictly larger than the noise we artificially add to the data set p . Thus, the performance of our classifiers would be even better if they were trained with a completely noiseless data set (p = 0) in the first place. In summary, the existence of residual noise in the actual data set means that the resilience against noise of our classifiers is better than the results here presented. Moreover, if the residual noise in our data set were statistically relevant, the AUC of our classifiers would become 0.5 (a random guess) for a level of noise strictly less than 0.5. We do not observe such behavior in our results. First, we have shown the feasibility of security bug report classification based solely on the title of the bug report. This is particularly relevant in scenarios where the entire bug report is not available due to privacy constraints. For example, in our case, the bug reports contained private information such as passwords and cryptographic keys and were not available for training the classifiers. Our result shows that SBR identification can be performed at high accuracy even when only report titles are available. Our classification model that utilizes a combination of TF-IDF and logistic regression performs at an AUC of 0.9831. Second, we analyzed the effect of mislabeled training and validation data. We compared three well-known machine learning classification techniques (naive Bayes, logistic regression and AdaBoost) in terms of their robustness against different noise types and noise levels. All three classifiers are robust to single-class noise. Noise in the training data has no significant effect in the resulting classifier. The decrease in AUC is very small ( 0.01) for a level of noise of 50%. For noise present in both classes and is class-independent, naive Bayes and AdaBoost models present significant variations in AUC only when trained with a data set with noise levels greater than 40%. Finally, class-dependent noise significantly impacts the AUC only when there is more then 35% noise in both classes. AdaBoost showed the most robustness. The impact in AUC is very small even when the positive class has 50% of its labels noisy, provided that the negative class contains 45% of noisy labels or less. In this case, the drop in AUC is less than 0.03. To the best of our knowledge, this is the first systematic study on the effect of noisy data sets for security bug report identification. In this paper we have started the systematic study of the effects of noise in the performance of machine learning classifiers for the identification of security bugs. There are several interesting sequels to this work, including: examining the effect of noisy data sets in determining the severity level of a security bug; understanding the effect of class imbalance on the resilience of the trained models against noise; understanding the effect of noise that is adversarially introduced in the data set. [ Microsoft recommends customers get ahead of this issue by removing TLS 1.0 dependencies in their environments and disabling TLS 1.0 at the operating system level where possible. Given the length of time TLS 1.0 has been supported by the software industry, it is highly recommended that any TLS 1.0 deprecation plan include the following: Code analysis to find/fix hardcoded instances of TLS 1.0 or older security protocols. Network endpoint scanning and traffic analysis to identify operating systems using TLS 1.0 or older protocols. Full regression testing through your entire application stack with TLS 1.0 disabled. Migration of legacy operating systems and development libraries/frameworks to versions capable of negotiating TLS 1.2 by default. Compatibility testing across operating systems used by your business to identify any TLS 1.2 support issues. Coordination with your own business partners and customers to notify them of your move to deprecate TLS 1.0. Understanding which clients may no longer be able to connect to your servers once TLS 1.0 is disabled. The goal of this document is to provide recommendations which can help remove technical blockers to disabling TLS 1.0 while at the same time increasing visibility into the impact of this change to your own customers. Completing such investigations can help reduce the business impact of the next security vulnerability in TLS 1.0. For the purposes of this document, references to the deprecation of TLS 1.0 also include TLS 1.1. Enterprise software developers have a strategic need to adopt more future-safe and agile solutions (otherwise known as Crypto Agility) to deal with future security protocol compromises. While this document proposes agile solutions to the elimination of TLS hardcoding, broader Crypto Agility solutions are beyond the scope of this The Current State of Microsoft's TLS 1.0 implementation Ensuring support for TLS 1.2 across deployed operating systems  Microsoft's TLS 1.0 implementation is free of known security vulnerabilities. Due to the potential for future protocol downgrade attacks and other TLS 1.0 vulnerabilities not specific to Microsoft's implementation, it is recommended that dependencies on all security protocols older than TLS 1.2 be removed where possible (TLS 1.1/1.0/ SSLv3/SSLv2). In planning for this migration to TLS 1.2+, developers and system administrators should be aware of the potential for protocol version hardcoding in applications developed by their employees and partners. Hardcoding here means that the TLS version is fixed to a version that is outdated and less secure than newer versions. TLS versions newer than the hardcoded version cannot be used without modifying the program in question. This class of problem cannot be addressed without source code changes and software update deployment. Protocol version hardcoding was commonplace in the past for testing and supportability purposes as many different browsers and operating systems had varying levels of TLS support. Microsoft's Engineering Improvements to eliminate TLS 1.0 dependencies Finding and fixing TLS 1.0 dependencies in code If not already complete, it is highly recommended to conduct an inventory of operating systems used by your enterprise, customers and partners (the latter two via outreach/communication or at least HTTP User-Agent string collection). This inventory can be further supplemented by traffic analysis at your enterprise network edge. In such a situation, traffic analysis will yield the TLS versions successfully negotiated by customers/partners connecting to your services, but the traffic itself will remain encrypted. Since the v1 release of this document, Microsoft has shipped a number of software updates and new features in support of TLS 1.0 deprecation. These include: IIS custom logging to correlate client IP/user agent string, service URI, TLS protocol version and cipher suite. With this logging, admins can finally quantify their customers' exposure to weak TLS. SecureScore -To help Office 365 tenant admins identify their own weak TLS usage, the SecureScore portal has been built to share this information as TLS 1.0 exited support in Office 365 in October 2018. This portal provides Office 365 tenant admins with the valuable information they need to reach out to their own customers who may be unaware of their own TLS 1.0 dependencies. Please visit https://securescore.microsoft.com/ for more information. .Net Framework updates to eliminate app-level hardcoding and prevent framework-inherited TLS 1.0 dependencies. Developer Guidance and software updates have been released to help customers identify and eliminate .Net dependencies on weak TLS: Transport Layer Security (TLS) best practices with the .NET Framework FYI: All apps targeting .NET 4.5 or below are likely going to have to be modified in order to support TLS 1.2. TLS 1.2 has been backported to Windows Server 2008 SP2 and XP POSReady 2009 to help customers with legacy obligations. More announcements will be made in early 2019 and communicated in subsequent updates of this document. For products using the Windows OS-provided cryptography libraries and security protocols, the following steps should help identify any hardcoded TLS 1.0 usage in your applications: 1. Identify all instances of AcquireCredentialsHandle(). This helps reviewers get closer proximity to code blocks where TLS may be hardcoded.
 Review any instances of the SecPkgContext_SupportedProtocols and SecPkgContext_ConnectionInfo structures for hardcoded TLS. 3. In native code, set any non-zero assignments of grbitEnabledProtocols to zero. This allows the operating system to use its default TLS version. Following the fixes recommended in the section above, products should be regression-tested for protocol negotiation errors and compatibility with other operating systems in your enterprise. The most common issue in this regression testing will be a TLS negotiation failure due to a client connection attempt from an operating system or browser that does not support TLS 1.2. For example, a Vista client will fail to negotiate TLS with a server configured for TLS 1.2+ as Vista's maximum supported TLS version is 1.0. That client should be either upgraded or decommissioned in a TLS 1.2+ environment. Products using certificate-based Mutual TLS authentication may require additional regression testing as the certificate-selection code associated with TLS 1.0 was less expressive than that for TLS 1.2. If a product negotiates MTLS with a certificate from a non-standard location (outside of the standard named certificate stores in Windows), then that code may need updating to ensure the certificate is acquired correctly. Service interdependencies should be reviewed for trouble spots. Any services which interoperate with 3 -party services should conduct additional interop testing with those 3 parties. rd rd Any non-Windows applications or server operating systems in use require investigation / confirmation that they can support TLS 1.2. Scanning is the easiest way to determine this. A simple blueprint for testing these changes in an online service consists of the following: 1. Conduct a scan of production environment systems to identify operating systems which do not support TLS 1.2. 2. Scan source code and online service configuration files for hardcoded TLS as described in "Finding and fixing TLS 1.0 dependencies in code" After TLS hardcoding is addressed and operating system/development framework updates are completed, should you opt to deprecate TLS 1.0 it will be necessary to coordinate with customers and partners: Early partner/customer outreach is essential to a successful TLS 1.0 deprecation rollout. At a minimum this should consist of blog postings, whitepapers or other web content. Partners each need to evaluate their own TLS 1.2 readiness through the operating system/code scanning/regression testing initiatives described in above sections. Removing TLS 1.0 dependencies is a complicated issue to drive end to end. Microsoft and industry partners are taking action on this today to ensure our entire product stack is more secure by default, from our OS components and development frameworks up to the applications/services built on top of them. Following the recommendations made in this document will help your enterprise chart the right course and know what challenges to expect. It will also help your own customers become more prepared for the transition. As engineers worldwide work to eliminate their own dependencies on TLS 1.0, they run into the complex challenge of balancing their own security needs with the migration readiness of their customers. Disable Legacy TLS also allows an online service to offer two distinct groupings of endpoints on the same hardware: one which allows only TLS 1.2+ traffic, and another which accommodates legacy TLS 1.0 traffic. The changes are implemented in HTTP.sys, and in conjunction with the issuance of additional certificates, allow traffic to be routed to the new endpoint with the appropriate TLS version. Prior to this change, deploying such capabilities would require an additional hardware investment because such settings were only configurable system-wide via registry.
 TLS version A common deployment scenario features one set of hardware in a datacenter with customers of mixed needs: some need TLS 1.2 as an enforced minimum right now and others aren't done removing TLS 1.0 dependencies. Figure  1  illustrates TLS version selection and certificate binding as distinctly separate actions. This is the default functionality:
 Feature deployment guidance Option #1: IIS UI configuration (Available April 2020) Option #1: IIS UI configuration (Available April 2020) https://legacy.contoso.com directs customers with legacy TLS 1.0 needs (like those still migrating to TLS 1.2) to an endpoint which supports TLS 1.0 for a limited time. This allows customers to finish readiness testing for TLS 1.2 without service disruption and without blocking other customers who are ready for TLS 1.2. Traditionally, you' d need two physically separate hosts to handle all the traffic and provide for TLS version enforcement, as servicing TLS requests with a minimum protocol version requires disabling weaker protocols via system-wide registry settings. We have made this functionality available higher up the stack, where the TLS session is bound to the certificate, so a specific minimum TLS version can be assigned as described in Figure  2  below. Create a site binding for the SSL Certificate "secure.contoso.com" as shown below, then check "Disable Legacy TLS" and click OK. HTTP_SERVICE_CONFIG_SSL_FLAG_ENABLE_SESSION_TICKET: Enable/Disable Session Ticket for a particular SSL endpoint. HTTP_SERVICE_CONFIG_SSL_FLAG_LOG_EXTENDED_EVENTS : Enable/Disable extended event logging for a particular SSL endpoint. Additional events are logged to Windows Event Log. There is only one event supported as of now which is logged when the SSL handshake fails. HTTP_SERVICE_CONFIG_SSL_FLAG_DISABLE_LEGACY_TLS: Enable/Disable legacy TLS versions for a particular SSL endpoint. Setting this flag will disable TLS1.0/1.1 for that endpoint and will also restrict cipher suites that can be used to HTTP2 cipher suites. HTTP_SERVICE_CONFIG_SSL_FLAG_DISABLE_TLS12 : Enable/Disable TLS1.2 for a particular SSL endpoint. HTTP_SERVICE_CONFIG_SSL_FLAG_DISABLE_HTTP2: Enable/Disable HTTP/2 for a particular SSL endpoint. The simplest way to enable/disable this functionality per certificate in C++ is with the HTTP_SERVICE_CONFIG_SSL_FLAG_DISABLE_LEGACY_TLS flag provided by the HttpSetServiceConfiguration HTTP.sys API. When Disable Legacy TLS is set, the following restrictions are enforced: Disable SSL2, SSL3, TLS1.0 and TLS1.1 protocols. Disable encryption ciphers DES, 3DES, and RC4 (so only AES is used). Disable encryption cipher AES with CBC chaining mode (so only AES GCM is used). Disable RSA key exchange. Disable DH key exchange with key size less than 2048. Disable ECDH key exchanges with key size less than 224. Official documentation of these changes on docs.Microsoft.com is forthcoming. Disable Legacy TLS provides powerful new capabilities for enforcing TLS version/cipher suite floors on specific certificate/endpoint bindings. It also requires you to plan out the naming of the certificates issued with this functionality enabled. Some of the considerations include: Do I want the default path to my service endpoint to enforce TLS 1.2 today, and provide a different certificate as a backup "legacy" access point for users who need TLS 1.0? Should my default, already-in-use www.contoso.com certification use Disable Legacy TLS? If so, I may need to provide a legacy.contoso.com certificate and bind it to an endpoint allowing TLS 1.0. How can I best communicate the recommended usage of these certificates to my customers? You can leverage this feature to meet the needs of large groups of customers -those with an obligation to use TLS 1.2+, and those still working on the migration away from TLS 1.0, all without additional hardware expenditure. In addition to today's availability of per-certificate TLS version binding in Windows Server 2019, Microsoft will look to make Disable Legacy TLS available across its online services based on customer demand. Program Overview
 Contact Us Since its launch in 2003, the Government Security Program (GSP) has provided governments and international organizations with the ability to access and inspect source code for a variety of Microsoft products. The Online Source offering provides access through our Code Center Premium (CCP) web site to the GSP participant's premises. Using two-factor authentication and an encrypted connection, the Online Source offering enables access to product source code such as Windows, Office, SharePoint Server, and Exchange Server. The site provides read-only access and a debugger that can be used to review code. The Online Source offering and CCP site enable GSP participants to evaluate individual system component functions, component interaction, and security and reliability capabilities. For example, CCP enables agencies to search source code trees as well as step through source code in a debugger, set breakpoints on routines and source lines, and examine variables and data structures in their source layout form. In advance of going to a Transparency Center, an agency may wish to review code in CCP to acquaint itself with it and therefore improve the value of visits and most efficiently use time at the Transparency Center.
 Access Enablement Technical Trips The mission of Microsoft's Government Security Program (GSP) is to build trust through transparency. Since the program's inception in 2003, Microsoft has provided visibility into our technology and security artifacts which governments and international organizations can use to help protect themselves and their citizens. The Technical Data offering provides access to a broad range of confidential technical information, exclusive of source code, which allows government agencies and international organizations to evaluate the trustworthiness of Microsoft products and services. Information shared through the Technical Data offering is tailored to help GSP participants address their priorities. It provides a platform for customers to ask questions about security, to build trust in our products and services, and can include access to: Note: This sample document is for illustration purposes only. The content presented below outlines basic criteria to consider when creating security processes. It is not an exhaustive list of activities or criteria and should not be treated as such. Please refer to the definitions of terms in this section.
 Server Client Definitions of Terms Please refer to the Denial of Service Matrix for a complete matrix of server DoS scenarios. The server bar is usually not appropriate when user interaction is part of the exploitation process. If a Critical vulnerability exists only on server products, and is exploited in a way that requires user interaction and results in the compromise of the server, the severity may be reduced from Critical to Important in accordance with the NEAT/data definition of extensive user interaction presented at the start of the client severity pivot. Cases where the attacker can easily read information on the system from specific specific locations locations, including system information, which was not intended/ designed to be exposed.
 Example: Targeted disclosure of anonymous data Targeted disclosure of the existence of a file Targeted disclosure of a file version number Spoofing An entity (computer, server, user, process) is able to masquerade as a different, random entity that cannot be specifically selected.
 Example: Client properly authenticates to server, but server hands back a session from another random user who happens to be connected to the server at the same time A security assurance is either a security feature or another product feature/function that customers expect to offer security protection. Communications have messaged (explicitly or implicitly) that customers can rely on the integrity of the feature, and that's what makes it a security assurance. Security bulletins will be released for a shortcoming in a security assurance that undermines the customer's reliance or trust.
 Examples: Processes running with normal "user" privileges cannot gain "admin" privileges unless admin password/credentials have been provided via intentionally authorized methods. Internet-based JavaScript running in Internet Explorer cannot control anything the host operating system unless the user has explicitly changed the default security settings. Information disclosure (untargeted)
 Runtime information Example:
 Leak of random heap memory Tampering Temporary modification of data in a specific scenario that does not persist after restarting the OS/application "User interaction" can only happen in client-driven scenario. Normal, simple user actions, like previewing mail, viewing local folders, or file shares, are not extensive user interaction. "Extensive" includes users manually navigating to a particular website (for example, typing in a URL) or by clicking through a yes/no decision. "Not extensive" includes users clicking through e-mail links. Critical Important NEAT NEAT qualifier (applies to warnings only). Demonstrably, the UX is: N Necessary (Does the user really need to be presented with the decision?) E Explained (Does the UX present all the information the user needs to make this decision?) A Actionable (Is there a set of steps users can take to make good decisions in both benign and malicious scenarios?) T Tested (Has the warning been reviewed by multiple people, to make sure people understand how to respond to the warning?) Clarification: Clarification: Note that the effect of extensive user interaction is not one level reduction in severity, but is and has been a reduction in severity in certain circumstances where the phrase extensive user interaction appears in the bug bar. The intent is to help customers differentiate fast-spreading and wormable attacks from those, where because the user interacts, the attack is slowed down. This bug bar does not allow you to reduce the Elevation of Privilege below Important because of user interaction. Cases where the attacker can locate and read information on the system, including system information that was not intended or designed to be exposed.
 Spoofing Ability for attacker to present a UI that is different from but visually identical to the UI that users must rely on to make valid trust decisions in a default/common scenario. A trust decision is defined as any time the user takes an action believing some information is being presented by a particular entity-either the system or some specific local or remote source. The target cannot perform normal operations due to an attack. The response to an attack is roughly the same magnitude as the size of the attack. The target returns to the normal level of functionality shortly after the attack is finished. The exact definition of "shortly" should be evaluated for each product. For example, a server is unresponsive while an attacker is constantly sending a stream of packets across a network, and the server returns to normal a few seconds after the packet stream stops. temporar y DoS with amplification temporar y DoS with amplification A temporary DoS with amplification is a situation where the following criteria are met: The target cannot perform normal operations due to an attack. The response to an attack is magnitudes beyond the size of the attack. The target returns to the normal level of functionality after the attack is finished, but it takes some time (perhaps a few minutes). For example, if you can send a malicious 10-byte packet and cause a 2048k response on the network, you are DoSing the bandwidth by amplifying our attack effort.
 permanent DoS permanent DoS A permanent DoS is one that requires an administrator to start, restart, or reinstall all or parts of the system. Any vulnerability that automatically restarts the system is also a permanent DoS.  All asymmetric keys should have a maximum five-year lifetime, recommended one-year lifetime. All symmetric keys should have a maximum three-year lifetime; recommended one-year lifetime. You should provide a mechanism or have a process for replacing keys to achieve the limited active lifetime. After the end of its active lifetime, a key should not be used to produce new data (for example, for encryption or signing), but may still be used to read data (for example, for decryption or verification). All products and services should use cryptographically secure random number generators when randomness is required. Use of the dual elliptic curve random number generator ("DUAL_EC_DRBG") algorithm is not recommended.
 CNG On the Windows platform, Microsoft recommends using the crypto APIs built into the operating system. On other platforms, developers may choose to evaluate non-platform crypto libraries for use. In general, platform crypto libraries will be updated more frequently since they ship as part of an operating system as opposed to being bundled with an application. Any usage decision regarding platform vs non-platform crypto should be guided by the following requirements: Key Derivation Functions SSL/TLS/DTLS: Use APIs defined under the System.Net namespace (for example, HttpWebRequest). Key derivation is the process of deriving cryptographic key material from a shared secret or a existing cryptographic key. Products should use recommended key derivation functions. Deriving keys from user-chosen passwords, or hashing passwords for storage in an authentication system is a special case not covered by this guidance; developers should consult an expert. The following standards specify KDF functions recommended for use: NIST SP 800-108: Recommendation For Key Derivation Using Pseudorandom Functions. In particular, the KDF in counter mode, with HMAC as a pseudorandom function NIST SP 800-56A (Revision 2): Recommendation for Pair-Wise Key Establishment Schemes Using Discrete Logarithm Cryptography. In particular, the "Single-Step Key Derivation Function" in Section 5.8.1 is recommended. To derive keys from existing keys, use the BCryptKeyDerivation API with one of the algorithms: Revocation status. Usage (for example, "Server Authentication" for servers, "Client Authentication" for clients). Trust chain. Certificates should chain to a root certification authority (CA) that is trusted by the platform or explicitly configured by the administrator. If any of these verification tests fail, the product should terminate the connection with the entity. Clients that trust "self-signed" certificates (for example, a mail client connecting to an Exchange server in a default configuration) may ignore certificate verification checks. However, self-signed certificates do not inherently convey trust, support revocation, or support key renewal. You should only trust selfsigned certificates if you have obtained them from another trusted source (for example, a trusted entity that provides the certificate over an authenticated and integrity-protected transport). Products should use the SHA-2 family of hash algorithms (SHA256, SHA384, and SHA512). Truncation of cryptographic hashes for security purposes to less than 128 bits is not recommended. A message authentication code (MAC) is a piece of information attached to a message that allows its recipient to verify both the authenticity of the sender and the integrity of the message using a secret key. The use of either a hash-based MAC (HMAC) or block-cipher-based MAC is recommended as long as all underlying hash or symmetric encryption algorithms are also recommended for use; currently this includes the HMAC-SHA2 functions (HMAC-SHA256, HMAC-SHA384 and HMAC-SHA512). Truncation of HMACs to less than 128 bits is not recommended. You should provide a mechanism for replacing cryptographic keys as needed. Keys should be replaced once they have reached the end of their active lifetime or if the cryptographic key is compromised. Whenever you renew a certificate, you should renew it with a new key. Products using cryptographic algorithms to protect data should include enough metadata along with that content to support migrating to different algorithms in the future. This should include the algorithm used, key sizes, initialization vectors, and padding modes. For more information on Cryptographic Agility, see Cryptographic Agility on MSDN. Where available, products should use established, platform-provided cryptographic protocols rather than re-implementing them. This includes signing formats (e.g. use a standard, existing format). Symmetric stream ciphers such as RC4 should not be used. Instead of symmetric stream ciphers, Encrypting Sensitive Data prior to Storage products should use a block cipher, specifically AES with a key length of at least 128 bits. Do not report cryptographic operation failures to end-users. When returning an error to a remote caller (e.g. web client, or client in a client-server scenario), use a generic error message only. Avoid providing any unnecessary information, such as directly reporting out-of-range or invalid length errors. Log verbose errors on the server only, and only if verbose logging is enabled. Additional security review is highly recommended for any design incorporating the following: A new protocol that is primarily focused on security (such as an authentication or authorization protocol) A new protocol that uses cryptography in a novel or non-standard way o Example considerations include: Will a product that implements the protocol call any crypto APIs or methods as part of the protocol implementation? Does the protocol depend on any other protocol used for authentication or authorization? Will the protocol define storage formats for cryptographic elements, such as keys? Self-signed certificates are not recommended for production environments. Use of a self-signed certificate, like use of a raw cryptographic key, does not inherently provide users or administrators any basis for making a trust decision. In contrast, use of a certificate rooted in a trusted certificate authority makes clear the basis for relying on the associated private key and enables revocation and updates in the event of a security failure.
 DPAPI/DPAPI-NG For data that needs to be persisted across system reboots: Microsoft is deeply committed to making the online world safer for everyone. Our company's cybersecurity strategies have evolved from the unique visibility we have into the rapidly evolving cyberthreat landscape.
 Cybersecurity threat actors and motivations Cybersecurity is a shared responsibility, which impacts us all. Today, a single breach, physical or virtual, can cause millions of dollars of damage to an organization and potentially billions in financial losses to the global economy. Every day, we see reports of cybercriminals targeting businesses and individuals for financial gain or socially-motivated purposes. Add to these threats those by nation-state actors seeking to disrupt operations, conduct espionage, or generally undermine trust. In this brief, we share the state of online security, threat actors and the sophisticated tactics they employ to advance their goals, and how Microsoft's Cyber Defense Operations Center combats these threats and helps customers protect their sensitive applications and data. Innovation in the attack space across people, places, and processes is a necessary and continual investment we all need to make, as adversaries continue to evolve in both determination and sophistication. In response to increased investments in defense strategies by many organizations, attackers are adapting and improving tactics at breakneck speed. Fortunately, cyberdefenders like Microsoft's global information security teams are also innovating and disrupting long-reliable attack methods with ongoing, advanced training and modern security technologies, tools, and processes. The Microsoft Cyber Defense Operations Center (CDOC) is one example of the more than $1 billion we invest each year on security, data protection, and risk management. The CDOC brings together cybersecurity specialists and data scientists in a 24x7 facility to combat threats in real-time. We are connected to more than 3,500 security professionals globally across our product development teams, information security groups, and legal teams to protect our cloud infrastructure and services, products and devices, and internal resources. Microsoft has invested more than $15 billion in our cloud infrastructure, with over 90 percent of Fortune 500 companies using the Microsoft cloud. Today, we own and operate one of the world's largest cloud footprints with more than 100 geo-distributed datacenters, 200 cloud services, millions of devices, and a billion customers around the globe. The first step to protecting people, devices, data, and critical infrastructure is to understand the different types of threat actors and their motivations. Cybercriminals Cybercriminals span several subcategories though they often share common motivations-financial, intelligence, and/or social or political gain. Their approach is usually direct-by infiltrating a financialdata system, skimming micro-amounts too small to detect and exiting before being discovered. Maintaining a persistent, clandestine presence is critical to meeting their objective. Their approach may be an intrusion that diverts a large financial payout through a labyrinth of accounts to evade tracking and intervention. At times, the goal is to steal intellectual property that the target possesses so the cybercriminal acts as an intermediary to deliver a product design, software source code, or other proprietary information that has value to a specific entity. More than half of these activities are perpetrated by organized criminal groups. Nation-state actors Nation-state actors work for a government to disrupt or compromise targeted governments, organizations, or individuals to gain access to valuable data or intelligence. They are engaged in international affairs to influence and drive an outcome that may benefit a country or countries. A nation-state actor's objective is to disrupt operations, conduct espionage against corporations, steal secrets from other governments, or otherwise undermine trust in institutions. They work with large resources at their disposal and without fear of legal retribution, with a toolkit that spans from simple to highly complex. Nation-state actors can attract some of the most sophisticated cyberhacking talent and may advance their tools to the point of weaponization. Their intrusion approach often involves an advanced persistent threat using supercomputing power to brute-force break credentials through millions of attempts at arriving at the correct password. They may also use hyper-targeted phishing attacks to attract an insider into revealing their credentials.
 Insider Insider threats are particularly challenging due to the unpredictability of human behavior. The motivation for an insider maybe opportunistic and for financial gain. However, there are multiple causes for potential insider threats, spanning from simple carelessness to sophisticated schemes. Many data breaches resulting from insider threats are completely unintentional due to accidental or negligent activity that puts an organization at risk without being aware of the vulnerability.
 Hacktivists Hacktivists focus on political and/or sociallymotivated attacks. They strive to be visible and recognized in the news to draw attention to themselves and their cause. Their tactics include distributed denial-of-service (DDoS) attacks, vulnerability exploits or defacing an online presence. A connection to a social or political issue can make Threat actor techniques any company or organization a target. Social media enables hacktivists to quickly evangelize their cause and recruit others to participate. Adversaries are skilled at finding ways to penetrate an organization's network despite the protections in place using various sophisticated techniques. Several tactics have been around since the early days of the Internet, though others reflect the creativity and increasing sophistication of today's adversaries.
 Social engineering Social engineering is a broad term for an attack that tricks users into acting or divulging information they would otherwise not do. Social engineering plays on the good intentions of most people and their willingness to be helpful, to avoid problems, to trust familiar sources, or to potentially gain a reward. Other attack vectors can fall under the umbrella of social engineering, but the following are some of the attributes that make social engineering tactics easier to recognize and defend against: Malware has been with us since the dawn of computing. Today, we're seeing a strong up-tick in ransomware and malicious code specifically intended to encrypt devices and data. Cybercriminals then demand payment in cryptocurrency for the keys to unlock and return control to the victim. This can happen at an individual level to your computer and data files, or now more frequently, to an entire enterprise. The use of ransomware is particularly pronounced in the healthcare field, as the life-ordeath consequences these organizations face make them highly sensitive to network downtime. Supply chain inser tion Supply chain inser tion is an example of a creative approach to injecting malware into a network. For
 Phishing emails Phishing emails are an effective tool because they play against the weakest link in the security chain-everyday users who don't think about network security as top-of-mind. A phishing campaign may invite or frighten a user to inadvertently share their credentials by tricking them into clicking a link they believe is a legitimate site or downloading a file that contains malicious code. Phishing emails used to be poorly written and easy to recognize. Today, adversaries have become adept at mimicking legitimate emails and landing sites that are difficult to identify as fraudulent. Identity spoofing Identity spoofing involves an adversary masquerading as another legitimate user by falsifying the information presented to an application or network resource. An example is an email that arrives seemingly bearing the address of a colleague requesting action, but the address is hiding the real source of the email sender. Similarly, a URL can be spoofed to appear as a legitimate site, but the actual IP address is actually pointing to a cybercriminal's site. example, by hijacking an application update process, an adversary circumvents anti-malware tools and protections. We are seeing this technique become more common and this threat will continue to grow until more comprehensive security protections are infused into software by application developers. Man-in-the-middle Man-in-the-middleattacks involve an adversary inserting themselves between a user and a resource they are accessing, thereby intercepting critical information such as a user's login credentials. For example, a cybercriminal in a coffee shop may employ key-logging software to capture a users' domain credentials as they join the wifi network. The threat actor can then gain access to the user's sensitive information, such as banking and personal information that they can use or sell on the dark web. Distributed Denial of Ser vice (DDoS) Distributed Denial of Ser vice (DDoS)attacks have been around more than a decade and are massive attacks are becoming more common with the rapid growth of the Internet of Things (IoT). When using this technique, an adversary overwhelms a site by bombarding it with malicious traffic that displaces legitimate queries. Previously planted malware is often used to hijack an IoT device such as a webcam or smart thermostat. In a DDoS attack, incoming traffic from different sources flood a network with numerous requests. This overwhelms servers and denies access from legitimate requests. Many attacks involve forging of IP sender addresses (IP address spoofing) also, so that the location of the attacking machines cannot easily be identified and defeated. Often a denial of service attack is used to cover or distract from a more deceptive effort to penetrate an organization. In most cases, the objective of the adversary is to gain access to a network using compromised credentials, then move laterally across the network to gain access to more "powerful" credentials that are the keys to the most sensitive and valuable information within the organization.
 The militarization of cyberspace The Microsoft cybersecurity posture
 PROTECT Protect The growing possibility of cyberwarfare is one of the leading concerns among governments and citizens today. It involves nation-states using and targeting computers and networks in warfare. Both offensive and defensive operations are used to conduct cyberattacks, espionage and sabotage. Nationstates have been developing their capabilities and engaged in cyberwarfare either as aggressors, defendants, or both for many years. New threat tools and tactics developed through advanced military investments may also be breached and cyberthreats can be shared online and weaponized by cybercriminals for further use. While security has always been a priority for Microsoft, we recognize that the digital world requires continuous advances in our commitment in how we protect, detect, and respond to cybersecurity threats. These three commitments define our approach to cyber defense and serve as a useful framework for our discussion of Microsoft's cyber defense strategies and capabilities. Microsoft's first commitment is to protect the computing environment used by our customers and employees to ensure the resiliency of our cloud infrastructure and services, products, devices, and the company's internal corporate resources from determined adversaries. The CDOC teams' protection measures span across all endpoints, from sensors and datacenters to identities and software-as-a-service (SaaS) applications. Defenseindepth-applying controls at multiple layers with overlapping safeguards and risk mitigation strategies-is a best-practice across the industry and it's the approach we take to protect our valuable customer and corporate assets. Microsoft's protection tactics include: Extensive monitoring and controls over the physical environment of our global datacenters including cameras, personnel screening, fences and barriers, and multiple identification methods for physical access. Software-defined networks that protect our cloud infrastructure from intrusions and DDoS attacks. Multi-factor authentication is employed across our infrastructure to control identity and access management. It ensures that critical resources and data are protected by at least two of the following: Something you know (password or PIN) Something you are (biometrics) Non-persistent administration employs just-in-time (JIT) and just-enough administrator (JEA) privileges to engineering staff who manage infrastructure and services. This provides a unique set of credentials for elevated access that automatically expires after a predesignated duration. Proper hygiene is rigorously maintained through upto-date, anti-malware software and adherence to strict patching and configuration management. Microsoft Malware Protection Center's team of researchers identify, reverse engineer, and develop malware signatures and then deploy them across our infrastructure for advanced detection and defense. These signatures are distributed to our responders, customers and the industry through Windows Updates and notifications to protect their devices. Microsoft Security Development Lifecycle (SDL) is a software development process that helps developers build more secure software and address security compliance requirements while reducing development cost. The SDL is used to harden all applications, online services and products, and to routinely validate its effectiveness through penetration testing and vulnerability scanning. Threat modeling and attack surface analysis ensures that potential threats are assessed, exposed aspects of the service are evaluated, and the attack surface is minimized by restricting services or eliminating unnecessary functions. Classifying data according to its sensitivity and taking the appropriate measures to protect it, including encryption in transit and at rest, and enforcing the principle of least-privilege access provides additional protection. • Awareness training that fosters a trust relationship between the user and the security team to develop an environment where users will report incidents and anomalies without fear of repercussion.
 Something you have (smartphone) Having a rich set of controls and a defense-in-depth strategy helps ensure that should any one area fail, there are compensating controls in other areas to help maintain the security and privacy of our customers, cloud services, and our own infrastructure. However, no environment is truly impenetrable, as people will make errors and determined adversaries will continue to look for vulnerabilities and exploit them. The significant investments we continue to make in these protection layers and baseline analysis enables us to rapidly detect when abnormal activity is present.
 DETECT
 Detect
 RESPOND Respond The CDOC teams employ automated software, machine learning, behavioral analysis, and forensic techniques to create an intelligent security graph of our environment. This signal is enriched with contextual metadata and behavioral models generated from sources such as Active Directory, asset and configuration management systems, and event logs. Our extensive investments in security analytics build rich behavioral profiles and predictive models that allow us to "connect the dots" and identify advanced threats that might otherwise have gone undetected, then counter with strong containment and coordinated remediation activities. Microsoft also employs custom-developed security software, along with industryleading tools and machine learning. Our threat intelligence is continually evolving, with automated data-enrichment to more rapidly detect malicious activity and report with high fidelity. Vulnerability scans are performed regularly to test and refine the effectiveness of protective measures. The breadth of Microsoft's investment in its security ecosystem and the variety of signals monitored by the CDOC teams provide a more comprehensive threat view than can be achieved by most service providers. Microsoft's detection tactics include: Monitoring network and physical environments 24x7x365 for potential cybersecurity events. Behavior profiling is based on usage patterns and an understanding of unique threats to our services. Identity and behavioral analytics are developed to highlight abnormal activity. Machine learning software tools and techniques are routinely used to discover and flag irregularities. Advanced analytical tools and processes are deployed to further identify anomalous activity and innovative correlation capabilities. This enables highlycontextualized detections to be created from the enormous volumes of data in near real-time. Automated software-based processes that are continuously audited and evolved for increased effectiveness. Data scientists and security experts routinely work side-by-side to address escalated events that exhibit unusual characteristics requiring further analysis of targets. They can then determine potential response and remediation efforts.
 Cyberdefense for our customers When Microsoft detects abnormal activity in our systems, it triggers our response teams to engage and quickly respond with precise force. Notifications from software-based detection systems flow through our automated response systems using risk-based algorithms to flag events requiring intervention from our response team. Mean-Time-to-Mitigate is paramount and our automation system provides responders with relevant, actionable information that accelerates triage, mitigation, and recovery. To manage security incidents at such a massive scale, we deploy a tiered system to efficiently assign response tasks to the right resource and facilitate a rational escalation path. Microsoft's response tactics include: Automated response systems use risk-based algorithms to flag events requiring human intervention. Automated response systems use risk-based algorithms to flag events requiring human intervention. Well-defined, documented, and scalable incident response processes within a continuous improvement model helps to keep us ahead of adversaries by making these available to all responders. Subject matter expertise across our teams, in multiple security areas, provides a diverse skill set for addressing incidents. Security expertise in incident response, forensics, and intrusion analysis; and a deep understanding of the platforms, services, and applications operating in our cloud datacenters. Wide enterprise searching across cloud, hybrid and on-premises data and systems to determine the scope of an incident. Deep forensic analysis for major threats are performed by specialists to understand incidents and to aid in their containment and eradication. • Microsoft's security software tools, automation and hyper-scale cloud infrastructure enable our security experts to reduce the time to detect, investigate, analyze, respond, and recover from cyberattacks. Penetration testing is employed across all Microsoft products and services through ongoing Red Team/Blue Team exercises to unearth vulnerabilities before a real adversary can leverage those weak points for an attack. We   SC EN A RIO N UM B ER SC EN A RIO N UM B ER AT TA C K AT TA C K O VERVIEW O VERVIEW VIO L AT ES T RA DIT IO N A L VIO L AT ES T RA DIT IO N A L T EC H N O LO GIC A L N OT IO N T EC H N O LO GIC A L N OT IO N O F O F A C C ESS/ A UT H O RIZ AT IO N ? A C C ESS/ A UT H O RIZ AT IO N ?
 [ 6 ] 6 Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training[19]: The authors propose Highly Confident Near Neighbor (HCNN), a framework that combines confidence information and nearest neighbor search, to reinforce adversarial robustness of a base model. This can help distinguish between right and wrong model predictions in a neighborhood of a point sampled from the underlying training distribution.
 can be used to harden DNN models by detecting adversarial examples. It reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample. By comparing a DNN model's prediction on the original input with that on the squeezed input, feature squeezing can help detect adversarial examples. If the original and squeezed examples produce substantially different outputs from the model, the input is likely to be adversarial. By measuring the disagreement among predictions and selecting a threshold value, system can output the correct prediction for legitimate examples and rejects adversarial inputs. d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s Se v e r i t y Se v e r i t y
 M i t i g a t i o n s M i t i g a t i o n s T r a d i t i o n a l P a r a l l e l s T r a d i t i o n a l P a r a l l e l s Se v e r i t y Se v e r i t y
 # 7 7 Adversarial Example in the Physical domain (bits->atoms) Ex a m p l e s Ex a m p l e s
 rd Strong client<->server mutual authentication and access control to model interfaces Takedown of the offending accounts.
 Vectors and Machine Learning Techniques A. Feature Vectors and Machine Learning Techniquesunsupervised machine learning techniques, and study how much data is needed to train their models.
 -dependent noise: Class-dependent noise refers to the noise that depends on the true class of the instances. In this type of noise, the probability of mislabeling in class SBR is p and the probability of mislabeling in class NSBR is p . We introduce class-dependent noise in our data set by flipping each entry in the data set for which the true label is SBR with probability p . Analogously, we flip the class label of NSBR instances with probability p . -class noise: Single-class noise is a special case of class-dependent noise, where p = 0 and p > 0. Note that for class-independent noise we have p = p = p . the impact of different noise types and levels in the training of SBR classifiers. In our experiments we set 25% of the data set as test data, 10% as validation and 65% as training data.
 Noise: class-independentWe compare the performance of our three classifiers for the case where the training set is corrupted by a classindependent noise. We measure the AUC for each model trained with different levels of p in the training data.br
 brFig. 1 . 1 Fig.1. Variation of AUC-ROC in class-independent noise. For a noise level p =0.5 the classifier acts like a random classifier, i.e. AUC≈0.5. But we can observe that for lower noise levels (p ≤0.30), the logistic regression learner presents a better performance compared to the other two models. However, for 0.35≤ p ≤0.45 naive Bayes learner presents better AUCROC metrics.br
 this paper is twofold.
 Figure 1 : 1 : 11 Figure 1: Security Protocol Support by OS Version Figure 1: Security Protocol Support by OS Version
 Figure 1 : 1 Figure 1: Default TLS Version selection and Certificate Binding Functionality https://secure.contoso.com directs your customers to a service endpoint supporting only TLS 1.2 and above.
 Figure 2 : 2 Figure 2: Disable Legacy TLS feature enforcing minimum TLS version for a selected certificate, Secure.contoso.com.
 Option # 2 :Option # 3 : 23 PowerShell (Available April 2020) Option #2: PowerShell (Available April 2020) [Microsoft.Web.Administration.SslFlags]::DisableLegacyTLS $Sni = [Microsoft.Web.Administration.SslFlags]::Sni $Sni\_CCS = [Microsoft.Web.Administration.SslFlags]::Sni + [Microsoft.Web.Administration.SslFlags]::CentralCertStore $CCS = [Microsoft.Web.Administration.SslFlags]::CentralCertStore $DisableLegacyTLS = [Microsoft.Web.Administration.SslFlags]::DisableLegacyTLS $storeLocation = "Cert:\LocalMachine\My" $BindingInformation = "\*:443:" $siteName = "contoso" $Thumbprint = $certificate.ThumbPrint New-IISSite $siteName "$env:systemdrive\inetpub\wwwroot" "\*:443:secure.contoso.com" https $certificate.Thumbprint $DisableLegacyTLS $storeLocation -passthru New-IISSiteBinding -Name "Default Web Site" -BindingInformation $BindingInformation -CertificateThumbPrint $certificate.Thumbprint -Protocol https -SslFlag $DisableLegacyTLS, $CCS -Force -verbose In PowerShell you can reference SSL flags like this: It's convenient to create shorter named variables for them: An example of creating a site binding to a new site and disabling legacy TLS: New-IISSite with Sslflag DisableLegacyTLS property value: An example of adding a site binding to an existing site and disabling legacy TLS: Additionally, one can troubleshoot and test this feature with Netsh: Adding a new binding: C++ HTTP.sys APIs (Available Now) Option #3: C++ HTTP.sys APIs (Available Now) Next steps for TLS version enforcement netsh http add sslcert <regular parameters> disablelegacytls=enable Updating an existing binding: netsh http update sslcert <regular parameters> disablelegacytls=enable Check whether it is set on a binding: netsh http show sslcert <regular parameters> Watch for Disable Legacy TLS Versions : Set/Not Set Along with Disable Legacy TLS, the following additions have been made to HTTP.sys: HTTP_SERVICE_CONFIG_SSL_PARAM.DefaultFlags now supports the following new values:
 Microsoft is committed to providing an unprecedented level of transparency through the Government Security Program (GSP), aimed at helping customers gain confidence in the integrity and assurance of the products and services on which they rely. To support those efforts, Microsoft has opened five Transparency Centers throughout the world: provide GSP participants with an opportunity to visit a secure facility to conduct deep levels of source code inspection and analysis. Participants have access to source code and an environment for in-depth inspection with advanced tools. In addition, participants can compile components and use additional tools for a deeper understanding of the source code. Currently, the Transparency Centers provide source code for products such as Windows, Windows Ser ver, Office, Exchange Ser ver, SQL Ser ver, and SharePoint Windows, Windows Ser ver, Office, Exchange Ser ver, SQL Ser ver, and SharePoint Ser ver Ser ver.Each visit to a Transparency Center is tailored to the unique goals of what an agency is looking to accomplish. Visits can last from one day to two weeks, depending on an agency's needs, and are scheduled based on facility availability. Face-to-face or teleconference exchanges with Microsoft engineers may also be available and can be beneficial during Transparency Center visits for agencies who have also joined the Technical Data portion of the program.The environment and tools for source code evaluation include:Private network with one server rack and eight clients OpenGrok open source search and cross-reference engine SysInternals Example use cases of Transparency Centers Contact Us Tools provided by the participant and approved by Microsoft HexRays IDA Disassembler and Decompiler PowerShell v4.0 MSDN Documentation Evaluation of Cryptography Next Generation implementation and preparation for national cryptography implementation Review of SSL and TCP/IP implementation Inspection of the source of random number generators Walkthrough of Microsoft's build process Assessment of individual binaries to compare against shipped binaries Contact your local Microsoft representative to learn more about the Government Security Program Online Access to Source Code 2/19/2022 • 2 minutes to read Example cases of Online Source:
 Check code for specific assurance questions Examine specific function calls in source code As a reference to understand how a feature or function works Contact your local Microsoft representative to learn more about the Government Security Program Technical Data 2/19/2022 • 2 minutes to read
 Written materials Direct dialogue with Microsoft engineers and security experts Early access to documentation about Microsoft's products and services In-person exchanges with Microsoft engineers and security experts are sometimes scheduled in Microsoft facilities and cover security questions about Microsoft products and services. These meetings, commonly referred to as technical trips, provide deep technical conversations about topics of interest. To plan and facilitate these, the Microsoft local GSP representative works closely with the agency to understand what it is looking to accomplish and to develop a customized agenda. Technical trips are accommodated based on Microsoft engineering team availability. One of the frequent uses of this offering is to review security content related to Microsoft's cloud services. This includes access to the Microsoft Service Trust Platform where we share important third-party security and compliance reports, as well as security details about Microsoft's cloud services. Technical documentation and written materials needed to understand a security question Early releases of white papers Common Criteria evaluation documents for Windows Engagements with engineers in many different formats including Technical Adoption Programs (TAP) Cloud service security, privacy and compliance documentation such as audit reports and risk assessments Technical trip sample topics: Azure Incident Response Process Bitlocker Encryption Windows Telemetry Cyber Threat Intelligence Contact your local Microsoft representative to learn more about the Government Security Program SDL Security Bug Bar ( Sample )
 A UT H EN T IC AT ED VS. A UT H EN T IC AT ED VS. A N O N Y M O US AT TA C K A N O N Y M O US AT TA C K DEFA ULT / C O M M O N VS. DEFA ULT / C O M
 from a shared secret (the output of a key agreement) use the BCryptDeriveKey API with one of the following algorithms: SSL, TLS, or DTLS should fully verify the X.509 certificates of the entities they connect to. This includes verification of the certificates': Domain name.Validity dates (both beginning and expiration dates).
 are often asked what tools and processes our customers can adopt for their own environment and how Microsoft might help in their implementation. Microsoft has consolidated many of the cyberdefense products and services we use in the CDOC into a range Best practices to protect your environmentI N V E S T I N Y O U R I N V E S T I N Y O U R P L AT F O R M P L AT F O R M I N V E S T I N Y O U R I N V E S T I N Y O U R I N S T R U M E N TAT I O N I N S T R U M E N TAT I O N I N V E S T I N Y O U R I N V E S T I N Y O U

 How much knowledge is required to mount this attack -blackbox or whitebox? In Blackbox style attacks., the attacker does NOT have direct access to the training data, no knowledge of the ML algorithm used and no access to the source code of the model. The attacker only queries the model and observes the response. In a whitebox style attack the attacker has knowledge of either ML algorithm or access to the model source code.3. Commentary on if the attacker is violating traditional technological notion of access/authorization. 2. Backdoor ML 1 Perturbation attacks In perturbation style Malicious ML provider Integrity Poisoning attacks The goal of the Integrity Yes Image: Noise is In a medical dataset backdoors algorithm to attacks, the attacker attacker is to added to an X-ray where the goal is to activate with a specific stealthily modifies contaminate the image, which makes predict the dosage of the query to get a machine model trigger the predictions go anticoagulant drug desired response generated in the from normal scan to Warfarin using training phase, so abnormal [1] demographic 11 Exploit Software that predictions on Attacker uses traditional Yes [Blackbox] information, etc. Dependencies new data will be software exploits like buffer Text translation: Researchers overflow to confuse/control modified in the Specific introduced malicious testing phase ML systems characters are samples at 8% Targeted: In manipulated to poisoning rate, which targeted result in incorrect changed dosage by poisoning Unintended Failures Summary attacks, the translation. The 75.06% for half of attack can patients[4][Blackbox] SC EN A RIO # SC EN A RIO # FA IL URE FA IL URE attacker wants to misclassify O VERVIEW suppress specific O VERVIEW word or can even 1 Perturbation attack specific examples Attacker modifies the query No remove the word 12 to get appropriate response Reward Hacking Reinforcement Learning (RL) systems act in unintended ways because of completely[2] [Blackbox and Indiscriminate: 2 Poisoning attack Attacker contaminates the mismatch between stated reward and Whitebox] No true reward training phase of ML Speech: 13 systems to get intended result Side Effects RL system disrupts the environment as Researchers showed how it tries to attain its goal given a speech 3 Model Inversion Attacker recovers the secret No waveform, 14 features used in the model Distributional shifts The system is tested in one kind of another by through careful queries environment, but is unable to adapt to waveform can be 4 5 15 16 Membership Inference Model Stealing Natural Adversarial Examples Attacker can infer if a given data record was part of the model's training dataset or not Attacker is able to recover the model through carefully-crafted queries changes in other kinds of environment No No Without attacker perturbations, the ML system fails owing to hard negative mining Common Corruption exactly replicated Model Inversion The private features Confidentiality; Researchers were but transcribes used in machine able to recover into a totally learning models can private training data different text[3] be recovered used to train the [Whitebox but algorithm[6] The may be extended authors were able to to blackbox] reconstruct faces, by The system is not able to handle just the name and common corruptions and access to the model perturbations such as tilting, zooming, to the point where or noisy images. Mechanical turks 6 Reprogramming ML system Repurpose the ML system No could use the photo 17 to perform an activity it was Incomplete Testing The ML system is not tested in the to identify an not programmed for realistic conditions that it is meant to individual from aline- operate in. up with 95% 7 Details on Intentionally-Motivated Failures Adversarial Example in Physical Domain Attacker brings adversarial examples into physical domain to subvertML system e.g: 3d printing special eyewear to fool facial recognition system SC EN A RIO # SC EN A RIO # AT TA C K C L A SS AT TA C K C L A SS DESC RIP T IO N DESC RIP T IO N T Y P E O F T Y P E O F C O M P RO M ISE C O M P RO M ISE accuracy. The authors No were also able to extract specific information. [Whitebox and SC EN A RIO SC EN A RIO Blackbox][12] 8 Malicious ML provider recovering training data Membership The attacker can Malicious ML provider can Confidentiality Inference attack determine whether a query the model used by given data record customer and recover was part of the customer's training data model's training Researchers were Yes able to predict a patient's main procedure(e.g: Surgery the patient dataset or not went through) based 9 Attacking the ML supply Attacker compromises the Yes on the attributes (e.g: chain ML models as it is being age,gender, hospital) downloaded for use [7][Blackbox]
 and Mitigations in this Document Related Threats and Mitigations in this Document Example Attacks Example Attacks If confidence levels of your model output suddenly drop, can you find out how/why, as well as the data that caused it?Have you defined a well-formed input for your model? What are you doing to ensure inputs meet this format and what do you do if they don't? If your outputs are wrong but not causing errors to be reported, how would you know? Do you know if your training algorithms are resilient to adversarial inputs on a mathematical level? How do you recover from adversarial contamination of your training data? -Can you isolate/quarantine adversarial content and re-train impacted models? -Can you roll back/recover to a model of a prior version for re-training? Are you using Reinforcement Learning on uncurated public content? Start thinking about the lineage of your data -were you to find a problem, could you track it to its introduction into the dataset? If not, is that a problem? Know where your training data comes from and identify statistical norms in order to begin understanding what anomalies look like -What elements of your training data are vulnerable to outside influence?Related ThreatsIdentify actions your model(s) or product/service could take which can cause customer harm online or in the physical domain
 Summary Summary Questions to Ask in a Security Review Questions to Ask in a Security Review Related Threats and Mitigations in this Document Related Threats and Mitigations in this Document -Who can contribute to the data sets you're training from? Example Attacks Example Attacks -How would you you attack your sources of training data to harm a competitor? Adversarial Perturbation (all variants) Data Poisoning (all variants) Forcing benign emails to be classified as spam or causing a malicious example to go undetected Attacker-crafted inputs that reduce the confidence level of correct classification, especially in high- consequence scenarios Attacker injects noise randomly into the source data being classified to reduce the likelihood of the Identify all sources of AI/ML dependencies as well as frontend correct classification being used in the future, effectively dumbing down the model presentation layers in your data/model supply chain Contamination of training data to force the misclassification of select data points, resulting in specific actions being taken or omitted by a system Can an attacker cause reputational damage or PR backlash to your product by forcing it to carry out specific actions? How do you handle properly formatted but overtly biased data, such as from trolls? For each way to interact with or query your model is exposed, can that method be interrogated to disclose training data or model functionality? Membership Inference Model Inversion Model Stealing Left unmitigated, attacks on AI/ML systems can find their way over to the physical world. Any scenario which can be twisted to psychologically or physically harm users is a catastrophic risk to your product/service. This extends to any sensitive data about your customers used for training and design choices that can leak those private data points.Do you train with adversarial examples? What impact do they have on your model output in the physical domain?What does trolling look like to your product/service? How can you detect and respond to it? What would it take to get your model to return a result that tricks your service into denying access to legitimate users? What is the impact of your model being copied/stolen? Can your model be used to infer membership of an individual person in a particular group, or simply in the training data?
 Summary Summary Questions to Ask in a Security Review Questions to Ask in a Security Review Related Threats and Mitigations in this Document Related Threats and Mitigations in this Document Reconstruction and extraction of training data by repeatedly querying the model for maximum confidence results Duplication of the model itself by exhaustive query/response matching Querying the model in a way that reveals a specific element of private data was included in the training set Neural Net Reprogramming Adversarial Examples in the physical domain Malicious ML Providers Recovering Training Data Attacking the ML Supply Chain Backdoored Model Compromised ML-specific dependencies Self-driving car being tricked to ignore stop signs/traffic lights Example Attacks Example Attacks Conversational bots manipulated to troll benign users Malicious MLaaS provider trojans your model with a specific bypass Adversary customer finds vulnerability in common OSS dependency you use, uploads crafted training data payload to compromise your service Unscrupulous partner uses facial recognition APIs and creates a presentation layer over your service to produce Deep Fakes. AI/ML-specific Threats and their Mitigations #1: Adversarial Perturbation Description Description What kind of telemetry do you need to prove the trustworthiness of your model output to your customers? Identify all 3 party dependencies in your ML/Training data supply chain -not just open source software, but data providers as well Variant #1a: Targeted misclassification Examples Examples Many attacks in AI and Machine Learning begin with legitimate access to APIs which are surfaced to provide query access to a model. Because of the rich sources of data and rich user experiences involved here, authenticated but "inappropriate" (there's a gray area here) 3 -party access to your models is a risk because of the ability to act as a presentation layer above a Microsoft-provided service.rd Which customers/partners are authenticated to access your model or service APIs? -Can they act as a presentation layer on top of your service? -Can you revoke their access promptly in case of compromise? -What is your recovery strategy in the event of malicious use of your service or dependencies? Can a 3 party build a façade around your model to re-purpose it and harm Microsoft or its customers? rd Do customers provide training data to you directly? -How do you secure that data? -What if it's malicious and your service is the target? What does a false-positive look like here? What is the impact of a false-negative? Can you track and measure deviation of True Positive vs False Positive rates across multiple models? rd -Why are you using them and how do you verify their trustworthiness? Are you using pre-built models from 3 parties or submitting training data to 3 party MLaaS providers? rd rdInventory news stories about attacks on similar products/services. Understanding that many AI/ML threats transfer between model types, what impact would these attacks have on your own products?
 Identifying security bug reports (SBRs) is a vital step in the software development life-cycle. In supervised machine learning based approaches, it is usual to assume that entire bug reports are available for training and that their labels are noise free. To the best of our knowledge, this is the first study to show that accurate label prediction is possible for SBRs even when solely the title is available and in the presence of label noise. 2/19/2022 • 22 minutes to read Mayana Pereira Scott Christiansen CELA Data Science Customer Security and Trust Microsoft Microsoft Abstract - I. INTRODUCTION
 in software built on top of Microsoft operating systems, following up with details on product changes and new features delivered by Microsoft to protect your own customers and online services. It is intended to be used as a starting point for building a migration plan to a TLS 1.2+ network environment. While the solutions discussed here may carry over and help with removing TLS 1.0 usage in non-Microsoft operating systems or crypto libraries, they are not a focus of this document. Solving the TLS 1.0 Problem, 2nd Edition 2/19/2022 • 11 minutes to read By Andrew Marshall Principal Security Program Manager Microsoft Corporation Executive Summary This document presents the latest guidance on rapidly identifying and removing Transport Layer Security (TLS) protocol version 1.0 dependencies 1] John Anvik, Lyndon Hiew, and Gail C Murphy. Who should fix this bug? In Proceedings of the 28th international conference on Software engineering, pages 361-370. ACM, 2006. [2]  Diksha Behl, Sahil Handa, and Anuja Arora. A bug mining tool to identify and analyze security bugs using naive bayes and tf-idf. In Optimization, Reliabilty, and Information Technology (ICROIT), 2014 International Conference on, pages 294-299. IEEE, 2014.[3] Nicolas Bettenburg, Rahul Premraj, Thomas Zimmermann, and Sunghun Kim. Duplicate bug reports considered harmful really? In Software maintenance, 2008. ICSM 2008. IEEE international conference on, pages 337-345. IEEE, 2008. TLS 1.0 is a security protocol first defined in 1999 for establishing encryption channels over computer networks. Microsoft has supported this protocol since Windows XP/Server 2003. While no longer the default security protocol in use by modern OSes, TLS 1.0 is still supported for backwards compatibility. Evolving regulatory requirements as well as new security vulnerabilities in TLS 1.0 provide corporations with the incentive to disable TLS 1.0 entirely.
 Many operating systems have outdated TLS version defaults or support ceilings that need to be accounted for. Usage of Windows 8/Server 2012 or later means that TLS 1.2 will be the default security protocol version: *TLS 1.1/1.2 can be enabled on Windows Server 2008 via this optional Windows Update package. For more information on TLS 1.0/1.1 deprecation in IE/Edge, see Modernizing TLS connections in Microsoft Edge and Internet Explorer 11, Site compatibility-impacting changes coming to Microsoft Edge and Disabling TLS/1.0 and TLS/1.1 in the new Edge Browser A quick way to determine what TLS version will be requested by various clients when connecting to your online services is by referring to the Handshake Simulation at Qualys SSL Labs. This simulation covers client OS/browser combinations across manufacturers. See Appendix A at the end of this document for a detailed example showing the TLS protocol versions negotiated by various simulated client OS/browser combinations when connecting to www.microsoft.com.
 . Applications must add code to support TLS 1.2 via WinHttpSetOption 6. To cover all the bases, scan source code and online service configuration files for the patterns below corresponding to enumerated type values commonly used in TLS hardcoding:The recommended solution in all cases above is to remove the hardcoded protocol version selection and defer to the operating system default. If you are using DevSkim, click here to see rules covering the above checks which you can use with your own code. a. SecurityProtocolType b. SSLv2, SSLv23, SSLv3, TLS1, TLS 10, TLS11 c. WINHTTP_FLAG_SECURE_PROTOCOL_ d. SP_PROT_ e. NSStreamSocketSecurityLevel f. PROTOCOL_SSL or PROTOCOL_TLS Testing with TLS 1.2+ reg add HKLM\SOFTWARE\Microsoft\.NETFramework4.0.30319 /v SystemDefaultTlsVersions /t REG_DWORD /d 1 /f /reg:64 reg add HKLM\SOFTWARE\Microsoft\.NETFramework4.0.30319 /v SystemDefaultTlsVersions /t REG_DWORD /d 1 /f /reg:32 Rebuild/retarget managed applications using the latest .Net Framework version Applications using .NET framework versions prior to 4.7 may have limitations effectively capping support to TLS 1.0 regardless of the underlying OS defaults. Refer to the below diagram and https://docs.microsoft.com/dotnet/framework/network-programming/tls for more information. 4. Disable FIPS Mode if it is enabled due to the potential for conflict with settings required for explicitly disabling TLS 1.0/1.1 in this document. See Appendix B for more information.5. Update and recompile any applications using WinHTTP hosted on Server 2012 or older. a. Managed apps -rebuild and retarget against the latest .NET Framework version Update Windows PowerShell scripts or related registry settings 1. Modify the script in question to include the following: [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12; 2. Add a system-wide registry key (e.g. via group policy) to any machine that needs to make TLS 1.2 connections from a .NET app. This will cause .NET to use the "System Default" TLS versions which adds TLS 1.2 as an available protocol AND it will allow the scripts to use future TLS Versions when the OS supports them. (e.g. TLS 1.3) bWindows PowerShell uses .NET Framework 4.5, which does not include TLS 1.2 as an available protocol. To work around this, two solutions are available: Solutions (1) and (2) are mutually-exclusive, meaning they need not be implemented together. SystemDefaultTLSVersion takes precedence over app-level targeting of TLS versions. The recommended best practice is to always defer to the OS default TLS version. It is also the only crypto-agile solution that lets your apps take advantage of future TLS 1.3 support. If you are targeting older versions of .NET Framework such as 4.5.2 or 3.5, then by default your application will use the older and not recommended protocols such as SSL 3.0 or TLS 1.0. It is strongly recommended that you upgrade to newer versions of .NET Framework such as .NET Framework 4.6 or set the appropriate registry keys for 'UseStrongCrypto'.
 enforcement capabilities now available per certificate binding on Windows Server 2019 2/19/2022 • 5 minutes to read This post is authored by Andrew Marshall, Principal Security Program Manager, Customer Security and Trust Gabriel Montenegro, Principal Program Manager, Core Networking Niranjan Inamdar, Senior Software Engineer, Core Networking Michael Brown, Senior Software Engineer, Internet Information Services Ivan Pashov, Principal Software Engineering Lead, Core Networking August 2019 Feature scenario details
 Government Security Program (GSP) is to build trust through transparency. Microsoft recognizes that people will only use technology they trust and we strive to demonstrate our commitment to building this trust through our transparency, privacy, compliance, and security principles. Since the program's inception in 2003, Microsoft has provided visibility into our technology which governments and international organizations can use to help protect themselves and their citizens.The GSP is designed to provide participants with the confidential security information and resources they need to trust Microsoft's products and services. Participants currently include over 40 countries and international organizations represented by more than 70 agencies. Participation enables controlled access to source code, exchange of threat and vulnerability information, engagement on technical content about Microsoft's products and services, and access to five globally-distributed Transparency Centers, which are located in the United States, Belgium, Singapore, Brazil, and China.Enabling trust and transparency Providing access to security information about Microsoft products and services Providing data to improve protection of government information technology against cyber threats Fostering collaboration between Microsoft security teams and government cybersecurity expertsThe program is designed to allow participants to focus on topics that are most important to them. Agencies work with local Microsoft representatives to ensure their goals are best supported by the right combination of GSP offerings. Microsoft offers four different offerings which address a variety of priorities, including access to source code, threat and vulnerability information, technical content about Microsoft's products and services, and access to Microsoft engineering and security experts. 2/19/2022 • 3 minutes to read Our commitment to trust and transparency The mission of Microsoft's Transparency Centers 2/19/2022 • 2 minutes to read Qualifying agencies participate at no charge. Program criteria include requirements that GSP participants must be A Secure Facility for Inspection and Analysis The Purpose of the GSP is to help governments protect themselves and their citizens by GSP Offerings Center Visits Environment and Tools A legal entity of a national government or recognized international organization. Able to sign an agreement on behalf of that government. Compliant with permitted law enforcement activities . Able to adequately protect intellectual property and confidential information.
 has to include authenticating by the network. This implies that logging of some type must be able to occur so that the attacker can be identified. Any features that are active out of the box or that reach more than 10 percent of users. scenario scenario Any features that require special customization or use cases to enable, reaching less than 10 percent of users. Examples: Computer that is configured to run software that awaits and fulfills requests from client processes that run on C L IEN T C L IEN T C L IEN T C L IEN T C L IEN T C L IEN T other computers. Displaying a different URL in the browser's address bar from the URL of the site that the browser is Denial of service Denial of service **Critical.** *A security vulnerability that would be rated as Moderate Low actually displaying in a Permanent DoS requires cold reboot or Temporary DoS requires restart of having the highest potential for damage.* default/common default/common causes Blue Screen/Bug Check. application. scenario scenario Example: Example: **Important.** *A security vulnerability that would be rated as Spoofing having significant potential for damage, but less than Critical.* Displaying a window over Opening a Word Opening a HTML the browser's address bar document causes the document causes Internet that looks identical to an machine to Blue Explorer to crash address bar but displays bogus data in a Screen/Bug Check. default/common default/common scenario Cases where the attacker can read different from but visually identical to the scenario Information disclosure (targeted) Ability for attacker to present a UI that is **Moderate.** *A security vulnerability that would be rated as Displaying a different file information on the system from known UI that is a single part of a bigger attack having moderate potential for damage, but less than Important.* name in a "Do you want to locations, including system information scenario. run this program?" dialog box than that of the file that was not intended or designed to be exposed. Example: that will actually be loaded in a default/common default/common "malicious" web site, click Examples: User has to go a **Low.** *A security vulnerability that would be rated as having low potential for damage.* targeted information disclosure targeted information disclosure Ability to intentionally select (target) desired information. Spoofing scenario scenario Display a "fake" login prompt to gather user or account credentials Targeted existence of file on a button in spoofed dialog box, and is then Targeted file version susceptible to a number vulnerability based on a different browser bug temporar y DoS temporar y DoS Tampering Ability for attacker to present a UI that is Tampering Permanent modification of any user data or data used to make trust decisions in a common or default scenario that persists after restarting the OS/application. different from but visually identical to the defined as anything a user is commonly OS/application. specific scenario. "Accustomed to trust" is does not persist after restarting the UI that users are accustomed to trust in a Temporary modification of any data that A temporary DoS is a situation where the following criteria are met: familiar with based on normal interaction Information disclosure (untargeted) Examples: Web browser cache with the operating system or application but does not typically think of as a "trust Example: poisoning decision." Leak of random heap Modification of significant Examples: memory OS/application settings Web browser cache without user consent poisoning Definition of Terms Modification of user data Security features: Breaking or bypassing any Modification of significant OS/application settings authenticated authenticated security feature provided without user consent Any attack which anonymous anonymous Examples: Disabling or bypassing a firewall with informing user or gaining consent Modification of user data Any attack which does not need to authenticate to complete. Reconfiguring a firewall and allowing connection to other processes Either software that runs locally on a single computer or software that accesses shared resources provided by a client client server over a network. Using weak encryption or keeping the keys stored in plain text default/common default/common AccessCheck bypass Bitlocker bypass; for example not encrypting part of the drive Denial of Service (Server) Matrix Syskey bypass, a way to decode the syskey without the password ser ver ser ver
 versions Symmetric Block Ciphers, Cipher Modes and Initialization Vectors Symmetric Block Ciphers, Cipher Modes and Initialization Vectors This documentation is not an exhaustive reference on the SDL practices at Microsoft. Additional assurance work may be performed by product teams (but not necessarily documented) at their discretion. As a result, this example should not be considered as the exact process that Microsoft follows to secure all products. document contains recommendations and best practices for using encryption on Microsoft platforms. Much of the content here is paraphrased or aggregated from Microsoft's own internal security standards used to create the Security Development Lifecycle. It is meant to be used as a reference when designing products to use the same APIs, algorithms, protocols and key lengths that Microsoft requires of its own products and services.Developers on non-Windows platforms may also benefit from these recommendations. While the API and library names may be different, the best practices involving algorithm choice, key length and data protection are similar across platforms. Some other cipher modes like those included below have implementation pitfalls that make them more likely to be used incorrectly. In particular, the Electronic Code Book (ECB) mode of operation should be avoided. Reusing the same initialization vector (IV) with block ciphers in "streaming ciphers modes" such as CTR may cause encrypted data to be revealed. Additional security review is recommended if any of the below modes are used:All symmetric block ciphers should also be used with a cryptographically strong random number as an initialization vector. Initialization vectors should never be a constant value. See Random Number Generators for recommendations on generating cryptographically strong random numbers.Initialization vectors should never be reused when performing multiple encryption operations, as this can reveal information about the data being encrypted, particularly when using streaming cipher modes like Output Feedback (OFB) or Counter (CTR). Ciphertext Stealing (CTS) Key Lifetimes SC EN A RIO SC EN A RIO M O N VS. T EM P O RA RY DO S VS. T EM P O RA RY DO S VS. P ERM A N EN T P ERM A N EN T RAT IN G RAT IN G XEX-Based Tweaked-Codebook with Ciphertext Stealing (XTS) Authenticated Default/Common Permanent Moderate Authenticated Default/Common Temporary DoS with Moderate amplification Authenticated Introduction Output Feedback (OFB) Default/Common Temporary DoS Low Authenticated Cipher Feedback (CFB) Random Number Generators Scenario Permanent Moderate Counter (CTR) Authenticated Scenario Temporary DoS with Low Counter with CBC-MAC (CCM) amplification Authenticated Galois/Counter Mode (GCM) Scenario Temporary DoS Low Anything else not on the "recommended" list above Anonymous Default/Common Permanent Important Initialization Vectors (IV) Anonymous Security Protocol, Algorithm and Key Length Recommendations Default/Common Temporary DoS with Important amplification Anonymous SSL/TLS versions SSL/TLS Products and services should use cryptographically secure versions of SSL/TLS: Default/Common Temporary DoS Moderate Anonymous TLS 1.2 should be enabled Scenario Permanent Important TLS 1.1 and TLS 1.0 should be enabled for backward compatibility only Anonymous Scenario Temporary DoS with Important amplification Asymmetric Algorithms, Key Lengths, and Padding Modes SSL 3 and SSL 2 should be disabled by default Asymmetric Algorithms, Key Lengths, and Padding Modes Anonymous RSA Scenario Temporary DoS Low Block Ciphers RSA should be used for encryption, key exchange and signatures. Content Disclaimer Content Disclaimer For products using symmetric block ciphers: RSA encryption should use the OAEP or RSA-PSS padding modes. Existing code should use PKCS #1 v1.5 Advanced Encryption Standard (AES) is recommended for new code. padding mode for compatibility only. Three-key triple Data Encryption Standard (3DES) is permissible in existing code for backward Use of null padding is not recommended. compatibility. Keys >= 2048 bits are recommended All other block ciphers, including RC2, DES, 2-Key 3DES, DESX, and Skipjack, should only be used for ECDSA decrypting old data, and should be replaced if used for encryption. ECDSA with >= 256 bit keys is recommended For symmetric block encryption algorithms, a minimum key length of 128 bits is recommended. The only block encryption algorithm recommended for new code is AES (AES-128, AES-192, and AES-256 are all acceptable, ECDSA-based signatures should use one of the three NIST-approved curves (P-256, P-384, or P521). noting that AES-192 lacks optimization on some processors). Three-key 3DES is currently acceptable if already ECDH in use in existing code; transition to AES is recommended. DES, DESX, RC2, and Skipjack are no longer considered secure. These algorithms should only be used for decrypting existing data for the sake of backward-ECDH with >= 256 bit keys is recommended compatibility, and data should be re-encrypted using a recommended block cipher. ECDH-based key exchange should use one of the three NIST-approved curves (P-256, P-384, or P521). Windows Platform-supported Crypto Libraries Cipher Modes Integer Diffie-Hellman Symmetric algorithms can operate in a variety of modes, most of which link together the encryption operations on successive blocks of plaintext and ciphertext. Key length >= 2048 bits is recommended Symmetric block ciphers should be used with one of the following cipher modes: The group parameters should either be a well-known named group (e.g., RFC 7919), or generated by a trusted party and authenticated before use Cipher Block Chaining (CBC) This
 1. The library should be a current in-support version free of known security vulnerabilities 2. The latest security protocols, algorithms and key lengths should be supported 3. (Optional) The library should be capable of supporting older security protocols/algorithms for backwards compatibility only Use the API defined in System.Security.Cryptography namespace---the CNG classes are preferred. Use the latest version of the .Net Framework available. At a minimum this should be .Net Framework version 4.6. If an older version is required, ensure the "SchUseStrongCrypto" regkey is set to enable TLS 1.2 for the application in question. Native Code Crypto Primitives: If your release is on Windows or Windows Phone, use CNG if possible. Otherwise, use the CryptoAPI (also called CAPI, which is supported as a legacy component on Windows from Windows Vista onward). SSL/TLS/DTLS: WinINet, WinHTTP, Schannel, IXMLHTTPRequest2, or IXMLHTTPRequest3. WinHTTP apps should be built with WinHttpSetOptionin order to support TLS 1.2 Code signature verification: WinVerifyTrust is the supported API for verifying code signatures on Windows platforms. Certificate Validation (as used in restricted certificate validation for code signing or SSL/TLS/DTLS): CAPI2 API; for example, CertGetCertificateChain and CertVerifyCertificateChainPolicy Managed Code Crypto Primitives: Certificate Validation: Use APIs defined under the System.Security.Cryptography.X509Certificates namespace.
 can use SQL Server Transparent Data Encryption (TDE) to protect sensitive data.You should use a TDE database encryption key (DEK) that meets the SDL cryptographic algorithm and key strength requirements. Currently, only AES_128, AES_192 and AES_256 are recommended; TRIPLE_DES_3KEY is not recommended.There are some important considerations for using SQL TDE that you should keep in mind: SQL Server does not support encryption for FILESTREAM data, even when TDE is enabled. TDE does not automatically provide encryption for data in transit to or from the database; you should also enable encrypted connections to the SQL Server database. Please see Enable Encrypted Connections to the Database Engine (SQL Server Configuration Manager) for guidance on enabling encrypted connections.If you move a TDE-protected database to a different SQL Server instance, you should also move the certificate that protects the TDE Data Encryption Key (DEK) and install it in the master database of the destination SQL Server instance. Please see the TechNet article Move a TDE Protected Database to Another SQL Server for more details.Credential ManagementUse the Windows Credential Manager API or Microsoft Azure KeyVault to protect password and credential data.Windows Store AppsUse the classes in the Windows.Security.Cryptography and Windows.Security.Cryptography.DataProtection namespaces to protect secrets and sensitive data. Microsoft Cybersecurity Defense Operations Center 2/19/2022 • 15 minutes to read ProtectAsync ProtectStreamAsync The Microsoft Cyber Defense Operations Center UnprotectAsync UnprotectStreamAsync Use the classes in the Windows.Security.Credentials namespace to protect password and credential data. .NET For data that needs to be persisted across system reboots: ProtectedData.Protect CryptProtectData ProtectedData.Unprotect CryptUnprotectData For data that does not need to be persisted across system reboots: NCryptProtectSecret (Windows 8 CNG DPAPI) ProtectedMemory.Protect For data that does not need to be persisted across system reboots: ProtectedMemory.Unprotect CryptProtectMemory For configuration files, use CryptUnprotectMemory either RSAProtectedConfigurationProvider or DPAPIProtectedConfigurationProvider to protect your For data that needs to be persisted and accessed by multiple domain accounts and computers: configuration, using either RSA encryption or DPAPI, respectively. NCryptProtectSecret (in CNG DPAPI, available as of Windows 8) The RSAProtectedConfigurationProvider can be used across multiple machines in a cluster. See Encrypting Configuration Information Using Protected Configuration for more information. Microsoft Azure KeyVault SQL Server TDE You
 services. The Microsoft Enterprise Cybersecurity Group and Microsoft Consulting Services teams engage with our customers to deliver the solutions most appropriate for their specific needs and requirements.One of the first steps that Microsoft highly recommends is to establish a security foundation. Our foundation services provide critical attack defenses and core identity-enablement services that help you to ensure assets are protected. The foundation helps you to accelerate your digital transformation journey to move towards a more secure modern enterprise.Building on this foundation, customers can then leverage solutions proven successful with other Microsoft customers and deployed in Microsoft's own IT and cloud services environments. For more information on our enterprise cybersecurity tools, capabilities and service offerings, please visit Microsoft.com/security and contact our teams at <EMAIL>. of products and Employ multi- Monitor for Enlist, educate, factor abnormal and empower authentication to account and users to strengthen credential activity recognize likely protection of to detect abuse threats and their accounts and own role in device protecting business data R P E O P L E P E O P L E Agility and Ensure you are Skilled analysts scalability exhaustively and data require planning measuring the scientists are the and building elements in your foundation of enabling platform defense, while platform users are the new security perimeter Maintain a well- Acquire and/or Establsih documented build the tools relationships and inventory of your needed to fully lines of assets monitor your communication network, hosts, between the and logs incident response team and other groups Have a well- Proactively Adopt least defined security maintain controls privilege policy with clear and measures, administrator standards and and regularly principles; guidance for test them for eliminate your accuracy and persistent organization effectiveness administrator rights Maintain proper Maintain tight Use the lessons- hygiene-most control over learned process attacks could be change to gain value prevented with management from every timely patches policies major incident and antivirus
			 Mitigations Mitigations
			 Contact your local Microsoft representative to learn more about the Government Security Program