From ce7fbce16c0e5c5e71ca86450382dedf4a9f9621 Mon Sep 17 00:00:00 2001 From: Keith Manville Date: Tue, 10 Nov 2020 12:51:53 -0500 Subject: [PATCH] added imitiation case-study --- images/AttackOnMT.png | Bin 0 -> 11260 bytes pages/case-studies-page.md | 23 +++++++++++++++++++++++ readme.md | 1 + 3 files changed, 24 insertions(+) create mode 100644 images/AttackOnMT.png diff --git a/images/AttackOnMT.png b/images/AttackOnMT.png new file mode 100644 index 0000000000000000000000000000000000000000..c074659e39414155052496e2dc0444c1f58565b3 GIT binary patch literal 11260 zcmeHtXH-<%vMvS?2_iWtS#oHFCP=aBzGy91|;&1t-w{$saesfMefz*&F@Em&VEJIMp^Vm zSk%dJ*bY$O32Ml$ni2(KKK$J z8(kO|T^Ju-7#@KNk3^X&io6Xem2+zG3@SUQNqhui1Mn@qt1Z227^umh?sW&X$r`SW znr@8;^(jESj5HK1bd--Ca9w&R!~+<5?AoS&sqodny&mdbZxqmDq9kb&*z^1nq#C9s z{VHzIJrM04g!Tz8_5U!2fN3ML#zVqU@1lxgqKe{CQwdIH8F6`;ad{s+Em485P^sBK z3w&-~sOnt)l7Xy?c5Ogrwk7$teOvGTnAV?@I+BeT`LR7+n1C%xz?QxLQ4~K@6#sK| ze;yULSQ5XAPWV-ov{jw9wR>_mjdT4r6C1~U5fQ0J z%F3G0Uc7Mh^!1O5&it5PSX|Z6(%Ie9(?2kcomp61TwGpWSy@|KTVLPU{I#{Uy}h%u zv%9;ue{gVccyxSxdU|?xc79dE!V-6YX{xC)4t1QIgYli|xy-h4&)dz&rK(PG2Z}bt8%XCR+5}K~Xhzx{BYH5QbY?{XO+mMB8cK zQ1Pkon`e6=pdcO3QunVlRyRDeHWWu=o_VUGJNvIKDOzPcXuyNUW7(sMr1o3ig+}Q| zy*XI(br_Odeo7XO3K2efb8(PeFpy+=t*FZ()=E!;`Ay^|Dr0M7{0cVvWUvT`p4@JG zXJb=E@o%Vtco5=Vq29El_zy#82QeTzn{0XS>65x;7A=G1Ae>!oM?H`a`;VMIoyimnxcojuCl-KO*>`Gi%K zZ#$pQw`&ANf<=$bCF@gz9bDQO?cYLFwoLjJGU;Pl-ISL#;MApD;G1ic-6XhU|ixWI9F zR{!K6#MGwQ>PzwZ>Z+>s$xh|gQ-&YE6ce)>UMXd(e>a&~(#tX9rxvvMi#;=%To1sW+uNd*XrQ@@0oYCO=jG;ij2 zKH*y%n3B;Yf8Xpfaz{!z5a-bDnSyvp$U{DHWPandXST?7BQnDPSRK@@c7H{6eSN!; z*yFK9wY|qd0hDxl=FM`IY`(x}rlRY(6rsaG(w_xnLidkSM;E{yT;NtT;(Unv-JEQ* zWtLZvN%<|+l&=vVA!VMm=68GGQ&A7Q(~NwEyA&*)k?T;hR0z+GEN6Vv50Odrn{-)- z!22r$-<@d;;q-1wRGHs`EVb|!5VK`3oHJ-4Wu$L!h!HyysERo5-ByEh39RsmxNNo> ziW8`MCw1+DaWb@o1N0(TYH3CJn;o=D`?rzA{D;0;_qLWLO4hYF`y|FKkR+~B*k%+WwUkRqc* zxH1z%nmdmjHF(D48cz(vkOpwknW%h6+`EvLGLjma5B33 zAiZJVUESgITt7#VS88zK2`Dx63r)Ij*Z6IyWi7aOfYubAPQWt7RcJpWYe}!hVU}O9 z>x06d648WT`)uJkiEusXBUa@Hux*IFdU?<6B2-TlQ|cGM|xj0`$#mhqg@8V=`(JLw3BVU{vn>cyai+hkV(XiGN&hMb8F zScS6PPwaE!b>j^K~NCz>DBndtKeHrR#}8a7BX>+g|7`yp3X+A)qRWwODucn zQ}`0!`LdC*gH7TysW z@GrFbaNgkhaBei!)`JFx%M`{9rjm1LqkyI!CYiEQK<(Qo=m;v*7Lw*uPyIvU{+8(PL4n*bm$!XXwi&&_y|XHK4)aK@DyH^U6LyfT6d~hc5mDx@HJisa-RGr~ff)J(P8U zB)=H2x$>DwV2I_iyd?wW?XqE(hq7w6-c$QG`E~UEII;}=E0Xo}$_($>fuPNXQSp*% z%JQF?_SLbc0P*zbU7BVy_ z>V|<8i5`CdgFNDI3~NSgW&Yvu2FrQP1@X`auc*Fh{`!H1veFwQADBX=4@4v$;Pr#F zJ$WzNWVB52V)DBBSA2T+-6OX!MGWvsy8$?#eV;R8_3*}>ZVJY3RJ7FA-s*tO(1|aR z^U}b0*k~v}_VfCUIh%kZ#3yynGxuH8DJY4~*xh+wVM|na(5vZ?&deJLM1x{D%gXa& zF3Z-Ak~)wXSjW2T`)&y}okE5<>P;lYN__5X59KLZp{Ud+VAawzrmQ_6Fwm|;Sw6F? znG90^pve~Qq}l=A-?qHmcR^lfLs=G4-Zif~M80_ejN~s?kfKDN(s-yYS5ebliM~JaYL^6j8?s)gT(|!%J`45B7#v zVxwx(QclG$2#>dOx*6ZB;vD9fp6*N)2mLA=S7{}$NZ*J$YdOg2p5I=Sz6d!z9f$%K zZ}R!f_B$_fbZ@@lVeQ#8ujhDhj+);*%3?t0K_}f^nN>Uw>dIEW`tY4^ZLsks0CgL9>Xt%`=KS;8>EFVz>(%~Ay(XL)ZF<$}fuWn) z&Wj|N^AdUV>1SIXxN!q9;d;!WPhjhO(9xB+mdz7?f}%9_esz;;<8NNs&Cli=H&@MbrAt~tmG?844mhZ-=ut)3I zmDe~Z3By_h!xT?*%K@p;zTm3)>O`j5Ab+Y=0;gExDG|xWfk-)8MR{R*06j*h5RT&X zjqf|boTF)Ya*y{eWl%;?(8n!iiFDAwOWfpF_bIvIF!EKQ8B z(WKiLdxTrp1i0gCI-3g|#g}VQY24&qBulWG?r1r$2$b5gTOJs)9HuWlN?iXE{%pXA z_aoptYfiVtn_x1T9oq+nA6)yM>ygK4P?D_VZ@{T1wzPb-;q+r<*yvp;_g9)563_Kn zrg+mHyQVM*$e4oq?g^5}n+7Y%P~y7EB~ULsO5+%O_gzwC+}t*uquatxpODsn=}39+ znd0+#wrB14S`kg;8V^&(x3=Y+J|7QyqZ9~r)9DSaJ!EEkGJG|@%9FTS^I7q8db|oc zgIWhkDVL|)gv6VZ$Fddu&K?=*ogRZy4~>Nhd&QJY*XX&vMGh82U-FSRepFT~Oq^D8y`P*e5;a(ZmI`>o5#AD3mh8ZmMGEIuEz0N+a zKBI(rBauuP<=2Z9A(MK1Nv&#o#!x|0y6Ng#w=-*X0v8S&6=Mfu|BR6@=)0WgTj2H_ zR%`f2iP_x|VdS#Tb2Cz%=4vql+nH*IYr$Ot`!^WM>jx{m4LhH<^!4M(c4X0UTdxwO zhimZi-E1`qAmYuap5x&$nCTkpex#Rxmj3*8tLr)aFxbb)fmk-y7+w*iDDu9bXKbFz zwl3MeWFI$uG+5;zqyq-SiE;&-S0>VV!yZpQeIp#*s$`YT6o zITG|>AMZd)BeUO@U#Ef!K*}0vU1#HGi>_jb_XGQH_L`V7Ok|SNJ^q+BY3gR=%7Ym3 zTR4P9Vs7UM*Tr;2NQN|pzURF;dDQJ$&GZxIaX&c%Vk2+iJ*fzv$X~UUg;(4uP&a11 z_ufgr*@jRlYviSi_54>^LJ5PQ*%*oZ$_%e}rtn<+-5~m?o;~7n(uv~{qfCuFFraU~FXiJpuzviB%Dk0)5Ac>m#H~){2pzbkmzZu5bHbOL!mp+|8kzj-1o*00(duS5D$$LX2JsPu2)#YFTs1+D z1D@bVixu)6SHcs&o*&zZHxgbgK~fhOOH@x(@3SD^sHu@5zFm7fO9A5FDN>^x6Zkpq z$V@SzX4|PA9tbIkJ-aG499mE;sLD*4GHfER6(B&q(EKa=u|Z<73QvL7!Q)Ob3hdSQ zwa;y6lCA{kY3=7`@D!qTxRrzrR_}U77j7)=Y}OSP9FLVv&%a*eqc*vEvRb(=2MP`q z5ES1>7fY`MiyTGn&{iUeiAS@1(OD=p<5I9ot?Y-QDL+rvJ!FhenVArNCbQd;z*#BNS>FT6Dy68q5B z__8scD8ff3r$1hLp<1~wz~0zb^;FMkl0zA$bsNfX$Z7P!$z|EOM+RB`WBU}HSSP?w zo-%JaeD6oRcT*4b+nz}081EsSxzd$p=bjQSa1T7SpCZ7Obk4rCSpex*6?4A<129E1 znU=sKL?EP_n2<#nsGWkvA!zBx|9>iL;}4qOUJC)ZiT5a}OgW_HcA;mf&yl^%Z0uF5~K=uOMfdXgP#6 zL2d1bIxzF*C%whABB#hbaGv6!^%`@UKm&?za~2nk7Ip$^bk(JsRDNuqBV-l zZaOJQ#kd}gYG>J_lTt5&psU3{k)}tO3u0AcX{Ekh*vRH^$j? z1`6%lK9cwj>#~4BEa{EKGuG5a$5eL7R)%NS=Puv07)e!L<7QXd-9`rdFanVQnHpms!s;;>#$NReVk)`R?pWMcnS< zBO)%a7n8GQ%~d4D0}E7B)Nb_Y{28$29i z_e|*{M+#Wq*csV37-_&uzGy)T)U1JNUSedIwJaRk$*F1VugXVl1u5HklDn>Bx$ne; zM?v>9+rifB{>OuE&doc%B2mpYHP&>;4DDk%D&O_5J+oiFo2-G2nx=L?$JYc;SrKV#+PBSL1*+A(4QX{PZii;+k0H6K+HT-hELDsO=|i@Z^9EpB+9iNx53= zBoRBriD&tVDx^7$Ma3}0w{<)$2AE1O_rImlfOq2eVr_@VSimReVW0WIlZr;{V6T$> z4}4c-&H5oa(0=hhh-saTZ0~poBJFAbhzOV{rW4OC~ z%25uJeX&A>e9@ZR;GMzb9GigX!vz_>>+#4UlEDpOOmC2E21Y#VNDZ7bkI>;HOsdLyk$8F)w=@ zJcz_&MuR+dXbD+F0f^@FogTe46}bKlDmhz#>;sDipP9^?Pa(2Ew)(fz9q|q!F(Tp4 zzX7#?69Chntnxi}pN}+&%^H&4tW>$W`N&JRZ{)WhWgsM*epm6+ny+)XY2?t80-cWo z0pDl$MQK&;h5Od(K#mv@bwa$wBhnPG>YFN@A_l4q{-~GTe7O2RvltDpqTe<@O}!$> z2vXlQacy@;I2bA-$Hnyk%gS zxl|daQicido`O-;z&E4CHZSCC{&&f#&j#dqyOj6GrErIK5j^UGpr zE#%Cb)U?8Ch0KFk@8fWgaw^3HrSA~}2Bd;l859j%R}@3J*>EGd5qNjU-HC;fdCE^6<`5Zo}ooOXO&aCOMd=f&&vsMjiJ`2O>@EljafTd=f zw!9HRj|9MY2Dt7(%EW6qSi$8VQ&pxGw7Jk+K##s0oKGm&!Q$#8yl5@0rg?}H@$!et zo5Th31^{&?nsa+Mcf z{dYe0pbB#JlE{DOl6@Ai%iyvWh#jU+Q{X_v(XXpd(_HS!RMaJ~=e5)|IWfZxvC287 zdyvgf2H{bJA6mL4H_cL&iWyF%V4NMRo6`Pd6Gmj7if#5s*R5uGmsp`!ix2C*m#gOn z;N9;8kuHOV15Lt-&L1Q6O!9kj{P2)wk4yjYCN;8AeowvO{fGRP)E6{0v^V+NtN76q z&||G9+>!w{<{Xf+u!j&`ZwGRu4E(9}KVB3J#KfF%j=R8T?uru!mf21s^sMs@;8j#b zjz2AgEK@NtlnViWW!jW$WKg>jD16v!$(1ri4Obk$$iz(T5B-#f!yeRa#?;_Kc#Pxm zi|R>?xzJ%G57#+zsVjBidjw8@x&jSEkk$-ssrsF!FaDzrWKwho)$nW34x)0;veH`1 z*1E!1o(26J@$fg%g%Kgg+cRKy>xRB}k}3LBl?_Hq>E66Vw`TlaVP9-I`II`Pp-t*k zqIK3;)k+{SV&&yYt)Qo3FCwAvKF9Cg9L@sP41G^56qmF}B-~WAp#zG?{1P(oyk$U~ zv#V1t;iU$5RgoArJ+d^g%bP|s3$7C7h<54`{4=`oAWSjU|0$>b_j~d`yW~IVyf*w$ zaduu>3roF^n&S~>fcgIZ2F!?0qotgidZR4}h>ftSqf5AUL!g3-vSK4Y$@R!Htvjh2 za5mmx1)x^qQKp!*S=GbRgz~!&!6mxz`Pl?Qy9{#Zn|Y*Q&ECR8PBa;n+~qo>TG9;(arL-kNQUvH?x%)~PpD_FZxC8h=BE*5jTF7XJoRt!`&3ZpsK^#MHi1NRgV^@qFPw`-~BpgUiPYJNRBoBPxVa*ARJeUd7Ry80rk;|fzo%uS~b+-rG{;j4P3G6~7BosEUk z>*5;mq)*E*wkCC1amBrvBco4uo+3W=A3IUd83Y|6*lRE^G4LC2;E!lvx4$oTdR~v# zfe6~^J_>-*)sW3pCNi3{tD#9*B@cIh7UqFms;z%6WNxnY{jU6bOvmNYC9vY; z=B&k*obe!e3wqx~Hz3%pf_HONR*J^c9GC> zf=e&!;4g@{uE&`EgFZPfbJn8})Ds=ZkW74!Xd>4v=p?iusz`XlJ~HQLmCh7e6~0YZ zNF%*hL$}ZIjICXzkfhQ+)65d^!XGHUuaWdKiMLXJFpv%>`|KMKsy^5Dma+XJQF4+C zvh_~;3i4VA12cZX_>$nL!eRYgIev8})IzBz`MC6_n(*iJ0H0t+j08v3ej@x|X2AbN z9ogqccLKBA|B(^-5BOehM1oPUZS2n&Yp<~?cLhhTcCW_Yb%Wiyw#fv457d2T@OM3&U8)q*npeVxeO%~_ z&rFv~@w8fS_b|0Ow1QW%Eoq8LHPyk*ujR0T!LT8IBlzl#oc%KQGf(TphT&BVLvrCPQ>98d&sXaH*Osg+EYFnx>uZn|A-Xi=!`kQsSCHccB>Ui9_ z!D&@ke)PDxRO*~`n%%W`39ktnKxA~ExG5%FpWcr6&EIH>wrL{?$&oY8QqWH39LD!e zN;|E#Z)H-Fdf%dd-hc_Zm>r3wsN8z!l@CGM1^rHo?;wVXmI?T({Vxr=bplAh#_|(! zZ=&W0qEC6FMAxl{v(+?RbQuZaddW{T-b*YiAdPW;yXS>ZEW#r_Vg@WUcosA@6fpN^ z^1Cp(2B^!vnl|N_lbOA@Gu0#|&{}pl9h9M0hRZN%vo^^jP@je2_x){JDuA7Z%oa!Y zrm8gCQ-;ZzY;@mghgQzStDFv{oSyC=zniM+Z8tR%C2rp@j6F(JiOo?J>RxWH`%{Qb z0$e}-k8`sBqLTe%-29s}um2DA|75}oY(hFv_8)60wU)ISPD=*enzQi(wkGvG)u};; z=1Q%EtwaAwN&z|oq`(f_ze=O1P?x$qty%vnC80)L>e{q6|Eu(X+Tj?Grna8_Cn?qD z2_QXbt^23c)Ki@jbmADiz}MV zO*yA`a7h>IX2Yf*29eB(lk5sxsGR$E1~JUeR@W=|3>&h*oBk4jFzRywvG#QPcN*8*GD`}`pREs9xOSb|i> zZuO3*-ljaEm8jFVr#lYRt)sQF=8cD!Gws?fucOpfH`HMeAE@5YbBAY|e&F5Vblpm3 zJZ^u~e4*nQNd;`_ysY<7ARMb%`%=b|0LP^u}rR4=}TLB|QE9-z1 zqso^FV8efmh#Z^9017>*cfti(ZEsg%c|zMQycl;R^o6Ass8g@Zovg&lYt~*Y*q~~6 zo8qhLM*j7sHG31%we*oxQ0jK}~ literal 0 HcmV?d00001 diff --git a/pages/case-studies-page.md b/pages/case-studies-page.md index 531fd15..a5a2ea1 100644 --- a/pages/case-studies-page.md +++ b/pages/case-studies-page.md @@ -7,6 +7,7 @@ - [Microsoft - Azure Service - Evasion](/pages/case-studies-page.md#microsoft---azure-service) - [Microsoft Edge AI - Evasion](/pages/case-studies-page.md#microsoft---edge-ai) - [MITRE - Physical Adversarial Attack on Face Identification](/pages/case-studies-page.md#mitre---physical-adversarial-attack-on-face-identification) + - [Attack on Machine Translation Service - Google Translate, Bing Translator, and Systran Translate](/pages/case-studies-page.md#attack-on-machine-translation-service---google-translate-bing-translator-and-systran-translate) Attacks on machine learning (ML) systems are being developed and released with increased regularity. Historically, attacks against ML systems have been performed in a controlled academic settings, but as these case-studies demonstrate, attacks are being seen in-the-wild. In production settings ML systems are trained on personally identifiable information (PII), trusted to make critical decisions with little oversight, and have little to no logging and alerting attached to their use. The case-studies were selected because of the impact to production ML systems, and each demonstrates one of the following characteristics. @@ -165,6 +166,28 @@ MITRE AI Red Team **Source:** None +---- +## Attack on Machine Translation Service - Google Translate, Bing Translator, and Systran Translate + +**Summary of Incident:** +Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create an "imitation model" with near-production, state-of-the-art translation quality. Beyond demonstrating that IP can be stolen from a black-box system, they used the imitation model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites. + +**Mapping to Adversarial Threat Matrix:** +- Using published research papers, the researchers gathered similar datasets and model architectures that these translation services used +- They abuse a public facing application to query the model and produce machine translated sentence pairs as training data +- Using these translated sentence pairs, researchers trained a substitute model (model replication) +- The replicated models were used to construct offline adversarial examples that successfully transferred to an online evasion attack + + + +**Reported by:** +- Work by Eric Wallace, Mitchell Stern, Dawn Song and reported by Kenny Song (@helloksong) + +**Source:** +- https://arxiv.org/abs/2004.15015 +- https://www.ericswallace.com/imitation + + ---- # Contributing New Case Studies diff --git a/readme.md b/readme.md index 0101868..2e56317 100644 --- a/readme.md +++ b/readme.md @@ -35,6 +35,7 @@ To see the Matrix in action, we recommend seeing the curated case studies - [Microsoft - Azure Service - Evasion](/pages/case-studies-page.md#microsoft---azure-service) - [Microsoft Edge AI - Evasion](/pages/case-studies-page.md#microsoft---edge-ai) - [MITRE - Physical Adversarial Attack on Face Identification](/pages/case-studies-page.md#mitre---physical-adversarial-attack-on-face-identification) + - [Attack on Machine Translation Service - Google Translate, Bing Translator, and Systran Translate](/pages/case-studies-page.md#attack-on-machine-translation-service---google-translate-bing-translator-and-systran-translate)