Git-Mirrors/DISARMframeworks
1
0
Fork 0
mirror of https://github.com/DISARMFoundation/DISARMframeworks.git synced 2025-05-09 10:15:13 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Copy AMITT repository, clean up and rebrand

Browse source 
Took a copy of the current AMITT github repository - we'll be updating this and merging the SPICE branch back in
Rebranded to DISARM
Moved generated pages to their own folder, to make looking at the repository less confusing
This commit is contained in:
Sara-Jayne Terp 2022-01-29 11:34:46 -05:00
commit 22abaf93d8
448 changed files with 58066 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
CODE/.DS_Store vendored Normal file
View file

Binary file not shown.

459
CODE/.ipynb_checkpoints/AMITT_create_counters_summaries-checkpoint.ipynb Normal file
View file

@ -0,0 +1,459 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Create counters summaries\n",
"Create summaries of disinfo countermeasures for use in dataset cleaning etc."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Writing ../counter_tactics/ALcounters.md\n",
"Writing ../counter_tactics/TA01counters.md\n",
"Writing ../counter_tactics/TA02counters.md\n",
"Writing ../counter_tactics/TA03counters.md\n",
"Writing ../counter_tactics/TA04counters.md\n",
"Writing ../counter_tactics/TA05counters.md\n",
"Writing ../counter_tactics/TA06counters.md\n",
"Writing ../counter_tactics/TA07counters.md\n",
"Writing ../counter_tactics/TA08counters.md\n",
"Writing ../counter_tactics/TA09counters.md\n",
"Writing ../counter_tactics/TA10counters.md\n",
"Writing ../counter_tactics/TA11counters.md\n",
"Writing ../counter_tactics/TA12counters.md\n",
"updated ../counter_tactic_counts.md\n",
"Writing ../counters_metatag/cleaningcounters.md\n",
"Writing ../counters_metatag/countermessagingcounters.md\n",
"Writing ../counters_metatag/data pollutioncounters.md\n",
"Writing ../counters_metatag/daylightcounters.md\n",
"Writing ../counters_metatag/dilutioncounters.md\n",
"Writing ../counters_metatag/diversioncounters.md\n",
"Writing ../counters_metatag/frictioncounters.md\n",
"Writing ../counters_metatag/metatechniquecounters.md\n",
"Writing ../counters_metatag/reduce resourcescounters.md\n",
"Writing ../counters_metatag/removalcounters.md\n",
"Writing ../counters_metatag/resiliencecounters.md\n",
"Writing ../counters_metatag/scoringcounters.md\n",
"Writing ../counters_metatag/targetingcounters.md\n",
"Writing ../counters_metatag/verificationcounters.md\n",
"updated ../counter_metatag_counts.md\n",
"Writing ../counter_resource/DHScounters.md\n",
"Writing ../counter_resource/NGOcounters.md\n",
"Writing ../counter_resource/activistscounters.md\n",
"Writing ../counter_resource/adtechcounters.md\n",
"Writing ../counter_resource/civil_societycounters.md\n",
"Writing ../counter_resource/community_groupscounters.md\n",
"Writing ../counter_resource/companiescounters.md\n",
"Writing ../counter_resource/content_creatorscounters.md\n",
"Writing ../counter_resource/data_scientistcounters.md\n",
"Writing ../counter_resource/datastreamscounters.md\n",
"Writing ../counter_resource/developerscounters.md\n",
"Writing ../counter_resource/educatorscounters.md\n",
"Writing ../counter_resource/elvescounters.md\n",
"Writing ../counter_resource/factcheckerscounters.md\n",
"Writing ../counter_resource/fundingcounters.md\n",
"Writing ../counter_resource/gamesdesignerscounters.md\n",
"Writing ../counter_resource/governmentcounters.md\n",
"Writing ../counter_resource/government:policymakerscounters.md\n",
"Writing ../counter_resource/influencerscounters.md\n",
"Writing ../counter_resource/influencers:trusted_authoritycounters.md\n",
"Writing ../counter_resource/infoseccounters.md\n",
"Writing ../counter_resource/librariescounters.md\n",
"Writing ../counter_resource/mediacounters.md\n",
"Writing ../counter_resource/militarycounters.md\n",
"Writing ../counter_resource/moneycounters.md\n",
"Writing ../counter_resource/platform_admincounters.md\n",
"Writing ../counter_resource/platform_admin:adtechcounters.md\n",
"Writing ../counter_resource/platform_admin:fundingsitescounters.md\n",
"Writing ../counter_resource/platform_admin:socialmediacounters.md\n",
"Writing ../counter_resource/platform_algorithmscounters.md\n",
"Writing ../counter_resource/platform_outreachcounters.md\n",
"Writing ../counter_resource/platformscounters.md\n",
"Writing ../counter_resource/publiccounters.md\n",
"Writing ../counter_resource/public:account_ownerscounters.md\n",
"Writing ../counter_resource/religious_organisationscounters.md\n",
"Writing ../counter_resource/schoolscounters.md\n",
"Writing ../counter_resource/server_admincounters.md\n",
"updated ../counter_resource_counts.md\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>metatechnique</th>\n",
" <th>Title</th>\n",
" <th>Details</th>\n",
" <th>Playbook(s)</th>\n",
" <th>Resources needed</th>\n",
" <th>How found</th>\n",
" <th>References</th>\n",
" <th>Incidents</th>\n",
" <th>Tactic</th>\n",
" <th>Response</th>\n",
" <th>Techniques</th>\n",
" <th>NOTES</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00043</td>\n",
" <td>cleaning</td>\n",
" <td>Detect hijacked accounts and reallocate them</td>\n",
" <td>NaN</td>\n",
" <td>In all playbooks the platform must force user ...</td>\n",
" <td>platform_admin,activists,civil_society,money</td>\n",
" <td>2019-11-workshop</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA03 Develop People</td>\n",
" <td>D3 Disrupt</td>\n",
" <td>T0011 - Hijack accounts</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00053</td>\n",
" <td>cleaning</td>\n",
" <td>Delete old accounts / Remove unused social med...</td>\n",
" <td>NaN</td>\n",
" <td>Social media companies remove inactive account...</td>\n",
" <td>platform_admin,platform_admin:socialmedia,publ...</td>\n",
" <td>2019-11-workshop,2019-11-search</td>\n",
" <td>NaN</td>\n",
" <td>I00004</td>\n",
" <td>TA04 Develop Networks</td>\n",
" <td>D4 Degrade</td>\n",
" <td>T0011 - Hijack accounts\\nTA06 - Develop Conten...</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>C00074</td>\n",
" <td>cleaning</td>\n",
" <td>Identify identical content and mass deplatform</td>\n",
" <td>\\n</td>\n",
" <td>In all cases some checks need to prevent depla...</td>\n",
" <td>platform_admin,platform_admin:socialmedia</td>\n",
" <td>2019-11-workshop</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA06 Develop Content</td>\n",
" <td>D2 Deny</td>\n",
" <td>T0022 - Conspiracy narratives\\nT0026 - Create ...</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>C00026</td>\n",
" <td>countermessaging</td>\n",
" <td>Shore up democracy based messages (peace, free...</td>\n",
" <td>Unsure</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2019-11-workshop</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA01 Strategic Planning</td>\n",
" <td>D4 Degrade</td>\n",
" <td>T0002 - Facilitate State Propaganda</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>C00082</td>\n",
" <td>countermessaging</td>\n",
" <td>Ground truthing as automated response to pollu...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2019-11-workshop</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA06 Develop Content</td>\n",
" <td>D3 Disrupt</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>181</th>\n",
" <td>C00214</td>\n",
" <td>metatechnique</td>\n",
" <td>Create policy that makes social media police d...</td>\n",
" <td>German model: facebook forced to police conten...</td>\n",
" <td>NaN</td>\n",
" <td>government:policymakers</td>\n",
" <td>2019-11-workshop</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA07 Channel Selection</td>\n",
" <td>D3 Disrupt</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>182</th>\n",
" <td>C00215</td>\n",
" <td>metatechnique</td>\n",
" <td>Use fraud legislation to clean up social media</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>government:policymakers</td>\n",
" <td>2019-11-workshop</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA07 Channel Selection</td>\n",
" <td>D3 Disrupt</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>183</th>\n",
" <td>C00217</td>\n",
" <td>daylight</td>\n",
" <td>Registries alert when large batches of newsy U...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>platform_admin</td>\n",
" <td>grugq</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA07 Channel Selection</td>\n",
" <td>D2 Deny</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>184</th>\n",
" <td>C00218</td>\n",
" <td>removal</td>\n",
" <td>Censorship</td>\n",
" <td>Alter and/or block the publication/disseminati...</td>\n",
" <td>NaN</td>\n",
" <td>platform_admin</td>\n",
" <td>grugq</td>\n",
" <td>Taylor81</td>\n",
" <td>NaN</td>\n",
" <td>TA09 Exposure</td>\n",
" <td>D2 Deny</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>185</th>\n",
" <td>C00219</td>\n",
" <td>daylight</td>\n",
" <td>Add metadata to content - out of the control o...</td>\n",
" <td>NaN</td>\n",
" <td>Add date and source to images</td>\n",
" <td>NaN</td>\n",
" <td>grugq</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>TA06 Develop Content</td>\n",
" <td>D3 Disrupt</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>186 rows × 13 columns</p>\n",
"</div>"
],
"text/plain": [
" ID metatechnique \\\n",
"0 C00043 cleaning \n",
"1 C00053 cleaning \n",
"2 C00074 cleaning \n",
"3 C00026 countermessaging \n",
"4 C00082 countermessaging \n",
".. ... ... \n",
"181 C00214 metatechnique \n",
"182 C00215 metatechnique \n",
"183 C00217 daylight \n",
"184 C00218 removal \n",
"185 C00219 daylight \n",
"\n",
" Title \\\n",
"0 Detect hijacked accounts and reallocate them \n",
"1 Delete old accounts / Remove unused social med... \n",
"2 Identify identical content and mass deplatform \n",
"3 Shore up democracy based messages (peace, free... \n",
"4 Ground truthing as automated response to pollu... \n",
".. ... \n",
"181 Create policy that makes social media police d... \n",
"182 Use fraud legislation to clean up social media \n",
"183 Registries alert when large batches of newsy U... \n",
"184 Censorship \n",
"185 Add metadata to content - out of the control o... \n",
"\n",
" Details \\\n",
"0 NaN \n",
"1 NaN \n",
"2 \\n \n",
"3 Unsure \n",
"4 NaN \n",
".. ... \n",
"181 German model: facebook forced to police conten... \n",
"182 NaN \n",
"183 NaN \n",
"184 Alter and/or block the publication/disseminati... \n",
"185 NaN \n",
"\n",
" Playbook(s) \\\n",
"0 In all playbooks the platform must force user ... \n",
"1 Social media companies remove inactive account... \n",
"2 In all cases some checks need to prevent depla... \n",
"3 NaN \n",
"4 NaN \n",
".. ... \n",
"181 NaN \n",
"182 NaN \n",
"183 NaN \n",
"184 NaN \n",
"185 Add date and source to images \n",
"\n",
" Resources needed \\\n",
"0 platform_admin,activists,civil_society,money \n",
"1 platform_admin,platform_admin:socialmedia,publ... \n",
"2 platform_admin,platform_admin:socialmedia \n",
"3 NaN \n",
"4 NaN \n",
".. ... \n",
"181 government:policymakers \n",
"182 government:policymakers \n",
"183 platform_admin \n",
"184 platform_admin \n",
"185 NaN \n",
"\n",
" How found References Incidents \\\n",
"0 2019-11-workshop NaN NaN \n",
"1 2019-11-workshop,2019-11-search NaN I00004 \n",
"2 2019-11-workshop NaN NaN \n",
"3 2019-11-workshop NaN NaN \n",
"4 2019-11-workshop NaN NaN \n",
".. ... ... ... \n",
"181 2019-11-workshop NaN NaN \n",
"182 2019-11-workshop NaN NaN \n",
"183 grugq NaN NaN \n",
"184 grugq Taylor81 NaN \n",
"185 grugq NaN NaN \n",
"\n",
" Tactic Response \\\n",
"0 TA03 Develop People D3 Disrupt \n",
"1 TA04 Develop Networks D4 Degrade \n",
"2 TA06 Develop Content D2 Deny \n",
"3 TA01 Strategic Planning D4 Degrade \n",
"4 TA06 Develop Content D3 Disrupt \n",
".. ... ... \n",
"181 TA07 Channel Selection D3 Disrupt \n",
"182 TA07 Channel Selection D3 Disrupt \n",
"183 TA07 Channel Selection D2 Deny \n",
"184 TA09 Exposure D2 Deny \n",
"185 TA06 Develop Content D3 Disrupt \n",
"\n",
" Techniques NOTES \n",
"0 T0011 - Hijack accounts NaN \n",
"1 T0011 - Hijack accounts\\nTA06 - Develop Conten... NaN \n",
"2 T0022 - Conspiracy narratives\\nT0026 - Create ... NaN \n",
"3 T0002 - Facilitate State Propaganda NaN \n",
"4 NaN NaN \n",
".. ... ... \n",
"181 NaN NaN \n",
"182 NaN NaN \n",
"183 NaN NaN \n",
"184 NaN NaN \n",
"185 NaN NaN \n",
"\n",
"[186 rows x 13 columns]"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import generate_amitt_counters\n",
"\n",
"counter = generate_amitt_counters.Counter()\n",
"counter.write_tactics_markdown()\n",
"counter.write_metacounts_markdown()\n",
"counter.write_resource_markdown()\n",
"counter.dfcounters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

268
CODE/.ipynb_checkpoints/AMITT_create_incident_visualisations-checkpoint.ipynb Normal file
View file

@ -0,0 +1,268 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Create AMITT incident visualisations\n",
"\n",
"Many thanks to https://python-graph-gallery.com/91-customize-seaborn-heatmap/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import seaborn as sns\n",
"import pandas as pd\n",
"import numpy as np\n",
"import generate_amitt_ttps\n",
"\n",
"# Check that heatmap works\n",
"df = pd.DataFrame(np.random.random((10,12)), columns=[\"a\",\"b\",\"c\",\"d\",\"e\",\"f\",\"g\",\"h\",\"i\",\"j\",\"k\",\"l\"])\n",
"sns.heatmap(df, annot=True, annot_kws={\"size\": 7})"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amitt = generate_amitt_ttps.Amitt()\n",
"redgrid = amitt.create_padded_framework_table('AMITT Red', 'technique_ids', False)\n",
"\n",
"techcounts = amitt.it[['id_incident','id_technique']].drop_duplicates().groupby('id_technique').count().to_dict()['id_incident']\n",
"techlabels = redgrid[2:][:]\n",
"nrows = len(techlabels)\n",
"ncols = len(techlabels[0])\n",
"techgrid = np.zeros([nrows, ncols], dtype = int)\n",
"\n",
"for row in range(nrows):\n",
" for col in range(ncols):\n",
" if techlabels[row][col] in techcounts:\n",
" techgrid[row][col] = techcounts[techlabels[row][col]]\n",
"\n",
"sns.heatmap(techgrid, annot=True, annot_kws={\"size\": 7})\n",
"techgrid"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amitt.df_tactics"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amitt.it"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ct = amitt.cross_counterid_techniqueid\n",
"ct[ct['technique_id'] != '']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ct[(ct['id'] == 'C00197') & (ct['technique_id'].isin(['T0002', 'T0007']))]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ct = ct[ct['technique_id'].isin(amitt.df_techniques['id'].to_list()) & ct['id'].isin(amitt.df_counters['id'].to_list())]\n",
"ct"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"technique_id_list = ['T0007', 'T0008', 'T0022', 'T0023', 'T0043', 'T0052', 'T0036', 'T0037', 'T0038']\n",
"counter_id_list = ['C00009', 'C00008', 'C00042', 'C00030', 'C00093', 'C00193', 'C00073', 'C000197', 'C00174', 'C00205']\n",
"possible_counters_for_techniques = ct[ct['technique_id'].isin(technique_id_list)] \n",
"possible_techniques_for_counters = ct[ct['id'].isin(counter_id_list)] \n",
"coverage = ct[(ct['id'].isin(counter_id_list)) & (ct['technique_id'].isin(technique_id_list))]\n",
"coverage"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"possible_techniques_for_counters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"possible_counters_for_techniques"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"clicked button T0045 8 7\n",
"clicked button T0046 9 7\n",
"clicked button T0049 4 8\n",
"clicked button T0057 2 9\n",
"clicked button T0060 4 10\n",
"clicked button T0029 2 6\n",
"clicked button T0016 2 4\n"
]
}
],
"source": [
"import tkinter as Tk\n",
"import numpy as np\n",
"import generate_amitt_ttps\n",
"\n",
"class Begueradj(Tk.Frame):\n",
" def __init__(self,parent):\n",
" amitt = generate_amitt_ttps.Amitt()\n",
" self.redgrid = amitt.create_padded_framework_table('AMITT Red', 'technique_ids', False)\n",
" self.bluegrid = amitt.create_padded_framework_table('AMITT Blue', 'counter_ids', False)\n",
"\n",
" Tk.Frame.__init__(self, parent)\n",
" self.parent = parent\n",
" self.button= ''\n",
" self.initialize()\n",
" \n",
" def initialize(self):\n",
" '''\n",
" Draw the GUI\n",
" '''\n",
" self.parent.title(\"AMITT FRAMEWORK COVERAGE\") \n",
" self.parent.grid_rowconfigure(1,weight=1)\n",
" self.parent.grid_columnconfigure(1,weight=1)\n",
"\n",
" self.frame = Tk.Frame(self.parent) \n",
" self.frame.pack(fill=Tk.X, padx=5, pady=5)\n",
"\n",
" # Create a 6x7 array of zeros as the one you used\n",
" numrows = len(self.redgrid) - 1\n",
" numcols = len(self.redgrid[0])\n",
" self.buttons = {}\n",
" for row in range(1,numrows):\n",
" for col in range(0,numcols):\n",
" button_id = self.redgrid[row][col]\n",
" self.button = Tk.Button(self.frame, text = button_id, bg='blue', \n",
" command= lambda bid=button_id, row=row, col=col: self.clicked(bid, row, col))\n",
" self.button.grid(row=row, column=col)\n",
" \n",
" def clicked(self, bid, row, col):\n",
" print('clicked button {} {} {}'.format(bid, row, col))\n",
" self.find_in_grid(self.frame, row, col)\n",
"\n",
" def find_in_grid(self, frame, row, column):\n",
" for children in frame.children.values():\n",
" info = children.grid_info()\n",
" #note that rows and column numbers are stored as string\n",
" if info['row'] == str(row) and info['column'] == str(column):\n",
" print('{}'.format(children.get()))\n",
" return None\n",
"\n",
"root=Tk.Tk()\n",
"app = Begueradj(root) \n",
"root.mainloop()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"redgrid"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for row in range(2,len(redgrid)):\n",
" print(len(redgrid[row]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"redgrid[1][2]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

8443
CODE/.ipynb_checkpoints/AMITT_create_website_sql-checkpoint.ipynb Normal file
View file

File diff suppressed because it is too large Load diff

608
CODE/.ipynb_checkpoints/AMITT_generate_github_files-checkpoint.ipynb Normal file
View file

@ -0,0 +1,608 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Generate AMITT github files\n",
"\n",
"Generate all the AMITT github files from the AMITT master spreadsheet, being careful to reatin any comments people have made below the \"don't write above this\" line in them. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"updated ../amitt_red_framework.md\n",
"updated ../amitt_red_framework_clickable.html\n",
"updated ../amitt_blue_framework.md\n",
"updated ../amitt_blue_framework_clickable.html\n",
"Temp: objecttype phase\n",
"updated ../phases_index.md\n",
"Temp: objecttype tactic\n",
"updated ../tactics_index.md\n",
"Temp: objecttype technique\n",
"updated ../techniques_index.md\n",
"Temp: objecttype task\n",
"updated ../tasks_index.md\n",
"Temp: objecttype incident\n",
"updated ../incidents_index.md\n",
"Temp: objecttype counter\n",
"updated ../counters_index.md\n",
"Temp: objecttype metatechnique\n",
"updated ../metatechniques_index.md\n",
"Temp: objecttype actortype\n",
"updated ../actortypes_index.md\n",
"Updating ../actortypes/A001.md\n",
"Updating ../actortypes/A002.md\n",
"Updating ../actortypes/A003.md\n",
"Updating ../actortypes/A004.md\n",
"Updating ../actortypes/A005.md\n",
"Updating ../actortypes/A006.md\n",
"Updating ../actortypes/A007.md\n",
"Updating ../actortypes/A008.md\n",
"Updating ../actortypes/A009.md\n",
"Updating ../actortypes/A010.md\n",
"Updating ../actortypes/A011.md\n",
"Updating ../actortypes/A012.md\n",
"Updating ../actortypes/A013.md\n",
"Updating ../actortypes/A014.md\n",
"Updating ../actortypes/A015.md\n",
"Updating ../actortypes/A016.md\n",
"Updating ../actortypes/A017.md\n",
"Updating ../actortypes/A018.md\n",
"Updating ../actortypes/A019.md\n",
"Updating ../actortypes/A020.md\n",
"Updating ../actortypes/A021.md\n",
"Updating ../actortypes/A022.md\n",
"Updating ../actortypes/A023.md\n",
"Updating ../actortypes/A024.md\n",
"Updating ../actortypes/A025.md\n",
"Updating ../actortypes/A026.md\n",
"Updating ../actortypes/A027.md\n",
"Updating ../actortypes/A028.md\n",
"Updating ../actortypes/A029.md\n",
"Updating ../actortypes/A030.md\n",
"Updating ../actortypes/A031.md\n",
"Updating ../actortypes/A032.md\n",
"Updating ../actortypes/A033.md\n",
"updated ../responsetype_index.md\n",
"updated ../detections_index.md\n",
"updated ../tactics_by_responsetype_table.md\n",
"updated ../metatechniques_by_responsetype_table.md\n"
]
}
],
"source": [
"import pandas as pd\n",
"from generate_amitt_ttps import Amitt\n",
"amitt = Amitt()\n",
"amitt.generate_and_write_datafiles()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>amitt_id</th>\n",
" <th>name</th>\n",
" <th>summary</th>\n",
" <th>sector_ids</th>\n",
" <th>framework_ids</th>\n",
" <th>longname</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A001</td>\n",
" <td>data scientist</td>\n",
" <td>Person who can wrangle data, implement machine...</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A001 - data scientist</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>A002</td>\n",
" <td>target</td>\n",
" <td>Person being targeted by disinformation campaign</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW02</td>\n",
" <td>A002 - target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A003</td>\n",
" <td>trusted authority</td>\n",
" <td>Influencer</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A003 - trusted authority</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A004</td>\n",
" <td>activist</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A004 - activist</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>A005</td>\n",
" <td>community group</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A005 - community group</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>A006</td>\n",
" <td>educator</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A006 - educator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>A007</td>\n",
" <td>factchecker</td>\n",
" <td>Someone with the skills to verify whether info...</td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A007 - factchecker</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>A008</td>\n",
" <td>library</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A008 - library</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A009</td>\n",
" <td>NGO</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A009 - NGO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A010</td>\n",
" <td>religious organisation</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A010 - religious organisation</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>A011</td>\n",
" <td>school</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A011 - school</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>A012</td>\n",
" <td>account owner</td>\n",
" <td>Anyone who owns an account online</td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A012 - account owner</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>A013</td>\n",
" <td>content creator</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A013 - content creator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>A014</td>\n",
" <td>elves</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW02</td>\n",
" <td>A014 - elves</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>A015</td>\n",
" <td>general public</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW02</td>\n",
" <td>A015 - general public</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>A016</td>\n",
" <td>influencer</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A016 - influencer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>A017</td>\n",
" <td>coordinating body</td>\n",
" <td>For example the DHS</td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A017 - coordinating body</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>A018</td>\n",
" <td>government</td>\n",
" <td>Government agencies</td>\n",
" <td>S003</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A018 - government</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>A019</td>\n",
" <td>military</td>\n",
" <td></td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A019 - military</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>A020</td>\n",
" <td>policy maker</td>\n",
" <td></td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A020 - policy maker</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>A021</td>\n",
" <td>media organisation</td>\n",
" <td></td>\n",
" <td>S010</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A021 - media organisation</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>A022</td>\n",
" <td>company</td>\n",
" <td></td>\n",
" <td>S009</td>\n",
" <td>FW02</td>\n",
" <td>A022 - company</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>A023</td>\n",
" <td>adtech provider</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A023 - adtech provider</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>A024</td>\n",
" <td>developer</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A024 - developer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>A025</td>\n",
" <td>funding_site_admin</td>\n",
" <td>Funding site admin</td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A025 - funding_site_admin</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>A026</td>\n",
" <td>games designer</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A026 - games designer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>A027</td>\n",
" <td>information security</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A027 - information security</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>A028</td>\n",
" <td>platform administrator</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A028 - platform administrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>A029</td>\n",
" <td>server admininistrator</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A029 - server admininistrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>A030</td>\n",
" <td>platforms</td>\n",
" <td></td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A030 - platforms</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>A031</td>\n",
" <td>social media platform adminstrator</td>\n",
" <td>Person with the authority to make changes to a...</td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A031 - social media platform adminstrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>A032</td>\n",
" <td>social media platform outreach</td>\n",
" <td></td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A032 - social media platform outreach</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>A033</td>\n",
" <td>social media platform owner</td>\n",
" <td>Person with authority to make changes to a soc...</td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A033 - social media platform owner</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" amitt_id name \\\n",
"0 A001 data scientist \n",
"1 A002 target \n",
"2 A003 trusted authority \n",
"3 A004 activist \n",
"4 A005 community group \n",
"5 A006 educator \n",
"6 A007 factchecker \n",
"7 A008 library \n",
"8 A009 NGO \n",
"9 A010 religious organisation \n",
"10 A011 school \n",
"11 A012 account owner \n",
"12 A013 content creator \n",
"13 A014 elves \n",
"14 A015 general public \n",
"15 A016 influencer \n",
"16 A017 coordinating body \n",
"17 A018 government \n",
"18 A019 military \n",
"19 A020 policy maker \n",
"20 A021 media organisation \n",
"21 A022 company \n",
"22 A023 adtech provider \n",
"23 A024 developer \n",
"24 A025 funding_site_admin \n",
"25 A026 games designer \n",
"26 A027 information security \n",
"27 A028 platform administrator \n",
"28 A029 server admininistrator \n",
"29 A030 platforms \n",
"30 A031 social media platform adminstrator \n",
"31 A032 social media platform outreach \n",
"32 A033 social media platform owner \n",
"\n",
" summary \\\n",
"0 Person who can wrangle data, implement machine... \n",
"1 Person being targeted by disinformation campaign \n",
"2 Influencer \n",
"3 \n",
"4 \n",
"5 \n",
"6 Someone with the skills to verify whether info... \n",
"7 \n",
"8 \n",
"9 \n",
"10 \n",
"11 Anyone who owns an account online \n",
"12 \n",
"13 \n",
"14 \n",
"15 \n",
"16 For example the DHS \n",
"17 Government agencies \n",
"18 \n",
"19 \n",
"20 \n",
"21 \n",
"22 \n",
"23 \n",
"24 Funding site admin \n",
"25 \n",
"26 \n",
"27 \n",
"28 \n",
"29 \n",
"30 Person with the authority to make changes to a... \n",
"31 \n",
"32 Person with authority to make changes to a soc... \n",
"\n",
" sector_ids framework_ids \\\n",
"0 S001, S002, S003, S004, S005, S006, S007, S008... FW01, FW02 \n",
"1 S001, S002, S003, S004, S005, S006, S007, S008... FW02 \n",
"2 S001, S002, S003, S004, S005, S006, S007, S008... FW01, FW02 \n",
"3 S002 FW02 \n",
"4 S002 FW02 \n",
"5 S002 FW02 \n",
"6 S002 FW02 \n",
"7 S002 FW02 \n",
"8 S002 FW02 \n",
"9 S002 FW02 \n",
"10 S002 FW02 \n",
"11 S006 FW01\\nFW02 \n",
"12 S006 FW01\\nFW02 \n",
"13 S006 FW02 \n",
"14 S006 FW02 \n",
"15 S006 FW01\\nFW02 \n",
"16 S003 FW02 \n",
"17 S003 FW01\\nFW02 \n",
"18 S003 FW02 \n",
"19 S003 FW02 \n",
"20 S010 FW01\\nFW02 \n",
"21 S009 FW02 \n",
"22 S008 FW02 \n",
"23 S008 FW02 \n",
"24 S008 FW02 \n",
"25 S008 FW01, FW02 \n",
"26 S008 FW02 \n",
"27 S008 FW02 \n",
"28 S008 FW02 \n",
"29 S007 FW02 \n",
"30 S007 FW02 \n",
"31 S007 FW02 \n",
"32 S007 FW02 \n",
"\n",
" longname \n",
"0 A001 - data scientist \n",
"1 A002 - target \n",
"2 A003 - trusted authority \n",
"3 A004 - activist \n",
"4 A005 - community group \n",
"5 A006 - educator \n",
"6 A007 - factchecker \n",
"7 A008 - library \n",
"8 A009 - NGO \n",
"9 A010 - religious organisation \n",
"10 A011 - school \n",
"11 A012 - account owner \n",
"12 A013 - content creator \n",
"13 A014 - elves \n",
"14 A015 - general public \n",
"15 A016 - influencer \n",
"16 A017 - coordinating body \n",
"17 A018 - government \n",
"18 A019 - military \n",
"19 A020 - policy maker \n",
"20 A021 - media organisation \n",
"21 A022 - company \n",
"22 A023 - adtech provider \n",
"23 A024 - developer \n",
"24 A025 - funding_site_admin \n",
"25 A026 - games designer \n",
"26 A027 - information security \n",
"27 A028 - platform administrator \n",
"28 A029 - server admininistrator \n",
"29 A030 - platforms \n",
"30 A031 - social media platform adminstrator \n",
"31 A032 - social media platform outreach \n",
"32 A033 - social media platform owner "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"amitt.df_actortypes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

425
CODE/.ipynb_checkpoints/DISARM_code_testbed-checkpoint.ipynb Normal file
View file

@ -0,0 +1,425 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Test area for DISARM code"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dict_keys(['df_phases', 'df_frameworks', 'df_techniques', 'df_tasks', 'df_incidents', 'df_counters', 'df_detections', 'df_actortypes', 'df_resources', 'df_responsetypes', 'df_metatechniques', 'it', 'df_tactics', 'df_techniques_per_tactic', 'df_counters_per_tactic', 'phases', 'tactics', 'techniques', 'counters', 'metatechniques', 'actortypes', 'resources', 'num_tactics', 'cross_counterid_techniqueid', 'cross_counterid_resourceid', 'cross_counterid_actortypeid'])\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>amitt_id</th>\n",
" <th>technique_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0007</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0043</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0053</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0025</td>\n",
" </tr>\n",
" <tr>\n",
" <th>136</th>\n",
" <td>C00220</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>137</th>\n",
" <td>C00221</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>138</th>\n",
" <td>C00222</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>139</th>\n",
" <td>C00223</td>\n",
" <td></td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>898 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" amitt_id technique_id\n",
"0 C00006 T0007\n",
"0 C00006 T0015\n",
"0 C00006 T0018\n",
"0 C00006 T0043\n",
"0 C00006 T0053\n",
".. ... ...\n",
"135 C00219 T0025\n",
"136 C00220 \n",
"137 C00221 \n",
"138 C00222 \n",
"139 C00223 \n",
"\n",
"[898 rows x 2 columns]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"import sqlite3 as sql\n",
"from generate_DISARM_pages import Disarm\n",
"\n",
"\n",
"# Generate AMITT datasets\n",
"disarm = Disarm()\n",
"\n",
"# Check which amitt variables we can see from here\n",
"print('{}'.format(vars(disarm).keys()))\n",
"vars(disarm)['cross_counterid_techniqueid']"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>actor_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>A033</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>A007</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>C00009</td>\n",
" <td>A016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>C00009</td>\n",
" <td>A006</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>C00010</td>\n",
" <td>A020</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>136</th>\n",
" <td>C00220</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>137</th>\n",
" <td>C00221</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>138</th>\n",
" <td>C00222</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>139</th>\n",
" <td>C00223</td>\n",
" <td></td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>166 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" id actor_id\n",
"0 C00006 A033\n",
"1 C00008 A007\n",
"2 C00009 A016\n",
"2 C00009 A006\n",
"3 C00010 A020\n",
".. ... ...\n",
"135 C00219 \n",
"136 C00220 \n",
"137 C00221 \n",
"138 C00222 \n",
"139 C00223 \n",
"\n",
"[166 rows x 2 columns]"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"disarm.cross_counterid_actorid"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>technique_id</th>\n",
" <th>Weight</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>TA01</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>TA06</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>TA08</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>T0006</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>T0009</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>134</th>\n",
" <td>C00216</td>\n",
" <td>T0018</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>134</th>\n",
" <td>C00216</td>\n",
" <td>T0057</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0024</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0026</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0025</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>717 rows × 3 columns</p>\n",
"</div>"
],
"text/plain": [
" id technique_id Weight\n",
"1 C00008 TA01 1\n",
"1 C00008 TA06 1\n",
"1 C00008 TA08 1\n",
"1 C00008 T0006 1\n",
"1 C00008 T0009 1\n",
".. ... ... ...\n",
"134 C00216 T0018 1\n",
"134 C00216 T0057 1\n",
"135 C00219 T0024 1\n",
"135 C00219 T0026 1\n",
"135 C00219 T0025 1\n",
"\n",
"[717 rows x 3 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ct = disarm.cross_counterid_techniqueid\n",
"ct['Weight'] = 1\n",
"ct = ct[ct['technique_id'].str.len() > 0]\n",
"ct.to_csv('../visualisations/cross_counterid_techniqueid.csv', index=False, header=['Source','Target', 'Weight'])\n",
"ct"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

110
CODE/.ipynb_checkpoints/check_DISARM_excel_changes-checkpoint.ipynb Normal file
View file

@ -0,0 +1,110 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# fix the problem with excelfile changes\n",
"\n",
"Background: DISARM's master dataset is in an excelfile. Changes in this dont' show up in github, so it's difficult to tell what's changed between versions. Code below checks for those differences - use this repeatedly until versions align. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import sqlite3 as sql\n",
"from generate_DISARM_pages import Disarm\n",
"import pandas as pd\n",
"import numpy as np\n",
"import os\n",
"from sklearn.feature_extraction.text import CountVectorizer\n",
"pd.set_option('display.max_rows', 1000)\n",
"pd.set_option('display.max_colwidth', -1)\n",
"\n",
"newfile = '../DISARM_MASTER_DATA/DISARM_FRAMEWORKS_MASTER.xlsx'\n",
"oldfile = '../DISARM_MASTER_DATA/DISARM_FRAMEWORKS_MASTER_previous_version.xlsx'\n",
"\n",
"# Load dfs from file\n",
"newdfs = {}\n",
"newxlsx = pd.ExcelFile(newfile)\n",
"for sheetname in newxlsx.sheet_names:\n",
" newdfs[sheetname] = newxlsx.parse(sheetname)\n",
" newdfs[sheetname].fillna('', inplace=True)\n",
"\n",
"olddfs = {}\n",
"oldxlsx = pd.ExcelFile(oldfile)\n",
"for sheetname in oldxlsx.sheet_names:\n",
" olddfs[sheetname] = oldxlsx.parse(sheetname)\n",
" olddfs[sheetname].fillna('', inplace=True)\n",
"\n",
"addedtables = newdfs.keys() - olddfs.keys()\n",
"losttables = olddfs.keys() - newdfs.keys()\n",
"if len(addedtables) + len(losttables) > 0:\n",
" print('Table changes: new tables are {}, lost tables are {}'.format(addedtables, losttables))\n",
"\n",
"def investigate_table(table):\n",
" print('\\n\\nTable {} is changed'.format(table))\n",
" # Column headings\n",
" coldiffs = set(newdfs[table].columns).symmetric_difference(set(olddfs[table].columns))\n",
" if len(coldiffs) > 0:\n",
" print('column differences: {}'.format(coldiffs))\n",
" # length\n",
" if len(newdfs[table]) != len(olddfs[table]):\n",
" print('length differences: new {} old {}'.format(len(newdfs[table]), len(olddfs[table])))\n",
"\n",
" # column by column\n",
" for column in newdfs[table].columns:\n",
" coldiffs = newdfs[table][column] != olddfs[table][column]\n",
" if len(newdfs[table][coldiffs]) > 0:\n",
" print('Differences in column {}'.format(column))\n",
" return\n",
"\n",
"for table in newdfs.keys():\n",
" if newdfs[table].equals(olddfs[table]) == False:\n",
" investigate_table(table)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Look at individual table differences\n",
"table = 'countermeasures'\n",
"column = 'summary'\n",
"coldiffs = newdfs[table][column] != olddfs[table][column]\n",
"diffcols = pd.DataFrame()\n",
"diffcols['amitt_id'] = newdfs[table][coldiffs]['amitt_id']\n",
"diffcols['new'] = newdfs[table][coldiffs][column]\n",
"diffcols['old'] = olddfs[table][coldiffs][column]\n",
"diffcols[diffcols['old'] != '']"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

556
CODE/.ipynb_checkpoints/generate_DISARM_pages-checkpoint.ipynb Normal file
View file

@ -0,0 +1,556 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Generate AMITT github files\n",
"\n",
"Generate all the AMITT github files from the AMITT master spreadsheet, being careful to reatin any comments people have made below the \"don't write above this\" line in them. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [
{
"ename": "ImportError",
"evalue": "cannot import name 'Disarm' from 'generate_DISARM_pages' (/Users/sara/Dropbox/SJT_Projects_current/DISARM_foundation/code_repositories/DISARM-Frameworks/CODE/generate_DISARM_pages.py)",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-2-25052c215084>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mgenerate_DISARM_pages\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDisarm\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mdisarm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mDisarm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mdisarm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgenerate_and_write_datafiles\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mImportError\u001b[0m: cannot import name 'Disarm' from 'generate_DISARM_pages' (/Users/sara/Dropbox/SJT_Projects_current/DISARM_foundation/code_repositories/DISARM-Frameworks/CODE/generate_DISARM_pages.py)"
]
}
],
"source": [
"import pandas as pd\n",
"from generate_DISARM_pages import Disarm\n",
"disarm = Disarm()\n",
"disarm.generate_and_write_datafiles()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>amitt_id</th>\n",
" <th>name</th>\n",
" <th>summary</th>\n",
" <th>sector_ids</th>\n",
" <th>framework_ids</th>\n",
" <th>longname</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A001</td>\n",
" <td>data scientist</td>\n",
" <td>Person who can wrangle data, implement machine...</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A001 - data scientist</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>A002</td>\n",
" <td>target</td>\n",
" <td>Person being targeted by disinformation campaign</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW02</td>\n",
" <td>A002 - target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A003</td>\n",
" <td>trusted authority</td>\n",
" <td>Influencer</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A003 - trusted authority</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A004</td>\n",
" <td>activist</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A004 - activist</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>A005</td>\n",
" <td>community group</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A005 - community group</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>A006</td>\n",
" <td>educator</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A006 - educator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>A007</td>\n",
" <td>factchecker</td>\n",
" <td>Someone with the skills to verify whether info...</td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A007 - factchecker</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>A008</td>\n",
" <td>library</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A008 - library</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A009</td>\n",
" <td>NGO</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A009 - NGO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A010</td>\n",
" <td>religious organisation</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A010 - religious organisation</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>A011</td>\n",
" <td>school</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A011 - school</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>A012</td>\n",
" <td>account owner</td>\n",
" <td>Anyone who owns an account online</td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A012 - account owner</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>A013</td>\n",
" <td>content creator</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A013 - content creator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>A014</td>\n",
" <td>elves</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW02</td>\n",
" <td>A014 - elves</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>A015</td>\n",
" <td>general public</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW02</td>\n",
" <td>A015 - general public</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>A016</td>\n",
" <td>influencer</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A016 - influencer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>A017</td>\n",
" <td>coordinating body</td>\n",
" <td>For example the DHS</td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A017 - coordinating body</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>A018</td>\n",
" <td>government</td>\n",
" <td>Government agencies</td>\n",
" <td>S003</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A018 - government</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>A019</td>\n",
" <td>military</td>\n",
" <td></td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A019 - military</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>A020</td>\n",
" <td>policy maker</td>\n",
" <td></td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A020 - policy maker</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>A021</td>\n",
" <td>media organisation</td>\n",
" <td></td>\n",
" <td>S010</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A021 - media organisation</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>A022</td>\n",
" <td>company</td>\n",
" <td></td>\n",
" <td>S009</td>\n",
" <td>FW02</td>\n",
" <td>A022 - company</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>A023</td>\n",
" <td>adtech provider</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A023 - adtech provider</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>A024</td>\n",
" <td>developer</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A024 - developer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>A025</td>\n",
" <td>funding_site_admin</td>\n",
" <td>Funding site admin</td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A025 - funding_site_admin</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>A026</td>\n",
" <td>games designer</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A026 - games designer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>A027</td>\n",
" <td>information security</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A027 - information security</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>A028</td>\n",
" <td>platform administrator</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A028 - platform administrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>A029</td>\n",
" <td>server admininistrator</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A029 - server admininistrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>A030</td>\n",
" <td>platforms</td>\n",
" <td></td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A030 - platforms</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>A031</td>\n",
" <td>social media platform adminstrator</td>\n",
" <td>Person with the authority to make changes to a...</td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A031 - social media platform adminstrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>A032</td>\n",
" <td>social media platform outreach</td>\n",
" <td></td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A032 - social media platform outreach</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>A033</td>\n",
" <td>social media platform owner</td>\n",
" <td>Person with authority to make changes to a soc...</td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A033 - social media platform owner</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" amitt_id name \\\n",
"0 A001 data scientist \n",
"1 A002 target \n",
"2 A003 trusted authority \n",
"3 A004 activist \n",
"4 A005 community group \n",
"5 A006 educator \n",
"6 A007 factchecker \n",
"7 A008 library \n",
"8 A009 NGO \n",
"9 A010 religious organisation \n",
"10 A011 school \n",
"11 A012 account owner \n",
"12 A013 content creator \n",
"13 A014 elves \n",
"14 A015 general public \n",
"15 A016 influencer \n",
"16 A017 coordinating body \n",
"17 A018 government \n",
"18 A019 military \n",
"19 A020 policy maker \n",
"20 A021 media organisation \n",
"21 A022 company \n",
"22 A023 adtech provider \n",
"23 A024 developer \n",
"24 A025 funding_site_admin \n",
"25 A026 games designer \n",
"26 A027 information security \n",
"27 A028 platform administrator \n",
"28 A029 server admininistrator \n",
"29 A030 platforms \n",
"30 A031 social media platform adminstrator \n",
"31 A032 social media platform outreach \n",
"32 A033 social media platform owner \n",
"\n",
" summary \\\n",
"0 Person who can wrangle data, implement machine... \n",
"1 Person being targeted by disinformation campaign \n",
"2 Influencer \n",
"3 \n",
"4 \n",
"5 \n",
"6 Someone with the skills to verify whether info... \n",
"7 \n",
"8 \n",
"9 \n",
"10 \n",
"11 Anyone who owns an account online \n",
"12 \n",
"13 \n",
"14 \n",
"15 \n",
"16 For example the DHS \n",
"17 Government agencies \n",
"18 \n",
"19 \n",
"20 \n",
"21 \n",
"22 \n",
"23 \n",
"24 Funding site admin \n",
"25 \n",
"26 \n",
"27 \n",
"28 \n",
"29 \n",
"30 Person with the authority to make changes to a... \n",
"31 \n",
"32 Person with authority to make changes to a soc... \n",
"\n",
" sector_ids framework_ids \\\n",
"0 S001, S002, S003, S004, S005, S006, S007, S008... FW01, FW02 \n",
"1 S001, S002, S003, S004, S005, S006, S007, S008... FW02 \n",
"2 S001, S002, S003, S004, S005, S006, S007, S008... FW01, FW02 \n",
"3 S002 FW02 \n",
"4 S002 FW02 \n",
"5 S002 FW02 \n",
"6 S002 FW02 \n",
"7 S002 FW02 \n",
"8 S002 FW02 \n",
"9 S002 FW02 \n",
"10 S002 FW02 \n",
"11 S006 FW01\\nFW02 \n",
"12 S006 FW01\\nFW02 \n",
"13 S006 FW02 \n",
"14 S006 FW02 \n",
"15 S006 FW01\\nFW02 \n",
"16 S003 FW02 \n",
"17 S003 FW01\\nFW02 \n",
"18 S003 FW02 \n",
"19 S003 FW02 \n",
"20 S010 FW01\\nFW02 \n",
"21 S009 FW02 \n",
"22 S008 FW02 \n",
"23 S008 FW02 \n",
"24 S008 FW02 \n",
"25 S008 FW01, FW02 \n",
"26 S008 FW02 \n",
"27 S008 FW02 \n",
"28 S008 FW02 \n",
"29 S007 FW02 \n",
"30 S007 FW02 \n",
"31 S007 FW02 \n",
"32 S007 FW02 \n",
"\n",
" longname \n",
"0 A001 - data scientist \n",
"1 A002 - target \n",
"2 A003 - trusted authority \n",
"3 A004 - activist \n",
"4 A005 - community group \n",
"5 A006 - educator \n",
"6 A007 - factchecker \n",
"7 A008 - library \n",
"8 A009 - NGO \n",
"9 A010 - religious organisation \n",
"10 A011 - school \n",
"11 A012 - account owner \n",
"12 A013 - content creator \n",
"13 A014 - elves \n",
"14 A015 - general public \n",
"15 A016 - influencer \n",
"16 A017 - coordinating body \n",
"17 A018 - government \n",
"18 A019 - military \n",
"19 A020 - policy maker \n",
"20 A021 - media organisation \n",
"21 A022 - company \n",
"22 A023 - adtech provider \n",
"23 A024 - developer \n",
"24 A025 - funding_site_admin \n",
"25 A026 - games designer \n",
"26 A027 - information security \n",
"27 A028 - platform administrator \n",
"28 A029 - server admininistrator \n",
"29 A030 - platforms \n",
"30 A031 - social media platform adminstrator \n",
"31 A032 - social media platform outreach \n",
"32 A033 - social media platform owner "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"disarm.df_actortypes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

2548
CODE/.ipynb_checkpoints/generate_DISARM_sql-checkpoint.ipynb Normal file
View file

File diff suppressed because it is too large Load diff

425
CODE/DISARM_code_testbed.ipynb Normal file
View file

@ -0,0 +1,425 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Test area for DISARM code"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dict_keys(['df_phases', 'df_frameworks', 'df_techniques', 'df_tasks', 'df_incidents', 'df_counters', 'df_detections', 'df_actortypes', 'df_resources', 'df_responsetypes', 'df_metatechniques', 'it', 'df_tactics', 'df_techniques_per_tactic', 'df_counters_per_tactic', 'phases', 'tactics', 'techniques', 'counters', 'metatechniques', 'actortypes', 'resources', 'num_tactics', 'cross_counterid_techniqueid', 'cross_counterid_resourceid', 'cross_counterid_actortypeid'])\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>amitt_id</th>\n",
" <th>technique_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0007</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0018</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0043</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>T0053</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0025</td>\n",
" </tr>\n",
" <tr>\n",
" <th>136</th>\n",
" <td>C00220</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>137</th>\n",
" <td>C00221</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>138</th>\n",
" <td>C00222</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>139</th>\n",
" <td>C00223</td>\n",
" <td></td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>898 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" amitt_id technique_id\n",
"0 C00006 T0007\n",
"0 C00006 T0015\n",
"0 C00006 T0018\n",
"0 C00006 T0043\n",
"0 C00006 T0053\n",
".. ... ...\n",
"135 C00219 T0025\n",
"136 C00220 \n",
"137 C00221 \n",
"138 C00222 \n",
"139 C00223 \n",
"\n",
"[898 rows x 2 columns]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"import sqlite3 as sql\n",
"from generate_DISARM_pages import Disarm\n",
"\n",
"\n",
"# Generate AMITT datasets\n",
"disarm = Disarm()\n",
"\n",
"# Check which amitt variables we can see from here\n",
"print('{}'.format(vars(disarm).keys()))\n",
"vars(disarm)['cross_counterid_techniqueid']"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>actor_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>C00006</td>\n",
" <td>A033</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>A007</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>C00009</td>\n",
" <td>A016</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>C00009</td>\n",
" <td>A006</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>C00010</td>\n",
" <td>A020</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>136</th>\n",
" <td>C00220</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>137</th>\n",
" <td>C00221</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>138</th>\n",
" <td>C00222</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <th>139</th>\n",
" <td>C00223</td>\n",
" <td></td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>166 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" id actor_id\n",
"0 C00006 A033\n",
"1 C00008 A007\n",
"2 C00009 A016\n",
"2 C00009 A006\n",
"3 C00010 A020\n",
".. ... ...\n",
"135 C00219 \n",
"136 C00220 \n",
"137 C00221 \n",
"138 C00222 \n",
"139 C00223 \n",
"\n",
"[166 rows x 2 columns]"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"disarm.cross_counterid_actorid"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>technique_id</th>\n",
" <th>Weight</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>TA01</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>TA06</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>TA08</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>T0006</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>C00008</td>\n",
" <td>T0009</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>134</th>\n",
" <td>C00216</td>\n",
" <td>T0018</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>134</th>\n",
" <td>C00216</td>\n",
" <td>T0057</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0024</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0026</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>C00219</td>\n",
" <td>T0025</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>717 rows × 3 columns</p>\n",
"</div>"
],
"text/plain": [
" id technique_id Weight\n",
"1 C00008 TA01 1\n",
"1 C00008 TA06 1\n",
"1 C00008 TA08 1\n",
"1 C00008 T0006 1\n",
"1 C00008 T0009 1\n",
".. ... ... ...\n",
"134 C00216 T0018 1\n",
"134 C00216 T0057 1\n",
"135 C00219 T0024 1\n",
"135 C00219 T0026 1\n",
"135 C00219 T0025 1\n",
"\n",
"[717 rows x 3 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ct = disarm.cross_counterid_techniqueid\n",
"ct['Weight'] = 1\n",
"ct = ct[ct['technique_id'].str.len() > 0]\n",
"ct.to_csv('../visualisations/cross_counterid_techniqueid.csv', index=False, header=['Source','Target', 'Weight'])\n",
"ct"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

BIN
CODE/__pycache__/generate_DISARM_pages.cpython-38.pyc Normal file
View file

Binary file not shown.

110
CODE/compare_DISARM_excel_versions.ipynb Normal file
View file

@ -0,0 +1,110 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# fix the problem with excelfile changes\n",
"\n",
"Background: DISARM's master dataset is in an excelfile. Changes in this dont' show up in github, so it's difficult to tell what's changed between versions. Code below checks for those differences - use this repeatedly until versions align. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import sqlite3 as sql\n",
"from generate_DISARM_pages import Disarm\n",
"import pandas as pd\n",
"import numpy as np\n",
"import os\n",
"from sklearn.feature_extraction.text import CountVectorizer\n",
"pd.set_option('display.max_rows', 1000)\n",
"pd.set_option('display.max_colwidth', -1)\n",
"\n",
"newfile = '../DISARM_MASTER_DATA/DISARM_FRAMEWORKS_MASTER.xlsx'\n",
"oldfile = '../DISARM_MASTER_DATA/DISARM_FRAMEWORKS_MASTER_previous_version.xlsx'\n",
"\n",
"# Load dfs from file\n",
"newdfs = {}\n",
"newxlsx = pd.ExcelFile(newfile)\n",
"for sheetname in newxlsx.sheet_names:\n",
" newdfs[sheetname] = newxlsx.parse(sheetname)\n",
" newdfs[sheetname].fillna('', inplace=True)\n",
"\n",
"olddfs = {}\n",
"oldxlsx = pd.ExcelFile(oldfile)\n",
"for sheetname in oldxlsx.sheet_names:\n",
" olddfs[sheetname] = oldxlsx.parse(sheetname)\n",
" olddfs[sheetname].fillna('', inplace=True)\n",
"\n",
"addedtables = newdfs.keys() - olddfs.keys()\n",
"losttables = olddfs.keys() - newdfs.keys()\n",
"if len(addedtables) + len(losttables) > 0:\n",
" print('Table changes: new tables are {}, lost tables are {}'.format(addedtables, losttables))\n",
"\n",
"def investigate_table(table):\n",
" print('\\n\\nTable {} is changed'.format(table))\n",
" # Column headings\n",
" coldiffs = set(newdfs[table].columns).symmetric_difference(set(olddfs[table].columns))\n",
" if len(coldiffs) > 0:\n",
" print('column differences: {}'.format(coldiffs))\n",
" # length\n",
" if len(newdfs[table]) != len(olddfs[table]):\n",
" print('length differences: new {} old {}'.format(len(newdfs[table]), len(olddfs[table])))\n",
"\n",
" # column by column\n",
" for column in newdfs[table].columns:\n",
" coldiffs = newdfs[table][column] != olddfs[table][column]\n",
" if len(newdfs[table][coldiffs]) > 0:\n",
" print('Differences in column {}'.format(column))\n",
" return\n",
"\n",
"for table in newdfs.keys():\n",
" if newdfs[table].equals(olddfs[table]) == False:\n",
" investigate_table(table)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Look at individual table differences\n",
"table = 'countermeasures'\n",
"column = 'summary'\n",
"coldiffs = newdfs[table][column] != olddfs[table][column]\n",
"diffcols = pd.DataFrame()\n",
"diffcols['amitt_id'] = newdfs[table][coldiffs]['amitt_id']\n",
"diffcols['new'] = newdfs[table][coldiffs][column]\n",
"diffcols['old'] = olddfs[table][coldiffs][column]\n",
"diffcols[diffcols['old'] != '']"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

268
CODE/generate_DISARM_incident_visualisations.ipynb Normal file
View file

@ -0,0 +1,268 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Create AMITT incident visualisations\n",
"\n",
"Many thanks to https://python-graph-gallery.com/91-customize-seaborn-heatmap/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import seaborn as sns\n",
"import pandas as pd\n",
"import numpy as np\n",
"import generate_amitt_ttps\n",
"\n",
"# Check that heatmap works\n",
"df = pd.DataFrame(np.random.random((10,12)), columns=[\"a\",\"b\",\"c\",\"d\",\"e\",\"f\",\"g\",\"h\",\"i\",\"j\",\"k\",\"l\"])\n",
"sns.heatmap(df, annot=True, annot_kws={\"size\": 7})"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amitt = generate_amitt_ttps.Amitt()\n",
"redgrid = amitt.create_padded_framework_table('AMITT Red', 'technique_ids', False)\n",
"\n",
"techcounts = amitt.it[['id_incident','id_technique']].drop_duplicates().groupby('id_technique').count().to_dict()['id_incident']\n",
"techlabels = redgrid[2:][:]\n",
"nrows = len(techlabels)\n",
"ncols = len(techlabels[0])\n",
"techgrid = np.zeros([nrows, ncols], dtype = int)\n",
"\n",
"for row in range(nrows):\n",
" for col in range(ncols):\n",
" if techlabels[row][col] in techcounts:\n",
" techgrid[row][col] = techcounts[techlabels[row][col]]\n",
"\n",
"sns.heatmap(techgrid, annot=True, annot_kws={\"size\": 7})\n",
"techgrid"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amitt.df_tactics"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amitt.it"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ct = amitt.cross_counterid_techniqueid\n",
"ct[ct['technique_id'] != '']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ct[(ct['id'] == 'C00197') & (ct['technique_id'].isin(['T0002', 'T0007']))]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ct = ct[ct['technique_id'].isin(amitt.df_techniques['id'].to_list()) & ct['id'].isin(amitt.df_counters['id'].to_list())]\n",
"ct"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"technique_id_list = ['T0007', 'T0008', 'T0022', 'T0023', 'T0043', 'T0052', 'T0036', 'T0037', 'T0038']\n",
"counter_id_list = ['C00009', 'C00008', 'C00042', 'C00030', 'C00093', 'C00193', 'C00073', 'C000197', 'C00174', 'C00205']\n",
"possible_counters_for_techniques = ct[ct['technique_id'].isin(technique_id_list)] \n",
"possible_techniques_for_counters = ct[ct['id'].isin(counter_id_list)] \n",
"coverage = ct[(ct['id'].isin(counter_id_list)) & (ct['technique_id'].isin(technique_id_list))]\n",
"coverage"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"possible_techniques_for_counters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"possible_counters_for_techniques"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"clicked button T0045 8 7\n",
"clicked button T0046 9 7\n",
"clicked button T0049 4 8\n",
"clicked button T0057 2 9\n",
"clicked button T0060 4 10\n",
"clicked button T0029 2 6\n",
"clicked button T0016 2 4\n"
]
}
],
"source": [
"import tkinter as Tk\n",
"import numpy as np\n",
"import generate_amitt_ttps\n",
"\n",
"class Begueradj(Tk.Frame):\n",
" def __init__(self,parent):\n",
" amitt = generate_amitt_ttps.Amitt()\n",
" self.redgrid = amitt.create_padded_framework_table('AMITT Red', 'technique_ids', False)\n",
" self.bluegrid = amitt.create_padded_framework_table('AMITT Blue', 'counter_ids', False)\n",
"\n",
" Tk.Frame.__init__(self, parent)\n",
" self.parent = parent\n",
" self.button= ''\n",
" self.initialize()\n",
" \n",
" def initialize(self):\n",
" '''\n",
" Draw the GUI\n",
" '''\n",
" self.parent.title(\"AMITT FRAMEWORK COVERAGE\") \n",
" self.parent.grid_rowconfigure(1,weight=1)\n",
" self.parent.grid_columnconfigure(1,weight=1)\n",
"\n",
" self.frame = Tk.Frame(self.parent) \n",
" self.frame.pack(fill=Tk.X, padx=5, pady=5)\n",
"\n",
" # Create a 6x7 array of zeros as the one you used\n",
" numrows = len(self.redgrid) - 1\n",
" numcols = len(self.redgrid[0])\n",
" self.buttons = {}\n",
" for row in range(1,numrows):\n",
" for col in range(0,numcols):\n",
" button_id = self.redgrid[row][col]\n",
" self.button = Tk.Button(self.frame, text = button_id, bg='blue', \n",
" command= lambda bid=button_id, row=row, col=col: self.clicked(bid, row, col))\n",
" self.button.grid(row=row, column=col)\n",
" \n",
" def clicked(self, bid, row, col):\n",
" print('clicked button {} {} {}'.format(bid, row, col))\n",
" self.find_in_grid(self.frame, row, col)\n",
"\n",
" def find_in_grid(self, frame, row, column):\n",
" for children in frame.children.values():\n",
" info = children.grid_info()\n",
" #note that rows and column numbers are stored as string\n",
" if info['row'] == str(row) and info['column'] == str(column):\n",
" print('{}'.format(children.get()))\n",
" return None\n",
"\n",
"root=Tk.Tk()\n",
"app = Begueradj(root) \n",
"root.mainloop()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"redgrid"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for row in range(2,len(redgrid)):\n",
" print(len(redgrid[row]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"redgrid[1][2]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

575
CODE/generate_DISARM_pages.ipynb Normal file
View file

@ -0,0 +1,575 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Generate AMITT github files\n",
"\n",
"Generate all the AMITT github files from the AMITT master spreadsheet, being careful to reatin any comments people have made below the \"don't write above this\" line in them. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"updated ../generated_pages/disarm_red_framework.md\n",
"updated ../generated_files/disarm_red_framework_clickable.html\n",
"updated ../generated_pages/disarm_blue_framework.md\n",
"updated ../generated_files/disarm_blue_framework_clickable.html\n",
"Temp: objecttype phase\n",
"updated ../generated_pages/phases_index.md\n",
"Temp: objecttype tactic\n",
"updated ../generated_pages/tactics_index.md\n",
"Temp: objecttype technique\n",
"updated ../generated_pages/techniques_index.md\n",
"Temp: objecttype task\n",
"updated ../generated_pages/tasks_index.md\n",
"Temp: objecttype incident\n",
"updated ../generated_pages/incidents_index.md\n",
"Temp: objecttype counter\n",
"updated ../generated_pages/counters_index.md\n",
"Temp: objecttype metatechnique\n",
"updated ../generated_pages/metatechniques_index.md\n",
"Temp: objecttype actortype\n",
"updated ../generated_pages/actortypes_index.md\n",
"updated ../generated_pages/responsetype_index.md\n",
"updated ../generated_pages/detections_index.md\n",
"updated ../generated_pages/tactics_by_responsetype_table.md\n",
"updated ../generated_pages/metatechniques_by_responsetype_table.md\n"
]
}
],
"source": [
"import pandas as pd\n",
"from generate_DISARM_pages import Disarm\n",
"disarm = Disarm()\n",
"disarm.generate_and_write_datafiles()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>disarm_id</th>\n",
" <th>name</th>\n",
" <th>summary</th>\n",
" <th>sector_ids</th>\n",
" <th>framework_ids</th>\n",
" <th>longname</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A001</td>\n",
" <td>data scientist</td>\n",
" <td>Person who can wrangle data, implement machine...</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A001 - data scientist</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>A002</td>\n",
" <td>target</td>\n",
" <td>Person being targeted by disinformation campaign</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW02</td>\n",
" <td>A002 - target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A003</td>\n",
" <td>trusted authority</td>\n",
" <td>Influencer</td>\n",
" <td>S001, S002, S003, S004, S005, S006, S007, S008...</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A003 - trusted authority</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A004</td>\n",
" <td>activist</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A004 - activist</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>A005</td>\n",
" <td>community group</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A005 - community group</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>A006</td>\n",
" <td>educator</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A006 - educator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>A007</td>\n",
" <td>factchecker</td>\n",
" <td>Someone with the skills to verify whether info...</td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A007 - factchecker</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>A008</td>\n",
" <td>library</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A008 - library</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A009</td>\n",
" <td>NGO</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A009 - NGO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A010</td>\n",
" <td>religious organisation</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A010 - religious organisation</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>A011</td>\n",
" <td>school</td>\n",
" <td></td>\n",
" <td>S002</td>\n",
" <td>FW02</td>\n",
" <td>A011 - school</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>A012</td>\n",
" <td>account owner</td>\n",
" <td>Anyone who owns an account online</td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A012 - account owner</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>A013</td>\n",
" <td>content creator</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A013 - content creator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>A014</td>\n",
" <td>elves</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW02</td>\n",
" <td>A014 - elves</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>A015</td>\n",
" <td>general public</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW02</td>\n",
" <td>A015 - general public</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>A016</td>\n",
" <td>influencer</td>\n",
" <td></td>\n",
" <td>S006</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A016 - influencer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>A017</td>\n",
" <td>coordinating body</td>\n",
" <td>For example the DHS</td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A017 - coordinating body</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>A018</td>\n",
" <td>government</td>\n",
" <td>Government agencies</td>\n",
" <td>S003</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A018 - government</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>A019</td>\n",
" <td>military</td>\n",
" <td></td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A019 - military</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>A020</td>\n",
" <td>policy maker</td>\n",
" <td></td>\n",
" <td>S003</td>\n",
" <td>FW02</td>\n",
" <td>A020 - policy maker</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>A021</td>\n",
" <td>media organisation</td>\n",
" <td></td>\n",
" <td>S010</td>\n",
" <td>FW01\\nFW02</td>\n",
" <td>A021 - media organisation</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>A022</td>\n",
" <td>company</td>\n",
" <td></td>\n",
" <td>S009</td>\n",
" <td>FW02</td>\n",
" <td>A022 - company</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>A023</td>\n",
" <td>adtech provider</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A023 - adtech provider</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>A024</td>\n",
" <td>developer</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A024 - developer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>A025</td>\n",
" <td>funding_site_admin</td>\n",
" <td>Funding site admin</td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A025 - funding_site_admin</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>A026</td>\n",
" <td>games designer</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW01, FW02</td>\n",
" <td>A026 - games designer</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>A027</td>\n",
" <td>information security</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A027 - information security</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>A028</td>\n",
" <td>platform administrator</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A028 - platform administrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>A029</td>\n",
" <td>server admininistrator</td>\n",
" <td></td>\n",
" <td>S008</td>\n",
" <td>FW02</td>\n",
" <td>A029 - server admininistrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>A030</td>\n",
" <td>platforms</td>\n",
" <td></td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A030 - platforms</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>A031</td>\n",
" <td>social media platform adminstrator</td>\n",
" <td>Person with the authority to make changes to a...</td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A031 - social media platform adminstrator</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>A032</td>\n",
" <td>social media platform outreach</td>\n",
" <td></td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A032 - social media platform outreach</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>A033</td>\n",
" <td>social media platform owner</td>\n",
" <td>Person with authority to make changes to a soc...</td>\n",
" <td>S007</td>\n",
" <td>FW02</td>\n",
" <td>A033 - social media platform owner</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" disarm_id name \\\n",
"0 A001 data scientist \n",
"1 A002 target \n",
"2 A003 trusted authority \n",
"3 A004 activist \n",
"4 A005 community group \n",
"5 A006 educator \n",
"6 A007 factchecker \n",
"7 A008 library \n",
"8 A009 NGO \n",
"9 A010 religious organisation \n",
"10 A011 school \n",
"11 A012 account owner \n",
"12 A013 content creator \n",
"13 A014 elves \n",
"14 A015 general public \n",
"15 A016 influencer \n",
"16 A017 coordinating body \n",
"17 A018 government \n",
"18 A019 military \n",
"19 A020 policy maker \n",
"20 A021 media organisation \n",
"21 A022 company \n",
"22 A023 adtech provider \n",
"23 A024 developer \n",
"24 A025 funding_site_admin \n",
"25 A026 games designer \n",
"26 A027 information security \n",
"27 A028 platform administrator \n",
"28 A029 server admininistrator \n",
"29 A030 platforms \n",
"30 A031 social media platform adminstrator \n",
"31 A032 social media platform outreach \n",
"32 A033 social media platform owner \n",
"\n",
" summary \\\n",
"0 Person who can wrangle data, implement machine... \n",
"1 Person being targeted by disinformation campaign \n",
"2 Influencer \n",
"3 \n",
"4 \n",
"5 \n",
"6 Someone with the skills to verify whether info... \n",
"7 \n",
"8 \n",
"9 \n",
"10 \n",
"11 Anyone who owns an account online \n",
"12 \n",
"13 \n",
"14 \n",
"15 \n",
"16 For example the DHS \n",
"17 Government agencies \n",
"18 \n",
"19 \n",
"20 \n",
"21 \n",
"22 \n",
"23 \n",
"24 Funding site admin \n",
"25 \n",
"26 \n",
"27 \n",
"28 \n",
"29 \n",
"30 Person with the authority to make changes to a... \n",
"31 \n",
"32 Person with authority to make changes to a soc... \n",
"\n",
" sector_ids framework_ids \\\n",
"0 S001, S002, S003, S004, S005, S006, S007, S008... FW01, FW02 \n",
"1 S001, S002, S003, S004, S005, S006, S007, S008... FW02 \n",
"2 S001, S002, S003, S004, S005, S006, S007, S008... FW01, FW02 \n",
"3 S002 FW02 \n",
"4 S002 FW02 \n",
"5 S002 FW02 \n",
"6 S002 FW02 \n",
"7 S002 FW02 \n",
"8 S002 FW02 \n",
"9 S002 FW02 \n",
"10 S002 FW02 \n",
"11 S006 FW01\\nFW02 \n",
"12 S006 FW01\\nFW02 \n",
"13 S006 FW02 \n",
"14 S006 FW02 \n",
"15 S006 FW01\\nFW02 \n",
"16 S003 FW02 \n",
"17 S003 FW01\\nFW02 \n",
"18 S003 FW02 \n",
"19 S003 FW02 \n",
"20 S010 FW01\\nFW02 \n",
"21 S009 FW02 \n",
"22 S008 FW02 \n",
"23 S008 FW02 \n",
"24 S008 FW02 \n",
"25 S008 FW01, FW02 \n",
"26 S008 FW02 \n",
"27 S008 FW02 \n",
"28 S008 FW02 \n",
"29 S007 FW02 \n",
"30 S007 FW02 \n",
"31 S007 FW02 \n",
"32 S007 FW02 \n",
"\n",
" longname \n",
"0 A001 - data scientist \n",
"1 A002 - target \n",
"2 A003 - trusted authority \n",
"3 A004 - activist \n",
"4 A005 - community group \n",
"5 A006 - educator \n",
"6 A007 - factchecker \n",
"7 A008 - library \n",
"8 A009 - NGO \n",
"9 A010 - religious organisation \n",
"10 A011 - school \n",
"11 A012 - account owner \n",
"12 A013 - content creator \n",
"13 A014 - elves \n",
"14 A015 - general public \n",
"15 A016 - influencer \n",
"16 A017 - coordinating body \n",
"17 A018 - government \n",
"18 A019 - military \n",
"19 A020 - policy maker \n",
"20 A021 - media organisation \n",
"21 A022 - company \n",
"22 A023 - adtech provider \n",
"23 A024 - developer \n",
"24 A025 - funding_site_admin \n",
"25 A026 - games designer \n",
"26 A027 - information security \n",
"27 A028 - platform administrator \n",
"28 A029 - server admininistrator \n",
"29 A030 - platforms \n",
"30 A031 - social media platform adminstrator \n",
"31 A032 - social media platform outreach \n",
"32 A033 - social media platform owner "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"disarm.df_actortypes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

794
CODE/generate_DISARM_pages.py Normal file
View file

@ -0,0 +1,794 @@
''' Manage DISARM metadata
The DISARM github repo at https://github.com/cDISARMFoundation/DISARMFrameworks serves multiple purposes:
* Holds the master copy of DISARM (in excel file DISARM_FRAMEWORK_MASTER.xlsx)
* Holds the master copy of DISARM data (in excel file DISARM_DATA_MASTER.xlsx)
* Holds notes on each DISARM object (in excel file DISARM_comments.xlsx)
* Holds a list of suggested changes to DISARM, in the github repo's issues list
* Provides a set of indexed views of DISARM objects, to make exploring DISARM easier
The file in this code updates the github repo contents, after the master spreadsheet is updated.
It creates this:
* A html page for each DISARM TTP object (creator and counter), if it doesn't already exist.
If a html page does exist, update the metadata on it, and preserve any hand-created
notes below the metadata area in it.
* A html page for each DISARM phase, tactic, and task.
* A html page for each incident used to create DISARM
* A grid view of all the DISARM creator techniques
* A grid view of all the DISARM counter techniques
* Indexes for the counter techniques, by tactic, resource and metatag
Here are the file inputs and outputs associated with that work:
Reads 1 excel file: MASTERDATA_DIR + 'DISARM_FRAMEWORKS_MASTER.xlsx' with sheets:
* phases
* techniques
* tasks
* incidents
* incidenttechniques
* tactics
* countermeasures
* actortypes
* resources
* responsetypes
Reads template files from directory page_templates:
* template_phase.md
* template_tactic.md
* template_task.md
* template_technique.md
* template_incident.md
* template_counter.md
Creates markdown files:
* GENERATED_PAGES_DIR + disarm_blue_framework.md
* GENERATED_PAGES_DIR + disarm_red_framework.md
* GENERATED_PAGES_DIR + disarm_red_framework_clickable.md
* GENERATED_PAGES_DIR + incidents_list.md
* GENERATED_PAGES_DIR + counter_tactic_counts.md
* GENERATED_PAGES_DIR + metatechniques_by_responsetype.md
* GENERATED_PAGES_DIR + resources_by_responsetype.md
* GENERATED_PAGES_DIR + tactics_by_responsetype.md
* GENERATED_PAGES_DIR + counter_tactics/*counters.md
* GENERATED_PAGES_DIR + metatechniques/*.md
* GENERATED_PAGES_DIR + resources_needed/*.md
Updates markdown files:
* GENERATED_PAGES_DIR + phases/*.md
* GENERATED_PAGES_DIR + tactics/*.md
* GENERATED_PAGES_DIR + techniques/*.md
* GENERATED_PAGES_DIR + incidents/*.md
* GENERATED_PAGES_DIR + tasks/*.md
* GENERATED_PAGES_DIR + counters/*.md
Creates CSVs
* GENERATED_FILES_DIR + generated_csvs/counters_tactics_table.csv
* GENERATED_FILES_DIR + generated_csvs/techniques_tactics_table.csv
todo:
* add all framework comments to the repo issues list
* add clickable blue framework
* add detections
'''
import pandas as pd
import numpy as np
import os
from sklearn.feature_extraction.text import CountVectorizer
GENERATED_PAGES_DIR = '../generated_pages/'
GENERATED_FILES_DIR = '../generated_files/'
MASTERDATA_DIR = '../DISARM_MASTER_DATA/'
class Disarm:
def __init__(self,
frameworkfile = MASTERDATA_DIR + 'DISARM_FRAMEWORKS_MASTER.xlsx',
datafile = MASTERDATA_DIR + 'DISARM_DATA_MASTER.xlsx',
commentsfile = MASTERDATA_DIR + 'DISARM_COMMENTS_MASTER.xlsx'):
# Load metadata from file
metadata = {}
xlsx = pd.ExcelFile(frameworkfile)
for sheetname in xlsx.sheet_names:
metadata[sheetname] = xlsx.parse(sheetname)
metadata[sheetname].fillna('', inplace=True)
xlsx = pd.ExcelFile(datafile)
for sheetname in xlsx.sheet_names:
metadata[sheetname] = xlsx.parse(sheetname)
metadata[sheetname].fillna('', inplace=True)
# Create individual tables and dictionaries
self.df_phases = metadata['phases']
self.df_frameworks = metadata['frameworks']
self.df_techniques = metadata['techniques']
self.df_tasks = metadata['tasks']
self.df_incidents = metadata['incidents']
self.df_groups = metadata['groups']
self.df_tools = metadata['tools']
self.df_examples = metadata['examples']
self.df_counters = metadata['countermeasures'].sort_values('disarm_id')
self.df_counters[['tactic_id', 'tactic_name']] = self.df_counters['tactic'].str.split(' ', 1, expand=True)
self.df_counters[['metatechnique_id', 'metatechnique_name']] = self.df_counters['metatechnique'].str.split(' ', 1, expand=True)
self.df_detections = metadata['detections']
self.df_detections[['tactic_id', 'tactic_name']] = self.df_detections['tactic'].str.split(' ', 1, expand=True)
# self.df_detections[['metatechnique_id', 'metatechnique_name']] = self.df_detections['metatechnique'].str.split(' ', 1, expand=True) #FIXIT
self.df_actortypes = metadata['actortypes']
self.df_resources = metadata['resources']
self.df_responsetypes = metadata['responsetypes']
self.df_metatechniques = metadata['metatechniques']
self.it = self.create_incident_technique_crosstable(metadata['incidenttechniques'])
self.df_tactics = metadata['tactics']
self.df_playbooks = metadata['playbooks']
# Add columns containing lists of techniques and counters to the tactics dataframe
self.df_techniques_per_tactic = self.df_techniques.groupby('tactic_id')['disarm_id'].apply(list).reset_index().rename({'disarm_id':'technique_ids'}, axis=1)
self.df_counters_per_tactic = self.df_counters.groupby('tactic_id')['disarm_id'].apply(list).reset_index().rename({'disarm_id':'counter_ids'}, axis=1)
self.df_tactics = self.df_tactics.merge(self.df_techniques_per_tactic, left_on='disarm_id', right_on='tactic_id', how='left').fillna('').drop('tactic_id', axis=1)
self.df_tactics = self.df_tactics.merge(self.df_counters_per_tactic, left_on='disarm_id', right_on='tactic_id', how='left').fillna('').drop('tactic_id', axis=1)
# Add simple dictionaries (id -> name) for objects
self.phases = self.make_object_dictionary(self.df_phases)
self.tactics = self.make_object_dictionary(self.df_tactics)
self.techniques = self.make_object_dictionary(self.df_techniques)
self.counters = self.make_object_dictionary(self.df_counters)
self.metatechniques = self.make_object_dictionary(self.df_metatechniques)
self.actortypes = self.make_object_dictionary(self.df_actortypes)
self.resources = self.make_object_dictionary(self.df_resources)
# Create the data table for each framework file
self.num_tactics = len(self.df_tactics)
# Create counters and detections cross-tables
self.cross_counterid_techniqueid = self.create_cross_table(self.df_counters[['disarm_id', 'techniques']],
'techniques', 'technique', '\n')
self.cross_counterid_resourceid = self.create_cross_table(self.df_counters[['disarm_id', 'resources_needed']],
'resources_needed', 'resource', ',')
self.cross_counterid_actortypeid = self.create_cross_table(self.df_counters[['disarm_id', 'actortypes']],
'actortypes', 'actortype', ',')
self.cross_detectionid_techniqueid = self.create_cross_table(self.df_detections[['disarm_id', 'techniques']],
'techniques', 'technique', '\n')
self.cross_detectionid_resourceid = self.create_cross_table(self.df_detections[['disarm_id', 'resources_needed']],
'resources_needed', 'resource', ',')
self.cross_detectionid_actortypeid = self.create_cross_table(self.df_detections[['disarm_id', 'actortypes']],
'actortypes', 'actortype', ',')
def create_incident_technique_crosstable(self, it_metadata):
# Generate full cross-table between incidents and techniques
it = it_metadata
it.index=it['disarm_id']
it = it['technique_ids'].str.split(',').apply(lambda x: pd.Series(x)).stack().reset_index(level=1, drop=True).to_frame('technique_id').reset_index().merge(it.drop('disarm_id', axis=1).reset_index()).drop('technique_ids', axis=1)
it = it.merge(self.df_incidents[['disarm_id','name']],
left_on='incident_id', right_on='disarm_id',
suffixes=['','_incident']).drop('incident_id', axis=1)
it = it.merge(self.df_techniques[['disarm_id','name']],
left_on='technique_id', right_on='disarm_id',
suffixes=['','_technique']).drop('technique_id', axis=1)
return(it)
def make_object_dictionary(self, df):
return(pd.Series(df.name.values,index=df.disarm_id).to_dict())
def create_cross_table(self, df, col, newcol, divider=','):
''' Convert a column with multiple values per cell into a crosstable
# Thanks https://stackoverflow.com/questions/17116814/pandas-how-do-i-split-text-in-a-column-into-multiple-rows?noredirect=1
'''
crosstable = df.join(df[col]
.str.split(divider, expand=True).stack()
.reset_index(drop=True,level=1)
.rename(newcol)).drop(col, axis=1)
crosstable = crosstable[crosstable[newcol].notnull()]
crosstable[newcol+'_id'] = crosstable[newcol].str.split(' ').str[0]
crosstable.drop(newcol, axis=1, inplace=True)
return crosstable
def create_technique_incidents_string(self, techniqueid):
incidentstr = '''
| Incident | Descriptions given for this incident |
| -------- | -------------------- |
'''
incirow = '| [{0} {1}]({2}incidents/{0}.md) | {3} |\n'
its = self.it[self.it['disarm_id_technique']==techniqueid]
for index, row in its[['disarm_id_incident', 'name_incident']].drop_duplicates().sort_values('disarm_id_incident').iterrows():
techstring = ', '.join(its[its['disarm_id_incident']==row['disarm_id_incident']]['name'].to_list())
incidentstr += incirow.format(row['disarm_id_incident'], row['name_incident'],
GENERATED_PAGES_DIR, techstring)
return incidentstr
def create_incident_techniques_string(self, incidentid):
techstr = '''
| Technique | Description given for this incident |
| --------- | ------------------------- |
'''
techrow = '| [{0} {1}]({2}techniques/{0}.md) | {3} {4} |\n'
techlist = self.it[self.it['disarm_id_incident'] == incidentid]
for index, row in techlist.sort_values('disarm_id_technique').iterrows():
techstr += techrow.format(row['disarm_id_technique'], row['name_technique'],
GENERATED_PAGES_DIR, row['disarm_id'], row['name'])
return techstr
def create_tactic_tasks_string(self, tactic_id):
table_string = '''
| Tasks |
| ----- |
'''
tactic_tasks = self.df_tasks[self.df_tasks['tactic_id']==tactic_id]
task_string = '| [{0} {1}]({2}tasks/{0}.md) |\n'
for index, row in tactic_tasks.sort_values('disarm_id').iterrows():
table_string += task_string.format(row['disarm_id'], row['name'], GENERATED_PAGES_DIR)
return table_string
def create_tactic_techniques_string(self, tactic_id):
table_string = '''
| Techniques |
| ---------- |
'''
tactic_techniques = self.df_techniques[self.df_techniques['tactic_id']==tactic_id]
row_string = '| [{0} {1}]({2}techniques/{0}.md) |\n'
for index, row in tactic_techniques.sort_values('disarm_id').iterrows():
table_string += row_string.format(row['disarm_id'], row['name'], GENERATED_PAGES_DIR)
return table_string
def create_object_counters_string(self, objectcolumn, object_id):
table_string = '''
| Counters | Response types |
| -------- | -------------- |
'''
object_counters = self.df_counters[self.df_counters[objectcolumn]==object_id]
row_string = '| [{0} {1}]({2}counters/{0}.md) | {3} |\n'
for index, row in object_counters.sort_values(['responsetype', 'disarm_id']).iterrows():
table_string += row_string.format(row['disarm_id'], row['name'], GENERATED_PAGES_DIR, row['responsetype'])
return table_string
def create_technique_counters_string(self, technique_id):
table_string = '''
| Counters | Response types |
| -------- | -------------- |
'''
technique_counters = self.cross_counterid_techniqueid[self.cross_counterid_techniqueid['technique_id']==technique_id]
technique_counters = pd.merge(technique_counters, self.df_counters[['disarm_id', 'name', 'responsetype']])
row_string = '| [{0} {1}]({2}counters/{0}.md) | {3} |\n'
for index, row in technique_counters.sort_values('disarm_id').iterrows():
table_string += row_string.format(row['disarm_id'], row['name'], GENERATED_PAGES_DIR, row['responsetype'])
return table_string
def create_counter_actortypes_string(self, counter_id):
table_string = '''
| Actor types | Sectors |
| ----------- | ------- |
'''
counter_actortypes = self.cross_counterid_actortypeid[self.cross_counterid_actortypeid['disarm_id']==counter_id]
counter_actortypes = pd.merge(counter_actortypes, self.df_actortypes[['disarm_id', 'name', 'sector_ids']], left_on='actortype_id', right_on='disarm_id')
row_string = '| [{0} {1}]({2}actortypes/{0}.md) | {3} |\n'
for index, row in counter_actortypes.sort_values('actortype_id').iterrows():
table_string += row_string.format(row['actortype_id'], row['name'], GENERATED_PAGES_DIR, row['sector_ids'])
return table_string
def create_actortype_counters_string(self, actortype_id):
table_string = '''
| Counters | Response types |
| -------- | -------------- |
'''
actortype_counters = self.cross_counterid_actortypeid[self.cross_counterid_actortypeid['actortype_id']==actortype_id]
actortype_counters = pd.merge(actortype_counters, self.df_counters[['disarm_id', 'name', 'responsetype']])
row_string = '| [{0} {1}]({2}counters/{0}.md) | {3} |\n'
for index, row in actortype_counters.sort_values('disarm_id').iterrows():
table_string += row_string.format(row['disarm_id'], row['name'], GENERATED_PAGES_DIR, row['responsetype'])
return table_string
def create_resource_counters_string(self, resource_id):
table_string = '''
| Counters | Response types |
| -------- | -------------- |
'''
resource_counters = self.cross_counterid_resourceid[self.cross_counterid_resourceid['resource_id']==resource_id]
resource_counters = pd.merge(resource_counters, self.df_counters[['disarm_id', 'name', 'responsetype']])
row_string = '| [{0} {1}]({2}counters/{0}.md) | {3} |\n'
for index, row in actortype_counters.sort_values('disarm_id').iterrows():
table_string += row_string.format(row['disarm_id'], row['name'], GENERATED_PAGES_DIR, row['responsetype'])
return table_string
def create_counter_tactics_string(self, counter_id):
table_string = '''
| Counters these Tactics |
| ---------------------- |
'''
# tactic_counters = self.df_counters[self.df_counters['tactic_id']==tactic_id]
# row_string = '| {0} | [{1} {2}]({3}counters/{1}.md) |\n'
# for index, row in tactic_counters.sort_values(['responsetype', 'disarm_id']).iterrows():
# table_string += row_string.format(row['responsetype'], row['disarm_id'], row['name'], GENERATED_PAGES_DIR)
return table_string
def create_counter_techniques_string(self, counter_id):
table_string = '''
| Counters these Techniques |
| ------------------------- |
'''
counter_techniques = self.cross_counterid_techniqueid[self.cross_counterid_techniqueid['disarm_id']==counter_id]
counter_techniques = pd.merge(counter_techniques, self.df_techniques[['disarm_id', 'name']].rename(columns={'disarm_id': 'technique_id'}))
row_string = '| [{0} {1}]({2}techniques/{0}.md) |\n'
for index, row in counter_techniques.sort_values('disarm_id').iterrows():
table_string += row_string.format(row['technique_id'], row['name'], GENERATED_PAGES_DIR)
return table_string
def create_counter_incidents_string(self, counter_id):
table_string = '''
| Seen in incidents |
| ----------------- |
'''
# tactic_counters = self.df_counters[self.df_counters['tactic_id']==tactic_id]
# row_string = '| {0} | [{1} {2}]({3}counters/{1}.md) |\n'
# for index, row in tactic_counters.sort_values(['responsetype', 'disarm_id']).iterrows():
# table_string += row_string.format(row['responsetype'], row['disarm_id'], row['name'], GENERATED_PAGES_DIR)
return table_string
def write_object_index_to_file(self, objectname, objectcols, dfobject, outfile):
''' Write HTML version of incident list to markdown file
Assumes that dfobject has columns named 'disarm_id' and 'name'
'''
html = '''# DISARM {}:
<table border="1">
<tr>
'''.format(objectname.capitalize())
# Create header row
html += '<th>{}</th>\n'.format('disarm_id')
html += ''.join(['<th>{}</th>\n'.format(col) for col in objectcols])
html += '</tr>\n'
# Add row for each object
for index, row in dfobject[dfobject['name'].notnull()].iterrows():
html += '<tr>\n'
html += '<td><a href="{0}/{1}.md">{1}</a></td>\n'.format(objectname, row['disarm_id'])
html += ''.join(['<td>{}</td>\n'.format(row[col]) for col in objectcols])
html += '</tr>\n'
html += '</table>\n'
# Write file
with open(outfile, 'w') as f:
f.write(html)
print('updated {}'.format(outfile))
return
def write_object_indexes_to_file(self):
''' Create an index file for each object type.
'''
self.write_object_index_to_file(
'response types', ['name', 'summary'],
self.df_responsetypes, GENERATED_PAGES_DIR + 'responsetype_index.md')
self.write_object_index_to_file(
'detections', ['name', 'summary', 'metatechnique', 'tactic', 'responsetype'],
self.df_detections, GENERATED_PAGES_DIR + 'detections_index.md')
return
def update_markdown_files(self):
''' Create or update all the editable markdown files in the repo
Reads in any user-written text before updating the header information above it
Does this for phase, tactic, technique, task, incident and counter objects
'''
warntext = 'DO NOT EDIT ABOVE THIS LINE - PLEASE ADD NOTES BELOW'
warnlen = len(warntext)
metadata = {
'phase': self.df_phases,
'tactic': self.df_tactics,
'technique': self.df_techniques,
'task': self.df_tasks,
'incident': self.df_incidents,
'counter': self.df_counters,
'metatechnique': self.df_metatechniques,
'actortype': self.df_actortypes,
#'resource': self.df_resources,
#'responsetype': self.df_responsetypes,
#'detection': self.df_detections
}
indexrows = {
'phase': ['name', 'summary'],
'tactic': ['name', 'summary', 'phase_id'],
'technique': ['name', 'summary', 'tactic_id'],
'task': ['name', 'summary', 'tactic_id'],
'incident': ['name', 'objecttype', 'year_started', 'found_in_country', 'found_via'],
'counter': ['name', 'summary', 'metatechnique', 'tactic', 'responsetype'],
'detection': ['name', 'summary', 'metatechnique', 'tactic', 'responsetype'],
'responsetype': ['name', 'summary'],
'metatechnique': ['name', 'summary'],
'actortype': ['name', 'summary', 'sector_ids'],
'resource': ['name', 'summary', 'resource type']
}
for objecttype, df in metadata.items():
print('Temp: objecttype {}'.format(objecttype))
# Create objecttype directory if needed. Create index file for objecttype
objecttypeplural = objecttype + 's'
objecttypedir = GENERATED_PAGES_DIR + '{}'.format(objecttypeplural)
if not os.path.exists(objecttypedir):
os.makedirs(objecttypedir)
self.write_object_index_to_file(objecttypeplural, indexrows[objecttype],
metadata[objecttype],
GENERATED_PAGES_DIR + '{}_index.md'.format(objecttypeplural))
# Update or create file for every object with this objecttype type
template = open('page_templates/template_{}.md'.format(objecttype)).read()
for index, row in df[df['name'].notnull()].iterrows():
# First read in the file - if it exists - and grab everything
# below the "do not write about this line". Will write this
# out below new metadata.
datafile = GENERATED_PAGES_DIR + '{}/{}.md'.format(objecttypeplural, row['disarm_id'])
oldmetatext = ''
if os.path.exists(datafile):
with open(datafile) as f:
filetext = f.read()
warnpos = filetext.find(warntext)
if warnpos == -1:
print('no warning text found in {}: adding to file'.format(datafile))
usertext = filetext
else:
oldmetatext = filetext[:warnpos+warnlen]
usertext = filetext[warnpos+warnlen:]
else:
usertext = ''
# Now populate datafiles with new metadata plus old userdata
if objecttype == 'phase':
metatext = template.format(type='Phase', id=row['disarm_id'], name=row['name'], summary=row['summary'])
if objecttype == 'tactic':
metatext = template.format(type = 'Tactic', id=row['disarm_id'], name=row['name'],
phase=row['phase_id'], summary=row['summary'],
tasks=self.create_tactic_tasks_string(row['disarm_id']),
techniques=self.create_tactic_techniques_string(row['disarm_id']),
counters=self.create_object_counters_string('tactic_id', row['disarm_id']))
if objecttype == 'task':
metatext = template.format(type='Task', id=row['disarm_id'], name=row['name'],
tactic=row['tactic_id'], summary=row['summary'])
if objecttype == 'technique':
metatext = template.format(type = 'Technique', id=row['disarm_id'], name=row['name'],
tactic=row['tactic_id'], summary=row['summary'],
incidents=self.create_technique_incidents_string(row['disarm_id']),
counters=self.create_technique_counters_string(row['disarm_id']))
if objecttype == 'counter':
metatext = template.format(type = 'Counter', id=row['disarm_id'], name=row['name'],
tactic=row['tactic_id'], summary=row['summary'],
playbooks='', metatechnique=row['metatechnique'],
actortypes=self.create_counter_actortypes_string(row['disarm_id']),
resources_needed=row['resources_needed'],
tactics=self.create_counter_tactics_string(row['disarm_id']),
techniques=self.create_counter_techniques_string(row['disarm_id']),
incidents=self.create_counter_incidents_string(row['disarm_id']))
if objecttype == 'incident':
metatext = template.format(type = 'Incident', id=row['disarm_id'], name=row['name'],
incidenttype=row['objecttype'], summary=row['summary'],
yearstarted=row['year_started'],
fromcountry=row['attributions_seen'],
tocountry=row['found_in_country'],
foundvia=row['found_via'],
dateadded=row['when_added'],
techniques=self.create_incident_techniques_string(row['disarm_id']))
if objecttype == 'actortype':
metatext = template.format(type = 'Actor', id=row['disarm_id'], name=row['name'],
summary=row['summary'], sector=row['sector_ids'],
viewpoint=row['framework_ids'],
counters=self.create_actortype_counters_string(row['disarm_id']))
if objecttype == 'resource':
metatext = template.format(type = 'Resource', id=row['disarm_id'], name=row['name'],
summary=row['summary'], resource_type=row['resource_type'],
counters=self.create_resource_counters_string(row['disarm_id']))
if objecttype == 'metatechnique':
metatext = template.format(type='Metatechnique', id=row['disarm_id'], name=row['name'],
summary=row['summary'],
counters=self.create_object_counters_string('metatechnique_id', row['disarm_id']))
# Make sure the user data goes in
if (metatext + warntext) != oldmetatext:
print('Updating {}'.format(datafile))
with open(datafile, 'w') as f:
f.write(metatext)
f.write(warntext)
f.write(usertext)
f.close()
return
def create_padded_framework_table(self, title, ttp_col, tocsv=True):
# Create the master grid that we make all the framework visuals from
# cols = number of tactics
# rows = max number of techniques per tactic + 2
numrows = max(self.df_tactics[ttp_col].apply(len)) + 2
arr = [['' for i in range(self.num_tactics)] for j in range(numrows)]
for index, tactic in self.df_tactics.iterrows():
arr[0][index] = tactic['phase_id']
arr[1][index] = tactic['disarm_id']
if tactic[ttp_col] == '':
continue
for index2, technique in enumerate(tactic[ttp_col]):
arr[index2+2][index] = technique
#Save grid to file
if tocsv:
snakecase_title = title.replace(' ', '_')
csvdir = GENERATED_FILES_DIR
if not os.path.exists(csvdir):
os.makedirs(csvdir)
pd.DataFrame(arr).to_csv('{0}/{1}_ids.csv'.format(csvdir, snakecase_title), index=False, header=False)
return(arr)
def write_disarm_frameworks(self):
self.write_disarm_framework_files("red framework", self.techniques, "techniques", 'technique_ids')
self.write_disarm_framework_files("blue framework", self.counters, "counters", 'counter_ids')
return
def write_disarm_framework_files(self, title, ttp_dictionary, ttp_dir, ttp_col):
# Write HTML version of framework diagram to markdown file
# Needs phases, tactics
snakecase_title = title.replace(' ', '_')
outfile = GENERATED_PAGES_DIR + 'disarm_{}.md'.format(snakecase_title)
clickable_file = GENERATED_FILES_DIR + 'disarm_{}_clickable.html'.format(snakecase_title)
# Create padded table to make the writing easier
padded_table = self.create_padded_framework_table(title, ttp_col)
html = '''# DISARM {}: Latest Framework
<table border="1">
<tr>
'''.format(title.capitalize())
# row with phase names in - removed because it makes the tables confusing
# for col in range(self.num_tactics):
# html += '<td><a href="phases/{0}.md">{0} {1}</a></td>\n'.format(
# padded_table[0][col], self.phases[padded_table[0][col]])
# html += '</tr>\n'
html += '<tr style="background-color:blue;color:white;">\n'
for col in range(self.num_tactics):
html += '<td><a href="tactics/{0}.md">{0} {1}</a></td>\n'.format(
padded_table[1][col], self.tactics[padded_table[1][col]])
html += '</tr>\n<tr>\n'
for row in range(2,len(padded_table)):
for col in range(self.num_tactics):
if padded_table[row][col] == '':
html += '<td> </td>\n'
else:
html += '<td><a href="{0}/{1}.md">{1} {2}</a></td>\n'.format(
ttp_dir, padded_table[row][col], ttp_dictionary[padded_table[row][col]])
html += '</tr>\n<tr>\n'
html += '</tr>\n</table>\n'
with open(outfile, 'w') as f:
f.write(html)
print('updated {}'.format(outfile))
# Clickable version
self.write_clickable_disarm_framework_file(title, padded_table, ttp_dictionary, clickable_file)
return
def write_clickable_disarm_framework_file(self, title, padded_table, ttp_dictionary, outfile):
# Write clickable html version of the matrix grid to html file
html = '''<!DOCTYPE html>
<html>
<head>
<title>DISARM {}</title>
</head>
<body>
<script>
function handleTechniqueClick(box) {{
var technique = document.getElementById(box);
var checkBox = document.getElementById(box+"check");
var text = document.getElementById(box+"text");
if (checkBox.checked == true){{
text.style.display = "block";
technique.bgColor = "Lime"
}} else {{
text.style.display = "none";
technique.bgColor = "Silver"
}}
}}
</script>
<h1>DISARM</h1>
<table border=1 bgcolor=silver>
'''.format(title.capitalize())
html += '<tr bgcolor=fuchsia>\n'
for col in range(self.num_tactics):
html += '<td>{0} {1}</td>\n'.format(padded_table[0][col], self.phases[padded_table[0][col]])
html += '</tr>\n'
html += '<tr bgcolor=aqua>\n'
for col in range(self.num_tactics):
html += '<td>{0} {1}</td>\n'.format(padded_table[1][col], self.tactics[padded_table[1][col]])
html += '</tr>\n'
liststr = ''
html += '<tr>\n'
for row in range(2,len(padded_table)):
for col in range(self.num_tactics):
techid = padded_table[row][col]
if techid == '':
html += '<td bgcolor=white> </td>\n'
else:
html += '<td id="{0}">{0} {1}<input type="checkbox" id="{0}check" onclick="handleTechniqueClick(\'{0}\')"></td>\n'.format(
techid, ttp_dictionary[techid])
liststr += '<li id="{0}text" style="display:none">{0}: {1}</li>\n'.format(
techid, ttp_dictionary[techid])
html += '</tr>\n<tr>\n'
html += '</tr>\n</table>\n<hr>\n'
html += '<ul>\n{}</ul>\n'.format(liststr)
html += '''
</body>
</html>
'''
with open(outfile, 'w') as f:
f.write(html)
print('updated {}'.format(outfile))
return
def print_technique_incidents(self):
for id_technique in self.df_techniques['disarm_id'].to_list():
print('{}\n{}'.format(id_technique,
self.create_incidentstring(id_technique)))
return
def print_incident_techniques(self):
for id_incident in self.df_incidents['disarm_id'].to_list():
print('{}\n{}'.format(id_incident,
self.create_techstring(id_incident)))
return
def analyse_counter_text(self, col='name'):
# Analyse text in counter descriptions
alltext = (' ').join(self.df_counters[col].to_list()).lower()
count_vect = CountVectorizer(stop_words='english')
word_counts = count_vect.fit_transform([alltext])
dfw = pd.DataFrame(word_counts.A, columns=count_vect.get_feature_names()).transpose()
dfw.columns = ['count']
dfw = dfw.sort_values(by='count', ascending=False)
return(dfw)
def analyse_coverage(self, technique_id_list, counter_id_list):
ct = self.cross_counterid_techniqueid.copy()
ct = ct[ct['technique_id'].isin(self.df_techniques['disarm_id'].to_list()) & ct['disarm_id'].isin(self.df_counters['disarm_id'].to_list())]
possible_counters_for_techniques = ct[ct['technique_id'].isin(technique_id_list)]
possible_techniques_for_counters = ct[ct['technique_id'].isin(counter_id_list)]
coverage = ct[(ct['disarm_id'].isin(counter_id_list)) & (ct['technique_id'].isin(technique_id_list))]
return coverage, possible_counters_for_techniques, possible_techniques_for_counters
def write_counts_table_to_file(self, objectname, objectdict, counts_table, outfile):
html = '''# DISARM {} courses of action
<table border="1">
<tr>
<td> </td>
'''.format(objectname.capitalize())
# Table heading row
for col in counts_table.columns.get_level_values(1)[:-1]:
html += '<td>{}</td>\n'.format(col)
html += '<td>TOTALS</td></tr><tr>\n'
# Data rows
for index, counts in counts_table.iterrows():
html += '<td><a href="{3}{0}s/{1}.md">{1} {2}</a></td>\n'.format(
objectname, index, objectdict[index], GENERATED_PAGES_DIR)
for val in counts.values:
html += '<td>{}</td>\n'.format(val)
html += '</tr>\n<tr>\n'
# Column sums
html += '<td>TOTALS</td>\n'
for val in counts_table.sum().values:
html += '<td>{}</td>\n'.format(val)
html += '</tr>\n</table>\n'
with open(outfile, 'w') as f:
f.write(html)
print('updated {}'.format(outfile))
return
def write_responsetype_tactics_table_file(self, outfile = GENERATED_PAGES_DIR + 'tactics_by_responsetype_table.md'):
''' Write course of action matrix for tactics vs responsetype
'''
counts_table = pd.pivot_table(self.df_counters[['responsetype', 'tactic_id','disarm_id']],
index='tactic_id', columns='responsetype', aggfunc=len,
fill_value=0)
counts_table['TOTALS'] = counts_table.sum(axis=1)
self.write_counts_table_to_file('tactic', self.tactics, counts_table, outfile)
return
def write_metatechniques_responsetype_table_file(self, outfile = GENERATED_PAGES_DIR + 'metatechniques_by_responsetype_table.md'):
counts_table = pd.pivot_table(self.df_counters[['responsetype', 'metatechnique_id','disarm_id']],
index='metatechnique_id', columns='responsetype', aggfunc=len,
fill_value=0)
counts_table['TOTALS'] = counts_table.sum(axis=1)
self.write_counts_table_to_file('metatechnique', self.metatechniques, counts_table, outfile)
return
def write_resources_responsetype_table_file(self, outfile = GENERATED_PAGES_DIR + 'resources_by_responsetype_table.md'):
# dirty hack because there are lots of -blanks?- in the cross-table that should have been filtered out
crosstable_with_responsetype = self.cross_counterid_resourceid.merge(self.df_counters[['disarm_id', 'responsetype']])
crosstable_with_responsetype = crosstable_with_responsetype[crosstable_with_responsetype['responsetype'].isin(self.resources.keys())]
counts_table = pd.pivot_table(crosstable_with_responsetype,
index='resource_id', columns='responsetype', aggfunc=len,
fill_value=0)
counts_table['TOTALS'] = counts_table.sum(axis=1)
self.write_counts_table_to_file('resource', self.resources, counts_table, outfile)
return
def generate_and_write_datafiles(self):
# Framework matrices
self.write_disarm_frameworks()
# Editable files
self.update_markdown_files()
self.write_object_indexes_to_file()
# Cross tables
self.write_responsetype_tactics_table_file()
self.write_metatechniques_responsetype_table_file()
# FIXIT - this is just giving trouble today self.write_resources_responsetype_table_file()
return
def main():
disarm = Disarm()
disarm.generate_and_write_datafiles()
if __name__ == "__main__":
main()

2548
CODE/generate_DISARM_sql.ipynb Normal file
View file

File diff suppressed because it is too large Load diff

10
CODE/page_templates/template_actortype.md Normal file
View file

@ -0,0 +1,10 @@
# {type} {id}: {name}
* **Summary:** {summary}
* **Sector:** {sector}
* **Viewpoint:** {viewpoint}
{counters}

20
CODE/page_templates/template_counter.md Normal file
View file

@ -0,0 +1,20 @@
# {type} {id}: {name}
* **Summary**: {summary}
* **Playbooks**: {playbooks}
* **Metatechnique**: {metatechnique}
* **Resources needed:** {resources_needed}
* **Belongs to tactic stage**: {tactic}
{actortypes}
{tactics}
{techniques}
{incidents}

16
CODE/page_templates/template_incident.md Normal file
View file

@ -0,0 +1,16 @@
# {type} {id}: {name}
* **Summary:** {summary}
* **incident type**: {incidenttype}
* **Year started:** {yearstarted}
* **Countries:** {fromcountry} , {tocountry}
* **Found via:** {foundvia}
* **Date added:** {dateadded}
{techniques}

7
CODE/page_templates/template_metatechnique.md Normal file
View file

@ -0,0 +1,7 @@
# {type} {id}: {name}
* **Summary:** {summary}
{counters}

4
CODE/page_templates/template_phase.md Normal file
View file

@ -0,0 +1,4 @@
# {type} {id}: {name}
* **Summary:** {summary}

8
CODE/page_templates/template_resource.md Normal file
View file

@ -0,0 +1,8 @@
# {type} {id}: {name}
* **Summary:** {summary}
* **Resource type:** {resource_type}
{counters}

13
CODE/page_templates/template_tactic.md Normal file
View file

@ -0,0 +1,13 @@
# {type} {id}: {name}
* **Summary:** {summary}
* **Belongs to phase:** {phase}
{tasks}
{techniques}
{counters}

6
CODE/page_templates/template_task.md Normal file
View file

@ -0,0 +1,6 @@
# {type} {id}: {name}
* **Summary:** {summary}
* **Belongs to tactic stage:** {tactic}

10
CODE/page_templates/template_technique.md Normal file
View file

@ -0,0 +1,10 @@
# {type} {id}: {name}
* **Summary**: {summary}
* **Belongs to tactic stage**: {tactic}
{incidents}
{counters}