🧆 update readme:

README.md

@@ -1,10 +1,14 @@
-# Curated Resources on Orchestration
+# ✨ Resources on Cloud and Orchestration
 
-* [References for K8s]().
-* [References for EKS]().
-* [References for Docker]().
-* [References for Golang]().
-* [References for kubectl]().
+
+* [References for AWS and CDK](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_aws.md).
+* [References for Lambda Functions](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_lambda.md).
+* [References for EKS](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_eks.md).
+* [References for Docker](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_docker.md).
+* [References for K8s](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_k8s.md).
+* [References for kubectl](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_kubectl.md).
+* [References for Golang](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_go.md).
+* [References for GCP](https://github.com/bt3gl/Curated_Cloud_and_Orchestration/blob/master/references_for_gcp.md).
 
 
 ----

cdk/.gitkeep

@@ -0,0 +1 @@
+

cdk/admin_guide_cdk.md

@@ -0,0 +1,416 @@
+# GitOps, Flux, and Deploying services with CDK
+
+An introduction to some concepts and tools we will be using. 
+
+## What's **GitOps**
+
+In general, there are two ways to deploy infrastructure changes:
+
+- **Procedural way**: telling some tools what to do, e.g.: Ansible. This is also known as a push model.
+- **Declarative way**: telling some tool what you want to have done, also known as infrastructure as code, e.g.: Terraform, Pulumi, CDK.
+
+[GitOps](https://www.weave.works/technologies/gitops/) is a term created by WeWorks and works by using Git as a source of truth for declarative infrastructure and applications. Automated CI/CD pipelines roll out changes to your infrastructure after commits are pushed and approved in Git. 
+
+The GitOps methodology consists in describing the desired state of the system using a **declarative** specification for each environment (e.g., our Kubernetes cluster for a specific environment):
+
+- A Git repo is the single source of truth for the desired state of the system
+- All changes to the desired state are Git commits
+- All specified properties of the cluster are also observable in the cluster so that we can detect if the desired and observed states are the same (converged) or different (diverged)
+
+In GitOps you only push code. The developer interacts with the source control, which triggers the CI/CD tool (CicleCI), and this pushes the docker image to the container register (e.g. docker hub). You see the Docker image as an artifact.
+
+To deploy that Docker image, you have a different config repository which contains the Kubernetes manifests. CircleCI sends a pull request, and when it is merged, a pod in the Kubernetes cluster pulls the image to the cluster (similar to `kubectl apply`, or even `helm update`). Everything is controlled through pull requests. You push code, not containers.
+
+The refereed pod runs a tool called [Flux](https://github.com/fluxcd/flux), which automatically ensures that the state of a cluster matches the config in Git. It uses an operator in the cluster to trigger deployments inside Kubernetes, which means you don't need a separated CircleCI. It monitors all relevant image repositories, detects new images, triggers deployments, and updates the desired running configuration based on that.
+
+## Kubernetes
+
+A Kubernetes cluster consists of a series of objects:
+
+- **Nodes**, which can be equated to servers, be they bare-metal or virtual machines running in a cloud.
+- Nodes run **Pods**, which are collections of Docker containers. A Pod is the unit of deployment in Kubernetes. All containers in a Pod share the same network and can refer to each other as if they were running on the same host. The Kubernetes object responsible for launching and maintaining the desired number of pods is called a **Deployment.**
+- For Pods to communicate with other Pods, Kubernetes provides another kind of object called a **Service.**
+- Services are tied to Deployments through **Selectors** and **Labels,** and are also exposed to external clients either by exposing a **NodePort** as a static port on each Kubernetes node or by creating a **LoadBalancer** object.
+
+## Kustomize
+
+Kustomize provides a **purely declarative approach** to configuration customization that adheres to and leverages the familiar and carefully designed Kubernetes API.
+
+Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is. Kustomize targets Kubernetes; it understands and can patch.
+
+### How Kustomize works
+
+For each service, there is two directories, where a `kustomization.yaml` file list all the `yaml` files inside them: 
+
+- `base/` - usually immutable.
+- `overlay/`- where you add customizations and new code.
+
+---
+
+# **Bootstrapping Services in an AWS EKS cluster**
+
+## **Pre-requisites**
+
+### **Install CLI tools**
+
+- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html).
+- [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
+- [sops](https://github.com/mozilla/sops).
+- [Kustomize](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/INSTALL.md).
+- [fluxctl](https://www.weave.works/blog/install-fluxctl-and-manage-your-deployments-easily).
+
+### **Get access to our AWS Cluster**
+
+We spin up clusters' resources using AWS CDK. This provisions a **developer EKS cluster** and **MSK cluster**, together with the following resources: a dedicated **VPC**, a **VPN**, **Elasticsearch cluster**, **Cloudwatch dashboards**, and an **RDS Postgres instance configured for Hydra.**
+
+This staging and dev clusters are already available for you in our AWS staging account. For full access you need:
+
+- AWS credentials (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
+- The VPN `.ovpn` file (can be downloaded from the dashboard) and VPN client private key.
+- Kubeconfig file.
+
+However, if you would like to bootstrap an entirely new cluster, follow the instructions below.
+
+
+## **Bootstrapping Step-by-step**
+
+### **Update Kubeconfig**
+
+Edit `./bootstrap/kubeconfig/aws-auth-configmap.yaml` with your account's `rolearn`.
+
+Set env variables:
+
+    export REGION=<aws region>
+
+Get kubectl config:
+
+    ./get_kubeconfig.sh
+
+Remember, you can always change your kubeconfig context with:
+
+    kubectl config use-context <context>
+
+You can also use [kubectx](https://github.com/ahmetb/kubectx) for this.
+
+### **Create Nginx ingress controller in the EKS cluster**
+
+Create Nginx ingress controller's namespaces, services, roless, deployments, etc. by running:
+
+    kubectl apply -f ./bootstrap/nginx-ingress-alb/all-in-one.yaml
+
+This is the output:
+
+    namespace/kube-ingress created
+    serviceaccount/nginx-ingress-controller created
+    clusterrole.rbac.authorization.k8s.io/nginx-ingress-controller created
+    role.rbac.authorization.k8s.io/nginx-ingress-controller created
+    clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-controller created
+    rolebinding.rbac.authorization.k8s.io/nginx-ingress-controller created
+    service/nginx-default-backend created
+    deployment.extensions/nginx-default-backend created
+    configmap/ingress-nginx created
+    service/ingress-nginx created
+    deployment.extensions/ingress-nginx created
+    priorityclass.scheduling.k8s.io/high-priority created
+
+Check whether all the pods created:
+
+    kubectl get pods --namespace kube-ingress
+
+Should result:
+
+    NAMESPACE      NAME                                    READY   STATUS    RESTARTS   AGE
+    kube-ingress   ingress-nginx-55966f5cf8-bpvwj          1/1     Running   0          7m53s
+    kube-ingress   ingress-nginx-55966f5cf8-vssfl          1/1     Running   0          7m53s
+    kube-ingress   ingress-nginx-55966f5cf8-xtkv9          1/1     Running   0          7m53s
+    kube-ingress   nginx-default-backend-c4bbbc8b7-j5cnh   1/1     Running   0          7m57s
+
+Check all the services created:
+
+    kubectl get services --namespace kube-ingress
+
+Should result:
+
+    NAMESPACE      NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
+    kube-ingress   ingress-nginx           NodePort    172.20.203.36   <none>        80:30080/TCP,443:30443/TCP   6m32s
+    kube-ingress   nginx-default-backend   ClusterIP   172.20.128.3    <none>        80/TCP                       6m35s
+
+Note that the `Service` type for the `ingress-nginx` service is `NodePort` and not `LoadBalancer`. We don't want AWS to create a new Load Balancer every time we recreate the ingress. Instead, we provision an ALB and send both HTTP and HTTPS traffic to a `Target Group` that targets port `30080` on the EKS worker nodes (which is the `nodePort` in the manifest above for HTTP traffic).
+
+### **Create a namespace in the EKS cluster**
+
+This step is necessary in case you are creating an entirely new cluster namespace (i.e., if it's not `dev` nor `staging`).
+
+To add a new namespace, just follow the current examples in `/bootstrap/namespaces/overlays/` and then apply the changes in the overlay:
+
+    cd ./bootstrap/namespaces/overlays/
+    kustomize build . |  kubectl apply -k <namespace>
+
+You should see something like:
+
+    namespace/<namespace> created
+    namespace/logging created
+    namespace/monitoring created
+    namespace/observability created
+
+Check whether it worked:
+
+    kubectl get ns
+
+### **Create secret for DockerHub credentials in EKS cluster**
+
+All right, if you are working on an AWS account that is not staging, hold tight, because this step is a trip.
+
+Currently, we use [sops](https://github.com/mozilla/sops) to manage secrets in Kubernetes.
+
+You have a file named `./bootstrap/dockerhub-creds-secret/docker-hub.yaml` that possess the secret for DockerHub credentials and it's encrypted. So the first thing we need to do is decrypt it so we can use the secret for our cluster. The caveat is that you need to set your AWS creds to the account `773713188930` (staging) account (or it won't be able to grab the key to decrypt):
+
+    sops -d  docker-hub.yaml > dec.yaml
+
+Take a look at `dec.yaml`, you will see something like this:
+
+    apiVersion: v1
+    type: kubernetes.io/dockerconfigjson
+    kind: Secret
+    metadata:
+        name: docker-hub
+    data:
+        .dockerconfigjson: <Base64 1337 password>
+
+Now, the next step is either to go to the [AWS KMS dashboard](https://us-east-2.console.aws.amazon.com/kms/home?region=us-east-2#/kms/keys) or run `aws kms create-custom-key-store` to create a `Customer managed keys`.
+
+KMS is a service that encrypts and decrypts data with AES_GCM, using keys that are never visible to users of the service. Each KMS master key has a set of role-based access controls, and individual roles are permitted to encrypt or decrypt using the master key. KMS helps solve the problem of distributing keys, by shifting it into an access control problem that can be solved using AWS's trust model.
+
+Once you have this ready, grab its ARN.
+
+Create a new encrypted file with your new KMS key:
+
+    sops --kms="ARN" --encrypt  dec.yaml > docker-hub-<MY CLUSTER>.yaml
+
+
+This Secret is created in several namespaces (default, monitoring, logging, flux-system).
+
+### **Apply overlay config for fluentd, jaeger-operator and prometheus-operator**
+
+Follow the same procedure for each of these services: `./bootstrap/fluentd`, `./bootstrap/jaeger-operator` and `./bootstrap/prometheus-operator`, copying an overlay subdirectory for your namespace, replacing your namespace string to anywhere where `staging` is, and running:
+
+    cd ./bootstrap/<service>/overlays/<namespace>
+    kustomize build . | kubectl apply -f -
+
+### **Install and configure Flux in EKS cluster**
+
+This part is a little longer. [Here](https://docs.fluxcd.io/en/latest/tutorials/get-started-kustomize.html) is the official Flux documentation with Kustomize.
+
+Flux (and memcached) is bootstrapped by following the instructions inside `bootstrap/flux/`. That directory should have the following structure:
+
+    ├── base
+    │   ├── flux
+    │   └── memcached
+    ├── overlays
+
+The first step is creating an `overlay/<namespace>` directory for your deployment, similar to `overlay/staging`.
+
+### **How Flux works**
+
+Flux runs by looking at `./.flux.yaml`. This calls `./generate_kustomize_output.sh` in a docker container and runs the following:
+
+1. Set the environment (e.g. `staging`).
+2. For each sub-directory in `kustomize/`, `cd` inside each `overlays/` for the environment and runs `kustomize build`.
+3. If there are `sops` secrets inside these directories, decrypts the secret as well.
+
+### **Setting up Flux Docker image**
+
+The default `Deployment` for Flux is using `weaveworks/flux` Docker image.
+
+You will need to push a docker image to DockerHub for your namespace. 
+
+Once you have a [docker image in Docker Hub] grab its tag (e.g. `staging-af87bcc`).
+
+
+### **Building and deploying**
+
+Inside your overlay directory, run:
+
+    cd bootstrap/flux/overlays/<namespace>
+    kustomize build . | kubectl apply -f -
+
+You should see the following:
+
+    namespace/flux-system created
+    serviceaccount/flux created
+    podsecuritypolicy.policy/flux created
+    role.rbac.authorization.k8s.io/flux created
+    clusterrole.rbac.authorization.k8s.io/flux-psp created
+    clusterrole.rbac.authorization.k8s.io/flux created
+    clusterrole.rbac.authorization.k8s.io/flux-readonly created
+    rolebinding.rbac.authorization.k8s.io/flux created
+    clusterrolebinding.rbac.authorization.k8s.io/flux-psp created
+    clusterrolebinding.rbac.authorization.k8s.io/flux created
+    configmap/flux-kube-config-hmbbmcb469 created
+    secret/flux-git-deploy created
+    service/flux-memcached created
+    deployment.apps/flux created
+    deployment.apps/flux-memcached created
+
+Wait for Flux and memcached to start:
+
+    kubectl -n flux-system rollout status deployment.apps/flux
+
+Check that the pods are up:
+
+    kubectl get pods --namespace flux-system
+
+You should see two pods, something like this:
+
+    NAME                              READY   STATUS    RESTARTS   AGE
+    flux-<some string>                1/1     Running   0          21m
+    flux-memcached-<some string>      1/1     Running   0          60m
+
+At any point you can debug your pod by running:
+
+    kubectl describe pod  flux-<some string> -n flux-system
+
+### **Adding key to Github**
+
+Generate a deployment key:
+
+    fluxctl --k8s-fwd-ns=flux-system identity
+
+
+Later on, when you have everything set, you can force Flux `git pull` with
+
+    fluxctl sync --k8s-fwd-ns flux-system
+
+
+### **Create k8s-developer-role in multiple namespaces in EKS cluster**
+
+Similarly to the previous step, create an overlay to your namespace (e.g. dev) in the [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) kustomize resources. You can do this by copying the files from `./bootstrap/rbac/overlays/staging`, and changing the namespace string from `staging` inside `k8s-developer-user.yaml`:
+
+    ...
+    metadata:
+      name: k8s-developer-role
+      namespace: <namespace>
+    ...
+    metadata:
+      name: k8s-developer-rolebinding
+      namespace: <namespace>
+
+Apply the changes with:
+
+    cd bootstrap/rbac/overlays/<namespace>
+    kustomize build . | kubectl apply -f -
+
+You should see the following:
+
+    role.rbac.authorization.k8s.io/k8s-developer-role-default created
+    role.rbac.authorization.k8s.io/k8s-developer-role created
+    role.rbac.authorization.k8s.io/k8s-developer-role-monitoring created
+    rolebinding.rbac.authorization.k8s.io/k8s-developer-rolebinding-default created
+    rolebinding.rbac.authorization.k8s.io/k8s-developer-rolebinding created
+    rolebinding.rbac.authorization.k8s.io/k8s-developer-rolebinding-monitoring created
+
+
+
+---
+
+# **Deploying Advanced services in an AWS EKS cluster**
+
+
+## **Porting Hydra**
+
+### **Customizing the overlay directory**
+
+Inside `./kustomize/hydra`, create `overlay/` subdirectory for your environment.
+
+Create a KMS key (the same way as in step *Create secret for DockerHub credentials in EKS cluster* in `./boostrap`). Grab its ARN and add it too `./kustomize/hydra/overlays/.sops.yaml`.
+
+Replace the `staging` string (and the correct host URLS) for your namespace, inside `kustomization.yaml`, `configmap.yaml`.
+
+### **Creating sops secrets for Hydra**
+
+We use sops to encrypt secret values for environment variables representing credentials, database connections, etc. so that Flux can pick these secrets when it needs.
+
+We place these files inside a `.sops/` directory inside the overlay environment directory.
+
+Grab the RDS Postgres data and create secret string:
+
+    echo -n "postgres://hydradbadmin:< hydra_passport>@<hydra_db_endpoint" > .sops/DATABASE_URL.enc
+    sops -e -i .sops/DATABASE_URL.enc
+
+Create password salt:
+
+    echo -n "<random_string" > .sops/OIDC_SUBJECT_TYPE_PAIRWISE_SALT.enc
+    sops -e -i .sops/OIDC_SUBJECT_TYPE_PAIRWISE_SALT.enc
+
+Create system secret:
+
+    echo -n "<random_string" > .sops/SYSTEM_SECRET.enc
+    sops -e -i .sops/SYSTEM_SECRET.enc
+
+Generate `secrets.yaml':
+
+    npx --quiet --package @reactioncommerce/merge-sops-secrets@1.2.1 sops-to-secret secret-stub.yaml > secret.yaml
+
+### **Building and applying**
+
+Now, just run:
+
+    cd ./kustomize/hydra/overlays/<namespace>
+    kustomize build . | kubectl apply -f -
+
+
+### **Create MongoDB database and user in Atlas**
+
+So that you can have MongDB URL and MongDB OPLOG URL for the next step.
+
+### **Creating sops secrets**
+
+Create MongDB URL secret:
+
+    echo -n "<atlas url>" .sops/MONGO_URL.enc
+    sops -e -i .sops/MONGO_URL.enc
+
+Create MongDB OPLOG URL secret:
+
+    echo -n "<atlas ops url>" .sops/MONGO_OPLOG_URL.enc
+    sops -e -i .sops/MONGO_OPLOG_URL.enc
+
+
+Generate `secrets.yaml':
+
+    npx --quiet --package @reactioncommerce/merge-sops-secrets@1.2.1 sops-to-secret secret-stub.yaml > secret.yaml
+
+### **Building and applying**
+
+Now, just run:
+
+    cd ./kustomize/hydra/overlays/<namespace>
+    kustomize build . | kubectl apply -f -
+
+
+### **Testing pod**
+
+    kubectl get pods -ntest
+
+Exec to the pod:
+
+    kubectl exec -it <cdc-toolbox-HASH> -ntest -- bash
+
+## **Setting DNS Records**
+
+
+### **Adding NS recorders**
+
+First, add the nameserver records for `ENV.doman.io` in [Route53](https://console.aws.amazon.com/route53/home?region=us-east-2#hosted-zones:).
+
+### **Adding Certificate**
+
+You might have to add a net certificate `*.ENV.domain.io/` to [ACM](https://us-east-2.console.aws.amazon.com/acm/home?region=us-east-2#/), then add its record in Route53 (as CNAME), and associate it to the load balancer.
+
+In the [load balancer dashboard](https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#LoadBalancers:sort=loadBalancerName), go to listeners and make sure `HTTPS : 443` uses that certificate. Make sure the load balancer has the correct security groups.
+
+### **Add All aliases**
+
+Then add all the URL aboves as IPv4 aliases pointing them to the load balancer.
+
+

cdk/python/PostgreSQL_example/README.md

@@ -0,0 +1,70 @@
+# Setting up a PostgreSQL RDS with CDK in Python
+
+### Create a virtual environment and install dependencies:
+
+```
+virtualenv .env
+source .env/bin/activate
+pip3 install -r requirements.txt
+```
+
+### Define You RDS DB 
+
+Add any constant variable in `cdk.json` and then define how you want your RDS instance in `postgre_sql_example/postgre_sql_example_stack.py`:
+
+```
+class PostgreSqlExampleStack(core.Stack):
+
+    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
+        super().__init__(scope, id, **kwargs)
+
+        # Database Instance
+        instance = rds.DatabaseInstance(self,
+            'examplepostgresdbinstance', 
+            master_username=master_username,
+            engine=rds.DatabaseInstanceEngine.POSTGRES, instance_class=ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE2, ec2.InstanceSize.MICRO), 
+            vpc=self.vpc,
+            auto_minor_version_upgrade=auto_minor_version_upgrade,
+            availability_zone=availability_zone,
+            database_name=database_name,
+            enable_performance_insights=enable_performance_insights,
+            storage_encrypted=storage_encrypted,
+            multi_az=multi_az,
+            backup_retention=backup_retention,
+            monitoring_interval=monitoring_interval,
+         )
+```
+
+### Create synthesized CloudFormation templates
+
+```
+cdk synth
+```
+
+You can check what changes are introduced into your current AWS resources with:
+```
+cdk diff --profile <AWS PROFILE>
+```
+
+
+### Deploy to AWS
+
+If everything looks OK, deploy with:
+
+```
+cdk deploy --profile <AWS PROFILE>
+```
+
+To check all the stacks in the app:
+
+```
+cdk ls
+```
+
+### Clean up
+
+To destroy/remove all the newly created resources, run:
+
+```
+cdk destroy --profile <AWS PROFILE>
+```

cdk/python/PostgreSQL_example/app.py

@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+from aws_cdk import core
+
+from postgre_sql_example.postgre_sql_example_stack import PostgreSqlExampleStack
+
+
+app = core.App()
+PostgreSqlExampleStack(app, "postgre-sql-example")
+
+app.synth()

cdk/python/PostgreSQL_example/cdk.json

@@ -0,0 +1,16 @@
+{
+    "app": "python3 app.py",
+    "context": {
+        "rds.auto_minor_version_upgrade": false,
+        "rds.availability_zone": 
+        "rds.backup_retention": 
+        "rds.database_name": "postgres_db",
+        "rds.enable_performance_insights": true,
+        "rds.master_username": "postgres",
+        "rds.monitoring_interval": 60,
+        "rds.multi_az": false,
+        "rds.storage_encrypted": false,
+        "vpc.cidr": "10.0.0.0/16",
+        "vpc.max_azs": 
+    }
+}

cdk/python/PostgreSQL_example/postgre_sql_example/postgre_sql_example_stack.py

@@ -0,0 +1,59 @@
+import json
+import sys
+from aws_cdk import (
+    aws_ec2 as ec2,
+    aws_rds as rds,
+    core as core,
+)
+
+# Python CDK does not have get_context yet.
+def _get_context():
+    CONTEXT_FILE = 'cdk.json'
+    try:
+        with open(CONTEXT_FILE, 'r') as f:
+            return json.load(f)['context']
+    except IOError:
+        print('Could not open context file {}. Exiting...'.format(CONTEXT_FILE))
+        sys.exit(1)
+    except KeyError as e:
+        print('Context file {0} is misconfigured {1}. Exiting...'.format(CONTEXT_FILE, e))
+        sys.exit(1)
+
+class PostgreSqlExampleStack(core.Stack):
+
+    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
+        super().__init__(scope, id, **kwargs)
+
+        # Grab variables from cdk.json
+        context = _get_context()
+        auto_minor_version_upgrade =  context["rds.auto_minor_version_upgrade"]
+        availability_zone = context["rds.availability_zone"]
+        backup_retention = core.Duration.days(context["rds.backup_retention"])
+        database_name = context["rds.database_name"]
+        enable_performance_insights = context["rds.enable_performance_insights"]
+        master_username = context["rds.master_username"]
+        monitoring_interval = core.Duration.seconds(context["rds.monitoring_interval"])
+        multi_az = context["rds.multi_az"]
+        storage_encrypted = context["rds.storage_encrypted"]
+        cidr = context["vpc.cidr"]
+        max_azs = context["vpc.max_azs"]
+
+        # Set VPC
+        self.vpc = ec2.Vpc(self, "VPCTest", cidr=cidr, max_azs=max_azs)
+
+        # Database Instance
+        instance = rds.DatabaseInstance(self,
+            'storefrontrdspostgresdbinstance', 
+            master_username=master_username,
+            engine=rds.DatabaseInstanceEngine.POSTGRES, instance_class=ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE2, ec2.InstanceSize.MICRO), 
+            vpc=self.vpc,
+            auto_minor_version_upgrade=auto_minor_version_upgrade,
+            availability_zone=availability_zone,
+            database_name=database_name,
+            enable_performance_insights=enable_performance_insights,
+            storage_encrypted=storage_encrypted,
+            multi_az=multi_az,
+            backup_retention=backup_retention,
+            monitoring_interval=monitoring_interval,
+         )
+

cdk/python/PostgreSQL_example/requirements.txt

@@ -0,0 +1,34 @@
+astroid==2.2.5
+attrs==19.1.0
+aws-cdk.assets==1.10.0
+aws-cdk.aws-cloudwatch==1.10.0
+aws-cdk.aws-ec2==1.10.0
+aws-cdk.aws-events==1.10.0
+aws-cdk.aws-iam==1.10.0
+aws-cdk.aws-kms==1.10.0
+aws-cdk.aws-lambda==1.10.0
+aws-cdk.aws-logs==1.10.0
+aws-cdk.aws-rds==1.10.0
+aws-cdk.aws-s3==1.10.0
+aws-cdk.aws-s3-assets==1.10.0
+aws-cdk.aws-sam==1.10.0
+aws-cdk.aws-secretsmanager==1.10.0
+aws-cdk.aws-sqs==1.10.0
+aws-cdk.aws-ssm==1.10.0
+aws-cdk.core==1.10.0
+aws-cdk.cx-api==1.10.0
+aws-cdk.region-info==1.10.0
+cattrs==0.9.0
+isort==4.3.21
+jsii==0.17.1
+lazy-object-proxy==1.4.2
+mccabe==0.6.1
+pep8==1.7.1
+publication==0.0.3
+pylint==2.3.1
+python-dateutil==2.8.0
+six==1.12.0
+typed-ast==1.4.0
+typing-extensions==3.7.4
+virtualenv==16.7.4
+wrapt==1.11.2

cdk/python/PostgreSQL_example/setup.py

@@ -0,0 +1,45 @@
+import setuptools
+
+
+with open("README.md") as fp:
+    long_description = fp.read()
+
+
+setuptools.setup(
+    name="postgre_sql_example",
+    version="0.0.1",
+
+    description="A postgres CDK Python example",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+
+    author="author",
+
+    package_dir={"": "postgre_sql_example"},
+    packages=setuptools.find_packages(where="postgre_sql_example"),
+
+    install_requires=[
+        "aws-cdk.core",
+    ],
+
+    python_requires=">=3.6",
+
+    classifiers=[
+        "Development Status :: 4 - Beta",
+
+        "Intended Audience :: Developers",
+
+        "License :: OSI Approved :: Apache Software License",
+
+        "Programming Language :: JavaScript",
+        "Programming Language :: Python :: 3 :: Only",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+
+        "Topic :: Software Development :: Code Generators",
+        "Topic :: Utilities",
+
+        "Typing :: Typed",
+    ],
+)

cdk/python/VPC_example/README.md

@@ -0,0 +1,75 @@
+# Setting up a VPC with CDK in Python
+
+[AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/home.html) is a very neat way to write infrastructure as code, enabling you to create and provision AWS infrastructure deployments predictably and repeatedly.
+
+You choose your favorite language to code what resources (stacks) you want, and CDK synthetizes them to CloudFormation and helps you to deploy them to AWS.
+
+In this example we see how to setup a VPC using CDK in Python. 
+
+### Install AWS CDK
+
+Follow [this instructions](https://github.com/aws/aws-cdk#at-a-glance).
+
+### Create a virtual environment and install dependencies:
+
+```
+virtualenv venv
+source venv/bin/activate
+pip install -r requirements.txt
+```
+
+Note: If you are starting from a blank project with `cdk init app --language=python` instead, you will we need to manually install resources, such as `pip install aws_cdk.aws_ec2`.
+
+### Define You VPC 
+
+Define how you want your VPC in the file `vpc_example/vpc_example_stack.py`:
+
+```
+class VpcExampleStack(core.Stack):
+
+    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
+        super().__init__(scope, id, **kwargs)
+
+        vpc = aws_ec2.Vpc(self, "MiaVPCTest", cidr="10.0.0.0/16", max_azs=3)
+```
+
+
+### Create synthesized CloudFormation template
+
+```
+cdk synth
+```
+
+You can check what changes this introduces into your AWS account:
+```
+cdk diff --profile <AWS PROFILE>
+```
+
+### Deploy to AWS
+
+If everything looks right, deploy:
+
+```
+cdk deploy --profile <AWS PROFILE>
+```
+
+These are the resources that will be created with this command:
+
+![vpc](https://github.com/bt3gl/AWS_Resources/blob/master/CDK_examples/VPC_example/imgs/vpc.png)
+
+
+
+To check all the stacks in the app:
+
+```
+cdk ls
+```
+
+### Clean up
+
+To destroy/remove all the newly created resources, run:
+
+```
+cdk destroy --profile <AWS PROFILE>
+```
+

cdk/python/VPC_example/app.py

@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+from aws_cdk import core
+
+from vpc_example.vpc_example_stack import VpcExampleStack
+
+
+app = core.App()
+VpcExampleStack(app, "vpc-example")
+
+app.synth()

cdk/python/VPC_example/cdk.json

@@ -0,0 +1,3 @@
+{
+    "app": "python3 app.py"
+}

cdk/python/VPC_example/requirements.txt

@@ -0,0 +1,14 @@
+attrs==19.1.0
+aws-cdk.aws-cloudwatch==1.10.0
+aws-cdk.aws-ec2==1.10.0
+aws-cdk.aws-iam==1.10.0
+aws-cdk.aws-ssm==1.10.0
+aws-cdk.core==1.10.0
+aws-cdk.cx-api==1.10.0
+aws-cdk.region-info==1.10.0
+cattrs==0.9.0
+jsii==0.17.1
+publication==0.0.3
+python-dateutil==2.8.0
+six==1.12.0
+typing-extensions==3.7.4

cdk/python/VPC_example/setup.py

@@ -0,0 +1,45 @@
+import setuptools
+
+
+with open("README.md") as fp:
+    long_description = fp.read()
+
+
+setuptools.setup(
+    name="vpc_example",
+    version="0.0.1",
+
+    description="A VPC CDK Python example",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+
+    author="author",
+
+    package_dir={"": "vpc_example"},
+    packages=setuptools.find_packages(where="vpc_example"),
+
+    install_requires=[
+        "aws-cdk.core",
+    ],
+
+    python_requires=">=3.6",
+
+    classifiers=[
+        "Development Status :: 4 - Beta",
+
+        "Intended Audience :: Developers",
+
+        "License :: OSI Approved :: Apache Software License",
+
+        "Programming Language :: JavaScript",
+        "Programming Language :: Python :: 3 :: Only",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+
+        "Topic :: Software Development :: Code Generators",
+        "Topic :: Utilities",
+
+        "Typing :: Typed",
+    ],
+)

cdk/python/VPC_example/vpc_example/vpc_example_stack.py

@@ -0,0 +1,8 @@
+from aws_cdk import core, aws_ec2
+
+class VpcExampleStack(core.Stack):
+
+    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
+        super().__init__(scope, id, **kwargs)
+
+        vpc = aws_ec2.Vpc(self, "MiaVPCTest", cidr="10.0.0.0/16", max_azs=3)

cdk/ts/.gitignore

@@ -0,0 +1,136 @@
+# CDK
+*.js
+!jest.config.js
+*.d.ts
+node_modules
+.cdk.staging
+cdk.out
+
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+.idea
+.DS_Store
+# Diagnostic reports (https://nodejs.org/api/report.html)
+report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+*.lcov
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# TypeScript v1 declaration files
+typings/
+
+# TypeScript cache
+*.tsbuildinfo
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variables file
+.env
+.env.test
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+
+# next.js build output
+.next
+
+# nuxt.js build output
+.nuxt
+
+# vuepress build output
+.vuepress/dist
+
+# Serverless directories
+.serverless/
+
+# FuseBox cache
+.fusebox/
+
+# DynamoDB Local files
+.dynamodb/
+
+# Kubernetes kubectl configuration
+kubeconfig
+
+# CDK
+cdk.out/
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Certs
+*.key
+*.crt
+easy-rsa/

cdk/ts/MSK_example/README.md

@@ -0,0 +1,61 @@
+# CDK MSK Example
+
+
+
+### Deploy VPC
+
+[Amazon VPC](https://aws.amazon.com/vpc/) lets you provision a logically isolated section of the AWS Cloud where you can all the resources as in a virtual network.
+
+These are the default values in `cdk.json`:
+
+```
+"vpc.cidr": "10.0.0.0/16",
+"vpc.maxAzs": 3,
+```
+
+Deploy with:
+
+```
+cdk deploy VPCStack
+```
+
+### Deploy MSK
+
+[Amazon MSK](https://aws.amazon.com/msk/) is a managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data.
+
+These are the default values in `cdk.json`:
+
+```
+"msk.DevMskCluster": "MskCluster",
+"msk.ClusterTag": "MSK cluster",
+"msk.brokerNodeGroupBrokerAzDistribution": "DEFAULT",
+"msk.enhancedMonitoring": "PER_BROKER",
+"msk.brokerNodeGroupEBSVolumeSize": 100,
+"msk.brokerNodeGroupInstanceType": "kafka.m5.large",
+"msk.brokerPort": 9092,
+"msk.kafkaVersion": "2.2.1",
+"msk.numberOfBrokerNodes": 3
+```
+
+Deploy with:
+
+```
+cdk deploy MskClusterStack
+```
+
+#### Kafka CLI 
+
+Note that the CLI commands for MKS are given by the keyword `kafka`, for example:
+
+```
+aws kafka list-clusters
+```
+
+To retrieve `BootstrapBrokerStringTls`, run:
+
+```
+ aws kafka get-bootstrap-brokers --cluster-arn <cluster ARN>
+ ```
+
+However, access and development within the cluster (e.g. creating topics, accessing brokers) need to be done while connected to the VPN.
+

cdk/ts/MSK_example/bin/example.ts

@@ -0,0 +1,15 @@
+#!/usr/bin/env node
+import 'source-map-support/register';
+import cdk = require('@aws-cdk/core');
+
+import { VPCStack } from "../lib/Vpc";
+import { MskClusterStack } from "../lib/MskCluster";
+
+const app = new cdk.App();
+const app_env = {
+    region: <account region>,
+    account: <account number>
+};
+
+const vpcStack = new VPCStack(app, 'VPCStack', {env: app_env});
+new MskClusterStack(app, 'MskClusterStack',{env: app_env, vpc: vpcStack.Vpc});

cdk/ts/MSK_example/cdk.context.json

@@ -0,0 +1,7 @@
+{
+  "availability-zones:account=<ACOUNT NUMBER>:region=us-west-1": [
+    "us-west-1a",
+    "us-west-1b",
+    "us-west-1c"
+  ]
+}

cdk/ts/MSK_example/cdk.json

@@ -0,0 +1,17 @@
+{
+  "app": "npx ts-node bin/example.ts",
+  "context": {
+    "env.type": "dev",
+    "vpc.cidr": "10.0.0.0/16",
+    "vpc.maxAzs": 3,
+    "msk.DevMskCluster": "MskCluster",
+    "msk.ClusterTag": "MSK cluster",
+    "msk.brokerNodeGroupBrokerAzDistribution": "DEFAULT",
+    "msk.enhancedMonitoring": "PER_BROKER",
+    "msk.brokerNodeGroupEBSVolumeSize": 100,
+    "msk.brokerNodeGroupInstanceType": "kafka.m5.large",
+    "msk.brokerPort": 9092,
+    "msk.kafkaVersion": "2.2.1",
+    "msk.numberOfBrokerNodes": 3
+  }
+}

cdk/ts/MSK_example/lib/MskCluster.ts

@@ -0,0 +1,75 @@
+import cdk = require("@aws-cdk/core");
+import ec2 = require("@aws-cdk/aws-ec2");
+import msk = require("@aws-cdk/aws-msk");
+
+
+interface MSKStackProps extends cdk.StackProps {
+  vpc: ec2.IVpc;
+}
+export class MskClusterStack extends cdk.Stack {
+  private vpc: ec2.IVpc;
+  
+  constructor(scope: cdk.Construct, id: string, props?: MSKStackProps) {
+    super(scope, id, props);
+    const current_env = this.node.tryGetContext("env.type");
+
+    //****************************** Context variables **************************************//
+    const clusterName = this.node.tryGetContext("msk.DevMskCluster");
+    const clusterTag = this.node.tryGetContext("msk.mskClusterTag");
+    const brokerNodeGroupBrokerAzDistribution = this.node.tryGetContext("msk.brokerNodeGroupBrokerAzDistribution");
+    const brokerNodeGroupEBSVolumeSize = this.node.tryGetContext("msk.brokerNodeGroupEBSVolumeSize");
+    const brokerNodeGroupInstanceType = this.node.tryGetContext("msk.brokerNodeGroupInstanceType");
+    const brokerPort = this.node.tryGetContext("msk.brokerPort");
+    const kafkaVersion = this.node.tryGetContext("msk.kafkaVersion");
+    const numberOfBrokerNodes = this.node.tryGetContext("msk.numberOfBrokerNodes");
+    const enhancedMonitoring = this.node.tryGetContext("msk.enhancedMonitoring");
+
+    //****************************************  VPC 
+    if (props)
+      this.vpc = props.vpc;
+    else
+      this.vpc = ec2.Vpc.fromLookup(this, current_env+"Vpc", {
+        vpcName: "VPCStack/"+current_env+"Vpc"
+      });
+
+    //****************************************  SG 
+    const description = "Allow access to "+current_env+" MSK Cluster";
+    const SecurityGroup = new ec2.SecurityGroup(
+        this,
+        current_env+"MskClusterSG",
+        {
+          vpc: this.vpc,
+          securityGroupName: current_env+"MskClusterSG",
+          description: description,
+          allowAllOutbound: true
+        }
+    );
+    SecurityGroup.addIngressRule(
+        ec2.Peer.anyIpv4(),
+        ec2.Port.tcp(brokerPort),
+        description
+    );
+
+    //*******************************  MSK Cluster **************************//
+    const cluster = new msk.CfnCluster(this, "MskCluster", {
+        brokerNodeGroupInfo: {
+          clientSubnets: this.vpc.privateSubnets.map(x => x.subnetId),
+          instanceType: brokerNodeGroupInstanceType,
+          brokerAzDistribution: brokerNodeGroupBrokerAzDistribution,
+          storageInfo: {
+            ebsStorageInfo: {
+              volumeSize: brokerNodeGroupEBSVolumeSize
+            }
+          }
+        },
+        clusterName: clusterName,
+        kafkaVersion: kafkaVersion,
+        numberOfBrokerNodes: numberOfBrokerNodes,
+        enhancedMonitoring: enhancedMonitoring,
+        tags: {
+          name: current_env+clusterTag,
+        }
+      });
+    }
+  }
+

cdk/ts/MSK_example/lib/Vpc.ts

@@ -0,0 +1,19 @@
+import cdk = require('@aws-cdk/core');
+import ec2 = require("@aws-cdk/aws-ec2");
+
+export class VPCStack extends cdk.Stack {
+  readonly Vpc: ec2.IVpc;
+
+  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
+    super(scope, id, props);
+    const current_env = this.node.tryGetContext("env.type");
+
+    const vpc_cidr = this.node.tryGetContext("vpc.cidr");
+    const vpc_maxAzs = this.node.tryGetContext("vpc.maxAzs");
+    const vpc = new ec2.Vpc(this, current_env+"Vpc", {
+      cidr: vpc_cidr,
+      maxAzs: vpc_maxAzs
+    });
+    this.Vpc = vpc;
+  }
+}

cdk/ts/MSK_example/package.json

@@ -0,0 +1,28 @@
+{
+  "name": "dev",
+  "version": "0.1.0",
+  "bin": {
+    "dev": "bin/dev.js"
+  },
+  "scripts": {
+    "build": "tsc",
+    "watch": "tsc -w",
+    "test": "jest",
+    "cdk": "cdk"
+  },
+  "devDependencies": {
+    "@aws-cdk/assert": "^1.12.0",
+    "@types/jest": "^24.0.18",
+    "aws-cdk": "^1.12.0",
+    "jest": "^24.9.0",
+    "ts-jest": "^24.1.0",
+    "ts-node": "^8.4.1",
+    "typescript": "~3.6.4"
+  },
+  "dependencies": {
+    "@aws-cdk/aws-ec2": "^1.12.0",
+    "@aws-cdk/aws-msk": "^1.12.0",
+    "@aws-cdk/core": "^1.12.0",
+    "source-map-support": "^0.5.13"
+  }
+}

cdk/ts/MSK_example/tsconfig.json

@@ -0,0 +1,23 @@
+{
+  "compilerOptions": {
+    "target":"ES2018",
+    "module": "commonjs",
+    "lib": ["es2016", "es2017.object", "es2017.string"],
+    "declaration": true,
+    "strict": true,
+    "noImplicitAny": true,
+    "strictNullChecks": true,
+    "noImplicitThis": true,
+    "alwaysStrict": true,
+    "noUnusedLocals": false,
+    "noUnusedParameters": false,
+    "noImplicitReturns": true,
+    "noFallthroughCasesInSwitch": false,
+    "inlineSourceMap": true,
+    "inlineSources": true,
+    "experimentalDecorators": true,
+    "strictPropertyInitialization":false,
+    "typeRoots": ["./node_modules/@types"]
+  },
+  "exclude": ["cdk.out"]
+}

cdk/ts/README.md

@@ -0,0 +1,3 @@
+# CDK TypeScript Examples
+
+* MSK + VPC

gpc/.gitignore

@@ -0,0 +1,104 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/

gpc/README.md

@@ -0,0 +1,6 @@
+# GCP Resources
+
+
+* `slides_data_engineering/`: private resources from [GPC data engineering training](https://cloud.google.com/training/data-ml).
+* `slides_infrastructure/`: private resources from [GCP infra training](https://cloud.google.com/training/cloud-infrastructure).
+* `labs/`: laboratory exercises from the resources above.

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/caff0d62ed

@@ -0,0 +1 @@
+NREUM.setToken({'stn':0,'err':0,'ins':0,'cap':0,'spa':0})

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/saved_resource(1).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/saved_resource(2).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/saved_resource(3).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/saved_resource(4).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/saved_resource(5).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/saved_resource(6).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Leveraging Unstructured Data _ Qwiklabs + roitraining_files/saved_resource.html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/caff0d62ed

@@ -0,0 +1 @@
+NREUM.setToken({'stn':0,'err':0,'ins':0,'cap':0,'spa':0})

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/saved_resource(1).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/saved_resource(2).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/saved_resource(3).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/saved_resource(4).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/saved_resource(5).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/saved_resource(6).html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Data Analysis (Python) _ Qwiklabs + roitraining_files/saved_resource.html

@@ -0,0 +1,3 @@
+
+<!-- saved from url=(0011)about:blank -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body></body></html>

gpc/labs/Serverless Machine Learning _ Qwiklabs + roitraining.htm

@@ -0,0 +1,903 @@
+<!DOCTYPE html>
+<!-- saved from url=(0054)https://roitraining.qwiklab.com/focuses/2773/materials -->
+<html class="mdl-js"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+<script type="text/javascript" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/caff0d62ed"></script><script src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/nr-1044.min.js"></script><script type="text/javascript">window.NREUM||(NREUM={});NREUM.info={"beacon":"bam.nr-data.net","errorBeacon":"bam.nr-data.net","licenseKey":"caff0d62ed","applicationID":"25010137","transactionName":"IQ1XRUEOVV1dFxlRXAEXSlRATkpZVxJpWlIWB0tYUg1K","queueTime":0,"applicationTime":382,"agent":""}</script>
+<script type="text/javascript">window.NREUM||(NREUM={}),__nr_require=function(e,n,t){function r(t){if(!n[t]){var o=n[t]={exports:{}};e[t][0].call(o.exports,function(n){var o=e[t][1][n];return r(o||n)},o,o.exports)}return n[t].exports}if("function"==typeof __nr_require)return __nr_require;for(var o=0;o<t.length;o++)r(t[o]);return r}({1:[function(e,n,t){function r(){}function o(e,n,t){return function(){return i(e,[c.now()].concat(u(arguments)),n?null:this,t),n?void 0:this}}var i=e("handle"),a=e(2),u=e(3),f=e("ee").get("tracer"),c=e("loader"),s=NREUM;"undefined"==typeof window.newrelic&&(newrelic=s);var p=["setPageViewName","setCustomAttribute","setErrorHandler","finished","addToTrace","inlineHit","addRelease"],d="api-",l=d+"ixn-";a(p,function(e,n){s[n]=o(d+n,!0,"api")}),s.addPageAction=o(d+"addPageAction",!0),s.setCurrentRouteName=o(d+"routeName",!0),n.exports=newrelic,s.interaction=function(){return(new r).get()};var m=r.prototype={createTracer:function(e,n){var t={},r=this,o="function"==typeof n;return i(l+"tracer",[c.now(),e,t],r),function(){if(f.emit((o?"":"no-")+"fn-start",[c.now(),r,o],t),o)try{return n.apply(this,arguments)}finally{f.emit("fn-end",[c.now()],t)}}}};a("setName,setAttribute,save,ignore,onEnd,getContext,end,get".split(","),function(e,n){m[n]=o(l+n)}),newrelic.noticeError=function(e){"string"==typeof e&&(e=new Error(e)),i("err",[e,c.now()])}},{}],2:[function(e,n,t){function r(e,n){var t=[],r="",i=0;for(r in e)o.call(e,r)&&(t[i]=n(r,e[r]),i+=1);return t}var o=Object.prototype.hasOwnProperty;n.exports=r},{}],3:[function(e,n,t){function r(e,n,t){n||(n=0),"undefined"==typeof t&&(t=e?e.length:0);for(var r=-1,o=t-n||0,i=Array(o<0?0:o);++r<o;)i[r]=e[n+r];return i}n.exports=r},{}],4:[function(e,n,t){n.exports={exists:"undefined"!=typeof window.performance&&window.performance.timing&&"undefined"!=typeof window.performance.timing.navigationStart}},{}],ee:[function(e,n,t){function r(){}function o(e){function n(e){return e&&e instanceof r?e:e?f(e,u,i):i()}function t(t,r,o,i){if(!d.aborted||i){e&&e(t,r,o);for(var a=n(o),u=m(t),f=u.length,c=0;c<f;c++)u[c].apply(a,r);var p=s[y[t]];return p&&p.push([b,t,r,a]),a}}function l(e,n){v[e]=m(e).concat(n)}function m(e){return v[e]||[]}function w(e){return p[e]=p[e]||o(t)}function g(e,n){c(e,function(e,t){n=n||"feature",y[t]=n,n in s||(s[n]=[])})}var v={},y={},b={on:l,emit:t,get:w,listeners:m,context:n,buffer:g,abort:a,aborted:!1};return b}function i(){return new r}function a(){(s.api||s.feature)&&(d.aborted=!0,s=d.backlog={})}var u="nr@context",f=e("gos"),c=e(2),s={},p={},d=n.exports=o();d.backlog=s},{}],gos:[function(e,n,t){function r(e,n,t){if(o.call(e,n))return e[n];var r=t();if(Object.defineProperty&&Object.keys)try{return Object.defineProperty(e,n,{value:r,writable:!0,enumerable:!1}),r}catch(i){}return e[n]=r,r}var o=Object.prototype.hasOwnProperty;n.exports=r},{}],handle:[function(e,n,t){function r(e,n,t,r){o.buffer([e],r),o.emit(e,n,t)}var o=e("ee").get("handle");n.exports=r,r.ee=o},{}],id:[function(e,n,t){function r(e){var n=typeof e;return!e||"object"!==n&&"function"!==n?-1:e===window?0:a(e,i,function(){return o++})}var o=1,i="nr@id",a=e("gos");n.exports=r},{}],loader:[function(e,n,t){function r(){if(!x++){var e=h.info=NREUM.info,n=d.getElementsByTagName("script")[0];if(setTimeout(s.abort,3e4),!(e&&e.licenseKey&&e.applicationID&&n))return s.abort();c(y,function(n,t){e[n]||(e[n]=t)}),f("mark",["onload",a()+h.offset],null,"api");var t=d.createElement("script");t.src="https://"+e.agent,n.parentNode.insertBefore(t,n)}}function o(){"complete"===d.readyState&&i()}function i(){f("mark",["domContent",a()+h.offset],null,"api")}function a(){return E.exists&&performance.now?Math.round(performance.now()):(u=Math.max((new Date).getTime(),u))-h.offset}var u=(new Date).getTime(),f=e("handle"),c=e(2),s=e("ee"),p=window,d=p.document,l="addEventListener",m="attachEvent",w=p.XMLHttpRequest,g=w&&w.prototype;NREUM.o={ST:setTimeout,SI:p.setImmediate,CT:clearTimeout,XHR:w,REQ:p.Request,EV:p.Event,PR:p.Promise,MO:p.MutationObserver};var v=""+location,y={beacon:"bam.nr-data.net",errorBeacon:"bam.nr-data.net",agent:"js-agent.newrelic.com/nr-1044.min.js"},b=w&&g&&g[l]&&!/CriOS/.test(navigator.userAgent),h=n.exports={offset:u,now:a,origin:v,features:{},xhrWrappable:b};e(1),d[l]?(d[l]("DOMContentLoaded",i,!1),p[l]("load",r,!1)):(d[m]("onreadystatechange",o),p[m]("onload",r)),f("mark",["firstbyte",u],null,"api");var x=0,E=e(4)},{}]},{},["loader"]);</script>
+<meta name="csrf-param" content="authenticity_token">
+<meta name="csrf-token" content="QagR46dQ6cz/fE7QLGPcaYEmMReiPhvcSK2AVVN0VomR6mjyKUkLtajGoQoNJlMN5Ta4t/fB1e8CeJeXaN7VfQ==">
+<title>Serverless Machine Learning | Qwiklabs + roitraining</title>
+<meta content="width=device-width, initial-scale=1.0, maximum-scale=1, user-scalable=0" name="viewport">
+<meta content="In this series of labs, you go from exploring a taxicab dataset to training and deploying a high-accuracy distributed model with Cloud ML Engine." name="description">
+<meta content="Learn AWS, AWS Training, AWS Labs, Learn Amazon Web Services, Amazon Web Services Training, Amazon Web Services Labs" name="keywords">
+<meta content="Qwiklabs" name="author">
+<meta content="Serverless Machine Learning | Qwiklabs + roitraining" property="og:title">
+<meta content="website" property="og:type">
+<meta content="/favicon.png" property="og:image">
+<meta content="https://www.qwiklabs.com" property="og:url">
+<meta content="Qwiklabs" property="og:site_name">
+<meta content="In this series of labs, you go from exploring a taxicab dataset to training and deploying a high-accuracy distributed model with Cloud ML Engine." property="og:description">
+<meta content="/qwiklabs_logo_900x887.png" property="og:logo" size="900x887">
+<meta content="/qwiklabs_logo_994x187.png" property="og:logo" size="994x187">
+<meta content="#3681E4" property="msapplication-TileColor">
+<meta content="/favicon-144.png" property="msapplication-TileImage">
+<link href="https://roitraining.qwiklab.com/favicon.ico" rel="shortcut icon">
+<link color="#3681E4" href="https://roitraining.qwiklab.com/favicon-svg.svg" rel="mask-icon">
+<link href="https://roitraining.qwiklab.com/favicon-180.png" rel="apple-touch-icon-precomposed">
+
+<!--[if lt IE 9]>
+<script src='http://html5shim.googlecode.com/svn/trunk/html5.js' type='text/javascript'></script>
+<![endif]-->
+<!--[endif]>  <![endif]-->
+<script>
+//<![CDATA[
+window.gon={};gon.current_user={"firstname":"marina","lastname":"steinkirch","fullname":"marina steinkirch","company":"etsy","email":"mvonsteinkirch@etsy.com","origin":"roitraining, direct","subscriptions":0,"id":"12ee659298eb15258fdeb4d43db52cb8","qlCreatedAt":"2017-11-28 14:06:23 UTC","optIn":false};gon.segment=null;gon.deployment="roitraining";
+//]]>
+</script>
+
+
+
+<link rel="stylesheet" media="all" href="./Serverless Machine Learning _ Qwiklabs + roitraining_files/application-6460790cbdd89c50da4755d15c7ef68fa373dd59daad1528c39815f8c2c4676d.css">
+<script src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/application-965286b1d75b8ed026adfefe5748f3ad70657330c97a79281c8bc1b35d341af9.js"></script>
+</head>
+<body class="focuses focuses-show_materials l-no-padding ilt-mode">
+<div class="header-container">
+<div class="header">
+<a class="mdl-button mdl-button--icon mdl-js-button mdl-js-ripple-effect header__button header__button--nav header__side-menu-button js-side-menu-button" data-upgraded=",MaterialButton,MaterialRipple">
+<i class="material-icons">menu</i>
+<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></a>
+<div class="header__title">
+<a class="mdl-button mdl-js-button mdl-button--icon mdl-js-ripple-effect header__button header__button--nav" href="https://roitraining.qwiklab.com/materials/252" data-upgraded=",MaterialButton,MaterialRipple"><i class="material-icons">arrow_back</i><span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></a>
+
+<h1>
+Serverless Machine Learning
+</h1>
+</div>
+<div class="header__actions">
+<div class="header__menu header__menu--my-account">
+<button class="mdl-button mdl-button--icon mdl-js-button mdl-js-ripple-effect" id="header_menu" data-upgraded=",MaterialButton,MaterialRipple">
+<i class="material-icons"><img class="avatar " src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/a835b0e3b23a9e319e795e2bf1bccaa8.png" alt="A835b0e3b23a9e319e795e2bf1bccaa8"></i>
+<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></button>
+<div class="mdl-menu__container is-upgraded"><div class="mdl-menu__outline mdl-menu--bottom-right"></div><ul class="mdl-menu mdl-menu--bottom-right mdl-js-menu mdl-js-ripple-effect mdl-js-ripple-effect--ignore-events" for="header_menu" data-upgraded=",MaterialMenu,MaterialRipple">
+<li class="mdl-menu__item header__menu__item mdl-js-ripple-effect" tabindex="-1" data-upgraded=",MaterialRipple"><a href="https://roitraining.qwiklab.com/my_account/profile">My Account</a><span class="mdl-menu__item-ripple-container"><span class="mdl-ripple"></span></span></li>
+<li class="mdl-menu__item header__menu__item mdl-js-ripple-effect" tabindex="-1" data-upgraded=",MaterialRipple"><a rel="nofollow" data-method="delete" href="https://roitraining.qwiklab.com/users/sign_out">Sign Out</a><span class="mdl-menu__item-ripple-container"><span class="mdl-ripple"></span></span></li>
+
+</ul></div>
+</div>
+
+</div>
+</div>
+</div>
+<div class="header__search-bar js-header-search-bar">
+<form action="https://roitraining.qwiklab.com/searches/lab" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="✓"><input type="hidden" name="authenticity_token" value="QagR46dQ6cz/fE7QLGPcaYEmMReiPhvcSK2AVVN0VomR6mjyKUkLtajGoQoNJlMN5Ta4t/fB1e8CeJeXaN7VfQ==">
+<input type="text" name="keywords" id="keywords" value="" placeholder="Search for labs">
+</form>
+
+<a class="mdl-button mdl-js-button mdl-button--icon mdl-js-ripple-effect header__button" data-upgraded=",MaterialButton,MaterialRipple">
+<i class="material-icons">close</i>
+<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></a>
+</div>
+
+<div class="l-flex">
+<div class="side-menu js-side-menu">
+<div class="side-menu__inner">
+<nav class="side-menu__nav">
+<a class="side-menu__item" href="https://roitraining.qwiklab.com/materials"><div class="side-menu__item__icon">
+<i class="material-icons">view_comfy</i>
+</div>
+<span class="side-menu__item__tooltip">Materials</span>
+<div class="side-menu__item__label">
+Materials
+</div>
+</a>
+<a class="side-menu__item" href="https://roitraining.qwiklab.com/dashboard"><div class="side-menu__item__icon">
+<i class="material-icons">history</i>
+</div>
+<span class="side-menu__item__tooltip">My Learning</span>
+<div class="side-menu__item__label">
+My Learning
+</div>
+</a>
+<hr>
+<a class="side-menu__item" href="https://roitraining.qwiklab.com/my_account/credits"><div class="side-menu__item__icon">
+<i class="material-icons">account_circle</i>
+</div>
+<span class="side-menu__item__tooltip">My Account</span>
+<div class="side-menu__item__label">
+My Account
+</div>
+</a>
+<a class="side-menu__item" href="https://qwiklab.zendesk.com/hc/en-us"><div class="side-menu__item__icon">
+<i class="material-icons">help</i>
+</div>
+<span class="side-menu__item__tooltip">Help</span>
+<div class="side-menu__item__label">
+Help
+</div>
+</a>
+</nav>
+<div class="side-menu__small-links">
+<a href="https://roitraining.qwiklab.com/privacy_policy">Privacy Policy</a>
+<br>
+<a href="https://roitraining.qwiklab.com/terms_of_service">Terms of Service</a>
+</div>
+</div>
+</div>
+<div class="side-menu__overlay js-side-menu-button"></div>
+
+<main>
+<div class="l-alert-wrapper alerts">
+<span class="hidden" id="flash-sibling-before"></span>
+
+</div>
+<div class="l-main-wrapper">
+
+
+
+<div class="l-lab-container js-lab_and_classroom_info" data-classroom-name="Data Engineering on Google Cloud Platform v1.1" data-deployment="roitraining" data-lab-name="Serverless Machine Learning" data-label="Serverless Machine Learning">
+<div class="l-lab-sidebar js-lab-sidebar-container">
+<div class="lab-sidebar js-lab-sidebar">
+<div class="lab-sidebar__header">
+<div class="lab-sidebar__header-row">
+<span class="small-label">
+4m setup
+·
+2880m access
+·
+2880m completion
+</span>
+</div>
+<div class="lab-sidebar__header-row">
+<div class="rateit l-mrm" data-rateit-readonly="true" data-rateit-value="3.3333"><div class="rateit-reset" style="display: none;"></div><div class="rateit-range" style="width: 80px; height: 16px;"><div class="rateit-selected" style="height: 16px; width: 53.3328px;"></div><div class="rateit-hover" style="height:16px"></div></div></div>
+
+<a class="small-label l-mrm" data-target="#lab-review-modal" data-toggle="modal">
+Rate Lab
+</a>
+<a class="small-label" data-target="#lab-details-modal" data-toggle="modal">
+Lab Details
+</a>
+</div>
+
+</div>
+<div class="lab-sidebar__tabs">
+<div class="tab-contents tab-contents--lab-sidebar">
+<div class="tab-content is-active">
+<h5 class="l-mbs">
+Connection Details
+</h5>
+<div class="form-row js-form-row">
+<a class="button button--full-width button--secondary js-connection-dns-link js-external-window" target="_blank" href="https://roitraining.qwiklab.com/lab_instances/run/6383">
+Open Google Console
+</a>
+</div>
+
+<div class="form-row js-form-row">
+<div class="control-group">
+<label class="label--console">
+Username 
+</label>
+<input class="input input--console js-connection-username-0" readonly="readonly" value="············">
+<button class="button button--copy button--copy-input js-copy-input-button" data-clipboard-target=".js-connection-username-0">
+<i class="fa fa-clipboard"></i>
+</button>
+<span style="opacity: 1; left: 274px; top: 18.5px; width: 19px; min-width: 19px; height: 13px; position: absolute; background-image: url(&quot;data:image/svg+xml;base64,PHN2ZyB4bWxucz0naHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmcnIHhtbG5zOnhsaW5rPSdodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rJyB3aWR0aD0nMTcnIGhlaWdodD0nMTInIHZpZXdCb3g9JzAgMCAxNyAxMic+IDxkZWZzPiA8cGF0aCBpZD0nYScgZD0nTTcuOTA5IDEuNDYybDIuMTIxLjg2NHMtLjY3MS4xMy0xLjIwOS4yOTRjMCAwIC40MzcuNjM0Ljc3LjkzOC4zOTEtLjE4LjY1Ny0uMjQ4LjY1Ny0uMjQ4LS44MTEgMS42NjgtMi45NzkgMi43MDMtNC41MyAyLjcwMy0uMDkzIDAtLjQ4Mi0uMDA2LS43MjcuMDE1LS40MzUuMDIxLS41ODEuMzgtLjM3NC40NzMuMzczLjIwMSAxLjE0My42NjIuOTU4IDEuMDA5QzUuMiA4LjAwMy45OTkgMTEgLjk5OSAxMWwuNjQ4Ljg4Nkw2LjEyOSA4LjYzQzguNjAyIDYuOTQ4IDEyLjAwNiA2IDE1IDZoM1Y1aC00LjAwMWMtMS4wNTggMC0yLjA0LjEyMi0yLjQ3My0uMDItLjQwMi0uMTMzLS41MDItLjY3OS0uNDU1LTEuMDM1YTcuODcgNy44NyAwIDAgMSAuMTg3LS43MjljLjAyOC0uMDk5LjA0Ni0uMDc3LjE1NS0uMDk5LjU0LS4xMTIuNzc3LS4wOTUuODIxLS4xNi4xNDYtLjI0NS4yNTQtLjk3NC4yNTQtLjk3NEw3LjU2OS4zODlzLjIwMiAxLjAxMy4zNCAxLjA3M3onLz4gPC9kZWZzPiA8dXNlIGZpbGw9JyMwMDdDOTcnIGZpbGwtcnVsZT0nZXZlbm9kZCcgdHJhbnNmb3JtPSd0cmFuc2xhdGUoLTEpJyB4bGluazpocmVmPScjYScvPiA8L3N2Zz4=&quot;); background-repeat: no-repeat; background-position: 0px 0px; border: none; display: inline; visibility: visible; z-index: auto;"></span></div>
+</div>
+
+<div class="form-row js-form-row">
+<div class="control-group">
+<label class="label--console">
+Password
+</label>
+<input class="input input--console js-connection-password" readonly="readonly" value="·············">
+<button class="button button--copy button--copy-input js-copy-input-button" data-clipboard-target=".js-connection-password">
+<i class="fa fa-clipboard"></i>
+</button>
+<span style="opacity: 1; left: 274px; top: 18.5px; width: 19px; min-width: 19px; height: 13px; position: absolute; background-image: url(&quot;data:image/svg+xml;base64,PHN2ZyB4bWxucz0naHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmcnIHhtbG5zOnhsaW5rPSdodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rJyB3aWR0aD0nMTcnIGhlaWdodD0nMTInIHZpZXdCb3g9JzAgMCAxNyAxMic+IDxkZWZzPiA8cGF0aCBpZD0nYScgZD0nTTcuOTA5IDEuNDYybDIuMTIxLjg2NHMtLjY3MS4xMy0xLjIwOS4yOTRjMCAwIC40MzcuNjM0Ljc3LjkzOC4zOTEtLjE4LjY1Ny0uMjQ4LjY1Ny0uMjQ4LS44MTEgMS42NjgtMi45NzkgMi43MDMtNC41MyAyLjcwMy0uMDkzIDAtLjQ4Mi0uMDA2LS43MjcuMDE1LS40MzUuMDIxLS41ODEuMzgtLjM3NC40NzMuMzczLjIwMSAxLjE0My42NjIuOTU4IDEuMDA5QzUuMiA4LjAwMy45OTkgMTEgLjk5OSAxMWwuNjQ4Ljg4Nkw2LjEyOSA4LjYzQzguNjAyIDYuOTQ4IDEyLjAwNiA2IDE1IDZoM1Y1aC00LjAwMWMtMS4wNTggMC0yLjA0LjEyMi0yLjQ3My0uMDItLjQwMi0uMTMzLS41MDItLjY3OS0uNDU1LTEuMDM1YTcuODcgNy44NyAwIDAgMSAuMTg3LS43MjljLjAyOC0uMDk5LjA0Ni0uMDc3LjE1NS0uMDk5LjU0LS4xMTIuNzc3LS4wOTUuODIxLS4xNi4xNDYtLjI0NS4yNTQtLjk3NC4yNTQtLjk3NEw3LjU2OS4zODlzLjIwMiAxLjAxMy4zNCAxLjA3M3onLz4gPC9kZWZzPiA8dXNlIGZpbGw9JyMwMDdDOTcnIGZpbGwtcnVsZT0nZXZlbm9kZCcgdHJhbnNmb3JtPSd0cmFuc2xhdGUoLTEpJyB4bGluazpocmVmPScjYScvPiA8L3N2Zz4=&quot;); background-repeat: no-repeat; background-position: 0px 0px; border: none; display: inline; visibility: visible; z-index: auto;"></span></div>
+</div>
+
+<div class="form-row js-form-row">
+<div class="control-group">
+<label class="label--console">
+GCP Project ID 
+</label>
+<input class="input input--console js-connection-project-0" readonly="readonly" value="···········">
+<button class="button button--copy button--copy-input js-copy-input-button" data-clipboard-target=".js-connection-project-0">
+<i class="fa fa-clipboard"></i>
+</button>
+</div>
+</div>
+
+
+<div class="form-row js-form-row" style="display: none;">
+<div class="control-group">
+<label class="label--console">
+Username
+</label>
+<input class="input input--console js-connection-username" disabled="disabled" readonly="readonly" value="··········" style="">
+<button class="button button--copy button--copy-input js-copy-input-button" data-clipboard-target=".js-connection-username">
+<i class="fa fa-clipboard"></i>
+</button>
+<span style="opacity: 1; left: 0px; top: -6.5px; width: 19px; min-width: 19px; height: 13px; position: absolute; background-image: url(&quot;data:image/svg+xml;base64,PHN2ZyB4bWxucz0naHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmcnIHhtbG5zOnhsaW5rPSdodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rJyB3aWR0aD0nMTcnIGhlaWdodD0nMTInIHZpZXdCb3g9JzAgMCAxNyAxMic+IDxkZWZzPiA8cGF0aCBpZD0nYScgZD0nTTcuOTA5IDEuNDYybDIuMTIxLjg2NHMtLjY3MS4xMy0xLjIwOS4yOTRjMCAwIC40MzcuNjM0Ljc3LjkzOC4zOTEtLjE4LjY1Ny0uMjQ4LjY1Ny0uMjQ4LS44MTEgMS42NjgtMi45NzkgMi43MDMtNC41MyAyLjcwMy0uMDkzIDAtLjQ4Mi0uMDA2LS43MjcuMDE1LS40MzUuMDIxLS41ODEuMzgtLjM3NC40NzMuMzczLjIwMSAxLjE0My42NjIuOTU4IDEuMDA5QzUuMiA4LjAwMy45OTkgMTEgLjk5OSAxMWwuNjQ4Ljg4Nkw2LjEyOSA4LjYzQzguNjAyIDYuOTQ4IDEyLjAwNiA2IDE1IDZoM1Y1aC00LjAwMWMtMS4wNTggMC0yLjA0LjEyMi0yLjQ3My0uMDItLjQwMi0uMTMzLS41MDItLjY3OS0uNDU1LTEuMDM1YTcuODcgNy44NyAwIDAgMSAuMTg3LS43MjljLjAyOC0uMDk5LjA0Ni0uMDc3LjE1NS0uMDk5LjU0LS4xMTIuNzc3LS4wOTUuODIxLS4xNi4xNDYtLjI0NS4yNTQtLjk3NC4yNTQtLjk3NEw3LjU2OS4zODlzLjIwMiAxLjAxMy4zNCAxLjA3M3onLz4gPC9kZWZzPiA8dXNlIGZpbGw9JyMwMDdDOTcnIGZpbGwtcnVsZT0nZXZlbm9kZCcgdHJhbnNmb3JtPSd0cmFuc2xhdGUoLTEpJyB4bGluazpocmVmPScjYScvPiA8L3N2Zz4=&quot;); background-repeat: no-repeat; background-position: 0px 0px; border: none; display: none; visibility: visible; z-index: auto;"></span></div>
+</div>
+
+<div class="form-row js-form-row">
+<div class="control-group">
+<label class="label--console">
+Password
+</label>
+<input class="input input--console js-connection-password" readonly="readonly" value="·········">
+<button class="button button--copy button--copy-input js-copy-input-button" data-clipboard-target=".js-connection-password">
+<i class="fa fa-clipboard"></i>
+</button>
+<span style="opacity: 1; left: 274px; top: 18.5px; width: 19px; min-width: 19px; height: 13px; position: absolute; background-image: url(&quot;data:image/svg+xml;base64,PHN2ZyB4bWxucz0naHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmcnIHhtbG5zOnhsaW5rPSdodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rJyB3aWR0aD0nMTcnIGhlaWdodD0nMTInIHZpZXdCb3g9JzAgMCAxNyAxMic+IDxkZWZzPiA8cGF0aCBpZD0nYScgZD0nTTcuOTA5IDEuNDYybDIuMTIxLjg2NHMtLjY3MS4xMy0xLjIwOS4yOTRjMCAwIC40MzcuNjM0Ljc3LjkzOC4zOTEtLjE4LjY1Ny0uMjQ4LjY1Ny0uMjQ4LS44MTEgMS42NjgtMi45NzkgMi43MDMtNC41MyAyLjcwMy0uMDkzIDAtLjQ4Mi0uMDA2LS43MjcuMDE1LS40MzUuMDIxLS41ODEuMzgtLjM3NC40NzMuMzczLjIwMSAxLjE0My42NjIuOTU4IDEuMDA5QzUuMiA4LjAwMy45OTkgMTEgLjk5OSAxMWwuNjQ4Ljg4Nkw2LjEyOSA4LjYzQzguNjAyIDYuOTQ4IDEyLjAwNiA2IDE1IDZoM1Y1aC00LjAwMWMtMS4wNTggMC0yLjA0LjEyMi0yLjQ3My0uMDItLjQwMi0uMTMzLS41MDItLjY3OS0uNDU1LTEuMDM1YTcuODcgNy44NyAwIDAgMSAuMTg3LS43MjljLjAyOC0uMDk5LjA0Ni0uMDc3LjE1NS0uMDk5LjU0LS4xMTIuNzc3LS4wOTUuODIxLS4xNi4xNDYtLjI0NS4yNTQtLjk3NC4yNTQtLjk3NEw3LjU2OS4zODlzLjIwMiAxLjAxMy4zNCAxLjA3M3onLz4gPC9kZWZzPiA8dXNlIGZpbGw9JyMwMDdDOTcnIGZpbGwtcnVsZT0nZXZlbm9kZCcgdHJhbnNmb3JtPSd0cmFuc2xhdGUoLTEpJyB4bGluazpocmVmPScjYScvPiA8L3N2Zz4=&quot;); background-repeat: no-repeat; background-position: 0px 0px; border: none; display: inline; visibility: visible; z-index: auto;"></span></div>
+</div>
+
+<div class="form-row js-form-row" style="display: none;">
+<div class="control-group">
+<label class="label--console">
+Endpoint
+</label>
+<input class="input input--console js-connection-endpoint" disabled="disabled" readonly="readonly" value="··············">
+<button class="button button--copy button--copy-input js-copy-input-button" data-clipboard-target=".js-connection-endpoint">
+<i class="fa fa-clipboard"></i>
+</button>
+</div>
+</div>
+
+
+<div class="lab-sidebar__resource lab-sidebar__resource--additional-details l-mtl is-hidden js-cf-connection-output"></div>
+<div class="lab-sidebar__resource lab-sidebar__resource--additional-details l-mtl is-hidden js-additional-connection-info"></div>
+
+</div>
+</div>
+</div>
+</div>
+<div class="lab-sidebar__slider js-sidebar-slider">
+<i class="fa fa-arrow-left"></i>
+<iframe class="l-ie-iframe-fix" kwframeid="1" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/saved_resource.html"></iframe>
+</div>
+
+</div>
+<div class="l-lab-main">
+<div class="l-lab-main-header">
+<header class="lab-header js-lab-header has-shadow">
+<div class="lab-header__section lab-header__section--flex">
+<div class="lab-header__progress js-progress is-hidden">
+<div class="lab-header__progress__bar js-progress-bar" style="width: 100%;"></div>
+</div>
+</div>
+<div class="lab-header__section lab-header__section--no-border">
+<span class="lab-header__progress-message js-progress-message">
+<div class="lab-header__progress-message__indicator js-progress-message-indicator is-complete"></div>
+<span class="js-progress-message-incomplete is-hidden">
+Lab Setting Up
+</span>
+<span class="js-progress-message-complete">
+Lab Running
+</span>
+</span>
+</div>
+<div class="lab-header__section">
+<a class="button button--start button--lab js-start-lab-button is-hidden" data-focus-id="2773" data-lab-access="None" data-lab-instance-id="">
+Start Lab
+</a>
+<a class="button button--wait button--lab js-waiting-lab-button is-hidden">
+<i class="fa fa-spinner fa-pulse"></i>
+</a>
+<a class="button button--end button--lab js-end-lab-button" data-lab-instance-id="6383">
+End Lab
+</a>
+</div>
+<div class="lab-header__section">
+<h3 class="text--sign js-timer" data-duration="172800">1 days</h3>
+</div>
+</header>
+
+</div>
+<div class="l-lab-main-body">
+<div class="lab-content js-lab-content">
+<div class="lab-content__markdown-wrapper">
+<div class="js-markdown-instructions lab-content__markdown markdown-lab-instructions" id="markdown-lab-instructions">
+
+<h1 id="serverless-machine-learning">Serverless Machine Learning</h1>
+
+<h1 id="overview">Overview</h1>
+
+<p><em>Duration is 1 min</em></p>
+
+<p>In this lab, you go from exploring a taxicab dataset to training and deploying a high-accuracy distributed model with Cloud ML Engine.</p>
+
+<h2 id="what-you-need"><strong>What you need</strong></h2>
+
+<h3 id="what-you-ll-need">What you'll need</h3>
+
+<p>To complete this lab, you'll need:</p>
+<ul><li>Access to a standard internet browser (Chrome browser recommended).</li>
+<li>Time. Note the lab's <strong>Duration</strong> in the <strong>Lab Details</strong> tab in Qwiklabs, which is an estimate of the time it should take to complete all steps.  Plan your schedule so you have time to complete the lab. Once you start the lab, you will not be able to pause and return later (you begin at step 1 every time you start a lab). </li>
+<li>You do NOT need a Google Cloud Platform account or project. An account, project and associated resources are provided to you as part of this lab.</li>
+<li>If you already have your own GCP account, make sure you do not use it for this lab. </li>
+<li>If your lab prompts you to log into the console, <strong>use only the student account provided to you by the lab</strong>. This prevents you from incurring charges for lab activities in your personal GCP account.</li>
+</ul>
+<div class="codelabs-infobox codelabs-infobox-warning"><p>Before accessing the Cloud Console, <strong>log out of all other Google / Gmail accounts you may be logged in with.</strong>  If this is not possible, <strong>use a new Incognito window (Chrome)</strong> or another browser for the Qwiklabs session.</p>
+</div>
+
+<h3 id="start-your-lab">Start your lab</h3>
+
+<p>Note the <strong>Setup time</strong> in the <strong>Lab Details</strong> tab in Qwiklabs.  That's how long it will take for the lab account to build its resources. You can track your lab's progress with the status bar at the top of your screen. </p>
+
+<p>When you are ready, click <strong>Start Lab</strong>. </p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/5e5165bd940f65c.png" alt="5e5165bd940f65c.png"></p>
+
+<div class="codelabs-infobox codelabs-infobox-warning"><p><strong>Important</strong>:  What is happening during this time?</p>
+<p>Your lab is spinning up GCP resources for you behind the scenes, including an account, a project, resources within the project, and permission for you to control the resources you will need to run the lab. This means that instead of spending time manually setting up a project and building resources from scratch as part of your lab, you can more quickly begin learning.</p>
+</div>
+
+<h3 id="find-your-lab-s-gcp-username-and-password">Find Your Lab's GCP Username and Password</h3>
+
+<p>To access the resources and console for this lab, locate the Connection Details panel in Qwiklabs.  Here you will find the account ID and password for the account you will use to log in to the Google Cloud Platform:</p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/56aa1b525796849d.png" alt="56aa1b525796849d.png"></p>
+
+<p>If your lab provides other resource identifiers or connection-related information, it will appear on this panel as well.</p>
+
+<h2 id="what-you-learn"><strong>What you learn</strong></h2>
+
+<p>In this series of labs, you go from exploring a taxicab dataset to training and deploying a high-accuracy distributed model with Cloud ML Engine.</p>
+
+<h1 id="setup">Setup</h1>
+
+<h3 id="activate-google-cloud-shell">Activate Google Cloud Shell</h3>
+
+<p>From the GCP Console click the icon (as depicted below) on the top right toolbar:</p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/5f504766b9b3be17.png" alt="5f504766b9b3be17.png"></p>
+
+<p>Then click "Start Cloud Shell" as shown here:</p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/c0726a5e7bd6ec39.png" alt="c0726a5e7bd6ec39.png"></p>
+
+<p>It should only take a few moments to provision and connect to the environment: <img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/718b09ed50d1e90.png" alt="718b09ed50d1e90.png"></p>
+
+<p>This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication.   Much if not all of your work in this lab can be done with simply a browser or your Google Chromebook.</p>
+
+<p>Once connected to the cloud shell, you should see that you are already authenticated and that the project is already set to your <em>PROJECT_ID</em>:</p>
+<pre class="highlight shell"><code>gcloud auth list
+</code><button class="button button--copy js-copy-button-0"><i class="fa fa-clipboard"></i></button></pre>
+<p><strong>Command output</strong></p>
+<pre class="highlight shell"><code>Credentialed accounts:
+ - &lt;myaccount&gt;@&lt;mydomain&gt;.com <span class="o">(</span>active<span class="o">)</span>
+</code><button class="button button--copy js-copy-button-1"><i class="fa fa-clipboard"></i></button></pre>
+<div class="codelabs-infobox codelabs-infobox-special"><p><strong>Note: </strong><code>gcloud</code> is the powerful and unified command-line tool for Google Cloud Platform. Full documentation is available from <a href="https://cloud.google.com/sdk/gcloud/" target="_blank">https://cloud.google.com/sdk/gcloud</a>. It comes pre-installed on CloudShell and you will surely enjoy its support for tab-completion.</p>
+</div>
+<pre class="highlight shell"><code>gcloud config list project
+</code><button class="button button--copy js-copy-button-2"><i class="fa fa-clipboard"></i></button></pre>
+<p><strong>Command output</strong></p>
+<pre class="highlight shell"><code><span class="o">[</span>core]
+project <span class="o">=</span> &lt;PROJECT_ID&gt;
+</code><button class="button button--copy js-copy-button-3"><i class="fa fa-clipboard"></i></button></pre>
+<p>If it is not, you can set it with this command:</p>
+<pre class="highlight shell"><code>gcloud config <span class="nb">set </span>project &lt;PROJECT_ID&gt;
+</code><button class="button button--copy js-copy-button-4"><i class="fa fa-clipboard"></i></button></pre>
+<p><strong>Command output</strong></p>
+<pre class="highlight shell"><code>Updated property <span class="o">[</span>core/project].
+</code><button class="button button--copy js-copy-button-5"><i class="fa fa-clipboard"></i></button></pre>
+<h1 id="launch-cloud-datalab">Launch Cloud Datalab</h1>
+
+<p><em>Duration is 2 min</em></p>
+
+<p>To launch Cloud Datalab:</p>
+
+<h2 id="step-1"><strong>Step 1</strong></h2>
+
+<p>In Cloud Shell, type:</p>
+<pre class="highlight shell"><code>gcloud compute zones list
+</code><button class="button button--copy js-copy-button-6"><i class="fa fa-clipboard"></i></button></pre>
+<p>Pick a zone in a geographically closeby region.</p>
+
+<h2 id="step-2"><strong>Step 2</strong></h2>
+
+<p>In Cloud Shell, type:</p>
+<pre class="highlight shell"><code>datalab create dataengvm --zone &lt;ZONE&gt;
+</code><button class="button button--copy js-copy-button-7"><i class="fa fa-clipboard"></i></button></pre>
+<p>Datalab will take about 5 minutes to start.</p>
+
+<p><input readonly="" class="copyable-inline-input" size="45" type="text" value="Note: follow the prompts during this process."></p>
+
+<p>If you are not yet familiar with Datalab, what follows is a graphical cheat sheet for the main Datalab functionality:</p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/369bf7e045b084ed.png" alt="369bf7e045b084ed.png"></p>
+
+<p><strong>Move on to the next step</strong>.</p>
+
+<h1 id="checkout-notebook-into-cloud-datalab">Checkout notebook into Cloud Datalab</h1>
+
+<p><em>Duration is 5 min</em></p>
+
+<h2 id="step-1-2"><strong>Step 1</strong></h2>
+
+<p>If necessary, wait for Datalab to finish launching. Datalab is ready when you see a message prompting you to do a "Web Preview".</p>
+
+<h2 id="step-2-2"><strong>Step 2</strong></h2>
+
+<p>Click on the <strong>Web Preview</strong> icon on the top-left corner of the Cloud Shell ribbon.  Switch to port <strong>8081</strong>.</p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/7eb159ad9b4d3d2d.png" alt="7eb159ad9b4d3d2d.png"></p>
+
+<p><input readonly="" class="copyable-inline-input" size="242" type="text" value="Note: The connection to your Datalab instance remains open for as long as the datalab command is active. If the cloud shell used for running the datalab command is closed or interrupted, the connection to your Cloud Datalab VM will terminate."></p>
+
+<h2 id="step-3"><strong>Step 3</strong></h2>
+
+<p>In Datalab, click on the icon for <strong>"Open ungit"</strong> in the top-right ribbon.</p>
+
+<h2 id="part-"><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/fc5c4f63c40f83f.png" alt="fc5c4f63c40f83f.png"></h2>
+
+<h2 id="step-4"><strong>Step 4</strong></h2>
+
+<p>In the Ungit window, select the text that reads /content/datalab/notebooks and remove the notebooks so that it reads <strong>/content/datalab</strong>, then hit enter.</p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/70652193140af6f3.png" alt="70652193140af6f3.png"></p>
+
+<p>In the panel that comes up, type the following as the GitHub repository to <strong>Clone from</strong>: </p>
+<pre class="highlight shell"><code>https://github.com/GoogleCloudPlatform/training-data-analyst
+</code><button class="button button--copy js-copy-button-8"><i class="fa fa-clipboard"></i></button></pre>
+<p>Then, click on <strong>Clone repository</strong>.</p>
+
+<h1 id="1-explore-dataset-create-ml-datasets-create-benchmark">1. Explore dataset, create ML datasets, create benchmark</h1>
+
+<p><em>Duration is 15 min</em></p>
+
+<p>In this lab, you will:</p>
+<ul><li>Explore a dataset using BigQuery and Datalab</li>
+<li>Sample the dataset and create training, validation, and testing datasets for local development of TensorFlow models</li>
+<li>Create a benchmark to evaluate the performance of ML against</li>
+</ul>
+<h2 id="step-1-3"><strong>Step 1</strong></h2>
+
+<p>In Cloud Datalab, click on the Home icon, and then navigate to <strong>training-data-analyst/courses/machine_learning/datasets/</strong> and open <strong>create_datasets.ipynb</strong>.</p>
+
+<h2 id="step-2-3"><strong>Step 2</strong></h2>
+
+<p>In Datalab, click on <strong>Clear | All Cells (</strong>click on__ Clear_<em>, then in the drop-down menu, select</em>_ All Cells)__. Now, read the narrative and execute each cell in turn.</p>
+
+<h1 id="2a-getting-started-with-tensorflow">2a. Getting Started with TensorFlow</h1>
+
+<p><em>Duration is 15 min</em></p>
+
+<p>In this lab, you will learn how the TensorFlow Python API works:</p>
+<ul><li>Building a graph</li>
+<li>Running a graph</li>
+<li>Feeding values into a graph</li>
+<li>Find area of a triangle using TensorFlow</li>
+</ul>
+<h2 id="step-1-4"><strong>Step 1</strong></h2>
+
+<p>In Cloud Datalab, click on the Home icon, and then navigate to <strong>training-data-analyst/courses/machine_learning/tensorflow</strong> and open <strong>a_tfstart.ipynb</strong></p>
+
+<h2 id="step-2-4"><strong>Step 2</strong></h2>
+
+<p>In Datalab, click on <strong>Clear | All Cells</strong>. Now read the narrative and execute each cell in turn.</p>
+
+<h1 id="2b-machine-learning-using-tf-learn">2b. Machine Learning using tf.learn</h1>
+
+<p><em>Duration is 15 min</em></p>
+
+<p>In this lab, you will implement a simple machine learning model using tf.learn:</p>
+<ul><li>Read .csv data into a Pandas dataframe</li>
+<li>Implement a Linear Regression model in TensorFlow</li>
+<li>Train the model</li>
+<li>Evaluate the model</li>
+<li>Predict with the model</li>
+<li>Repeat with a Deep Neural Network model in TensorFlow</li>
+</ul>
+<h2 id="step-1-5"><strong>Step 1</strong></h2>
+
+<p>In Cloud Datalab, click on the Home icon, and then navigate to <strong>training-data-analyst/courses/machine_learning/tensorflow</strong> and open <strong>b_tflearn.ipynb</strong></p>
+
+<h2 id="step-2-5"><strong>Step 2</strong></h2>
+
+<p>In Datalab, click on <strong>Clear | All Cells</strong>. Now read the narrative and execute each cell in turn.</p>
+
+<h1 id="2c-tensorflow-on-big-data">2c. TensorFlow on Big Data</h1>
+
+<p><em>Duration is 15 min</em></p>
+
+<p>In this lab, you will learn how to:</p>
+<ul><li>Read from a potentially large file in batches</li>
+<li>Do a wildcard match on filenames</li>
+<li>Break the one-to-one relationship between inputs and features</li>
+</ul>
+<h2 id="step-1-6"><strong>Step 1</strong></h2>
+
+<p>In Cloud Datalab, click on the Home icon, and then navigate to <strong>training-data-analyst/courses/machine_learning/tensorflow</strong> and open__ c_batched.ipynb__.</p>
+
+<h2 id="step-2-6"><strong>Step 2</strong></h2>
+
+<p>In Datalab, click on <strong>Clear | All Cells</strong>. Now read the narrative and execute each cell in turn.</p>
+
+<h1 id="2d-refactor-for-distributed-training-and-monitoring">2d. Refactor for Distributed training and monitoring</h1>
+
+<p><em>Duration is 15 min</em></p>
+
+<p>In this lab, you will learn how to:</p>
+<ul><li>Use the Experiment class</li>
+<li>Monitor training using TensorBoard</li>
+</ul>
+<h2 id="step-1-7"><strong>Step 1</strong></h2>
+
+<p>In Cloud Datalab, click on the Home icon, and then navigate to <strong>training-data-analyst/courses/machine_learning/tensorflow</strong> and open <strong>d_experiment.ipynb</strong>.</p>
+
+<h2 id="step-2-7"><strong>Step 2</strong></h2>
+
+<p>In Datalab, click on <strong>Clear | All Cells</strong>. Now read the narrative and execute each cell in turn.</p>
+
+<h1 id="3-getting-started-with-cloud-ml-engine">3. Getting Started with Cloud ML Engine</h1>
+
+<p><em>Duration is 30 min</em></p>
+
+<p>In this lab, you will learn how to:</p>
+<ul><li>Package up TensorFlow model</li>
+<li>Run training locally</li>
+<li>Run training on cloud</li>
+<li>Deploy model to cloud</li>
+<li>Invoke model to carry out predictions</li>
+</ul>
+<h2 id="step-1-8"><strong>Step 1</strong></h2>
+
+<p>If you don't already have a bucket on Cloud Storage, create one from the  <a href="http://console.cloud.google.com/storage" target="_blank">Storage section of the GCP console</a>. Bucket names have to be globally unique.</p>
+
+<h2 id="step-2-8">Step 2</h2>
+
+<p>In Cloud Datalab, click on the Home icon, and then navigate to <strong>training-data-analyst/courses/machine_learning/cloudmle</strong> and open__ cloudmle.ipynb__.</p>
+
+<h2 id="step-3-2"><strong>Step 3</strong></h2>
+
+<p>In Datalab, click on <strong>Clear | All Cells</strong>. Now read the narrative and execute each cell in turn.</p>
+
+<h1 id="4-feature-engineering">4. Feature Engineering</h1>
+
+<p><em>Duration is 30 min</em></p>
+
+<p>In this lab, you will improve the ML model using feature engineering. In the process, you will learn how to:</p>
+<ul><li>Work with feature columns</li>
+<li>Add feature crosses in TensorFlow</li>
+<li>Read data from BigQuery</li>
+<li>Create datasets using Dataflow</li>
+<li>Use a wide-and-deep model</li>
+</ul>
+<h2 id="step-1-9"><strong>Step 1</strong></h2>
+
+<p>In Cloud Datalab, click on the Home icon, and then navigate to <strong>training-data-analyst/courses/machine_learning/feateng</strong> and open__ feateng.ipynb__.</p>
+
+<h2 id="step-2-9"><strong>Step 2</strong></h2>
+
+<p>In Datalab, click on <strong>Clear | All Cells</strong>. Now read the narrative and execute each cell in turn.</p>
+
+<p>Your instructor will demo notebooks that contain hyper-parameter tuning and training on 500 million rows of data. The changes to the model are minor -- essentially just command-line parameters, but the impact on model accuracy is huge:</p>
+
+<p><img src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/21d055d73bf7974e.png" alt="21d055d73bf7974e.png"></p>
+
+<p>©Google, Inc. or its affiliates. All rights reserved. Do not distribute.</p>
+
+<p><a href="https://docs.google.com/forms/d/11o8tVDrCnJm3v1eKMaIGNH4ODBY_bFpmCYqwm_g3Dm8/viewform" target="_blank">Provide Feedback on this Lab</a></p>
+
+</div>
+
+<div class="lab-content__outline js-lab-content-outline">
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#what-you-need" class=""><strong>What you need</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#what-you-learn" class=""><strong>What you learn</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1" class=""><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2" class=""><strong>Step 2</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-2" class=""><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-2" class=""><strong>Step 2</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-3" class=""><strong>Step 3</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#fc5c4f63c40f83f-png-roitraining-qwiklab-website-prod-s3-amazonaws-com-instructions-documents-963-original-img-fc5c4f63c40f83f-png">![fc5c4f63c40f83f.png](//roitraining-qwiklab-website-prod.s3.amazonaws.com/instructions/documents/963/original/img/fc5c4f63c40f83f.png)</a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-4" class="is-active"><strong>Step 4</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-3"><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-3"><strong>Step 2</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-4"><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-4"><strong>Step 2</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-5"><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-5"><strong>Step 2</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-6"><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-6"><strong>Step 2</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-7"><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-7"><strong>Step 2</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-8"><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-8">Step 2</a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-3-2"><strong>Step 3</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-1-9"><strong>Step 1</strong></a>
+<a href="https://roitraining.qwiklab.com/focuses/2773/materials#step-2-9"><strong>Step 2</strong></a>
+
+</div>
+
+</div>
+</div>
+
+<div class="lab-resource js-lab-resource-area">
+<div class="lab-resource__close js-lab-resource-area-close">
+×
+</div>
+<div class="js-lab-resource"></div>
+</div>
+<div class="lab-resource__background js-lab-resource-background">
+<iframe class="l-ie-iframe-fix" kwframeid="2" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/saved_resource(1).html"></iframe>
+</div>
+
+</div>
+</div>
+<div class="lab-buttons">
+<a class="mdl-button mdl-js-button mdl-button--fab mdl-button--large-fab mdl-js-ripple-effect mdl-button--accent mdl-shadow--8dp help-button" data-target="#lab-help-modal" data-toggle="modal" data-upgraded=",MaterialButton,MaterialRipple">
+<i class="material-icons">help</i>
+<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></a>
+</div>
+</div>
+<div class="modal fade" id="lab-help-modal">
+<div class="modal-container">
+<div class="mdl-shadow--24dp modal-content">
+<h4 class="modal-header">Get Help</h4>
+<form action="https://roitraining.qwiklab.com/contact_support" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="✓"><input type="hidden" name="authenticity_token" value="QagR46dQ6cz/fE7QLGPcaYEmMReiPhvcSK2AVVN0VomR6mjyKUkLtajGoQoNJlMN5Ta4t/fB1e8CeJeXaN7VfQ==">
+<div class="modal-body">
+<div class="control-group l-mbl">
+<label for="Question">Question</label>
+<input type="text" name="question" id="question" placeholder="Briefly describe your question">
+</div>
+<div class="control-group l-mbl">
+<label for="Details">Details</label>
+<textarea name="description" id="description" rows="5" placeholder="Fill in the details here. Please try to be as specific as possible.
+"></textarea>
+</div>
+<div class="control-group l-mbl">
+<label for="Your_Name">Your name</label>
+<input type="text" name="name" id="name" value="marina steinkirch">
+</div>
+<div class="control-group l-mbl">
+<label for="Your_Email">Your email</label>
+<input type="text" name="email" id="email" value="mvonsteinkirch@etsy.com">
+</div>
+<div class="control-group l-mbl">
+<label for="Severity">Severity</label>
+<select name="severity" id="severity"><option value="0">-</option>
+<option value="severity_1">Severity 1 (Highest)</option>
+<option value="severity_2">Severity 2</option>
+<option value="severity_3">Severity 3</option>
+<option value="severity_4">Severity 4</option>
+<option value="severity_5">Severity 5 (Lowest)</option></select>
+</div>
+<div class="control-group">
+<div class="control-label"></div>
+We will get back to you within 24 hours.
+</div>
+</div>
+<div class="modal-actions">
+<a class="mdl-button mdl-button--primary mdl-js-button mdl-js-ripple-effect" data-dismiss="modal" data-upgraded=",MaterialButton,MaterialRipple">
+Cancel
+<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></a>
+<input type="submit" name="commit" value="Submit" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--primary" data-upgraded=",MaterialButton,MaterialRipple"><span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></input>
+</div>
+</form>
+
+
+
+</div>
+</div>
+<iframe class="l-ie-iframe-fix" kwframeid="3" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/saved_resource(2).html"></iframe>
+</div>
+<div class="modal fade" id="lab-details-modal">
+<div class="modal-container">
+<div class="modal-content mdl-shadow--24dp">
+<a class="modal-close" data-dismiss="modal">×</a>
+<h4 class="modal-header">Serverless Machine Learning</h4>
+<div class="modal-body">
+<p class="l-mbm">
+In this series of labs, you go from exploring a taxicab dataset to training and deploying a high-accuracy distributed model with Cloud ML Engine.
+</p>
+<p class="small-label l-mbs">
+<strong>
+Duration:
+</strong>
+4m setup
+·
+2880m access
+·
+2880m completion
+</p>
+<p class="small-label l-mbs">
+
+</p>
+<p class="small-label">
+<span><strong>Levels: <a href="https://roitraining.qwiklab.com/tags/introductory/level">introductory</a></strong></span>
+</p>
+</div>
+<div class="modal-actions">
+<a class="mdl-button mdl-button--primary mdl-js-button mdl-js-ripple-effect" data-dismiss="modal" data-upgraded=",MaterialButton,MaterialRipple">
+Got It
+<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></a>
+</div>
+
+</div>
+</div>
+<iframe class="l-ie-iframe-fix" kwframeid="4" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/saved_resource(3).html"></iframe>
+</div>
+
+<div class="modal fade" id="lab-review-modal">
+<div class="modal-container">
+<div class="modal-content mdl-shadow--24dp">
+<a class="modal-close" data-dismiss="modal">×</a>
+<h4 class="modal-header">Rate Lab</h4>
+<form class="simple_form js-lab-review-form" id="new_lab_review" action="https://roitraining.qwiklab.com/lab_reviews" accept-charset="UTF-8" data-remote="true" method="post"><input name="utf8" type="hidden" value="✓"><div class="modal-body">
+<p class="label">
+How satisfied are you with this lab?
+</p>
+<div class="rateit js-rateit" data-rateit-max="5" data-rateit-min="0" data-rateit-resetable="false" data-rateit-step="1" data-rateit-value="0"><div class="rateit-reset" style="display: none;"></div><div class="rateit-range" style="width: 80px; height: 16px;"><div class="rateit-selected" style="height: 16px; width: 0px;"></div><div class="rateit-hover" style="height:16px"></div></div></div>
+<div class="l-mtm">
+
+<div class="control-group hidden lab_review_user_id"><div class="controls"><input class="hidden" type="hidden" value="942" name="lab_review[user_id]" id="lab_review_user_id"></div></div>
+<div class="control-group hidden lab_review_classroom_id"><div class="controls"><input class="hidden" type="hidden" value="252" name="lab_review[classroom_id]" id="lab_review_classroom_id"></div></div>
+<div class="control-group hidden lab_review_lab_id"><div class="controls"><input class="hidden" type="hidden" value="51" name="lab_review[lab_id]" id="lab_review_lab_id"></div></div>
+<div class="control-group hidden lab_review_focus_id"><div class="controls"><input class="hidden" type="hidden" value="2773" name="lab_review[focus_id]" id="lab_review_focus_id"></div></div>
+<div class="control-group hidden lab_review_rating"><div class="controls"><input class="hidden js-rating-input" type="hidden" name="lab_review[rating]" id="lab_review_rating"></div></div>
+<div class="control-group text optional lab_review_comment"><label class="text optional control-label" for="lab_review_comment">Comment</label><div class="controls"><textarea class="text optional" name="lab_review[comment]" id="lab_review_comment"></textarea></div></div>
+</div>
+</div>
+<div class="modal-actions">
+<a class="mdl-button mdl-button--primary mdl-js-button mdl-js-ripple-effect" data-dismiss="modal" data-upgraded=",MaterialButton,MaterialRipple">
+Cancel
+<span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></a>
+<input type="submit" name="commit" value="Submit" class="btn mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--primary" data-upgraded=",MaterialButton,MaterialRipple"><span class="mdl-button__ripple-container"><span class="mdl-ripple"></span></span></input>
+</div>
+</form>
+
+</div>
+</div>
+<iframe class="l-ie-iframe-fix" kwframeid="5" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/saved_resource(4).html"></iframe>
+</div>
+
+<div class="modal fade" id="lab-access-modal">
+<div class="modal-container">
+<div class="modal-content mdl-shadow--24dp">
+<a class="modal-close" data-dismiss="modal">×</a>
+<h4 class="modal-header">Lab Access</h4>
+<form class="js-lab-access-form" action="https://roitraining.qwiklab.com/lab_onetime_coupons/activate" accept-charset="UTF-8" data-remote="true" method="post"><input name="utf8" type="hidden" value="✓">
+<div class="modal-body">
+<div class="lab-access-modal">
+<input type="hidden" name="id" id="id" value="2773">
+<input type="hidden" name="classroom_id" id="classroom_id" value="252">
+<input type="hidden" name="user_id" id="user_id" value="942">
+<input type="hidden" name="launch_with_credits" id="launch_with_credits" value="0" class="js-launch-with-credits-input">
+<input type="hidden" name="launch_with_subs" id="launch_with_subs" value="0" class="js-launch-with-subscription-input">
+<div class="lab-access-modal__method">
+<p>
+Enter Lab Access Code:
+</p>
+<div class="lab-access-modal__code js-access-code">
+<input type="text" name="uuid_1" id="uuid_1" value="" maxlength="4" placeholder="1234">
+<input type="text" name="uuid_2" id="uuid_2" value="" maxlength="4" placeholder="1234">
+<input type="text" name="uuid_3" id="uuid_3" value="" maxlength="4" placeholder="1234">
+<input type="text" name="uuid_4" id="uuid_4" value="" maxlength="4" placeholder="1234">
+</div>
+<a class="button js-launch-with-access-code-button">
+Launch with Access Code
+</a>
+</div>
+</div>
+</div>
+</form>
+
+
+</div>
+</div>
+<iframe class="l-ie-iframe-fix" kwframeid="6" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/saved_resource(5).html"></iframe>
+</div>
+
+
+
+</div>
+</main>
+<div class="bottom-menu">
+<a class="side-menu__item" href="https://roitraining.qwiklab.com/materials"><div class="side-menu__item__icon">
+<i class="material-icons">view_comfy</i>
+</div>
+<span class="side-menu__item__tooltip">Materials</span>
+<div class="side-menu__item__label">
+Materials
+</div>
+</a>
+<a class="side-menu__item" href="https://roitraining.qwiklab.com/dashboard"><div class="side-menu__item__icon">
+<i class="material-icons">history</i>
+</div>
+<span class="side-menu__item__tooltip">My Learning</span>
+<div class="side-menu__item__label">
+My Learning
+</div>
+</a>
+<a class="side-menu__item" href="https://roitraining.qwiklab.com/my_account/credits"><div class="side-menu__item__icon">
+<i class="material-icons">account_circle</i>
+</div>
+<span class="side-menu__item__tooltip">My Account</span>
+<div class="side-menu__item__label">
+My Account
+</div>
+</a>
+<a class="side-menu__item js-side-menu-button">
+<div class="side-menu__item__icon">
+<i class="material-icons">menu</i>
+</div>
+<span class="side-menu__item__tooltip">More</span>
+<div class="side-menu__item__label">
+More
+</div>
+</a>
+</div>
+
+</div>
+
+
+<div class="modal fade" id="support-modal">
+<div class="modal-container">
+<div class="modal-content mdl-shadow--24dp">
+<a class="modal-close" data-dismiss="modal">×</a>
+<h4 class="modal-header">How can we help you?</h4>
+<p class="l-mbl">
+We will get back to you within 24 hours.
+</p>
+<form action="https://roitraining.qwiklab.com/contact_support" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="✓"><input type="hidden" name="authenticity_token" value="QagR46dQ6cz/fE7QLGPcaYEmMReiPhvcSK2AVVN0VomR6mjyKUkLtajGoQoNJlMN5Ta4t/fB1e8CeJeXaN7VfQ==">
+<div class="form-row">
+<div class="control-group">
+<label for="Question">Question</label>
+<input type="text" name="question" id="question" placeholder="Briefly describe your question">
+</div>
+</div>
+<div class="form-row">
+<div class="control-group">
+<label for="Details">Details</label>
+<textarea name="description" id="description" rows="5" placeholder="Fill in the details here. Please try to be as specific as possible.
+"></textarea>
+</div>
+</div>
+<div class="form-row">
+<div class="control-group">
+<label for="Your_Name">Your name</label>
+<input type="text" name="name" id="name" value="marina steinkirch">
+</div>
+<div class="control-group">
+<label for="Your_Email">Your email</label>
+<input type="text" name="email" id="email" value="mvonsteinkirch@etsy.com">
+</div>
+</div>
+<div class="form-row">
+<div class="control-group">
+<input type="submit" name="commit" value="Submit" class="button">
+</div>
+</div>
+</form>
+
+
+</div>
+</div>
+<iframe class="l-ie-iframe-fix" kwframeid="7" src="./Serverless Machine Learning _ Qwiklabs + roitraining_files/saved_resource(6).html"></iframe>
+</div>
+
+
+<script>
+  $( function() {
+    ql.initMaterialInputs();
+    initChosen();
+    initSearch();
+    initTabs();
+    initTooltips();
+    initLabSidebar();
+  ql.labOutline.init("2773");
+  initLabContent( );
+  initLabResource();
+  initLabReviewModal();
+  initLabAccessModal();
+  initLabTranslations( {"are_you_sure":"All done? If you end this lab, you will lose all your work. You may not be able to restart the lab if there is a quota limit. Are you sure you want to end this lab?\n","in_progress":"*In Progress*","ending":"*Ending*","starting":"*Starting, please wait*","end_concurrent_labs":"Sorry, you can only run one lab at a time. To start this lab, please confirm that you want all of your existing labs to end.\n","copied":"Copied","no_resource":"Error retrieving resource.","no_support":"No Support :(","mac_press":"Press ⌘-C to copy","thanks_review":"Thanks for reviewing this lab.","windows_press":"Press Ctrl-C to copy","days":"days"} );
+  initLabRun();
+  ql.initHeader();
+  ql.sideMenu.init();
+  
+  });
+</script>
+
+
+</body></html>