🛌 Commit progress before sleep break

code/kafka/AWS-MSK-commands.md

@@ -0,0 +1,41 @@
+## AWS MSK Commands
+
+### List clusters
+
+Returns a list of all the MSK clusters in the current Region:
+
+```
+aws kafka list-clusters
+```
+
+### Describing the cluster
+
+Returns a description of the MSK cluster whose (ARN) is specified in the request:
+
+```
+aws kafka describe-cluster --cluster-arn <arn>
+```
+
+### Listing nodes
+
+Returns a list of the broker nodes in the cluste:
+
+```
+aws kafka list-nodes --cluster-arn <arn>
+```
+
+### Listing bootstrap brokers urls
+
+A list of brokers that a client application can use to bootstrap:
+
+```
+aws kafka  get-bootstrap-brokers --cluster-arn <arn>
+```
+
+### List clusters operation
+
+Returns a list of all the operations that have been performed on the specified MSK cluster:
+
+```
+aws kafka  list-cluster-operations --cluster-arn <arn>
+```

code/kafka/README.md

@@ -0,0 +1,30 @@
+## resources on kafka and the streaming processing
+
+<br>
+
+## tutorials
+
+* [Apache Guide for Kafka Basic Architecture](https://iteritory.com/beginners-guide-apache-kafka-basic-architecture-components-concepts/)
+* [Confluent Tutorials](https://docs.confluent.io/current/tutorials/index.html)
+* [Kafka streams](https://docs.confluent.io/current/streams/index.html)
+* [Kafka Streams, the Clojure way](https://clojure-conundrums.co.uk/posts/kafka-streams-the-clojure-way/)
+
+<br>
+
+## articles
+
+* [Apache Kafka for Beginners](http://blog.cloudera.com/blog/2014/09/apache-kafka-for-beginners/)
+* [Introduction to Kafka](http://sysadvent.blogspot.com.br/2014/12/day-4-introduction-to-kafka.html)
+* [Introducing Kafka Streams: Stream Processing Made Simple](http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple)
+* [Apache Kafka, Samza, and the Unix Philosophy of Distributed Data](http://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data)
+* [Bottled Water: Real-time integration of PostgreSQL and Kafka](http://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/)
+* [Netflix tech blog posts on Kafka](http://techblog.netflix.com/search/label/kafka)
+* [Linkedin tech blog posts on Kafka](https://engineering.linkedin.com/blog/topic/kafka)
+* [Understanding Kafka with Factorio](https://hackernoon.com/understanding-kafka-with-factorio-74e8fc9bf181)
+* [Event Streams Essentials](https://blog.reactioncommerce.com/event-streams-essentials/)
+
+<br>
+
+## books
+
+* [Making sense of stream processing](http://www.confluent.io/making-sense-of-stream-processing-ebook)

code/kafka/kafka_streams.md

@@ -0,0 +1,5 @@
+## kafka streams
+
+<br>
+
+* kafka streams is a client library for building applications and microservices, where the input and output data are stored in kafka clusters. 

code/kafka/topics-cheatsheet.md

@@ -0,0 +1,52 @@
+## Kafka Topics Cheatsheet
+
+#### Create a temporary interactive container
+
+```
+kubectl exec -it -n <ns> kafka-tools -- bash
+```
+
+### List consumer groups
+
+```
+kafka-consumer-groups --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" --list
+```
+
+### Inspect a consumer group
+
+```
+kafka-consumer-groups --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" --describe --group GROUP
+```
+
+
+### Remove Lag - Reset a consumer group to latest
+
+Reset offsets to latest (speeds up reset in Grafana):
+
+```
+kafka-consumer-groups --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" --group GROUP \
+--reset-offsets --topic TOPIC --to-earliest --dry-run
+```
+
+### Delete a Consumer Group
+
+Reset offsets to latest (speeds up reset in Grafana):
+
+```
+kafka-consumer-groups --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" --group GROUP \
+--reset-offsets --topic TOPIC --to-earliest --dry-run
+```
+
+### Delete group.
+
+```
+kafka-consumer-groups --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" --group GROUP --delete 
+Shift consumer back 10k
+```
+
+### Specify the partition to shift, else all will be shifted
+
+```
+kafka-consumer-groups --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" --group GROUP \
+--topic TOPIC:0 --reset-offsets --shift-by -10000
+```