kafka
21 pagesAccessing topics from an application
This guide shows you how to access Kafka topics from your application. You need an existing topic to access. See Create a Kafka topic for how to create a topic.
Accessing topics from an application outside Nais
This guide will show you how to access a Kafka topic from an application outside Nais clusters. Make the values available to your application.
Avro and schema
This guide will show you how to do various schema operations on your Kafka topics. To register the first version of a schema under the subject "team.test-key" using Avro schema:
Create a Kafka topic
This guide will show you how to create a Kafka topic. The fully qualified topic name is the name of the Topic resource prefixed with your team namespace:
Delete Kafka topic and data
When you want to delete a Kafka topic and it's data, the Topic resource in Nais needs to be annotated.
Environment variables for Kafka
These variables are made available to your application when Kafka is enabled. Aiven has written several articles on how to configure your application. We use SSL, so ignore the SASL-SSL examples:
How to turn on Tiered Storage for your Kafka topic
This guide shows you how to enable tiered storage for your Kafka topic. You need to own an existing topic in nais. Check Create a Kafka topic for how to create a topic.
Kafka
Kafka is a distributed streaming platform that can be used to publish and subscribe to streams of records. It is a good alternative to synchronous communication between services if you need to decouple services.
Kafka metrics
This guide will show you how to monitor your Kafka topics with Grafana. This is a user-generated list of metrics that can be used with Grafana to monitor your Kafka topics.
Kafka Pools
This is a list of available Kafka pools for topics and applications. Topic resources can only be specified in GCP clusters. Applications can access topics from any cluster, including on-premises.
Manage access
This guide will show you how to manage access to your topic. Example of various ACLs:
Nais Postgres example YAML
This is a complete example of a Postgres resource, commonly known as the postgres.yaml file. For an in-depth explanation of each field, head over to the reference documentation.
Nais Postgres reference
This document describes all possible configuration values in the Postgres spec, commonly known as the postgres.yaml file.
Nais Topic example YAML
This is a complete example of an Topic resource, commonly known as the topic.yaml file. For an in-depth explanation of each field, head over to the reference documentation.
Nais Topic reference
This document describes all possible configuration values in the Topic spec, commonly known as the topic.yaml file.
Remove access to topics from an application
This guide will show you how to remove your application's access to a Kafka topic. Remove the ACL that grants your application access to the topic.
Renew credentials for non-Nais applications
Eventually the credentials created in Accessing topics from an application outside Nais will expire. Well in advance of this, Aiven will issue a notification to the technical contacts, and we route…
Tiered storage
Tiered storage is a way to offload rarely used messages to a cheaper, remote storage. Using tiered storage allow storing more data in a more cost-effective way. New messages can be stored locally on…
Troubleshooting
When you get a topic authorization failed error in your application, it means that the application has authenticated correctly with the cluster, but does not have the necessary permissions to access…
Using Kafka Streams with internal topics
This guide will show you how to use Kafka Streams with internal topics. Select a pool from one of the available pools.
Working with Kafka Offsets
The Kafka offset is the index of the last read message for a consumer. On Aiven Kafka, we retain consumer offsets for a period of 7 days. This is the period recommended by Aiven and the default for…