Example Kafka data source
This example is intended for use with the fbsql loader command and provides:
- valid SQL to create a table then insert data
- TOML key/values to save as a configuration file
Before you begin
Step 1 - Create data source
- Open a terminal then run
nano data-kafka-msg.json - Enter the following content then save.
{
"event_id":30,
"name":"name-30",
"qty":4,
"demo":{
"categories":["A", "C"],
"level":24.56
},
"ts":"2023-02-27T12:26:25.091284-06:00"
}
Step 2 - Create TOML configuration file
- Open a terminal then run
nano example-config.toml - Add the following keys and values:
hosts = ["localhost:9092"]
group = "grp"
topics = "events"
table = "loader-target"
batch-size = 300
batch-max-staleness = "5s"
timeout = "5s"
[[fields]]
name = "event_id"
source-type = "id"
primary-key = true
[[fields]]
name = "name"
source-type = "string"
[[fields]]
name = "qty"
source-type = "int"
[[fields]]
name = "categories"
source-type = "stringset"
source-path = ["demo", "categories"]
[[fields]]
name = "level"
source-type = "decimal(2)"
source-path = ["demo", "level"]
[[fields]]
name = "ts"
source-type = "timestamp"
- Save then exit nano.