Welcome to ShenZhenJia Knowledge Sharing Community for programmer and developer-Open, Learning and Share
menu search
person
Welcome To Ask or Share your Answers For Others

Categories

I have an outbox postgresql table and debezium connector in kafka connect that creates kafka messages based on the added rows to the table.

The problem I am facing is with the message format. This is the created message value:

{
  "schema": {
    "type": "string",
    "optional": true,
    "name": "io.debezium.data.Json",
    "version": 1
  },
  "payload": "{"foo": "bar"}"
}

But (because of consumer) I need the message to contain only the payload, like this:

{
  ""foo": "bar""
}

This is part of my kafka connector configuration:

"transforms": "outbox",
"transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
"transforms.outbox.route.topic.replacement": "${routedByValue}",
"transforms.outbox.route.by.field": "aggregate_type",
"transforms.outbox.table.field.event.payload.id": "aggregate_id",
"transforms.outbox.table.fields.additional.placement": "payload_type:header:__TypeId__"

Is there any way to achieve this without creating custom transformer?


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
thumb_up_alt 0 like thumb_down_alt 0 dislike
617 views
Welcome To Ask or Share your Answers For Others

1 Answer

It looks like you're using org.apache.kafka.connect.json.JsonConverter with schemas.enable=true for your value converter. When you do this it embeds the schema alongside the payload in the message.

If you set value.converter.schemas.enable=false you should get just the payload in your message.

Ref: Kafka Connect: Converters and Serialization Explained — JSON and Schemas


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
thumb_up_alt 0 like thumb_down_alt 0 dislike
Welcome to ShenZhenJia Knowledge Sharing Community for programmer and developer-Open, Learning and Share
...