How to send data stored in Kafka from beginning to a new consumer using NiFi?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to send data stored in Kafka from beginning to a new consumer using NiFi?

mayank rathi

Hello All,

How can I add a new NiFi consumer to a Kafka topic and send data from beginning? I do not see any setting in ConsumerKafkaRecord_0_10 processor to accomplish this.

I am looking for functionality similar to --from-beginning parameter in Kafka Console Consumer.

"./bin/kafka-console-consumer --zookeeper zk1:2181,zk2:2181,zk3:2181/kafka --topic LCL_LANG_DSTRBTN1 --from-beginning"

Thanks!!



--
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to send data stored in Kafka from beginning to a new consumer using NiFi?

Mark Payne
Mayank,

You would just set the "Offset Reset" property to "earliest". Please note, though, that if you have already
used the processor to consume from Kafka, you would need to change your "Group ID" also because if
the Group ID has already been used to consume from Kafka, then the offsets are not reset.

Thanks
-Mark


On Aug 7, 2017, at 10:02 AM, mayank rathi <[hidden email]> wrote:

Hello All,

How can I add a new NiFi consumer to a Kafka topic and send data from beginning? I do not see any setting in ConsumerKafkaRecord_0_10 processor to accomplish this.

I am looking for functionality similar to --from-beginning parameter in Kafka Console Consumer.

"./bin/kafka-console-consumer --zookeeper zk1:2181,zk2:2181,zk3:2181/kafka --topic LCL_LANG_DSTRBTN1 --from-beginning"

Thanks!!



--
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

Loading...