integration with Zeppelin

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

integration with Zeppelin

Wojciech Indyk
Hi All!
I am new here. I find NiFi as a great project for data routing. However, extendability of NiFi give us a great potential to expand application of NiFi routing. Recently I looked at Kylo- a software for management of data lake (based on NiFi and its templates). In this composition of NiFi and Kylo I feel lack of a component able to do custom data-analytics. I asked the Kylo community on possible integration of Kylo with Zeppelin here: https://groups.google.com/forum/#!topic/kylo-community/e6JdzneAnV0

The Kylo developers advised me to asked about possible integration here. I haven't found any NiFi processor for Zeppelin, that enables to work on the input from NiFi and handle the output from Zeppelin. The idea of such integration comes from functionality I've seen in Dataiku solution. There is a processor, that is able to run arbitrary code in the middle of processing by specific input and output variables. Having such opportunity the data governance in area of collecting data done by NiFi can be extended on data governance regarding data processing, like training of Machine Learning models, custom visualizations, etc.

What do you think of this idea? Is any NiFi processor, that supports such integration? Is it in line with a NiFi roadmap?

--
Kind regards/ Pozdrawiam,
Wojciech Indyk
Reply | Threaded
Open this post in threaded view
|

Re: integration with Zeppelin

James Wing
I do not believe NiFi has any specific features for Zeppelin yet, but it is possible to write custom Zeppelin code paragraphs that communicate with the NiFi API to pull data or inspect flow status.  For an example, I recommend Pierre Villard's US Presidential Election: tweet analysis using HDF/NiFi, Spark, Hive and Zeppelin (https://community.hortonworks.com/articles/30213/us-presidential-election-tweet-analysis-using-hdfn.html).

The most typical usage is to have NiFi write to a durable store like HDFS, HBase, SQL, etc., and then have Zeppelin read from the data store.  Would that not work for you?

Thanks,

James

On Thu, Jun 8, 2017 at 7:09 AM, Wojciech Indyk <[hidden email]> wrote:
Hi All!
I am new here. I find NiFi as a great project for data routing. However, extendability of NiFi give us a great potential to expand application of NiFi routing. Recently I looked at Kylo- a software for management of data lake (based on NiFi and its templates). In this composition of NiFi and Kylo I feel lack of a component able to do custom data-analytics. I asked the Kylo community on possible integration of Kylo with Zeppelin here: https://groups.google.com/forum/#!topic/kylo-community/e6JdzneAnV0

The Kylo developers advised me to asked about possible integration here. I haven't found any NiFi processor for Zeppelin, that enables to work on the input from NiFi and handle the output from Zeppelin. The idea of such integration comes from functionality I've seen in Dataiku solution. There is a processor, that is able to run arbitrary code in the middle of processing by specific input and output variables. Having such opportunity the data governance in area of collecting data done by NiFi can be extended on data governance regarding data processing, like training of Machine Learning models, custom visualizations, etc.

What do you think of this idea? Is any NiFi processor, that supports such integration? Is it in line with a NiFi roadmap?

--
Kind regards/ Pozdrawiam,
Wojciech Indyk