Nifi deployment versioning and automation

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Nifi deployment versioning and automation

Maciej Strzelecki

Hello, while it is certainly possible to automate the workflow deployment in nifi using the exposed API it is a non-trivial and lengthy process. 


  • Versioning the workflows (process groups)
  • deployment itself
    • managing staging/production promotion
    • gracefully stopping (and flushing the processors (e.g have one that waits for 16MB chunks before writing it to google storage, how to tell it "write-everything-you-have-now-and-shut-down)
    • starting the new flow and checking for errors
    • removing the old, stopped, flow
    • optional - doing a failover back to the old workflow, if errors spotted 
What is the most mature nifi deployment tool on the market? So far i have just seen very basic scripted solution that are not production and definitely not something i want to use in production and stick my neck up for.


Maciej Strzelecki
Senior Operations Engineer
crealytics GmbH – Semantic Advertising Technology
Brunngasse 1 • 94032 Passau • Germany
Oranienstraße 185 • 10999 Berlin • Germany
1 Bath Street · EC1V 9LB London · United Kingdom

Managing directors: Andreas Reiffen, Dr. Markus Kurch
Register court: Amtsgericht Passau, HRB 7466
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Nifi deployment versioning and automation

Giovanni Lanzani

Also super interested in this one if anyone knows!

 

Giovanni Lanzani
Chief Science Officer GoDataDriven
T: @gglanzani
M: +31 6 5120 6163

 

From: [hidden email]
Sent: Monday, June 19, 2017 12:42 PM
To: [hidden email]
Subject: Nifi deployment versioning and automation

 

Hello, while it is certainly possible to automate the workflow deployment in nifi using the exposed API it is a non-trivial and lengthy process. 


  • Versioning the workflows (process groups)
  • deployment itself
    • managing staging/production promotion
    • gracefully stopping (and flushing the processors (e.g have one that waits for 16MB chunks before writing it to google storage, how to tell it "write-everything-you-have-now-and-shut-down)
    • starting the new flow and checking for errors
    • removing the old, stopped, flow
    • optional - doing a failover back to the old workflow, if errors spotted 
What is the most mature nifi deployment tool on the market? So far i have just seen very basic scripted solution that are not production and definitely not something i want to use in production and stick my neck up for.


Maciej Strzelecki
Senior Operations Engineer
crealytics GmbH – Semantic Advertising Technology
Brunngasse 1 • 94032 Passau • Germany
Oranienstraße 185 • 10999 Berlin • Germany
1 Bath Street · EC1V 9LB London · United Kingdom

Managing directors: Andreas Reiffen, Dr. Markus Kurch
Register court: Amtsgericht Passau, HRB 7466
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Nifi deployment versioning and automation

Bryan Bende
Hello,

I can't speak for other third party tools, but this area has typically
been referred to as the "software development life-cycle" or
"configuration management of flows", and is definitely an area that
the community is planning to work on and improve.

There is a detailed feature proposal [1] that discusses some of the
design concepts, as well as a couple of JIRAs that show some of the
initial ideas [2], and there is a registry sub-project that was
created to eventually encompass this work [3].

Feel free to provide any feedback on the current design through email,
the wiki page, or JIRA.

Thanks,

Bryan

[1] https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
[2] https://issues.apache.org/jira/browse/NIFIREG-3
[3] https://nifi.apache.org/registry.html


On Mon, Jun 19, 2017 at 5:03 PM, Giovanni Lanzani
<[hidden email]> wrote:

> Also super interested in this one if anyone knows!
>
>
>
> Giovanni Lanzani
> Chief Science Officer GoDataDriven
> T: @gglanzani
> M: +31 6 5120 6163
>
>
>
> From: Maciej Strzelecki
> Sent: Monday, June 19, 2017 12:42 PM
> To: [hidden email]
> Subject: Nifi deployment versioning and automation
>
>
>
> Hello, while it is certainly possible to automate the workflow deployment in
> nifi using the exposed API it is a non-trivial and lengthy process.
>
>
> Versioning the workflows (process groups)
> deployment itself
>
> managing staging/production promotion
> gracefully stopping (and flushing the processors (e.g have one that waits
> for 16MB chunks before writing it to google storage, how to tell it
> "write-everything-you-have-now-and-shut-down)
> starting the new flow and checking for errors
> removing the old, stopped, flow
> optional - doing a failover back to the old workflow, if errors spotted
>
> What is the most mature nifi deployment tool on the market? So far i have
> just seen very basic scripted solution that are not production and
> definitely not something i want to use in production and stick my neck up
> for.
>
>
> Maciej Strzelecki
> Senior Operations Engineer
>
> crealytics GmbH – Semantic Advertising Technology
>
> Brunngasse 1 • 94032 Passau • Germany
>
> Oranienstraße 185 • 10999 Berlin • Germany
> 1 Bath Street · EC1V 9LB London · United Kingdom
>
> Managing directors: Andreas Reiffen, Dr. Markus Kurch
> Register court: Amtsgericht Passau, HRB 7466
Loading...