Parameterizing the nifi flow

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Parameterizing the nifi flow

More, Vikram (CONT)

Hi,

 

I have a nifi flow which pulls/extracts from source database table and loads into target database table. This flow will run several times in a day to get delta records from source table (more like batch process running every 3-4 hrs). Now I need to replicate this same process for 100+ different source tables. So rather than creating 100+ nifi flows for each separate table, can I create main flow (let's say template) and pass parameter like source extract sql, target load sql to main flow. And repeat these steps for each source table . Has anyone tried parameterizing the nifi flows, can you please advice . We are using NiFi 1.1.0

 

Appreciate any thoughts here.

 

 

Thanks & Regards,

Vikram

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameterizing the nifi flow

Andrew Grande-2

Hi,

Read up on the variable registry in the docs, that sounds like a good fit. I don't remember if it were available in 1.1 though.

Andrew


On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) <[hidden email]> wrote:

Hi,

 

I have a nifi flow which pulls/extracts from source database table and loads into target database table. This flow will run several times in a day to get delta records from source table (more like batch process running every 3-4 hrs). Now I need to replicate this same process for 100+ different source tables. So rather than creating 100+ nifi flows for each separate table, can I create main flow (let's say template) and pass parameter like source extract sql, target load sql to main flow. And repeat these steps for each source table . Has anyone tried parameterizing the nifi flows, can you please advice . We are using NiFi 1.1.0

 

Appreciate any thoughts here.

 

 

Thanks & Regards,

Vikram

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameterizing the nifi flow

Andy LoPresto-2
The variable registry is a great tool for parameterizing values that differ between environments/deployments. In this case it sounds like setting up a flow that read from a flowfile attribute/content to determine the source table’s name is a better fit. You can create a master list of all source tables and store it as a plaintext file, then on your schedule, read the contents of that file, split the content by line, and send each 100+ resulting flowfiles into the part of the flow that reads from the database. Each would provide the name of the source table to read at that time. 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 11, 2017, at 3:26 PM, Andrew Grande <[hidden email]> wrote:

Hi,

Read up on the variable registry in the docs, that sounds like a good fit. I don't remember if it were available in 1.1 though.

Andrew


On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) <[hidden email]> wrote:
Hi,
 
I have a nifi flow which pulls/extracts from source database table and loads into target database table. This flow will run several times in a day to get delta records from source table (more like batch process running every 3-4 hrs). Now I need to replicate this same process for 100+ different source tables. So rather than creating 100+ nifi flows for each separate table, can I create main flow (let's say template) and pass parameter like source extract sql, target load sql to main flow. And repeat these steps for each source table . Has anyone tried parameterizing the nifi flows, can you please advice . We are using NiFi 1.1.0
 
Appreciate any thoughts here.

 

 

Thanks & Regards,

Vikram

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Parameterizing the nifi flow

Carlos Manuel Fernandes (DSI)

Hi Vikram,

 

I had the same requirements like you , and my solution is a service:

 

HandleHttpRequest (prepared to handle http://host:8085/ods?sourceTable=tableName&targetTable=tableName&truncate=Y)  ->

ExecuteScript (Script reads the http parameters and/or read  extra parameters from a database table and make  the  synchronization) ->

HandleHttpResponse (return 200 if ok )

 

After this service are ready, you can create a scheduled invoker , like this:

 

ExecuteScript (get all the tables names you need to synchronize, in my case based on a query : select source_table,target_Table,,truncate from ods_tables ) ->

invokeHttp (http://host:8085/ods?sourceTable=${source_table}&targetTable=${target_table}&truncate=${truncate}) ->

putEmail (sending the result of the synchronization).

 

 

With this you don’t need to repeat flows, I hope this help.

 

Carlos Fernandes

 

 

 

 

From: Andy LoPresto [mailto:[hidden email]]
Sent: sábado, 12 de agosto de 2017 01:36
To: [hidden email]
Subject: Re: Parameterizing the nifi flow

 

The variable registry is a great tool for parameterizing values that differ between environments/deployments. In this case it sounds like setting up a flow that read from a flowfile attribute/content to determine the source table’s name is a better fit. You can create a master list of all source tables and store it as a plaintext file, then on your schedule, read the contents of that file, split the content by line, and send each 100+ resulting flowfiles into the part of the flow that reads from the database. Each would provide the name of the source table to read at that time. 

 

Andy LoPresto

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 

On Aug 11, 2017, at 3:26 PM, Andrew Grande <[hidden email]> wrote:

 

Hi,

Read up on the variable registry in the docs, that sounds like a good fit. I don't remember if it were available in 1.1 though.

Andrew

 

On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) <[hidden email]> wrote:

Hi,

 

I have a nifi flow which pulls/extracts from source database table and loads into target database table. This flow will run several times in a day to get delta records from source table (more like batch process running every 3-4 hrs). Now I need to replicate this same process for 100+ different source tables. So rather than creating 100+ nifi flows for each separate table, can I create main flow (let's say template) and pass parameter like source extract sql, target load sql to main flow. And repeat these steps for each source table . Has anyone tried parameterizing the nifi flows, can you please advice . We are using NiFi 1.1.0

 

Appreciate any thoughts here.

 

 

Thanks & Regards,

Vikram

 

 


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameterizing the nifi flow

dan young
We've done something similar as Carlos outlines here.  Works really great

Dano

On Sun, Aug 13, 2017, 9:39 AM Carlos Manuel Fernandes (DSI) <[hidden email]> wrote:

Hi Vikram,

 

I had the same requirements like you , and my solution is a service:

 

HandleHttpRequest (prepared to handle http://host:8085/ods?sourceTable=tableName&targetTable=tableName&truncate=Y)  ->

ExecuteScript (Script reads the http parameters and/or read  extra parameters from a database table and make  the  synchronization) ->

HandleHttpResponse (return 200 if ok )

 

After this service are ready, you can create a scheduled invoker , like this:

 

ExecuteScript (get all the tables names you need to synchronize, in my case based on a query : select source_table,target_Table,,truncate from ods_tables ) ->

invokeHttp (http://host:8085/ods?sourceTable=${source_table}&targetTable=${target_table}&truncate=${truncate}) ->

putEmail (sending the result of the synchronization).

 

 

With this you don’t need to repeat flows, I hope this help.

 

Carlos Fernandes

 

 

 

 

From: Andy LoPresto [mailto:[hidden email]]
Sent: sábado, 12 de agosto de 2017 01:36
To: [hidden email]
Subject: Re: Parameterizing the nifi flow

 

The variable registry is a great tool for parameterizing values that differ between environments/deployments. In this case it sounds like setting up a flow that read from a flowfile attribute/content to determine the source table’s name is a better fit. You can create a master list of all source tables and store it as a plaintext file, then on your schedule, read the contents of that file, split the content by line, and send each 100+ resulting flowfiles into the part of the flow that reads from the database. Each would provide the name of the source table to read at that time. 

 

Andy LoPresto

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 

On Aug 11, 2017, at 3:26 PM, Andrew Grande <[hidden email]> wrote:

 

Hi,

Read up on the variable registry in the docs, that sounds like a good fit. I don't remember if it were available in 1.1 though.

Andrew

 

On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) <[hidden email]> wrote:

Hi,

 

I have a nifi flow which pulls/extracts from source database table and loads into target database table. This flow will run several times in a day to get delta records from source table (more like batch process running every 3-4 hrs). Now I need to replicate this same process for 100+ different source tables. So rather than creating 100+ nifi flows for each separate table, can I create main flow (let's say template) and pass parameter like source extract sql, target load sql to main flow. And repeat these steps for each source table . Has anyone tried parameterizing the nifi flows, can you please advice . We are using NiFi 1.1.0

 

Appreciate any thoughts here.

 

 

Thanks & Regards,

Vikram

 

 


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameterizing the nifi flow

milind parikh
How do you guys handle schedule (one source every 4 hrs, other source every 24 hrs,...) as a parameter?

Thanks
Milind

On Aug 13, 2017 11:32 AM, "dan young" <[hidden email]> wrote:
We've done something similar as Carlos outlines here.  Works really great

Dano

On Sun, Aug 13, 2017, 9:39 AM Carlos Manuel Fernandes (DSI) <[hidden email]> wrote:

Hi Vikram,

 

I had the same requirements like you , and my solution is a service:

 

HandleHttpRequest (prepared to handle http://host:8085/ods?sourceTable=tableName&targetTable=tableName&truncate=Y)  ->

ExecuteScript (Script reads the http parameters and/or read  extra parameters from a database table and make  the  synchronization) ->

HandleHttpResponse (return 200 if ok )

 

After this service are ready, you can create a scheduled invoker , like this:

 

ExecuteScript (get all the tables names you need to synchronize, in my case based on a query : select source_table,target_Table,,truncate from ods_tables ) ->

invokeHttp (http://host:8085/ods?sourceTable=${source_table}&targetTable=${target_table}&truncate=${truncate}) ->

putEmail (sending the result of the synchronization).

 

 

With this you don’t need to repeat flows, I hope this help.

 

Carlos Fernandes

 

 

 

 

From: Andy LoPresto [mailto:[hidden email]]
Sent: sábado, 12 de agosto de 2017 01:36
To: [hidden email]
Subject: Re: Parameterizing the nifi flow

 

The variable registry is a great tool for parameterizing values that differ between environments/deployments. In this case it sounds like setting up a flow that read from a flowfile attribute/content to determine the source table’s name is a better fit. You can create a master list of all source tables and store it as a plaintext file, then on your schedule, read the contents of that file, split the content by line, and send each 100+ resulting flowfiles into the part of the flow that reads from the database. Each would provide the name of the source table to read at that time. 

 

Andy LoPresto

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 

On Aug 11, 2017, at 3:26 PM, Andrew Grande <[hidden email]> wrote:

 

Hi,

Read up on the variable registry in the docs, that sounds like a good fit. I don't remember if it were available in 1.1 though.

Andrew

 

On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) <[hidden email]> wrote:

Hi,

 

I have a nifi flow which pulls/extracts from source database table and loads into target database table. This flow will run several times in a day to get delta records from source table (more like batch process running every 3-4 hrs). Now I need to replicate this same process for 100+ different source tables. So rather than creating 100+ nifi flows for each separate table, can I create main flow (let's say template) and pass parameter like source extract sql, target load sql to main flow. And repeat these steps for each source table . Has anyone tried parameterizing the nifi flows, can you please advice . We are using NiFi 1.1.0

 

Appreciate any thoughts here.

 

 

Thanks & Regards,

Vikram

 

 


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

 

Loading...