how to execute code when processor is stopping

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

how to execute code when processor is stopping

尹文才
Hi guys, I'm trying to execute some code in my processor when the processor
is asked to stop in the NIFI UI by the user, I checked the developer guide
and only find OnUnscheduled will be called when the processor is no long
scheduled to run. I've tested this OnUnscheduled, it will also be called
after timed out executing OnScheduled task. So is there a way to execute
some code only when the processor is stopping?

Regards,
Ben
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

Pierre Villard
Hi Ben,

You might want to have a look here:
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#component-lifecycle

Pierre

2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:

> Hi guys, I'm trying to execute some code in my processor when the processor
> is asked to stop in the NIFI UI by the user, I checked the developer guide
> and only find OnUnscheduled will be called when the processor is no long
> scheduled to run. I've tested this OnUnscheduled, it will also be called
> after timed out executing OnScheduled task. So is there a way to execute
> some code only when the processor is stopping?
>
> Regards,
> Ben
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

尹文才
Hi Pierre, I've checked the developer guide before I sent the email and
according to the developer guide, the method annotated with OnUnScheduled
will be called in 2 cases according to my understanding, please correct me
if I'm wrong:
1. when user tries to stop the processor in the NIFI UI, thus the processor
is no longer scheduled to run in this case, and the method will be called.
2. when method annotated with OnScheduled throws exceptions, for example
time out execution exception, the OnUnScheduled method will also be called.

My question is how to tell the first scenario from the second one? Thanks.

Regards,
Ben

2017-08-11 15:51 GMT+08:00 Pierre Villard <[hidden email]>:

> Hi Ben,
>
> You might want to have a look here:
> https://nifi.apache.org/docs/nifi-docs/html/developer-
> guide.html#component-lifecycle
>
> Pierre
>
> 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
>
> > Hi guys, I'm trying to execute some code in my processor when the
> processor
> > is asked to stop in the NIFI UI by the user, I checked the developer
> guide
> > and only find OnUnscheduled will be called when the processor is no long
> > scheduled to run. I've tested this OnUnscheduled, it will also be called
> > after timed out executing OnScheduled task. So is there a way to execute
> > some code only when the processor is stopping?
> >
> > Regards,
> > Ben
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

Pierre Villard
Oh OK, get it now!

Not sure what's your use case, but I don't think you can do that unless you
set some information when the process actually executes onTrigger for the
first time and you then check this value in your OnUnscheduled annotated
method.

Pierre

2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:

> Hi Pierre, I've checked the developer guide before I sent the email and
> according to the developer guide, the method annotated with OnUnScheduled
> will be called in 2 cases according to my understanding, please correct me
> if I'm wrong:
> 1. when user tries to stop the processor in the NIFI UI, thus the processor
> is no longer scheduled to run in this case, and the method will be called.
> 2. when method annotated with OnScheduled throws exceptions, for example
> time out execution exception, the OnUnScheduled method will also be called.
>
> My question is how to tell the first scenario from the second one? Thanks.
>
> Regards,
> Ben
>
> 2017-08-11 15:51 GMT+08:00 Pierre Villard <[hidden email]>:
>
> > Hi Ben,
> >
> > You might want to have a look here:
> > https://nifi.apache.org/docs/nifi-docs/html/developer-
> > guide.html#component-lifecycle
> >
> > Pierre
> >
> > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
> >
> > > Hi guys, I'm trying to execute some code in my processor when the
> > processor
> > > is asked to stop in the NIFI UI by the user, I checked the developer
> > guide
> > > and only find OnUnscheduled will be called when the processor is no
> long
> > > scheduled to run. I've tested this OnUnscheduled, it will also be
> called
> > > after timed out executing OnScheduled task. So is there a way to
> execute
> > > some code only when the processor is stopping?
> > >
> > > Regards,
> > > Ben
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

尹文才
thanks Pierre, my case is that I need to implement a database connection
retry logic inside my OnScheduled method, when the database is not
available I will retry until the connection is back online.
The problem is when the database is offline it will throw timed out
execution exception inside OnScheduled and then call OnUnscheduled. But
when I manually stop the processor the OnUnsheduled
will also get called. I know my logic sounds a little weird but I need to
set some flag in the OnUnscheduled method to stop the retry logic inside
OnScheduled in order to be able to stop the processor,
otherwise the processor is not able to be stopped unless I restart the
whole NIFI.

Regards,
Ben

2017-08-11 17:18 GMT+08:00 Pierre Villard <[hidden email]>:

> Oh OK, get it now!
>
> Not sure what's your use case, but I don't think you can do that unless you
> set some information when the process actually executes onTrigger for the
> first time and you then check this value in your OnUnscheduled annotated
> method.
>
> Pierre
>
> 2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:
>
> > Hi Pierre, I've checked the developer guide before I sent the email and
> > according to the developer guide, the method annotated with OnUnScheduled
> > will be called in 2 cases according to my understanding, please correct
> me
> > if I'm wrong:
> > 1. when user tries to stop the processor in the NIFI UI, thus the
> processor
> > is no longer scheduled to run in this case, and the method will be
> called.
> > 2. when method annotated with OnScheduled throws exceptions, for example
> > time out execution exception, the OnUnScheduled method will also be
> called.
> >
> > My question is how to tell the first scenario from the second one?
> Thanks.
> >
> > Regards,
> > Ben
> >
> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <[hidden email]>:
> >
> > > Hi Ben,
> > >
> > > You might want to have a look here:
> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
> > > guide.html#component-lifecycle
> > >
> > > Pierre
> > >
> > > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
> > >
> > > > Hi guys, I'm trying to execute some code in my processor when the
> > > processor
> > > > is asked to stop in the NIFI UI by the user, I checked the developer
> > > guide
> > > > and only find OnUnscheduled will be called when the processor is no
> > long
> > > > scheduled to run. I've tested this OnUnscheduled, it will also be
> > called
> > > > after timed out executing OnScheduled task. So is there a way to
> > execute
> > > > some code only when the processor is stopping?
> > > >
> > > > Regards,
> > > > Ben
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

Bryan Bende
Ben,

I apologize if I am not understanding the situation, but...

In the case where your OnScheduled code is in a retry loop, if someone
stops the processor it will call your OnUnscheduled code which will
set the flag to bounce out of the loop. This sounds like what you
want, right?

In the case where OnScheduled times out, the framework is calling
OnUnscheduled which would call your code to set the flag, but wouldn't
that not matter at this point because you aren't looping anymore
anyway?

If the framework calls OnScheduled again, your code should set the
flag back to whatever it needs to be to start looping again right?

An alternative that might avoid some of this would be to lazily
initialize the connection in the onTrigger method of the processor.

-Bryan


On Fri, Aug 11, 2017 at 9:16 AM, 尹文才 <[hidden email]> wrote:

> thanks Pierre, my case is that I need to implement a database connection
> retry logic inside my OnScheduled method, when the database is not
> available I will retry until the connection is back online.
> The problem is when the database is offline it will throw timed out
> execution exception inside OnScheduled and then call OnUnscheduled. But
> when I manually stop the processor the OnUnsheduled
> will also get called. I know my logic sounds a little weird but I need to
> set some flag in the OnUnscheduled method to stop the retry logic inside
> OnScheduled in order to be able to stop the processor,
> otherwise the processor is not able to be stopped unless I restart the
> whole NIFI.
>
> Regards,
> Ben
>
> 2017-08-11 17:18 GMT+08:00 Pierre Villard <[hidden email]>:
>
>> Oh OK, get it now!
>>
>> Not sure what's your use case, but I don't think you can do that unless you
>> set some information when the process actually executes onTrigger for the
>> first time and you then check this value in your OnUnscheduled annotated
>> method.
>>
>> Pierre
>>
>> 2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:
>>
>> > Hi Pierre, I've checked the developer guide before I sent the email and
>> > according to the developer guide, the method annotated with OnUnScheduled
>> > will be called in 2 cases according to my understanding, please correct
>> me
>> > if I'm wrong:
>> > 1. when user tries to stop the processor in the NIFI UI, thus the
>> processor
>> > is no longer scheduled to run in this case, and the method will be
>> called.
>> > 2. when method annotated with OnScheduled throws exceptions, for example
>> > time out execution exception, the OnUnScheduled method will also be
>> called.
>> >
>> > My question is how to tell the first scenario from the second one?
>> Thanks.
>> >
>> > Regards,
>> > Ben
>> >
>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <[hidden email]>:
>> >
>> > > Hi Ben,
>> > >
>> > > You might want to have a look here:
>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
>> > > guide.html#component-lifecycle
>> > >
>> > > Pierre
>> > >
>> > > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
>> > >
>> > > > Hi guys, I'm trying to execute some code in my processor when the
>> > > processor
>> > > > is asked to stop in the NIFI UI by the user, I checked the developer
>> > > guide
>> > > > and only find OnUnscheduled will be called when the processor is no
>> > long
>> > > > scheduled to run. I've tested this OnUnscheduled, it will also be
>> > called
>> > > > after timed out executing OnScheduled task. So is there a way to
>> > execute
>> > > > some code only when the processor is stopping?
>> > > >
>> > > > Regards,
>> > > > Ben
>> > > >
>> > >
>> >
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

Matt Burgess-2
I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
(such as database connection retries), I recommend putting the
connection logic in the onTrigger() method. If you can check
connectivity, then your onTrigger() would know whether it needs to try
to reconnect before it does any work. If it tries to reconnect and is
unsuccessful, you can yield the processor if you want, so as not to
hammer the DB with connection attempts. The CaptureChangeMySQL
processor does this, it has a retry loop for trying various nodes in a
MySQL cluster, but once it's connected, it goes on about its work, and
if a connection fails, it will retry the connection loop before it
does any more work. It only uses onTrigger and none of the scheduling
stuff.

Regards,
Matt

On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende <[hidden email]> wrote:

> Ben,
>
> I apologize if I am not understanding the situation, but...
>
> In the case where your OnScheduled code is in a retry loop, if someone
> stops the processor it will call your OnUnscheduled code which will
> set the flag to bounce out of the loop. This sounds like what you
> want, right?
>
> In the case where OnScheduled times out, the framework is calling
> OnUnscheduled which would call your code to set the flag, but wouldn't
> that not matter at this point because you aren't looping anymore
> anyway?
>
> If the framework calls OnScheduled again, your code should set the
> flag back to whatever it needs to be to start looping again right?
>
> An alternative that might avoid some of this would be to lazily
> initialize the connection in the onTrigger method of the processor.
>
> -Bryan
>
>
> On Fri, Aug 11, 2017 at 9:16 AM, 尹文才 <[hidden email]> wrote:
>> thanks Pierre, my case is that I need to implement a database connection
>> retry logic inside my OnScheduled method, when the database is not
>> available I will retry until the connection is back online.
>> The problem is when the database is offline it will throw timed out
>> execution exception inside OnScheduled and then call OnUnscheduled. But
>> when I manually stop the processor the OnUnsheduled
>> will also get called. I know my logic sounds a little weird but I need to
>> set some flag in the OnUnscheduled method to stop the retry logic inside
>> OnScheduled in order to be able to stop the processor,
>> otherwise the processor is not able to be stopped unless I restart the
>> whole NIFI.
>>
>> Regards,
>> Ben
>>
>> 2017-08-11 17:18 GMT+08:00 Pierre Villard <[hidden email]>:
>>
>>> Oh OK, get it now!
>>>
>>> Not sure what's your use case, but I don't think you can do that unless you
>>> set some information when the process actually executes onTrigger for the
>>> first time and you then check this value in your OnUnscheduled annotated
>>> method.
>>>
>>> Pierre
>>>
>>> 2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:
>>>
>>> > Hi Pierre, I've checked the developer guide before I sent the email and
>>> > according to the developer guide, the method annotated with OnUnScheduled
>>> > will be called in 2 cases according to my understanding, please correct
>>> me
>>> > if I'm wrong:
>>> > 1. when user tries to stop the processor in the NIFI UI, thus the
>>> processor
>>> > is no longer scheduled to run in this case, and the method will be
>>> called.
>>> > 2. when method annotated with OnScheduled throws exceptions, for example
>>> > time out execution exception, the OnUnScheduled method will also be
>>> called.
>>> >
>>> > My question is how to tell the first scenario from the second one?
>>> Thanks.
>>> >
>>> > Regards,
>>> > Ben
>>> >
>>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <[hidden email]>:
>>> >
>>> > > Hi Ben,
>>> > >
>>> > > You might want to have a look here:
>>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
>>> > > guide.html#component-lifecycle
>>> > >
>>> > > Pierre
>>> > >
>>> > > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
>>> > >
>>> > > > Hi guys, I'm trying to execute some code in my processor when the
>>> > > processor
>>> > > > is asked to stop in the NIFI UI by the user, I checked the developer
>>> > > guide
>>> > > > and only find OnUnscheduled will be called when the processor is no
>>> > long
>>> > > > scheduled to run. I've tested this OnUnscheduled, it will also be
>>> > called
>>> > > > after timed out executing OnScheduled task. So is there a way to
>>> > execute
>>> > > > some code only when the processor is stopping?
>>> > > >
>>> > > > Regards,
>>> > > > Ben
>>> > > >
>>> > >
>>> >
>>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

尹文才
Hi Bryan and Matt, thanks for all your suggestions, I was trying to make
sure that the OnUnscheduled method was not called too frequently when the
connection is offline.
You guys were right, these sort of logic should not be placed inside the
scheduling methods, I need to refactor my code to place them into onTrigger.

Regards,
Ben

2017-08-12 0:53 GMT+08:00 Matt Burgess <[hidden email]>:

> I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
> (such as database connection retries), I recommend putting the
> connection logic in the onTrigger() method. If you can check
> connectivity, then your onTrigger() would know whether it needs to try
> to reconnect before it does any work. If it tries to reconnect and is
> unsuccessful, you can yield the processor if you want, so as not to
> hammer the DB with connection attempts. The CaptureChangeMySQL
> processor does this, it has a retry loop for trying various nodes in a
> MySQL cluster, but once it's connected, it goes on about its work, and
> if a connection fails, it will retry the connection loop before it
> does any more work. It only uses onTrigger and none of the scheduling
> stuff.
>
> Regards,
> Matt
>
> On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende <[hidden email]> wrote:
> > Ben,
> >
> > I apologize if I am not understanding the situation, but...
> >
> > In the case where your OnScheduled code is in a retry loop, if someone
> > stops the processor it will call your OnUnscheduled code which will
> > set the flag to bounce out of the loop. This sounds like what you
> > want, right?
> >
> > In the case where OnScheduled times out, the framework is calling
> > OnUnscheduled which would call your code to set the flag, but wouldn't
> > that not matter at this point because you aren't looping anymore
> > anyway?
> >
> > If the framework calls OnScheduled again, your code should set the
> > flag back to whatever it needs to be to start looping again right?
> >
> > An alternative that might avoid some of this would be to lazily
> > initialize the connection in the onTrigger method of the processor.
> >
> > -Bryan
> >
> >
> > On Fri, Aug 11, 2017 at 9:16 AM, 尹文才 <[hidden email]> wrote:
> >> thanks Pierre, my case is that I need to implement a database connection
> >> retry logic inside my OnScheduled method, when the database is not
> >> available I will retry until the connection is back online.
> >> The problem is when the database is offline it will throw timed out
> >> execution exception inside OnScheduled and then call OnUnscheduled. But
> >> when I manually stop the processor the OnUnsheduled
> >> will also get called. I know my logic sounds a little weird but I need
> to
> >> set some flag in the OnUnscheduled method to stop the retry logic inside
> >> OnScheduled in order to be able to stop the processor,
> >> otherwise the processor is not able to be stopped unless I restart the
> >> whole NIFI.
> >>
> >> Regards,
> >> Ben
> >>
> >> 2017-08-11 17:18 GMT+08:00 Pierre Villard <[hidden email]
> >:
> >>
> >>> Oh OK, get it now!
> >>>
> >>> Not sure what's your use case, but I don't think you can do that
> unless you
> >>> set some information when the process actually executes onTrigger for
> the
> >>> first time and you then check this value in your OnUnscheduled
> annotated
> >>> method.
> >>>
> >>> Pierre
> >>>
> >>> 2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:
> >>>
> >>> > Hi Pierre, I've checked the developer guide before I sent the email
> and
> >>> > according to the developer guide, the method annotated with
> OnUnScheduled
> >>> > will be called in 2 cases according to my understanding, please
> correct
> >>> me
> >>> > if I'm wrong:
> >>> > 1. when user tries to stop the processor in the NIFI UI, thus the
> >>> processor
> >>> > is no longer scheduled to run in this case, and the method will be
> >>> called.
> >>> > 2. when method annotated with OnScheduled throws exceptions, for
> example
> >>> > time out execution exception, the OnUnScheduled method will also be
> >>> called.
> >>> >
> >>> > My question is how to tell the first scenario from the second one?
> >>> Thanks.
> >>> >
> >>> > Regards,
> >>> > Ben
> >>> >
> >>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <
> [hidden email]>:
> >>> >
> >>> > > Hi Ben,
> >>> > >
> >>> > > You might want to have a look here:
> >>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
> >>> > > guide.html#component-lifecycle
> >>> > >
> >>> > > Pierre
> >>> > >
> >>> > > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
> >>> > >
> >>> > > > Hi guys, I'm trying to execute some code in my processor when the
> >>> > > processor
> >>> > > > is asked to stop in the NIFI UI by the user, I checked the
> developer
> >>> > > guide
> >>> > > > and only find OnUnscheduled will be called when the processor is
> no
> >>> > long
> >>> > > > scheduled to run. I've tested this OnUnscheduled, it will also be
> >>> > called
> >>> > > > after timed out executing OnScheduled task. So is there a way to
> >>> > execute
> >>> > > > some code only when the processor is stopping?
> >>> > > >
> >>> > > > Regards,
> >>> > > > Ben
> >>> > > >
> >>> > >
> >>> >
> >>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

尹文才
Hi guys, about my case, I have another question, if I implement the retry
logic inside the ontrigger method and I need to retry until the database
connection is back online, in case user needs to stop the processor in NIFI
UI while the database is still offline, according to my understanding
ontrigger will keep executing the retry logic and the processor couldn't be
stopped even if user tries to stop it, is there any way to solve this
problem? Thanks.

Regards,
Ben

2017-08-12 6:30 GMT+08:00 尹文才 <[hidden email]>:

> Hi Bryan and Matt, thanks for all your suggestions, I was trying to make
> sure that the OnUnscheduled method was not called too frequently when the
> connection is offline.
> You guys were right, these sort of logic should not be placed inside the
> scheduling methods, I need to refactor my code to place them into onTrigger.
>
> Regards,
> Ben
>
> 2017-08-12 0:53 GMT+08:00 Matt Burgess <[hidden email]>:
>
>> I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
>> (such as database connection retries), I recommend putting the
>> connection logic in the onTrigger() method. If you can check
>> connectivity, then your onTrigger() would know whether it needs to try
>> to reconnect before it does any work. If it tries to reconnect and is
>> unsuccessful, you can yield the processor if you want, so as not to
>> hammer the DB with connection attempts. The CaptureChangeMySQL
>> processor does this, it has a retry loop for trying various nodes in a
>> MySQL cluster, but once it's connected, it goes on about its work, and
>> if a connection fails, it will retry the connection loop before it
>> does any more work. It only uses onTrigger and none of the scheduling
>> stuff.
>>
>> Regards,
>> Matt
>>
>> On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende <[hidden email]> wrote:
>> > Ben,
>> >
>> > I apologize if I am not understanding the situation, but...
>> >
>> > In the case where your OnScheduled code is in a retry loop, if someone
>> > stops the processor it will call your OnUnscheduled code which will
>> > set the flag to bounce out of the loop. This sounds like what you
>> > want, right?
>> >
>> > In the case where OnScheduled times out, the framework is calling
>> > OnUnscheduled which would call your code to set the flag, but wouldn't
>> > that not matter at this point because you aren't looping anymore
>> > anyway?
>> >
>> > If the framework calls OnScheduled again, your code should set the
>> > flag back to whatever it needs to be to start looping again right?
>> >
>> > An alternative that might avoid some of this would be to lazily
>> > initialize the connection in the onTrigger method of the processor.
>> >
>> > -Bryan
>> >
>> >
>> > On Fri, Aug 11, 2017 at 9:16 AM, 尹文才 <[hidden email]> wrote:
>> >> thanks Pierre, my case is that I need to implement a database
>> connection
>> >> retry logic inside my OnScheduled method, when the database is not
>> >> available I will retry until the connection is back online.
>> >> The problem is when the database is offline it will throw timed out
>> >> execution exception inside OnScheduled and then call OnUnscheduled. But
>> >> when I manually stop the processor the OnUnsheduled
>> >> will also get called. I know my logic sounds a little weird but I need
>> to
>> >> set some flag in the OnUnscheduled method to stop the retry logic
>> inside
>> >> OnScheduled in order to be able to stop the processor,
>> >> otherwise the processor is not able to be stopped unless I restart the
>> >> whole NIFI.
>> >>
>> >> Regards,
>> >> Ben
>> >>
>> >> 2017-08-11 17:18 GMT+08:00 Pierre Villard <[hidden email]
>> >:
>> >>
>> >>> Oh OK, get it now!
>> >>>
>> >>> Not sure what's your use case, but I don't think you can do that
>> unless you
>> >>> set some information when the process actually executes onTrigger for
>> the
>> >>> first time and you then check this value in your OnUnscheduled
>> annotated
>> >>> method.
>> >>>
>> >>> Pierre
>> >>>
>> >>> 2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:
>> >>>
>> >>> > Hi Pierre, I've checked the developer guide before I sent the email
>> and
>> >>> > according to the developer guide, the method annotated with
>> OnUnScheduled
>> >>> > will be called in 2 cases according to my understanding, please
>> correct
>> >>> me
>> >>> > if I'm wrong:
>> >>> > 1. when user tries to stop the processor in the NIFI UI, thus the
>> >>> processor
>> >>> > is no longer scheduled to run in this case, and the method will be
>> >>> called.
>> >>> > 2. when method annotated with OnScheduled throws exceptions, for
>> example
>> >>> > time out execution exception, the OnUnScheduled method will also be
>> >>> called.
>> >>> >
>> >>> > My question is how to tell the first scenario from the second one?
>> >>> Thanks.
>> >>> >
>> >>> > Regards,
>> >>> > Ben
>> >>> >
>> >>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <
>> [hidden email]>:
>> >>> >
>> >>> > > Hi Ben,
>> >>> > >
>> >>> > > You might want to have a look here:
>> >>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
>> >>> > > guide.html#component-lifecycle
>> >>> > >
>> >>> > > Pierre
>> >>> > >
>> >>> > > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
>> >>> > >
>> >>> > > > Hi guys, I'm trying to execute some code in my processor when
>> the
>> >>> > > processor
>> >>> > > > is asked to stop in the NIFI UI by the user, I checked the
>> developer
>> >>> > > guide
>> >>> > > > and only find OnUnscheduled will be called when the processor
>> is no
>> >>> > long
>> >>> > > > scheduled to run. I've tested this OnUnscheduled, it will also
>> be
>> >>> > called
>> >>> > > > after timed out executing OnScheduled task. So is there a way to
>> >>> > execute
>> >>> > > > some code only when the processor is stopping?
>> >>> > > >
>> >>> > > > Regards,
>> >>> > > > Ben
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

Koji Kawamura
Hi Ben,

AbstractSessionFactoryProcessor has a protected isScheduled() method,
that can be used by a processor implementation class to check whether
it is still being scheduled (not being stopped).
For an example, ConsumeKafka_0_10.onTrigger uses it with while loop:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumeKafka_0_10.java#L316

Thanks,
Koji

On Mon, Aug 14, 2017 at 11:12 AM, 尹文才 <[hidden email]> wrote:

> Hi guys, about my case, I have another question, if I implement the retry
> logic inside the ontrigger method and I need to retry until the database
> connection is back online, in case user needs to stop the processor in NIFI
> UI while the database is still offline, according to my understanding
> ontrigger will keep executing the retry logic and the processor couldn't be
> stopped even if user tries to stop it, is there any way to solve this
> problem? Thanks.
>
> Regards,
> Ben
>
> 2017-08-12 6:30 GMT+08:00 尹文才 <[hidden email]>:
>
>> Hi Bryan and Matt, thanks for all your suggestions, I was trying to make
>> sure that the OnUnscheduled method was not called too frequently when the
>> connection is offline.
>> You guys were right, these sort of logic should not be placed inside the
>> scheduling methods, I need to refactor my code to place them into onTrigger.
>>
>> Regards,
>> Ben
>>
>> 2017-08-12 0:53 GMT+08:00 Matt Burgess <[hidden email]>:
>>
>>> I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
>>> (such as database connection retries), I recommend putting the
>>> connection logic in the onTrigger() method. If you can check
>>> connectivity, then your onTrigger() would know whether it needs to try
>>> to reconnect before it does any work. If it tries to reconnect and is
>>> unsuccessful, you can yield the processor if you want, so as not to
>>> hammer the DB with connection attempts. The CaptureChangeMySQL
>>> processor does this, it has a retry loop for trying various nodes in a
>>> MySQL cluster, but once it's connected, it goes on about its work, and
>>> if a connection fails, it will retry the connection loop before it
>>> does any more work. It only uses onTrigger and none of the scheduling
>>> stuff.
>>>
>>> Regards,
>>> Matt
>>>
>>> On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende <[hidden email]> wrote:
>>> > Ben,
>>> >
>>> > I apologize if I am not understanding the situation, but...
>>> >
>>> > In the case where your OnScheduled code is in a retry loop, if someone
>>> > stops the processor it will call your OnUnscheduled code which will
>>> > set the flag to bounce out of the loop. This sounds like what you
>>> > want, right?
>>> >
>>> > In the case where OnScheduled times out, the framework is calling
>>> > OnUnscheduled which would call your code to set the flag, but wouldn't
>>> > that not matter at this point because you aren't looping anymore
>>> > anyway?
>>> >
>>> > If the framework calls OnScheduled again, your code should set the
>>> > flag back to whatever it needs to be to start looping again right?
>>> >
>>> > An alternative that might avoid some of this would be to lazily
>>> > initialize the connection in the onTrigger method of the processor.
>>> >
>>> > -Bryan
>>> >
>>> >
>>> > On Fri, Aug 11, 2017 at 9:16 AM, 尹文才 <[hidden email]> wrote:
>>> >> thanks Pierre, my case is that I need to implement a database
>>> connection
>>> >> retry logic inside my OnScheduled method, when the database is not
>>> >> available I will retry until the connection is back online.
>>> >> The problem is when the database is offline it will throw timed out
>>> >> execution exception inside OnScheduled and then call OnUnscheduled. But
>>> >> when I manually stop the processor the OnUnsheduled
>>> >> will also get called. I know my logic sounds a little weird but I need
>>> to
>>> >> set some flag in the OnUnscheduled method to stop the retry logic
>>> inside
>>> >> OnScheduled in order to be able to stop the processor,
>>> >> otherwise the processor is not able to be stopped unless I restart the
>>> >> whole NIFI.
>>> >>
>>> >> Regards,
>>> >> Ben
>>> >>
>>> >> 2017-08-11 17:18 GMT+08:00 Pierre Villard <[hidden email]
>>> >:
>>> >>
>>> >>> Oh OK, get it now!
>>> >>>
>>> >>> Not sure what's your use case, but I don't think you can do that
>>> unless you
>>> >>> set some information when the process actually executes onTrigger for
>>> the
>>> >>> first time and you then check this value in your OnUnscheduled
>>> annotated
>>> >>> method.
>>> >>>
>>> >>> Pierre
>>> >>>
>>> >>> 2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:
>>> >>>
>>> >>> > Hi Pierre, I've checked the developer guide before I sent the email
>>> and
>>> >>> > according to the developer guide, the method annotated with
>>> OnUnScheduled
>>> >>> > will be called in 2 cases according to my understanding, please
>>> correct
>>> >>> me
>>> >>> > if I'm wrong:
>>> >>> > 1. when user tries to stop the processor in the NIFI UI, thus the
>>> >>> processor
>>> >>> > is no longer scheduled to run in this case, and the method will be
>>> >>> called.
>>> >>> > 2. when method annotated with OnScheduled throws exceptions, for
>>> example
>>> >>> > time out execution exception, the OnUnScheduled method will also be
>>> >>> called.
>>> >>> >
>>> >>> > My question is how to tell the first scenario from the second one?
>>> >>> Thanks.
>>> >>> >
>>> >>> > Regards,
>>> >>> > Ben
>>> >>> >
>>> >>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <
>>> [hidden email]>:
>>> >>> >
>>> >>> > > Hi Ben,
>>> >>> > >
>>> >>> > > You might want to have a look here:
>>> >>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
>>> >>> > > guide.html#component-lifecycle
>>> >>> > >
>>> >>> > > Pierre
>>> >>> > >
>>> >>> > > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
>>> >>> > >
>>> >>> > > > Hi guys, I'm trying to execute some code in my processor when
>>> the
>>> >>> > > processor
>>> >>> > > > is asked to stop in the NIFI UI by the user, I checked the
>>> developer
>>> >>> > > guide
>>> >>> > > > and only find OnUnscheduled will be called when the processor
>>> is no
>>> >>> > long
>>> >>> > > > scheduled to run. I've tested this OnUnscheduled, it will also
>>> be
>>> >>> > called
>>> >>> > > > after timed out executing OnScheduled task. So is there a way to
>>> >>> > execute
>>> >>> > > > some code only when the processor is stopping?
>>> >>> > > >
>>> >>> > > > Regards,
>>> >>> > > > Ben
>>> >>> > > >
>>> >>> > >
>>> >>> >
>>> >>>
>>>
>>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to execute code when processor is stopping

尹文才
Thanks Koji, this is exactly what I'm looking for.😁

Regards,
Ben

2017-08-14 12:21 GMT+08:00 Koji Kawamura <[hidden email]>:

> Hi Ben,
>
> AbstractSessionFactoryProcessor has a protected isScheduled() method,
> that can be used by a processor implementation class to check whether
> it is still being scheduled (not being stopped).
> For an example, ConsumeKafka_0_10.onTrigger uses it with while loop:
> https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/
> src/main/java/org/apache/nifi/processors/kafka/pubsub/
> ConsumeKafka_0_10.java#L316
>
> Thanks,
> Koji
>
> On Mon, Aug 14, 2017 at 11:12 AM, 尹文才 <[hidden email]> wrote:
> > Hi guys, about my case, I have another question, if I implement the retry
> > logic inside the ontrigger method and I need to retry until the database
> > connection is back online, in case user needs to stop the processor in
> NIFI
> > UI while the database is still offline, according to my understanding
> > ontrigger will keep executing the retry logic and the processor couldn't
> be
> > stopped even if user tries to stop it, is there any way to solve this
> > problem? Thanks.
> >
> > Regards,
> > Ben
> >
> > 2017-08-12 6:30 GMT+08:00 尹文才 <[hidden email]>:
> >
> >> Hi Bryan and Matt, thanks for all your suggestions, I was trying to make
> >> sure that the OnUnscheduled method was not called too frequently when
> the
> >> connection is offline.
> >> You guys were right, these sort of logic should not be placed inside the
> >> scheduling methods, I need to refactor my code to place them into
> onTrigger.
> >>
> >> Regards,
> >> Ben
> >>
> >> 2017-08-12 0:53 GMT+08:00 Matt Burgess <[hidden email]>:
> >>
> >>> I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
> >>> (such as database connection retries), I recommend putting the
> >>> connection logic in the onTrigger() method. If you can check
> >>> connectivity, then your onTrigger() would know whether it needs to try
> >>> to reconnect before it does any work. If it tries to reconnect and is
> >>> unsuccessful, you can yield the processor if you want, so as not to
> >>> hammer the DB with connection attempts. The CaptureChangeMySQL
> >>> processor does this, it has a retry loop for trying various nodes in a
> >>> MySQL cluster, but once it's connected, it goes on about its work, and
> >>> if a connection fails, it will retry the connection loop before it
> >>> does any more work. It only uses onTrigger and none of the scheduling
> >>> stuff.
> >>>
> >>> Regards,
> >>> Matt
> >>>
> >>> On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende <[hidden email]>
> wrote:
> >>> > Ben,
> >>> >
> >>> > I apologize if I am not understanding the situation, but...
> >>> >
> >>> > In the case where your OnScheduled code is in a retry loop, if
> someone
> >>> > stops the processor it will call your OnUnscheduled code which will
> >>> > set the flag to bounce out of the loop. This sounds like what you
> >>> > want, right?
> >>> >
> >>> > In the case where OnScheduled times out, the framework is calling
> >>> > OnUnscheduled which would call your code to set the flag, but
> wouldn't
> >>> > that not matter at this point because you aren't looping anymore
> >>> > anyway?
> >>> >
> >>> > If the framework calls OnScheduled again, your code should set the
> >>> > flag back to whatever it needs to be to start looping again right?
> >>> >
> >>> > An alternative that might avoid some of this would be to lazily
> >>> > initialize the connection in the onTrigger method of the processor.
> >>> >
> >>> > -Bryan
> >>> >
> >>> >
> >>> > On Fri, Aug 11, 2017 at 9:16 AM, 尹文才 <[hidden email]> wrote:
> >>> >> thanks Pierre, my case is that I need to implement a database
> >>> connection
> >>> >> retry logic inside my OnScheduled method, when the database is not
> >>> >> available I will retry until the connection is back online.
> >>> >> The problem is when the database is offline it will throw timed out
> >>> >> execution exception inside OnScheduled and then call OnUnscheduled.
> But
> >>> >> when I manually stop the processor the OnUnsheduled
> >>> >> will also get called. I know my logic sounds a little weird but I
> need
> >>> to
> >>> >> set some flag in the OnUnscheduled method to stop the retry logic
> >>> inside
> >>> >> OnScheduled in order to be able to stop the processor,
> >>> >> otherwise the processor is not able to be stopped unless I restart
> the
> >>> >> whole NIFI.
> >>> >>
> >>> >> Regards,
> >>> >> Ben
> >>> >>
> >>> >> 2017-08-11 17:18 GMT+08:00 Pierre Villard <
> [hidden email]
> >>> >:
> >>> >>
> >>> >>> Oh OK, get it now!
> >>> >>>
> >>> >>> Not sure what's your use case, but I don't think you can do that
> >>> unless you
> >>> >>> set some information when the process actually executes onTrigger
> for
> >>> the
> >>> >>> first time and you then check this value in your OnUnscheduled
> >>> annotated
> >>> >>> method.
> >>> >>>
> >>> >>> Pierre
> >>> >>>
> >>> >>> 2017-08-11 10:11 GMT+02:00 尹文才 <[hidden email]>:
> >>> >>>
> >>> >>> > Hi Pierre, I've checked the developer guide before I sent the
> email
> >>> and
> >>> >>> > according to the developer guide, the method annotated with
> >>> OnUnScheduled
> >>> >>> > will be called in 2 cases according to my understanding, please
> >>> correct
> >>> >>> me
> >>> >>> > if I'm wrong:
> >>> >>> > 1. when user tries to stop the processor in the NIFI UI, thus the
> >>> >>> processor
> >>> >>> > is no longer scheduled to run in this case, and the method will
> be
> >>> >>> called.
> >>> >>> > 2. when method annotated with OnScheduled throws exceptions, for
> >>> example
> >>> >>> > time out execution exception, the OnUnScheduled method will also
> be
> >>> >>> called.
> >>> >>> >
> >>> >>> > My question is how to tell the first scenario from the second
> one?
> >>> >>> Thanks.
> >>> >>> >
> >>> >>> > Regards,
> >>> >>> > Ben
> >>> >>> >
> >>> >>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <
> >>> [hidden email]>:
> >>> >>> >
> >>> >>> > > Hi Ben,
> >>> >>> > >
> >>> >>> > > You might want to have a look here:
> >>> >>> > > https://nifi.apache.org/docs/nifi-docs/html/developer-
> >>> >>> > > guide.html#component-lifecycle
> >>> >>> > >
> >>> >>> > > Pierre
> >>> >>> > >
> >>> >>> > > 2017-08-11 9:06 GMT+02:00 尹文才 <[hidden email]>:
> >>> >>> > >
> >>> >>> > > > Hi guys, I'm trying to execute some code in my processor when
> >>> the
> >>> >>> > > processor
> >>> >>> > > > is asked to stop in the NIFI UI by the user, I checked the
> >>> developer
> >>> >>> > > guide
> >>> >>> > > > and only find OnUnscheduled will be called when the processor
> >>> is no
> >>> >>> > long
> >>> >>> > > > scheduled to run. I've tested this OnUnscheduled, it will
> also
> >>> be
> >>> >>> > called
> >>> >>> > > > after timed out executing OnScheduled task. So is there a
> way to
> >>> >>> > execute
> >>> >>> > > > some code only when the processor is stopping?
> >>> >>> > > >
> >>> >>> > > > Regards,
> >>> >>> > > > Ben
> >>> >>> > > >
> >>> >>> > >
> >>> >>> >
> >>> >>>
> >>>
> >>
> >>
>
Loading...