NiFi 1.3.0 Remote ports "disabled" after a while?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi 1.3.0 Remote ports "disabled" after a while?

Joe Gresock
Before I investigate further, has anyone encountered NiFi RPGs "disabling"
after a while in 1.3.0?  Their status is reported as enabled/transmitting,
but the flow files just sit there until I "enable transmission" again.
This happens for nearly all of the RPGs I have in my flow.  I just have to
keep "kick starting" them periodically to keep things flowing.

This behavior did not happen in 1.2.0.
--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.3.0 Remote ports "disabled" after a while?

Koji Kawamura
Hi Joe,

I haven't encountered that issue. Would you tell us more details? So
that we can investigate it.

1. Which transport protocol are you using? RAW or HTTP
2. Is this a pull or push transfer? Pull = client NiFi pulls from
remote OutputPort, Push = sends to remote InputPort
3. Is compression enabled?
4. How many ports are configured? If there are more than one, RPG
indicates that it's 'Transmitting' as long as at least one port is
transmitting, while others can be disabled. Right click a RPG then
select 'Remote Ports' to see each port status.
5. How often does it happen? Stops once per hour/day/week??
6. Please share your environment, OS, Java version ... etc
7. If possible, please take stack-trace while you are facing that
issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
stack-trace will be written to nifi-bootstrap.log
8. Is proxy used?

Thanks,
Koji


On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <[hidden email]> wrote:

> Before I investigate further, has anyone encountered NiFi RPGs "disabling"
> after a while in 1.3.0?  Their status is reported as enabled/transmitting,
> but the flow files just sit there until I "enable transmission" again.
> This happens for nearly all of the RPGs I have in my flow.  I just have to
> keep "kick starting" them periodically to keep things flowing.
>
> This behavior did not happen in 1.2.0.
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.    *-Philippians 4:12-13*
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.3.0 Remote ports "disabled" after a while?

Joe Gresock
Great questions, Koji:

1. RAW
2. Push
3. No, compression is not enabled
4. Several, but it has the same effect to "Enable transmission" as it does to specifically disable and then enable a specific port.  The key being that all the ports are marked as enabled to start, even when not sending.
5. This is anecdotal, but it feels like 1+ times per day.
6. CentOS 6.9, Java 1.8.0_131.  This is a 4-node NiFi cluster.  There are multiple self-RPGs, including this one, at various points on the graph.
7. Attached.  The relevant queue that is inexplicably blocked shows this in the log:
"Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90: https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258 TIMED_WAITING  on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2edb0217
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

8. No proxy

One strange thing I noticed was that the <scheduledState> is not consistent in the flow.xml.gz between some of the nodes for other instances of this self-RPG (but not for the one that is currently backing up):
1. On disk, some of the nodes have this input port set to STOPPED, and some have it set to RUNNING.
2. In the UI, the input port is displayed as RUNNING.



On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <[hidden email]> wrote:
Hi Joe,

I haven't encountered that issue. Would you tell us more details? So
that we can investigate it.

1. Which transport protocol are you using? RAW or HTTP
2. Is this a pull or push transfer? Pull = client NiFi pulls from
remote OutputPort, Push = sends to remote InputPort
3. Is compression enabled?
4. How many ports are configured? If there are more than one, RPG
indicates that it's 'Transmitting' as long as at least one port is
transmitting, while others can be disabled. Right click a RPG then
select 'Remote Ports' to see each port status.
5. How often does it happen? Stops once per hour/day/week??
6. Please share your environment, OS, Java version ... etc
7. If possible, please take stack-trace while you are facing that
issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
stack-trace will be written to nifi-bootstrap.log
8. Is proxy used?

Thanks,
Koji


On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <[hidden email]> wrote:
> Before I investigate further, has anyone encountered NiFi RPGs "disabling"
> after a while in 1.3.0?  Their status is reported as enabled/transmitting,
> but the flow files just sit there until I "enable transmission" again.
> This happens for nearly all of the RPGs I have in my flow.  I just have to
> keep "kick starting" them periodically to keep things flowing.
>
> This behavior did not happen in 1.2.0.
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.    *-Philippians 4:12-13*



--
I know what it is to be in need, and I know what it is to have plenty.  I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want.  I can do all this through him who gives me strength.    -Philippians 4:12-13

nifi-bootstrap.log (660K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.3.0 Remote ports "disabled" after a while?

Joe Gresock
Btw, I was wrong about one statement in my last email.. this particular RPG
(705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes are
STOPPED and some are RUNNING.  However, even on the nodes that show RUNNING
in the flow.xml.gz, the queue is still blocked.



On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <[hidden email]> wrote:

> Great questions, Koji:
>
> 1. RAW
> 2. Push
> 3. No, compression is not enabled
> 4. Several, but it has the same effect to "Enable transmission" as it does
> to specifically disable and then enable a specific port.  The key being
> that all the ports are marked as enabled to start, even when not sending.
> 5. This is anecdotal, but it feels like 1+ times per day.
> 6. CentOS 6.9, Java 1.8.0_131.  This is a 4-node NiFi cluster.  There are
> multiple self-RPGs, including this one, at various points on the graph.
> 7. Attached.  The relevant queue that is inexplicably blocked shows this
> in the log:
> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
> TIMED_WAITING  on java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject@2edb0217
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:215)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:748)
>
> 8. No proxy
>
> One strange thing I noticed was that the <scheduledState> is not
> consistent in the flow.xml.gz between some of the nodes for other instances
> of this self-RPG (but not for the one that is currently backing up):
> 1. On disk, some of the nodes have this input port set to STOPPED, and
> some have it set to RUNNING.
> 2. In the UI, the input port is displayed as RUNNING.
>
>
>
> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <[hidden email]>
> wrote:
>
>> Hi Joe,
>>
>> I haven't encountered that issue. Would you tell us more details? So
>> that we can investigate it.
>>
>> 1. Which transport protocol are you using? RAW or HTTP
>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
>> remote OutputPort, Push = sends to remote InputPort
>> 3. Is compression enabled?
>> 4. How many ports are configured? If there are more than one, RPG
>> indicates that it's 'Transmitting' as long as at least one port is
>> transmitting, while others can be disabled. Right click a RPG then
>> select 'Remote Ports' to see each port status.
>> 5. How often does it happen? Stops once per hour/day/week??
>> 6. Please share your environment, OS, Java version ... etc
>> 7. If possible, please take stack-trace while you are facing that
>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
>> stack-trace will be written to nifi-bootstrap.log
>> 8. Is proxy used?
>>
>> Thanks,
>> Koji
>>
>>
>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <[hidden email]> wrote:
>> > Before I investigate further, has anyone encountered NiFi RPGs
>> "disabling"
>> > after a while in 1.3.0?  Their status is reported as
>> enabled/transmitting,
>> > but the flow files just sit there until I "enable transmission" again.
>> > This happens for nearly all of the RPGs I have in my flow.  I just have
>> to
>> > keep "kick starting" them periodically to keep things flowing.
>> >
>> > This behavior did not happen in 1.2.0.
>> > --
>> > I know what it is to be in need, and I know what it is to have plenty.
>> I
>> > have learned the secret of being content in any and every situation,
>> > whether well fed or hungry, whether living in plenty or in want.  I can
>> do
>> > all this through him who gives me strength.    *-Philippians 4:12-13*
>>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can
> do all this through him who gives me strength.    *-Philippians 4:12-13*
>



--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.3.0 Remote ports "disabled" after a while?

Bryan Bende
Joe,

Thanks for the info.

I think this might be similar to
https://issues.apache.org/jira/browse/NIFI-3900, but the fix didn't
account for RPGs.

I created this JIRA - https://issues.apache.org/jira/browse/NIFI-4075

-Bryan

On Thu, Jun 15, 2017 at 10:14 AM, Joe Gresock <[hidden email]> wrote:

> Btw, I was wrong about one statement in my last email.. this particular RPG
> (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes are
> STOPPED and some are RUNNING.  However, even on the nodes that show RUNNING
> in the flow.xml.gz, the queue is still blocked.
>
>
>
> On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <[hidden email]> wrote:
>
>> Great questions, Koji:
>>
>> 1. RAW
>> 2. Push
>> 3. No, compression is not enabled
>> 4. Several, but it has the same effect to "Enable transmission" as it does
>> to specifically disable and then enable a specific port.  The key being
>> that all the ports are marked as enabled to start, even when not sending.
>> 5. This is anecdotal, but it feels like 1+ times per day.
>> 6. CentOS 6.9, Java 1.8.0_131.  This is a 4-node NiFi cluster.  There are
>> multiple self-RPGs, including this one, at various points on the graph.
>> 7. Attached.  The relevant queue that is inexplicably blocked shows this
>> in the log:
>> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
>> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
>> TIMED_WAITING  on java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject@2edb0217
>>         at sun.misc.Unsafe.park(Native Method)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:215)
>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>         at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>         at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>         at java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1067)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1127)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>> 8. No proxy
>>
>> One strange thing I noticed was that the <scheduledState> is not
>> consistent in the flow.xml.gz between some of the nodes for other instances
>> of this self-RPG (but not for the one that is currently backing up):
>> 1. On disk, some of the nodes have this input port set to STOPPED, and
>> some have it set to RUNNING.
>> 2. In the UI, the input port is displayed as RUNNING.
>>
>>
>>
>> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <[hidden email]>
>> wrote:
>>
>>> Hi Joe,
>>>
>>> I haven't encountered that issue. Would you tell us more details? So
>>> that we can investigate it.
>>>
>>> 1. Which transport protocol are you using? RAW or HTTP
>>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
>>> remote OutputPort, Push = sends to remote InputPort
>>> 3. Is compression enabled?
>>> 4. How many ports are configured? If there are more than one, RPG
>>> indicates that it's 'Transmitting' as long as at least one port is
>>> transmitting, while others can be disabled. Right click a RPG then
>>> select 'Remote Ports' to see each port status.
>>> 5. How often does it happen? Stops once per hour/day/week??
>>> 6. Please share your environment, OS, Java version ... etc
>>> 7. If possible, please take stack-trace while you are facing that
>>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
>>> stack-trace will be written to nifi-bootstrap.log
>>> 8. Is proxy used?
>>>
>>> Thanks,
>>> Koji
>>>
>>>
>>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <[hidden email]> wrote:
>>> > Before I investigate further, has anyone encountered NiFi RPGs
>>> "disabling"
>>> > after a while in 1.3.0?  Their status is reported as
>>> enabled/transmitting,
>>> > but the flow files just sit there until I "enable transmission" again.
>>> > This happens for nearly all of the RPGs I have in my flow.  I just have
>>> to
>>> > keep "kick starting" them periodically to keep things flowing.
>>> >
>>> > This behavior did not happen in 1.2.0.
>>> > --
>>> > I know what it is to be in need, and I know what it is to have plenty.
>>> I
>>> > have learned the secret of being content in any and every situation,
>>> > whether well fed or hungry, whether living in plenty or in want.  I can
>>> do
>>> > all this through him who gives me strength.    *-Philippians 4:12-13*
>>>
>>
>>
>>
>> --
>> I know what it is to be in need, and I know what it is to have plenty.  I
>> have learned the secret of being content in any and every situation,
>> whether well fed or hungry, whether living in plenty or in want.  I can
>> do all this through him who gives me strength.    *-Philippians 4:12-13*
>>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.    *-Philippians 4:12-13*
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.3.0 Remote ports "disabled" after a while?

Joe Gresock
Great, thanks Bryan.

On Thu, Jun 15, 2017 at 2:38 PM, Bryan Bende <[hidden email]> wrote:

> Joe,
>
> Thanks for the info.
>
> I think this might be similar to
> https://issues.apache.org/jira/browse/NIFI-3900, but the fix didn't
> account for RPGs.
>
> I created this JIRA - https://issues.apache.org/jira/browse/NIFI-4075
>
> -Bryan
>
> On Thu, Jun 15, 2017 at 10:14 AM, Joe Gresock <[hidden email]> wrote:
> > Btw, I was wrong about one statement in my last email.. this particular
> RPG
> > (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes
> are
> > STOPPED and some are RUNNING.  However, even on the nodes that show
> RUNNING
> > in the flow.xml.gz, the queue is still blocked.
> >
> >
> >
> > On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <[hidden email]> wrote:
> >
> >> Great questions, Koji:
> >>
> >> 1. RAW
> >> 2. Push
> >> 3. No, compression is not enabled
> >> 4. Several, but it has the same effect to "Enable transmission" as it
> does
> >> to specifically disable and then enable a specific port.  The key being
> >> that all the ports are marked as enabled to start, even when not
> sending.
> >> 5. This is anecdotal, but it feels like 1+ times per day.
> >> 6. CentOS 6.9, Java 1.8.0_131.  This is a 4-node NiFi cluster.  There
> are
> >> multiple self-RPGs, including this one, at various points on the graph.
> >> 7. Attached.  The relevant queue that is inexplicably blocked shows this
> >> in the log:
> >> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
> >> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
> >> TIMED_WAITING  on java.util.concurrent.locks.
> AbstractQueuedSynchronizer$
> >> ConditionObject@2edb0217
> >>         at sun.misc.Unsafe.park(Native Method)
> >>         at java.util.concurrent.locks.LockSupport.parkNanos(
> >> LockSupport.java:215)
> >>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> >> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> >>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> >>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> >>         at java.util.concurrent.ThreadPoolExecutor.getTask(
> >> ThreadPoolExecutor.java:1067)
> >>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> >> ThreadPoolExecutor.java:1127)
> >>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> >> ThreadPoolExecutor.java:617)
> >>         at java.lang.Thread.run(Thread.java:748)
> >>
> >> 8. No proxy
> >>
> >> One strange thing I noticed was that the <scheduledState> is not
> >> consistent in the flow.xml.gz between some of the nodes for other
> instances
> >> of this self-RPG (but not for the one that is currently backing up):
> >> 1. On disk, some of the nodes have this input port set to STOPPED, and
> >> some have it set to RUNNING.
> >> 2. In the UI, the input port is displayed as RUNNING.
> >>
> >>
> >>
> >> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <[hidden email]
> >
> >> wrote:
> >>
> >>> Hi Joe,
> >>>
> >>> I haven't encountered that issue. Would you tell us more details? So
> >>> that we can investigate it.
> >>>
> >>> 1. Which transport protocol are you using? RAW or HTTP
> >>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
> >>> remote OutputPort, Push = sends to remote InputPort
> >>> 3. Is compression enabled?
> >>> 4. How many ports are configured? If there are more than one, RPG
> >>> indicates that it's 'Transmitting' as long as at least one port is
> >>> transmitting, while others can be disabled. Right click a RPG then
> >>> select 'Remote Ports' to see each port status.
> >>> 5. How often does it happen? Stops once per hour/day/week??
> >>> 6. Please share your environment, OS, Java version ... etc
> >>> 7. If possible, please take stack-trace while you are facing that
> >>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
> >>> stack-trace will be written to nifi-bootstrap.log
> >>> 8. Is proxy used?
> >>>
> >>> Thanks,
> >>> Koji
> >>>
> >>>
> >>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <[hidden email]>
> wrote:
> >>> > Before I investigate further, has anyone encountered NiFi RPGs
> >>> "disabling"
> >>> > after a while in 1.3.0?  Their status is reported as
> >>> enabled/transmitting,
> >>> > but the flow files just sit there until I "enable transmission"
> again.
> >>> > This happens for nearly all of the RPGs I have in my flow.  I just
> have
> >>> to
> >>> > keep "kick starting" them periodically to keep things flowing.
> >>> >
> >>> > This behavior did not happen in 1.2.0.
> >>> > --
> >>> > I know what it is to be in need, and I know what it is to have
> plenty.
> >>> I
> >>> > have learned the secret of being content in any and every situation,
> >>> > whether well fed or hungry, whether living in plenty or in want.  I
> can
> >>> do
> >>> > all this through him who gives me strength.    *-Philippians 4:12-13*
> >>>
> >>
> >>
> >>
> >> --
> >> I know what it is to be in need, and I know what it is to have plenty.
> I
> >> have learned the secret of being content in any and every situation,
> >> whether well fed or hungry, whether living in plenty or in want.  I can
> >> do all this through him who gives me strength.    *-Philippians 4:12-13*
> >>
> >
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.    *-Philippians 4:12-13*
>



--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*