NiFi 1.1.1 & 1.2.0 with PostgreSql 9.5

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

NiFi 1.1.1 & 1.2.0 with PostgreSql 9.5

Raymond Rogers-2
We are having a performance issue where we are inserting a 2000+ rows into a PostreSql database.

The database server is basically idle, disk writes 200-300 KB per second.
We are using the postgresql-42.1.1.jar JDBC driver.
The NiFi server has over 10,000 flow files waiting in queue for the PutSQL processor.
It appears that the PutSQL has completely stopped processing flow-files.
The two servers are sitting in the same rack and network traffic is less the 300 KB.
I can't find where anything is logging any errors.

Can anyone make any suggestions?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NiFi 1.1.1 & 1.2.0 with PostgreSql 9.5

Koji Kawamura
Hello Raymond,

Does the PutSQL has some number (probably 1) shown on its top right
corner in NiFi UI? If so, the execution thread is stuck at some state,
then looking at a thread dump may help to investigate where it stops.
Could you share "$NIFI_HOME/bin/nifi.sh dump" result? The output will
be written into logs/nifi-bootstrap.log.

Or if PutSQL has downstream flow such as 'success' or 'failure' and if
those relationships have 10,000 FlowFiles queued in it, then 'Back
Pressure' is enabled for those. In that case, NiFi won't schedule
PutSQL until those queued FlowFiles move forward or removed and number
of queued FlowFile and total size become less than configured
back-pressure threshold, 10,000 FlowFiles or 1GB in size by default.

Hope this helps.

Thanks,
Koji

On Sat, Jun 10, 2017 at 5:36 AM, Raymond Rogers
<[hidden email]> wrote:

> We are having a performance issue where we are inserting a 2000+ rows into a
> PostreSql database.
>
> The database server is basically idle, disk writes 200-300 KB per second.
> We are using the postgresql-42.1.1.jar JDBC driver.
> The NiFi server has over 10,000 flow files waiting in queue for the PutSQL
> processor.
> It appears that the PutSQL has completely stopped processing flow-files.
> The two servers are sitting in the same rack and network traffic is less the
> 300 KB.
> I can't find where anything is logging any errors.
>
> Can anyone make any suggestions?
Loading...