FlowFile Logging

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

FlowFile Logging

James Farrington
Hi everyone,

Whenever there is an error in a flow and some flow file is not processed
properly, I need to retrieve the Flow File UUID and the size of that file.
I checked the nifi-app.log file and it seemed that most (if not all) of the
time this information was being sent there on a processor error. Does
anyone know if this will always be the case? Or if there is a better way to
capture all flow files that failed to process?

Any help would be great!

Thank you.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: FlowFile Logging

Andy LoPresto-2
Hi James,

The app log will definitely contain a lot of relevant information about flowfile failure, but you can also make this easier for yourself by routing the failure connection of the relevant processor to a PutFile/PutEmail processor which outputs the flowfile UUID and content claim size to a special error destination. This means you won’t have to parse the app log and filter the non-error information. You may also want to look at the SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to transform the metadata (errors = bulletins, provenance for failure events) into raw data that you can then operate on directly. 

Obviously, you can also have that failure connection go to other processors, back to the original to retry, etc. 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 7, 2017, at 12:09 PM, James Farrington <[hidden email]> wrote:

Hi everyone,

Whenever there is an error in a flow and some flow file is not processed
properly, I need to retrieve the Flow File UUID and the size of that file.
I checked the nifi-app.log file and it seemed that most (if not all) of the
time this information was being sent there on a processor error. Does
anyone know if this will always be the case? Or if there is a better way to
capture all flow files that failed to process?

Any help would be great!

Thank you.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: FlowFile Logging

James Farrington
Hi Andy,

I am trying to catch any error that happens from any processor in my flow.
So adding a PutFile for each and every processor would not be ideal. And I
don't need the data to be transformed into a usable type. I am passing this
data to Logstash as a csv file (so raw data is just fine). Any thoughts?

Thank you,
James

On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto <[hidden email]> wrote:

> Hi James,
>
> The app log will definitely contain a lot of relevant information about
> flowfile failure, but you can also make this easier for yourself by routing
> the failure connection of the relevant processor to a PutFile/PutEmail
> processor which outputs the flowfile UUID and content claim size to a
> special error destination. This means you won’t have to parse the app log
> and filter the non-error information. You may also want to look at the
> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
> transform the metadata (errors = bulletins, provenance for failure events)
> into raw data that you can then operate on directly.
>
> Obviously, you can also have that failure connection go to other
> processors, back to the original to retry, etc.
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 7, 2017, at 12:09 PM, James Farrington <[hidden email]> wrote:
>
> Hi everyone,
>
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
>
> Any help would be great!
>
> Thank you.
>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: FlowFile Logging

Andy LoPresto-2
You can add an appender in the conf/logback.xml file which handles “org.apache.nifi” and anything of ERROR level and writes to a separate log file, which you can then monitor/parse/send wherever you like. This will filter out the normal operations logging from the failures you are interested in. Unfortunately, by default the full stacktraces and some other error information will be included here as well. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 8, 2017, at 1:09 PM, James Farrington <[hidden email]> wrote:

Hi Andy,

I am trying to catch any error that happens from any processor in my flow.
So adding a PutFile for each and every processor would not be ideal. And I
don't need the data to be transformed into a usable type. I am passing this
data to Logstash as a csv file (so raw data is just fine). Any thoughts?

Thank you,
James

On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto <[hidden email]> wrote:

Hi James,

The app log will definitely contain a lot of relevant information about
flowfile failure, but you can also make this easier for yourself by routing
the failure connection of the relevant processor to a PutFile/PutEmail
processor which outputs the flowfile UUID and content claim size to a
special error destination. This means you won’t have to parse the app log
and filter the non-error information. You may also want to look at the
SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
transform the metadata (errors = bulletins, provenance for failure events)
into raw data that you can then operate on directly.

Obviously, you can also have that failure connection go to other
processors, back to the original to retry, etc.

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 7, 2017, at 12:09 PM, James Farrington <[hidden email]> wrote:

Hi everyone,

Whenever there is an error in a flow and some flow file is not processed
properly, I need to retrieve the Flow File UUID and the size of that file.
I checked the nifi-app.log file and it seemed that most (if not all) of the
time this information was being sent there on a processor error. Does
anyone know if this will always be the case? Or if there is a better way to
capture all flow files that failed to process?

Any help would be great!

Thank you.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: FlowFile Logging

James Farrington
ah that sounds perfect for what I need to do. Thank you!

On Tue, Aug 8, 2017 at 7:20 PM, Andy LoPresto <[hidden email]> wrote:

> You can add an appender in the conf/logback.xml file which handles
> “org.apache.nifi” and anything of ERROR level and writes to a separate log
> file, which you can then monitor/parse/send wherever you like. This will
> filter out the normal operations logging from the failures you are
> interested in. Unfortunately, by default the full stacktraces and some
> other error information will be included here as well.
>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 8, 2017, at 1:09 PM, James Farrington <[hidden email]> wrote:
>
> Hi Andy,
>
> I am trying to catch any error that happens from any processor in my flow.
> So adding a PutFile for each and every processor would not be ideal. And I
> don't need the data to be transformed into a usable type. I am passing this
> data to Logstash as a csv file (so raw data is just fine). Any thoughts?
>
> Thank you,
> James
>
> On Mon, Aug 7, 2017 at 11:35 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> Hi James,
>
> The app log will definitely contain a lot of relevant information about
> flowfile failure, but you can also make this easier for yourself by routing
> the failure connection of the relevant processor to a PutFile/PutEmail
> processor which outputs the flowfile UUID and content claim size to a
> special error destination. This means you won’t have to parse the app log
> and filter the non-error information. You may also want to look at the
> SiteToSiteBulletinReportingTask and SiteToSiteProvenanceReportingTask to
> transform the metadata (errors = bulletins, provenance for failure events)
> into raw data that you can then operate on directly.
>
> Obviously, you can also have that failure connection go to other
> processors, back to the original to retry, etc.
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 7, 2017, at 12:09 PM, James Farrington <[hidden email]> wrote:
>
> Hi everyone,
>
> Whenever there is an error in a flow and some flow file is not processed
> properly, I need to retrieve the Flow File UUID and the size of that file.
> I checked the nifi-app.log file and it seemed that most (if not all) of the
> time this information was being sent there on a processor error. Does
> anyone know if this will always be the case? Or if there is a better way to
> capture all flow files that failed to process?
>
> Any help would be great!
>
> Thank you.
>
>
>
Loading...