How to count the number of lines in csv?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to count the number of lines in csv?

prabhu Mahendran
Hi All,

i try to count the number of lines in csv.

Following Reference,
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#count

with help of below method.I have stored the entire flow file in "input" attribute and then delimited by using LineFeed character.

${allDelineatedValues(${input}, "\n"):count()}

i have used extracttext processor for store entire flowfile into those attribute.

I can able to get number of lines in smaller csv files only it cannot work for huge files(10 GB files) due to extract text only holds 2147463746 bytes in Buffer size only.it leads wrong number of count stored in attribute.

Is there is any other way to count number of lines in csv without use Script?
Reply | Threaded
Open this post in threaded view
|

Re: How to count the number of lines in csv?

Venkat Williams
Write custom processor to with csv parser either using uniocity lib or open csv lib and do no.of records.. 

I don't aware if there is something along with CSVRecord processor.

Regards,
Venkat

On Fri, Jun 9, 2017 at 1:02 PM, prabhu Mahendran <[hidden email]> wrote:
Hi All,

i try to count the number of lines in csv.

Following Reference,
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#count

with help of below method.I have stored the entire flow file in "input" attribute and then delimited by using LineFeed character.

${allDelineatedValues(${input}, "\n"):count()}

i have used extracttext processor for store entire flowfile into those attribute.

I can able to get number of lines in smaller csv files only it cannot work for huge files(10 GB files) due to extract text only holds <a href="tel:(214)%20746-3746" value="+12147463746" target="_blank">2147463746 bytes in Buffer size only.it leads wrong number of count stored in attribute.

Is there is any other way to count number of lines in csv without use Script?

Reply | Threaded
Open this post in threaded view
|

Re: How to count the number of lines in csv?

prabhu Mahendran
I have found that we can easily calculate lines using SplitText processor.

It having "fragment.count" attribute which say number of splits.

So no need for create custom processor.

Thanks for your mail

On Fri, Jun 9, 2017 at 1:14 PM, Venkat Williams <[hidden email]> wrote:
Write custom processor to with csv parser either using uniocity lib or open csv lib and do no.of records.. 

I don't aware if there is something along with CSVRecord processor.

Regards,
Venkat

On Fri, Jun 9, 2017 at 1:02 PM, prabhu Mahendran <[hidden email]> wrote:
Hi All,

i try to count the number of lines in csv.

Following Reference,
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#count

with help of below method.I have stored the entire flow file in "input" attribute and then delimited by using LineFeed character.

${allDelineatedValues(${input}, "\n"):count()}

i have used extracttext processor for store entire flowfile into those attribute.

I can able to get number of lines in smaller csv files only it cannot work for huge files(10 GB files) due to extract text only holds <a href="tel:(214)%20746-3746" value="+12147463746" target="_blank">2147463746 bytes in Buffer size only.it leads wrong number of count stored in attribute.

Is there is any other way to count number of lines in csv without use Script?