Set priority to files based on date time value stored on attribute

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Set priority to files based on date time value stored on attribute

Manojkumar Ravichandran
Hi All,

I need to process the files based on the date time value stored on the attribute

For example:

If the incoming files contains the following date time attribute values

2017/06/07  16:57:02
2017/06/06  12:49:49
2017/06/06  11:09:28
2017/06/06  06:37:45

I need to process the files based on the order of time that is oldest one from the current time 

First I want to access the file that contains below date time attribute which is the oldest one among them from the current time 
i.e 2017/06/06  06:37:45
and then below one,
2017/06/06  11:09:28
and then this
2017/06/06  12:49:49
so on .... 

How can I achieve the above mentioned scenario ?

Regards,
Manoj kumar R
Reply | Threaded
Open this post in threaded view
|

Re: Set priority to files based on date time value stored on attribute

Pierre Villard
Hi Manoj,

You may want ot have a look at EnforceOrder processor [1] or simply the prioritizers [2] of the connections (it depends of how your workflow is working). The idea would be to extract the date as an attribute of your flow file, convert into an integer (using expression language) and use it to ensure order.

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
[2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization

Hope this helps.


2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <[hidden email]>:
Hi All,

I need to process the files based on the date time value stored on the attribute

For example:

If the incoming files contains the following date time attribute values

2017/06/07  16:57:02
2017/06/06  12:49:49
2017/06/06  11:09:28
2017/06/06  06:37:45

I need to process the files based on the order of time that is oldest one from the current time 

First I want to access the file that contains below date time attribute which is the oldest one among them from the current time 
i.e 2017/06/06  06:37:45
and then below one,
2017/06/06  11:09:28
and then this
2017/06/06  12:49:49
so on .... 

How can I achieve the above mentioned scenario ?

Regards,
Manoj kumar R

Reply | Threaded
Open this post in threaded view
|

Re: Set priority to files based on date time value stored on attribute

Koji Kawamura
Hi Manoj,

I think EnforceOrder would not be useful in your case, as it expects
the order to increases one by one (without skip).
As Pierre suggested, I'd suggest using PriorityAttributePrioritizer.

Thanks,
Koji

On Thu, Jun 8, 2017 at 3:50 PM, Pierre Villard
<[hidden email]> wrote:

> Hi Manoj,
>
> You may want ot have a look at EnforceOrder processor [1] or simply the
> prioritizers [2] of the connections (it depends of how your workflow is
> working). The idea would be to extract the date as an attribute of your flow
> file, convert into an integer (using expression language) and use it to
> ensure order.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization
>
> Hope this helps.
>
>
> 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <[hidden email]>:
>>
>> Hi All,
>>
>> I need to process the files based on the date time value stored on the
>> attribute
>>
>> For example:
>>
>> If the incoming files contains the following date time attribute values
>>
>> 2017/06/07  16:57:02
>> 2017/06/06  12:49:49
>> 2017/06/06  11:09:28
>> 2017/06/06  06:37:45
>>
>> I need to process the files based on the order of time that is oldest one
>> from the current time
>>
>> First I want to access the file that contains below date time attribute
>> which is the oldest one among them from the current time
>> i.e 2017/06/06  06:37:45
>> and then below one,
>> 2017/06/06  11:09:28
>> and then this
>> 2017/06/06  12:49:49
>> so on ....
>>
>> How can I achieve the above mentioned scenario ?
>>
>> Regards,
>> Manoj kumar R
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Set priority to files based on date time value stored on attribute

Andre
Koji, 

One could convert date to epoch format which is incremental in nature. Would that help? 

On 8 Jun 2017 19:33, "Koji Kawamura" <[hidden email]> wrote:
Hi Manoj,

I think EnforceOrder would not be useful in your case, as it expects
the order to increases one by one (without skip).
As Pierre suggested, I'd suggest using PriorityAttributePrioritizer.

Thanks,
Koji

On Thu, Jun 8, 2017 at 3:50 PM, Pierre Villard
<[hidden email]> wrote:
> Hi Manoj,
>
> You may want ot have a look at EnforceOrder processor [1] or simply the
> prioritizers [2] of the connections (it depends of how your workflow is
> working). The idea would be to extract the date as an attribute of your flow
> file, convert into an integer (using expression language) and use it to
> ensure order.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization
>
> Hope this helps.
>
>
> 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <[hidden email]>:
>>
>> Hi All,
>>
>> I need to process the files based on the date time value stored on the
>> attribute
>>
>> For example:
>>
>> If the incoming files contains the following date time attribute values
>>
>> 2017/06/07  16:57:02
>> 2017/06/06  12:49:49
>> 2017/06/06  11:09:28
>> 2017/06/06  06:37:45
>>
>> I need to process the files based on the order of time that is oldest one
>> from the current time
>>
>> First I want to access the file that contains below date time attribute
>> which is the oldest one among them from the current time
>> i.e 2017/06/06  06:37:45
>> and then below one,
>> 2017/06/06  11:09:28
>> and then this
>> 2017/06/06  12:49:49
>> so on ....
>>
>> How can I achieve the above mentioned scenario ?
>>
>> Regards,
>> Manoj kumar R
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Set priority to files based on date time value stored on attribute

Manojkumar Ravichandran
In reply to this post by Pierre Villard
Hi Pierre,


After converting those date time format in to integer (using expression language),I can able to process the file as per the requirement
by setting those integer values to the priority attribute and process those files based on that priority.

Thanks for your guidance


Regards,
Manoj kumar R 


On Thu, Jun 8, 2017 at 8:50 AM, Pierre Villard <[hidden email]> wrote:
Hi Manoj,

You may want ot have a look at EnforceOrder processor [1] or simply the prioritizers [2] of the connections (it depends of how your workflow is working). The idea would be to extract the date as an attribute of your flow file, convert into an integer (using expression language) and use it to ensure order.

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
[2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization

Hope this helps.


2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <[hidden email]>:
Hi All,

I need to process the files based on the date time value stored on the attribute

For example:

If the incoming files contains the following date time attribute values

2017/06/07  16:57:02
2017/06/06  12:49:49
2017/06/06  11:09:28
2017/06/06  06:37:45

I need to process the files based on the order of time that is oldest one from the current time 

First I want to access the file that contains below date time attribute which is the oldest one among them from the current time 
i.e 2017/06/06  06:37:45
and then below one,
2017/06/06  11:09:28
and then this
2017/06/06  12:49:49
so on .... 

How can I achieve the above mentioned scenario ?

Regards,
Manoj kumar R


Reply | Threaded
Open this post in threaded view
|

Re: Set priority to files based on date time value stored on attribute

Koji Kawamura
Hi Andre,

Epoch format is indeed incremental, but EnforceOrder needs every
number without skip.
Unless the use-case guarantees incoming FlowFiles has a perfect set of
incremental numbers in successful flow, EnforceOrder is not helpful.
E.g. 1, 5, 4, 2, 3 can be enforced by EnforceOrder
but 100, 500, 400, 200, 300 can NOT. But PriorityAttributePrioritizer
can sort these FlowFiles.

However, a caveat is, PriorityAttributePrioritizer does not wait if
some incoming FlowFiles are delayed.
If 1, 5 are queued when a processor is scheduled,
PriorityAttributePrioritizer picks 1 and 5. It does not care missing
numbers.

On the other hand, EnforceOrder does wait.
If 1, 5 are queued, EnforceOrder transfers 1, then keep 5 wait.
Then if 4, 2, 3 arrive, EnforceOrder transfers 2, 3, 4, then 5

Thanks,
Koji

On Thu, Jun 8, 2017 at 9:28 PM, Manojkumar Ravichandran
<[hidden email]> wrote:

> Hi Pierre,
>
>
> After converting those date time format in to integer (using expression
> language),I can able to process the file as per the requirement
> by setting those integer values to the priority attribute and process those
> files based on that priority.
>
> Thanks for your guidance
>
>
> Regards,
> Manoj kumar R
>
>
> On Thu, Jun 8, 2017 at 8:50 AM, Pierre Villard <[hidden email]>
> wrote:
>>
>> Hi Manoj,
>>
>> You may want ot have a look at EnforceOrder processor [1] or simply the
>> prioritizers [2] of the connections (it depends of how your workflow is
>> working). The idea would be to extract the date as an attribute of your flow
>> file, convert into an integer (using expression language) and use it to
>> ensure order.
>>
>> [1]
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
>> [2]
>> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization
>>
>> Hope this helps.
>>
>>
>> 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran
>> <[hidden email]>:
>>>
>>> Hi All,
>>>
>>> I need to process the files based on the date time value stored on the
>>> attribute
>>>
>>> For example:
>>>
>>> If the incoming files contains the following date time attribute values
>>>
>>> 2017/06/07  16:57:02
>>> 2017/06/06  12:49:49
>>> 2017/06/06  11:09:28
>>> 2017/06/06  06:37:45
>>>
>>> I need to process the files based on the order of time that is oldest one
>>> from the current time
>>>
>>> First I want to access the file that contains below date time attribute
>>> which is the oldest one among them from the current time
>>> i.e 2017/06/06  06:37:45
>>> and then below one,
>>> 2017/06/06  11:09:28
>>> and then this
>>> 2017/06/06  12:49:49
>>> so on ....
>>>
>>> How can I achieve the above mentioned scenario ?
>>>
>>> Regards,
>>> Manoj kumar R
>>
>>
>