How to get DBCP service inside ScriptedLookupService

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to get DBCP service inside ScriptedLookupService

Eric Chaves
Hi Folks,

I need to get an instance of DBCPService inside my ScriptedLookupService and for that I'm following Matt's post http://funnifi.blogspot.com.br/2016/04/sql-in-nifi-with-executescript.html 

In my groovy class I've overrided the initialize method and performing the lookup there but I'm getting the following error: 

java.lang.UnsupportedOperationException: Cannot obtain Controller Service Identifiers for service type interface org.apache.nifi.controller.ControllerService without providing a Process Group Identifier


@Override
  void initialize(ControllerServiceInitializationContext context) throws InitializationException {
    log = context.logger
    /* Get sql-connection */
    def lookup = context.controllerServiceLookup
    def dbcpServiceId = lookup.getControllerServiceIdentifiers(ControllerService).find {
      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
    }
    conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
    log.info("sql conn {}", conn)
  }

Is there other way to find service identifiers?

Regards,
Reply | Threaded
Open this post in threaded view
|

Re: How to get DBCP service inside ScriptedLookupService

Matt Burgess-2
Eric,

So I just learned ALOT about the bowels of the context and
initialization framework while digging into this issue, and needless
to say we will need a better way of making this available to scripts.
Here's some info:

1) The ControllerServiceInitializationContext object passed into
initialize() is an anonymous object that passes along the
ScriptedLookupService's context objects, such as the
ControllerServiceLookup.
2) The ControllerServiceLookup interface does not have a method
signature for getControllerServiceIdentifiers(Class, String) to pass
in the process group id.
3) The ControllerServiceLookup object returned by the
ControllerServiceInitializationContext.getControllerServiceLookup()
method is a StandardControllerServiceInitializationContext
4) Note that the context object passed into the initialize() method
and the one returned by context.getControllerServiceLookup() are
different (but both are ControllerServiceInitializationContext impls)
5) The StandardControllerServiceInitializationContext object contains
a private ControllerServiceProvider called serviceProvider of type
StandardControllerServiceProvider, the anonymous context object does
not
6) The StandardControllerServiceInitializationContext object delegates
the getControllerServiceIdentifiers(Class) method to the
serviceProvider
7) serviceProvider (a StandardControllerServiceProvider) does not
allow the call to the getControllerServiceIdentifiers(Class)
signature, and instead throws the error you're seeing
8) None of these objects can get at the process group ID. This is
because they are not associated with a ConfigurableComponent
9) ScriptedLookupService, after calling the script's initialize()
method, will then call the script's onEnabled(ConfigurationContext)
method if it exists. This is currently undocumented [1]
10) The script's onEnabled(ConfigurationContext) method will get a
StandardConfigurationContext object
11) The StandardConfigurationContext object has a private
ConfiguredComponent named component, it is actually a
StandardControllerServiceNode object
12) You can get the process group ID by calling the component's
getProcessGroupIdentifier() method
13) The StandardConfigurationContext object also has a private
ControllerServiceLookup named serviceLookup, it is actually a
StandardControllerServiceProvider object
14) Since we can get a process group ID from #11-12, we can now call
the supported method on the ControllerServiceProvider interface,
namely getControllerServiceIdentifiers(Class, String)
15) Getting at private members (#11 &13) is allowed in Groovy, but
IIRC only works if you don't have a security manager/policies on the
JVM.

TL;DR You can't currently get controller services by name in the
initialize() method, you have to implement onEnabled instead. If you
want to use logging, however, you'll need to save off the logger in
the initialize() method. Here's a working version of onEnabled:

void onEnabled(ConfigurationContext context) {
    lookup = context.serviceLookup
    processGroupId = context.component?.processGroupIdentifier
    /* Get sql-connection */
    def dbcpServiceId =
lookup.getControllerServiceIdentifiers(ControllerService,
processGroupId).find {
      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
    }
    def conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
  }

Hope this helps. I will think some more on how to make everything
fluid and legit -- Mark Payne, could use your help here :)

Regards,
Matt

On Tue, Nov 14, 2017 at 6:13 AM, Eric Chaves <[hidden email]> wrote:

> Hi Folks,
>
> I need to get an instance of DBCPService inside my ScriptedLookupService and
> for that I'm following Matt's post
> http://funnifi.blogspot.com.br/2016/04/sql-in-nifi-with-executescript.html
>
> In my groovy class I've overrided the initialize method and performing the
> lookup there but I'm getting the following error:
>
> java.lang.UnsupportedOperationException: Cannot obtain Controller Service
> Identifiers for service type interface
> org.apache.nifi.controller.ControllerService without providing a Process
> Group Identifier
>
>
> @Override
>   void initialize(ControllerServiceInitializationContext context) throws
> InitializationException {
>     log = context.logger
>     /* Get sql-connection */
>     def lookup = context.controllerServiceLookup
>     def dbcpServiceId =
> lookup.getControllerServiceIdentifiers(ControllerService).find {
>       cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>     }
>     conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>     log.info("sql conn {}", conn)
>   }
>
> Is there other way to find service identifiers?
>
> Regards,
Reply | Threaded
Open this post in threaded view
|

Re: How to get DBCP service inside ScriptedLookupService

Mark Payne
Matt, Eric,

The typical pattern that you would follow for obtaining a Controller Service would be to
return a property that uses the identifiesControllerService() method. For example:

static final PropertyDescriptor MYSQL_CONNECTION_POOL = new PropertyDescriptor.Builder()
.name("Connection Pool")
.identifiesControllerService(DBCPService.class)
.required(true)
.build();

Then, to obtain that controller service, you would access it as:

final DBCPService connectionPoolService = context.getProperty(MYSQL_CONNECTION_POOL).asControllerService(DBCPService.class)

This allows the user to explicitly choose which controller service that they want to use.

Attempting to obtain a Controller Service by name will certainly cause some problems,
as you have already learned :) The API was not intended to work that way, so you see
that doing so can become difficult. There are a few reasons that we don't want to retrieve the service
by name:

1. It would require that the user know that they need to have a service with
that name. Then they would have to leave the configuration of your service, and they would
have to create a service with that name. Plus it would likely not be obvious that they need to do this.
2. The framework would not know that your service is referencing the connection pool service, so if
the connection pool service is disabled, your service would still be valid and the lifecycle management
would not work as intended.
3. Controller Service names are not unique. So you may get the wrong one if there are multiple with
the same name. In fact, over 10 different iterations you could get 10 different services instead of always
getting the same service.

So I guess the question is: Is there a reason that the typical approach of identifying the service in a
Property Descriptor doesn't work for your use case?

Thanks
-Mark





> On Nov 14, 2017, at 12:11 PM, Matt Burgess <[hidden email]> wrote:
>
> Eric,
>
> So I just learned ALOT about the bowels of the context and
> initialization framework while digging into this issue, and needless
> to say we will need a better way of making this available to scripts.
> Here's some info:
>
> 1) The ControllerServiceInitializationContext object passed into
> initialize() is an anonymous object that passes along the
> ScriptedLookupService's context objects, such as the
> ControllerServiceLookup.
> 2) The ControllerServiceLookup interface does not have a method
> signature for getControllerServiceIdentifiers(Class, String) to pass
> in the process group id.
> 3) The ControllerServiceLookup object returned by the
> ControllerServiceInitializationContext.getControllerServiceLookup()
> method is a StandardControllerServiceInitializationContext
> 4) Note that the context object passed into the initialize() method
> and the one returned by context.getControllerServiceLookup() are
> different (but both are ControllerServiceInitializationContext impls)
> 5) The StandardControllerServiceInitializationContext object contains
> a private ControllerServiceProvider called serviceProvider of type
> StandardControllerServiceProvider, the anonymous context object does
> not
> 6) The StandardControllerServiceInitializationContext object delegates
> the getControllerServiceIdentifiers(Class) method to the
> serviceProvider
> 7) serviceProvider (a StandardControllerServiceProvider) does not
> allow the call to the getControllerServiceIdentifiers(Class)
> signature, and instead throws the error you're seeing
> 8) None of these objects can get at the process group ID. This is
> because they are not associated with a ConfigurableComponent
> 9) ScriptedLookupService, after calling the script's initialize()
> method, will then call the script's onEnabled(ConfigurationContext)
> method if it exists. This is currently undocumented [1]
> 10) The script's onEnabled(ConfigurationContext) method will get a
> StandardConfigurationContext object
> 11) The StandardConfigurationContext object has a private
> ConfiguredComponent named component, it is actually a
> StandardControllerServiceNode object
> 12) You can get the process group ID by calling the component's
> getProcessGroupIdentifier() method
> 13) The StandardConfigurationContext object also has a private
> ControllerServiceLookup named serviceLookup, it is actually a
> StandardControllerServiceProvider object
> 14) Since we can get a process group ID from #11-12, we can now call
> the supported method on the ControllerServiceProvider interface,
> namely getControllerServiceIdentifiers(Class, String)
> 15) Getting at private members (#11 &13) is allowed in Groovy, but
> IIRC only works if you don't have a security manager/policies on the
> JVM.
>
> TL;DR You can't currently get controller services by name in the
> initialize() method, you have to implement onEnabled instead. If you
> want to use logging, however, you'll need to save off the logger in
> the initialize() method. Here's a working version of onEnabled:
>
> void onEnabled(ConfigurationContext context) {
>    lookup = context.serviceLookup
>    processGroupId = context.component?.processGroupIdentifier
>    /* Get sql-connection */
>    def dbcpServiceId =
> lookup.getControllerServiceIdentifiers(ControllerService,
> processGroupId).find {
>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>    }
>    def conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>  }
>
> Hope this helps. I will think some more on how to make everything
> fluid and legit -- Mark Payne, could use your help here :)
>
> Regards,
> Matt
>
> On Tue, Nov 14, 2017 at 6:13 AM, Eric Chaves <[hidden email]> wrote:
>> Hi Folks,
>>
>> I need to get an instance of DBCPService inside my ScriptedLookupService and
>> for that I'm following Matt's post
>> http://funnifi.blogspot.com.br/2016/04/sql-in-nifi-with-executescript.html
>>
>> In my groovy class I've overrided the initialize method and performing the
>> lookup there but I'm getting the following error:
>>
>> java.lang.UnsupportedOperationException: Cannot obtain Controller Service
>> Identifiers for service type interface
>> org.apache.nifi.controller.ControllerService without providing a Process
>> Group Identifier
>>
>>
>> @Override
>>  void initialize(ControllerServiceInitializationContext context) throws
>> InitializationException {
>>    log = context.logger
>>    /* Get sql-connection */
>>    def lookup = context.controllerServiceLookup
>>    def dbcpServiceId =
>> lookup.getControllerServiceIdentifiers(ControllerService).find {
>>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>>    }
>>    conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>>    log.info("sql conn {}", conn)
>>  }
>>
>> Is there other way to find service identifiers?
>>
>> Regards,

Reply | Threaded
Open this post in threaded view
|

Re: How to get DBCP service inside ScriptedLookupService

Matt Burgess-2
Mark,

Good point, I forgot the ScriptedLookupService is itself a
ConfigurableComponent and can add its own properties. The original
script from my blog comes from ExecuteScript, where you can't define
your own properties. I was just trying to make that work instead of
thinking about the actual problem, d'oh!

Eric, rather than trying to get at a DBCPConnectionPool defined
elsewhere, you can add a property from your ScriptedLookupService that
is itself a reference to DBCPService. Then the user will see in the
properties a dropdown list of DBCPConnectionPool instances, just like
the other processors that use them (ExecuteSQL, e.g.). Mark outlined
that approach, and it is definitely way better. Sorry for the wild
goose chase, although I guess it was only me that wasted my time :P
Guess it's time to add a new post using this technique instead!

Thanks,
Matt


On Tue, Nov 14, 2017 at 12:43 PM, Mark Payne <[hidden email]> wrote:

> Matt, Eric,
>
> The typical pattern that you would follow for obtaining a Controller Service would be to
> return a property that uses the identifiesControllerService() method. For example:
>
> static final PropertyDescriptor MYSQL_CONNECTION_POOL = new PropertyDescriptor.Builder()
> .name("Connection Pool")
> .identifiesControllerService(DBCPService.class)
> .required(true)
> .build();
>
> Then, to obtain that controller service, you would access it as:
>
> final DBCPService connectionPoolService = context.getProperty(MYSQL_CONNECTION_POOL).asControllerService(DBCPService.class)
>
> This allows the user to explicitly choose which controller service that they want to use.
>
> Attempting to obtain a Controller Service by name will certainly cause some problems,
> as you have already learned :) The API was not intended to work that way, so you see
> that doing so can become difficult. There are a few reasons that we don't want to retrieve the service
> by name:
>
> 1. It would require that the user know that they need to have a service with
> that name. Then they would have to leave the configuration of your service, and they would
> have to create a service with that name. Plus it would likely not be obvious that they need to do this.
> 2. The framework would not know that your service is referencing the connection pool service, so if
> the connection pool service is disabled, your service would still be valid and the lifecycle management
> would not work as intended.
> 3. Controller Service names are not unique. So you may get the wrong one if there are multiple with
> the same name. In fact, over 10 different iterations you could get 10 different services instead of always
> getting the same service.
>
> So I guess the question is: Is there a reason that the typical approach of identifying the service in a
> Property Descriptor doesn't work for your use case?
>
> Thanks
> -Mark
>
>
>
>
>
>> On Nov 14, 2017, at 12:11 PM, Matt Burgess <[hidden email]> wrote:
>>
>> Eric,
>>
>> So I just learned ALOT about the bowels of the context and
>> initialization framework while digging into this issue, and needless
>> to say we will need a better way of making this available to scripts.
>> Here's some info:
>>
>> 1) The ControllerServiceInitializationContext object passed into
>> initialize() is an anonymous object that passes along the
>> ScriptedLookupService's context objects, such as the
>> ControllerServiceLookup.
>> 2) The ControllerServiceLookup interface does not have a method
>> signature for getControllerServiceIdentifiers(Class, String) to pass
>> in the process group id.
>> 3) The ControllerServiceLookup object returned by the
>> ControllerServiceInitializationContext.getControllerServiceLookup()
>> method is a StandardControllerServiceInitializationContext
>> 4) Note that the context object passed into the initialize() method
>> and the one returned by context.getControllerServiceLookup() are
>> different (but both are ControllerServiceInitializationContext impls)
>> 5) The StandardControllerServiceInitializationContext object contains
>> a private ControllerServiceProvider called serviceProvider of type
>> StandardControllerServiceProvider, the anonymous context object does
>> not
>> 6) The StandardControllerServiceInitializationContext object delegates
>> the getControllerServiceIdentifiers(Class) method to the
>> serviceProvider
>> 7) serviceProvider (a StandardControllerServiceProvider) does not
>> allow the call to the getControllerServiceIdentifiers(Class)
>> signature, and instead throws the error you're seeing
>> 8) None of these objects can get at the process group ID. This is
>> because they are not associated with a ConfigurableComponent
>> 9) ScriptedLookupService, after calling the script's initialize()
>> method, will then call the script's onEnabled(ConfigurationContext)
>> method if it exists. This is currently undocumented [1]
>> 10) The script's onEnabled(ConfigurationContext) method will get a
>> StandardConfigurationContext object
>> 11) The StandardConfigurationContext object has a private
>> ConfiguredComponent named component, it is actually a
>> StandardControllerServiceNode object
>> 12) You can get the process group ID by calling the component's
>> getProcessGroupIdentifier() method
>> 13) The StandardConfigurationContext object also has a private
>> ControllerServiceLookup named serviceLookup, it is actually a
>> StandardControllerServiceProvider object
>> 14) Since we can get a process group ID from #11-12, we can now call
>> the supported method on the ControllerServiceProvider interface,
>> namely getControllerServiceIdentifiers(Class, String)
>> 15) Getting at private members (#11 &13) is allowed in Groovy, but
>> IIRC only works if you don't have a security manager/policies on the
>> JVM.
>>
>> TL;DR You can't currently get controller services by name in the
>> initialize() method, you have to implement onEnabled instead. If you
>> want to use logging, however, you'll need to save off the logger in
>> the initialize() method. Here's a working version of onEnabled:
>>
>> void onEnabled(ConfigurationContext context) {
>>    lookup = context.serviceLookup
>>    processGroupId = context.component?.processGroupIdentifier
>>    /* Get sql-connection */
>>    def dbcpServiceId =
>> lookup.getControllerServiceIdentifiers(ControllerService,
>> processGroupId).find {
>>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>>    }
>>    def conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>>  }
>>
>> Hope this helps. I will think some more on how to make everything
>> fluid and legit -- Mark Payne, could use your help here :)
>>
>> Regards,
>> Matt
>>
>> On Tue, Nov 14, 2017 at 6:13 AM, Eric Chaves <[hidden email]> wrote:
>>> Hi Folks,
>>>
>>> I need to get an instance of DBCPService inside my ScriptedLookupService and
>>> for that I'm following Matt's post
>>> http://funnifi.blogspot.com.br/2016/04/sql-in-nifi-with-executescript.html
>>>
>>> In my groovy class I've overrided the initialize method and performing the
>>> lookup there but I'm getting the following error:
>>>
>>> java.lang.UnsupportedOperationException: Cannot obtain Controller Service
>>> Identifiers for service type interface
>>> org.apache.nifi.controller.ControllerService without providing a Process
>>> Group Identifier
>>>
>>>
>>> @Override
>>>  void initialize(ControllerServiceInitializationContext context) throws
>>> InitializationException {
>>>    log = context.logger
>>>    /* Get sql-connection */
>>>    def lookup = context.controllerServiceLookup
>>>    def dbcpServiceId =
>>> lookup.getControllerServiceIdentifiers(ControllerService).find {
>>>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>>>    }
>>>    conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>>>    log.info("sql conn {}", conn)
>>>  }
>>>
>>> Is there other way to find service identifiers?
>>>
>>> Regards,
>
Reply | Threaded
Open this post in threaded view
|

Re: How to get DBCP service inside ScriptedLookupService

Eric Chaves
Matt, Mark, thanks for the great explanations! I'm learning a lot! :)

So I went down the road described but I'm getting another error:  groovy.lang.MissingMethodException: No signature of method: org.apache.nifi.lookup.script.ScriptedLookupService$1.getProperty() is applicable for argument types: (org.apache.nifi.components.PropertyDescriptor) values: [PropertyDescriptor[Database Connection Pool Services]]

Basically I declared a final static PropertyDescriptor DBCP_SERVICE and inside the initialize method I tried to get the DBCPService as outlined. Comparing my code to the QueryDatabaseTable processor I noticed when QDT grabs the DBCPService instance it's context is a ProcessContext while in my GroovyLookupClass's initialize method context is a ControllerServiceInitializationContext so it's seem I'm using the wrong object right? Where perform the call for context.getProperty(DBCP_SERVICE).asControllerService(DBCPService)?


A minor notice for future reference by other users: it took me a while to get the PropertyDescriptor working because declaring it was not enough to make it shown at Properties dialog. I had to enable and than disable the ScriptedProcessor it at least once to have it shown (guessing the code was not executed).

Thanks again for all the support.


2017-11-14 15:52 GMT-02:00 Matt Burgess <[hidden email]>:
Mark,

Good point, I forgot the ScriptedLookupService is itself a
ConfigurableComponent and can add its own properties. The original
script from my blog comes from ExecuteScript, where you can't define
your own properties. I was just trying to make that work instead of
thinking about the actual problem, d'oh!

Eric, rather than trying to get at a DBCPConnectionPool defined
elsewhere, you can add a property from your ScriptedLookupService that
is itself a reference to DBCPService. Then the user will see in the
properties a dropdown list of DBCPConnectionPool instances, just like
the other processors that use them (ExecuteSQL, e.g.). Mark outlined
that approach, and it is definitely way better. Sorry for the wild
goose chase, although I guess it was only me that wasted my time :P
Guess it's time to add a new post using this technique instead!

Thanks,
Matt


On Tue, Nov 14, 2017 at 12:43 PM, Mark Payne <[hidden email]> wrote:
> Matt, Eric,
>
> The typical pattern that you would follow for obtaining a Controller Service would be to
> return a property that uses the identifiesControllerService() method. For example:
>
> static final PropertyDescriptor MYSQL_CONNECTION_POOL = new PropertyDescriptor.Builder()
> .name("Connection Pool")
> .identifiesControllerService(DBCPService.class)
> .required(true)
> .build();
>
> Then, to obtain that controller service, you would access it as:
>
> final DBCPService connectionPoolService = context.getProperty(MYSQL_CONNECTION_POOL).asControllerService(DBCPService.class)
>
> This allows the user to explicitly choose which controller service that they want to use.
>
> Attempting to obtain a Controller Service by name will certainly cause some problems,
> as you have already learned :) The API was not intended to work that way, so you see
> that doing so can become difficult. There are a few reasons that we don't want to retrieve the service
> by name:
>
> 1. It would require that the user know that they need to have a service with
> that name. Then they would have to leave the configuration of your service, and they would
> have to create a service with that name. Plus it would likely not be obvious that they need to do this.
> 2. The framework would not know that your service is referencing the connection pool service, so if
> the connection pool service is disabled, your service would still be valid and the lifecycle management
> would not work as intended.
> 3. Controller Service names are not unique. So you may get the wrong one if there are multiple with
> the same name. In fact, over 10 different iterations you could get 10 different services instead of always
> getting the same service.
>
> So I guess the question is: Is there a reason that the typical approach of identifying the service in a
> Property Descriptor doesn't work for your use case?
>
> Thanks
> -Mark
>
>
>
>
>
>> On Nov 14, 2017, at 12:11 PM, Matt Burgess <[hidden email]> wrote:
>>
>> Eric,
>>
>> So I just learned ALOT about the bowels of the context and
>> initialization framework while digging into this issue, and needless
>> to say we will need a better way of making this available to scripts.
>> Here's some info:
>>
>> 1) The ControllerServiceInitializationContext object passed into
>> initialize() is an anonymous object that passes along the
>> ScriptedLookupService's context objects, such as the
>> ControllerServiceLookup.
>> 2) The ControllerServiceLookup interface does not have a method
>> signature for getControllerServiceIdentifiers(Class, String) to pass
>> in the process group id.
>> 3) The ControllerServiceLookup object returned by the
>> ControllerServiceInitializationContext.getControllerServiceLookup()
>> method is a StandardControllerServiceInitializationContext
>> 4) Note that the context object passed into the initialize() method
>> and the one returned by context.getControllerServiceLookup() are
>> different (but both are ControllerServiceInitializationContext impls)
>> 5) The StandardControllerServiceInitializationContext object contains
>> a private ControllerServiceProvider called serviceProvider of type
>> StandardControllerServiceProvider, the anonymous context object does
>> not
>> 6) The StandardControllerServiceInitializationContext object delegates
>> the getControllerServiceIdentifiers(Class) method to the
>> serviceProvider
>> 7) serviceProvider (a StandardControllerServiceProvider) does not
>> allow the call to the getControllerServiceIdentifiers(Class)
>> signature, and instead throws the error you're seeing
>> 8) None of these objects can get at the process group ID. This is
>> because they are not associated with a ConfigurableComponent
>> 9) ScriptedLookupService, after calling the script's initialize()
>> method, will then call the script's onEnabled(ConfigurationContext)
>> method if it exists. This is currently undocumented [1]
>> 10) The script's onEnabled(ConfigurationContext) method will get a
>> StandardConfigurationContext object
>> 11) The StandardConfigurationContext object has a private
>> ConfiguredComponent named component, it is actually a
>> StandardControllerServiceNode object
>> 12) You can get the process group ID by calling the component's
>> getProcessGroupIdentifier() method
>> 13) The StandardConfigurationContext object also has a private
>> ControllerServiceLookup named serviceLookup, it is actually a
>> StandardControllerServiceProvider object
>> 14) Since we can get a process group ID from #11-12, we can now call
>> the supported method on the ControllerServiceProvider interface,
>> namely getControllerServiceIdentifiers(Class, String)
>> 15) Getting at private members (#11 &13) is allowed in Groovy, but
>> IIRC only works if you don't have a security manager/policies on the
>> JVM.
>>
>> TL;DR You can't currently get controller services by name in the
>> initialize() method, you have to implement onEnabled instead. If you
>> want to use logging, however, you'll need to save off the logger in
>> the initialize() method. Here's a working version of onEnabled:
>>
>> void onEnabled(ConfigurationContext context) {
>>    lookup = context.serviceLookup
>>    processGroupId = context.component?.processGroupIdentifier
>>    /* Get sql-connection */
>>    def dbcpServiceId =
>> lookup.getControllerServiceIdentifiers(ControllerService,
>> processGroupId).find {
>>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>>    }
>>    def conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>>  }
>>
>> Hope this helps. I will think some more on how to make everything
>> fluid and legit -- Mark Payne, could use your help here :)
>>
>> Regards,
>> Matt
>>
>> On Tue, Nov 14, 2017 at 6:13 AM, Eric Chaves <[hidden email]> wrote:
>>> Hi Folks,
>>>
>>> I need to get an instance of DBCPService inside my ScriptedLookupService and
>>> for that I'm following Matt's post
>>> http://funnifi.blogspot.com.br/2016/04/sql-in-nifi-with-executescript.html
>>>
>>> In my groovy class I've overrided the initialize method and performing the
>>> lookup there but I'm getting the following error:
>>>
>>> java.lang.UnsupportedOperationException: Cannot obtain Controller Service
>>> Identifiers for service type interface
>>> org.apache.nifi.controller.ControllerService without providing a Process
>>> Group Identifier
>>>
>>>
>>> @Override
>>>  void initialize(ControllerServiceInitializationContext context) throws
>>> InitializationException {
>>>    log = context.logger
>>>    /* Get sql-connection */
>>>    def lookup = context.controllerServiceLookup
>>>    def dbcpServiceId =
>>> lookup.getControllerServiceIdentifiers(ControllerService).find {
>>>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>>>    }
>>>    conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>>>    log.info("sql conn {}", conn)
>>>  }
>>>
>>> Is there other way to find service identifiers?
>>>
>>> Regards,
>

Reply | Threaded
Open this post in threaded view
|

Re: How to get DBCP service inside ScriptedLookupService

Eric Chaves
Hi Folks, after thinking about my scripted components I decided to ditch the ScriptedLookup in favor of writing a InvokeScriptedProcessor that seems more aligned with the proper use explained so far.

I've implemented the base script as outlined and added some Properties into my InvokeScriptedProcessor but one of them is keeping the processor in an invalid state claiming that the property is invalid because it's not a supported property. I've declared the property using the following code:

class GroovyProcessor implements Processor {

final static PropertyDescriptor LOOKUP_FIELD = new PropertyDescriptor.Builder()
    .name("lookup-field")
    .displayName("Lookup field")
    .description("Field used in lookup queries")
    .dynamic(true) // have tried with both dynamic true/false with same outcome. 
    .defaultValue("id")
    .build()

...}

What is my mistake here? 

Cheers,

Eric

2017-11-15 12:24 GMT-02:00 Eric Chaves <[hidden email]>:
Matt, Mark, thanks for the great explanations! I'm learning a lot! :)

So I went down the road described but I'm getting another error:  groovy.lang.MissingMethodException: No signature of method: org.apache.nifi.lookup.script.ScriptedLookupService$1.getProperty() is applicable for argument types: (org.apache.nifi.components.PropertyDescriptor) values: [PropertyDescriptor[Database Connection Pool Services]]

Basically I declared a final static PropertyDescriptor DBCP_SERVICE and inside the initialize method I tried to get the DBCPService as outlined. Comparing my code to the QueryDatabaseTable processor I noticed when QDT grabs the DBCPService instance it's context is a ProcessContext while in my GroovyLookupClass's initialize method context is a ControllerServiceInitializationContext so it's seem I'm using the wrong object right? Where perform the call for context.getProperty(DBCP_SERVICE).asControllerService(DBCPService)?


A minor notice for future reference by other users: it took me a while to get the PropertyDescriptor working because declaring it was not enough to make it shown at Properties dialog. I had to enable and than disable the ScriptedProcessor it at least once to have it shown (guessing the code was not executed).

Thanks again for all the support.


2017-11-14 15:52 GMT-02:00 Matt Burgess <[hidden email]>:
Mark,

Good point, I forgot the ScriptedLookupService is itself a
ConfigurableComponent and can add its own properties. The original
script from my blog comes from ExecuteScript, where you can't define
your own properties. I was just trying to make that work instead of
thinking about the actual problem, d'oh!

Eric, rather than trying to get at a DBCPConnectionPool defined
elsewhere, you can add a property from your ScriptedLookupService that
is itself a reference to DBCPService. Then the user will see in the
properties a dropdown list of DBCPConnectionPool instances, just like
the other processors that use them (ExecuteSQL, e.g.). Mark outlined
that approach, and it is definitely way better. Sorry for the wild
goose chase, although I guess it was only me that wasted my time :P
Guess it's time to add a new post using this technique instead!

Thanks,
Matt


On Tue, Nov 14, 2017 at 12:43 PM, Mark Payne <[hidden email]> wrote:
> Matt, Eric,
>
> The typical pattern that you would follow for obtaining a Controller Service would be to
> return a property that uses the identifiesControllerService() method. For example:
>
> static final PropertyDescriptor MYSQL_CONNECTION_POOL = new PropertyDescriptor.Builder()
> .name("Connection Pool")
> .identifiesControllerService(DBCPService.class)
> .required(true)
> .build();
>
> Then, to obtain that controller service, you would access it as:
>
> final DBCPService connectionPoolService = context.getProperty(MYSQL_CONNECTION_POOL).asControllerService(DBCPService.class)
>
> This allows the user to explicitly choose which controller service that they want to use.
>
> Attempting to obtain a Controller Service by name will certainly cause some problems,
> as you have already learned :) The API was not intended to work that way, so you see
> that doing so can become difficult. There are a few reasons that we don't want to retrieve the service
> by name:
>
> 1. It would require that the user know that they need to have a service with
> that name. Then they would have to leave the configuration of your service, and they would
> have to create a service with that name. Plus it would likely not be obvious that they need to do this.
> 2. The framework would not know that your service is referencing the connection pool service, so if
> the connection pool service is disabled, your service would still be valid and the lifecycle management
> would not work as intended.
> 3. Controller Service names are not unique. So you may get the wrong one if there are multiple with
> the same name. In fact, over 10 different iterations you could get 10 different services instead of always
> getting the same service.
>
> So I guess the question is: Is there a reason that the typical approach of identifying the service in a
> Property Descriptor doesn't work for your use case?
>
> Thanks
> -Mark
>
>
>
>
>
>> On Nov 14, 2017, at 12:11 PM, Matt Burgess <[hidden email]> wrote:
>>
>> Eric,
>>
>> So I just learned ALOT about the bowels of the context and
>> initialization framework while digging into this issue, and needless
>> to say we will need a better way of making this available to scripts.
>> Here's some info:
>>
>> 1) The ControllerServiceInitializationContext object passed into
>> initialize() is an anonymous object that passes along the
>> ScriptedLookupService's context objects, such as the
>> ControllerServiceLookup.
>> 2) The ControllerServiceLookup interface does not have a method
>> signature for getControllerServiceIdentifiers(Class, String) to pass
>> in the process group id.
>> 3) The ControllerServiceLookup object returned by the
>> ControllerServiceInitializationContext.getControllerServiceLookup()
>> method is a StandardControllerServiceInitializationContext
>> 4) Note that the context object passed into the initialize() method
>> and the one returned by context.getControllerServiceLookup() are
>> different (but both are ControllerServiceInitializationContext impls)
>> 5) The StandardControllerServiceInitializationContext object contains
>> a private ControllerServiceProvider called serviceProvider of type
>> StandardControllerServiceProvider, the anonymous context object does
>> not
>> 6) The StandardControllerServiceInitializationContext object delegates
>> the getControllerServiceIdentifiers(Class) method to the
>> serviceProvider
>> 7) serviceProvider (a StandardControllerServiceProvider) does not
>> allow the call to the getControllerServiceIdentifiers(Class)
>> signature, and instead throws the error you're seeing
>> 8) None of these objects can get at the process group ID. This is
>> because they are not associated with a ConfigurableComponent
>> 9) ScriptedLookupService, after calling the script's initialize()
>> method, will then call the script's onEnabled(ConfigurationContext)
>> method if it exists. This is currently undocumented [1]
>> 10) The script's onEnabled(ConfigurationContext) method will get a
>> StandardConfigurationContext object
>> 11) The StandardConfigurationContext object has a private
>> ConfiguredComponent named component, it is actually a
>> StandardControllerServiceNode object
>> 12) You can get the process group ID by calling the component's
>> getProcessGroupIdentifier() method
>> 13) The StandardConfigurationContext object also has a private
>> ControllerServiceLookup named serviceLookup, it is actually a
>> StandardControllerServiceProvider object
>> 14) Since we can get a process group ID from #11-12, we can now call
>> the supported method on the ControllerServiceProvider interface,
>> namely getControllerServiceIdentifiers(Class, String)
>> 15) Getting at private members (#11 &13) is allowed in Groovy, but
>> IIRC only works if you don't have a security manager/policies on the
>> JVM.
>>
>> TL;DR You can't currently get controller services by name in the
>> initialize() method, you have to implement onEnabled instead. If you
>> want to use logging, however, you'll need to save off the logger in
>> the initialize() method. Here's a working version of onEnabled:
>>
>> void onEnabled(ConfigurationContext context) {
>>    lookup = context.serviceLookup
>>    processGroupId = context.component?.processGroupIdentifier
>>    /* Get sql-connection */
>>    def dbcpServiceId =
>> lookup.getControllerServiceIdentifiers(ControllerService,
>> processGroupId).find {
>>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>>    }
>>    def conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>>  }
>>
>> Hope this helps. I will think some more on how to make everything
>> fluid and legit -- Mark Payne, could use your help here :)
>>
>> Regards,
>> Matt
>>
>> On Tue, Nov 14, 2017 at 6:13 AM, Eric Chaves <[hidden email]> wrote:
>>> Hi Folks,
>>>
>>> I need to get an instance of DBCPService inside my ScriptedLookupService and
>>> for that I'm following Matt's post
>>> http://funnifi.blogspot.com.br/2016/04/sql-in-nifi-with-executescript.html
>>>
>>> In my groovy class I've overrided the initialize method and performing the
>>> lookup there but I'm getting the following error:
>>>
>>> java.lang.UnsupportedOperationException: Cannot obtain Controller Service
>>> Identifiers for service type interface
>>> org.apache.nifi.controller.ControllerService without providing a Process
>>> Group Identifier
>>>
>>>
>>> @Override
>>>  void initialize(ControllerServiceInitializationContext context) throws
>>> InitializationException {
>>>    log = context.logger
>>>    /* Get sql-connection */
>>>    def lookup = context.controllerServiceLookup
>>>    def dbcpServiceId =
>>> lookup.getControllerServiceIdentifiers(ControllerService).find {
>>>      cs -> lookup.getControllerServiceName(cs) == 'MySQLConnectionPool'
>>>    }
>>>    conn = lookup.getControllerService(dbcpServiceId)?.getConnection()
>>>    log.info("sql conn {}", conn)
>>>  }
>>>
>>> Is there other way to find service identifiers?
>>>
>>> Regards,
>