NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

arunodhaya80
Hi,

Very Good morning.

I would like to make an attempt at resolving NIFI-4198
<https://issues.apache.org/jira/browse/NIFI-4198>.  Looking at the code, I
would think by introducing the proxy url and authentication properties and
delegating them to the OkHttpClient would be a good way to do it.

https://stackoverflow.com/questions/35554380/okhttpclient-proxy-authentication-how-to

The question I have is beside the test cases, I would like to test it
against an actual elastic server to be absolutely sure of the fix.  What's
the easiest/good way to run an elastic instance behind a proxy and test the
modified component?

Thanks in advance.

Cheers,
Arun
Reply | Threaded
Open this post in threaded view
|

Re: NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

Joe Witt
Arun

Very cool that you are planning to jump in on this.  Your approach
sounds like a good start.  As far as system testing you're hitting on
one of the more challenging parts of the equation here.  Your unit
tests of course won't integrate with a real ES instance but manual
testing can be done against a system as you mention.  I think a lot of
folks use things like Docker to do such testing or sometimes these
systems offer quick start configurations.  You could put a squid proxy
instance/container in front as well.  Perhaps this one can help
https://github.com/sameersbn/docker-squid

Thanks
Joe

On Sat, Aug 12, 2017 at 11:49 AM, Arun Manivannan <[hidden email]> wrote:

> Hi,
>
> Very Good morning.
>
> I would like to make an attempt at resolving NIFI-4198
> <https://issues.apache.org/jira/browse/NIFI-4198>.  Looking at the code, I
> would think by introducing the proxy url and authentication properties and
> delegating them to the OkHttpClient would be a good way to do it.
>
> https://stackoverflow.com/questions/35554380/okhttpclient-proxy-authentication-how-to
>
> The question I have is beside the test cases, I would like to test it
> against an actual elastic server to be absolutely sure of the fix.  What's
> the easiest/good way to run an elastic instance behind a proxy and test the
> modified component?
>
> Thanks in advance.
>
> Cheers,
> Arun
Reply | Threaded
Open this post in threaded view
|

Re: NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

arunodhaya80
Great.  Thanks a lot, Joe.  Appreciate your help.

Will get it up and running before I assign the issue to myself.

Regards,
Arun

On Sun, Aug 13, 2017 at 3:24 AM Joe Witt <[hidden email]> wrote:

> Arun
>
> Very cool that you are planning to jump in on this.  Your approach
> sounds like a good start.  As far as system testing you're hitting on
> one of the more challenging parts of the equation here.  Your unit
> tests of course won't integrate with a real ES instance but manual
> testing can be done against a system as you mention.  I think a lot of
> folks use things like Docker to do such testing or sometimes these
> systems offer quick start configurations.  You could put a squid proxy
> instance/container in front as well.  Perhaps this one can help
> https://github.com/sameersbn/docker-squid
>
> Thanks
> Joe
>
> On Sat, Aug 12, 2017 at 11:49 AM, Arun Manivannan <[hidden email]> wrote:
> > Hi,
> >
> > Very Good morning.
> >
> > I would like to make an attempt at resolving NIFI-4198
> > <https://issues.apache.org/jira/browse/NIFI-4198>.  Looking at the
> code, I
> > would think by introducing the proxy url and authentication properties
> and
> > delegating them to the OkHttpClient would be a good way to do it.
> >
> >
> https://stackoverflow.com/questions/35554380/okhttpclient-proxy-authentication-how-to
> >
> > The question I have is beside the test cases, I would like to test it
> > against an actual elastic server to be absolutely sure of the fix.
> What's
> > the easiest/good way to run an elastic instance behind a proxy and test
> the
> > modified component?
> >
> > Thanks in advance.
> >
> > Cheers,
> > Arun
>
Reply | Threaded
Open this post in threaded view
|

Re: NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

Joe Witt
Thanks Arun.  If you can document the steps you went through the setup
a representative testing environment that would *greatly* help
whomever reviews to do the same/similar.  The review pipeline for
contributions like this can be tough and it is almost always due to
helping the reviewer setup the environment.

Thanks

On Sat, Aug 12, 2017 at 12:48 PM, Arun Manivannan <[hidden email]> wrote:

> Great.  Thanks a lot, Joe.  Appreciate your help.
>
> Will get it up and running before I assign the issue to myself.
>
> Regards,
> Arun
>
> On Sun, Aug 13, 2017 at 3:24 AM Joe Witt <[hidden email]> wrote:
>
>> Arun
>>
>> Very cool that you are planning to jump in on this.  Your approach
>> sounds like a good start.  As far as system testing you're hitting on
>> one of the more challenging parts of the equation here.  Your unit
>> tests of course won't integrate with a real ES instance but manual
>> testing can be done against a system as you mention.  I think a lot of
>> folks use things like Docker to do such testing or sometimes these
>> systems offer quick start configurations.  You could put a squid proxy
>> instance/container in front as well.  Perhaps this one can help
>> https://github.com/sameersbn/docker-squid
>>
>> Thanks
>> Joe
>>
>> On Sat, Aug 12, 2017 at 11:49 AM, Arun Manivannan <[hidden email]> wrote:
>> > Hi,
>> >
>> > Very Good morning.
>> >
>> > I would like to make an attempt at resolving NIFI-4198
>> > <https://issues.apache.org/jira/browse/NIFI-4198>.  Looking at the
>> code, I
>> > would think by introducing the proxy url and authentication properties
>> and
>> > delegating them to the OkHttpClient would be a good way to do it.
>> >
>> >
>> https://stackoverflow.com/questions/35554380/okhttpclient-proxy-authentication-how-to
>> >
>> > The question I have is beside the test cases, I would like to test it
>> > against an actual elastic server to be absolutely sure of the fix.
>> What's
>> > the easiest/good way to run an elastic instance behind a proxy and test
>> the
>> > modified component?
>> >
>> > Thanks in advance.
>> >
>> > Cheers,
>> > Arun
>>
Reply | Threaded
Open this post in threaded view
|

Re: NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

arunodhaya80
Absolutely, I will.

Is uploading the steps as a word document to the JIRA an appropriate way? I
understand that these steps are specific to a set of JIRAs and the nifi dev
guide may or may not be a good home for this. (I can look up how to do that
if this is the case).

Thanks again, Joe.
Cheers
Arun

On Sun, Aug 13, 2017, 03:54 Joe Witt <[hidden email]> wrote:

> Thanks Arun.  If you can document the steps you went through the setup
> a representative testing environment that would *greatly* help
> whomever reviews to do the same/similar.  The review pipeline for
> contributions like this can be tough and it is almost always due to
> helping the reviewer setup the environment.
>
> Thanks
>
> On Sat, Aug 12, 2017 at 12:48 PM, Arun Manivannan <[hidden email]> wrote:
> > Great.  Thanks a lot, Joe.  Appreciate your help.
> >
> > Will get it up and running before I assign the issue to myself.
> >
> > Regards,
> > Arun
> >
> > On Sun, Aug 13, 2017 at 3:24 AM Joe Witt <[hidden email]> wrote:
> >
> >> Arun
> >>
> >> Very cool that you are planning to jump in on this.  Your approach
> >> sounds like a good start.  As far as system testing you're hitting on
> >> one of the more challenging parts of the equation here.  Your unit
> >> tests of course won't integrate with a real ES instance but manual
> >> testing can be done against a system as you mention.  I think a lot of
> >> folks use things like Docker to do such testing or sometimes these
> >> systems offer quick start configurations.  You could put a squid proxy
> >> instance/container in front as well.  Perhaps this one can help
> >> https://github.com/sameersbn/docker-squid
> >>
> >> Thanks
> >> Joe
> >>
> >> On Sat, Aug 12, 2017 at 11:49 AM, Arun Manivannan <[hidden email]>
> wrote:
> >> > Hi,
> >> >
> >> > Very Good morning.
> >> >
> >> > I would like to make an attempt at resolving NIFI-4198
> >> > <https://issues.apache.org/jira/browse/NIFI-4198>.  Looking at the
> >> code, I
> >> > would think by introducing the proxy url and authentication properties
> >> and
> >> > delegating them to the OkHttpClient would be a good way to do it.
> >> >
> >> >
> >>
> https://stackoverflow.com/questions/35554380/okhttpclient-proxy-authentication-how-to
> >> >
> >> > The question I have is beside the test cases, I would like to test it
> >> > against an actual elastic server to be absolutely sure of the fix.
> >> What's
> >> > the easiest/good way to run an elastic instance behind a proxy and
> test
> >> the
> >> > modified component?
> >> >
> >> > Thanks in advance.
> >> >
> >> > Cheers,
> >> > Arun
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

arunodhaya80
Hi,

I notice that the AbstractElasticsearchHttpProcessor already has support
for the proxy host and port.  I suppose all I need to do now is to

1. Add the properties to the PropertyDescriptors list
2. Add support for proxy authentication
3. System test.

Can I get away with just this?

Cheers,
Arun



On Sun, Aug 13, 2017 at 9:06 AM Arun Manivannan <[hidden email]> wrote:

> Absolutely, I will.
>
> Is uploading the steps as a word document to the JIRA an appropriate way?
> I understand that these steps are specific to a set of JIRAs and the nifi
> dev guide may or may not be a good home for this. (I can look up how to do
> that if this is the case).
>
> Thanks again, Joe.
> Cheers
> Arun
>
> On Sun, Aug 13, 2017, 03:54 Joe Witt <[hidden email]> wrote:
>
>> Thanks Arun.  If you can document the steps you went through the setup
>> a representative testing environment that would *greatly* help
>> whomever reviews to do the same/similar.  The review pipeline for
>> contributions like this can be tough and it is almost always due to
>> helping the reviewer setup the environment.
>>
>> Thanks
>>
>> On Sat, Aug 12, 2017 at 12:48 PM, Arun Manivannan <[hidden email]>
>> wrote:
>> > Great.  Thanks a lot, Joe.  Appreciate your help.
>> >
>> > Will get it up and running before I assign the issue to myself.
>> >
>> > Regards,
>> > Arun
>> >
>> > On Sun, Aug 13, 2017 at 3:24 AM Joe Witt <[hidden email]> wrote:
>> >
>> >> Arun
>> >>
>> >> Very cool that you are planning to jump in on this.  Your approach
>> >> sounds like a good start.  As far as system testing you're hitting on
>> >> one of the more challenging parts of the equation here.  Your unit
>> >> tests of course won't integrate with a real ES instance but manual
>> >> testing can be done against a system as you mention.  I think a lot of
>> >> folks use things like Docker to do such testing or sometimes these
>> >> systems offer quick start configurations.  You could put a squid proxy
>> >> instance/container in front as well.  Perhaps this one can help
>> >> https://github.com/sameersbn/docker-squid
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Sat, Aug 12, 2017 at 11:49 AM, Arun Manivannan <[hidden email]>
>> wrote:
>> >> > Hi,
>> >> >
>> >> > Very Good morning.
>> >> >
>> >> > I would like to make an attempt at resolving NIFI-4198
>> >> > <https://issues.apache.org/jira/browse/NIFI-4198>.  Looking at the
>> >> code, I
>> >> > would think by introducing the proxy url and authentication
>> properties
>> >> and
>> >> > delegating them to the OkHttpClient would be a good way to do it.
>> >> >
>> >> >
>> >>
>> https://stackoverflow.com/questions/35554380/okhttpclient-proxy-authentication-how-to
>> >> >
>> >> > The question I have is beside the test cases, I would like to test it
>> >> > against an actual elastic server to be absolutely sure of the fix.
>> >> What's
>> >> > the easiest/good way to run an elastic instance behind a proxy and
>> test
>> >> the
>> >> > modified component?
>> >> >
>> >> > Thanks in advance.
>> >> >
>> >> > Cheers,
>> >> > Arun
>> >>
>>
>