MINIFI-350 CMake target for docker integration tests

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

MINIFI-350 CMake target for docker integration tests

Andy Christianson
MiNiFi cpp team,


I am currently working on MINIFI-350. I have an integration test framework plus a few integration tests which are invoked via a new script called docker/DockerVerify.sh (https://github.com/achristianson/nifi-minifi-cpp/blob/MINIFI-350/docker/DockerVerify.sh?). This is intended to be consistent in naming and structure to the extant DockerBuild.sh.


My question resides in a final bit of integration. I see that there is a custom CMake target in cmake/DockerConfig.cmake that calls DockerBuild.sh. It seems like the best thing to integrate DockerVerify.sh is to add a sibling CMake custom target, perhaps called something like 'docker-verify.'


A few stylistic/structural questions come to mind. Do we want these integration tests to be invoked directly via a custom target, or do we want that custom target to be an intermediate target which is called by some existing 'make test' or 'make verify' target? The biggest risk, from what I can see, is that if we hooked into 'make test,' then it would fail if the user hadn't already run 'make docker.'


Please advise on preferred style and structure with regard to integration of the new docker integration tests with CMake.


-Andy I.C.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: MINIFI-350 CMake target for docker integration tests

Marc
Andy,
  This is great stuff. To facilitate use by developers we should probably
come to agreement on terminology. In my opinion, it seems that you are
facilitating more system integration testing (SIT) based on our
discussions. While what you are doing doesn't preclude you from integrating
one or two components it does cause confusion on what to use when I want to
test something. I think the same exists for our unit and integration tests
that we have now, but those are a function of design decisions made vice a
focus on testability. I think most of the tests that use the docker
framework will be system tests, and the importance of this is that we
should probably isolate these to a separate target. I think better
isolation of what these test facilities do will end users, and limiting
what ctest executes to pure unit and modular integration tests will be very
useful.
   I think that also helps sell the story that one must install docker if
they want to run system integration tests, but you can still integrate
several components with ctest if you want to limit what you are testing. I
personally look forward to these changes as I think having multi host tests
will help find more bugs, but I wouldn't want this to run when I type make
test.

On Wed, Aug 9, 2017 at 12:39 PM, Andy Christianson <
[hidden email]> wrote:

> MiNiFi cpp team,
>
>
> I am currently working on MINIFI-350. I have an integration test framework
> plus a few integration tests which are invoked via a new script called
> docker/DockerVerify.sh (https://github.com/achristianson/nifi-minifi-cpp/
> blob/MINIFI-350/docker/DockerVerify.sh?). This is intended to be
> consistent in naming and structure to the extant DockerBuild.sh.
>
>
> My question resides in a final bit of integration. I see that there is a
> custom CMake target in cmake/DockerConfig.cmake that calls DockerBuild.sh.
> It seems like the best thing to integrate DockerVerify.sh is to add a
> sibling CMake custom target, perhaps called something like 'docker-verify.'
>
>
> A few stylistic/structural questions come to mind. Do we want these
> integration tests to be invoked directly via a custom target, or do we want
> that custom target to be an intermediate target which is called by some
> existing 'make test' or 'make verify' target? The biggest risk, from what I
> can see, is that if we hooked into 'make test,' then it would fail if the
> user hadn't already run 'make docker.'
>
>
> Please advise on preferred style and structure with regard to integration
> of the new docker integration tests with CMake.
>
>
> -Andy I.C.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: MINIFI-350 CMake target for docker integration tests

Andy Christianson
These are good points. Having a separate target makes sense, plus it'll reduce risk of interfering with the existing native development workflow.

What shall we call the new target? Some possibilities:

- make sip (system integration tests)
- make docker-verify
- make verify-docker
- make integration

I'm open to ideas.

-Andy I.C.
________________________________________
From: Marc <[hidden email]>
Sent: Wednesday, August 09, 2017 1:16 PM
To: [hidden email]
Subject: Re: MINIFI-350 CMake target for docker integration tests

Andy,
  This is great stuff. To facilitate use by developers we should probably
come to agreement on terminology. In my opinion, it seems that you are
facilitating more system integration testing (SIT) based on our
discussions. While what you are doing doesn't preclude you from integrating
one or two components it does cause confusion on what to use when I want to
test something. I think the same exists for our unit and integration tests
that we have now, but those are a function of design decisions made vice a
focus on testability. I think most of the tests that use the docker
framework will be system tests, and the importance of this is that we
should probably isolate these to a separate target. I think better
isolation of what these test facilities do will end users, and limiting
what ctest executes to pure unit and modular integration tests will be very
useful.
   I think that also helps sell the story that one must install docker if
they want to run system integration tests, but you can still integrate
several components with ctest if you want to limit what you are testing. I
personally look forward to these changes as I think having multi host tests
will help find more bugs, but I wouldn't want this to run when I type make
test.

On Wed, Aug 9, 2017 at 12:39 PM, Andy Christianson <
[hidden email]> wrote:

> MiNiFi cpp team,
>
>
> I am currently working on MINIFI-350. I have an integration test framework
> plus a few integration tests which are invoked via a new script called
> docker/DockerVerify.sh (https://github.com/achristianson/nifi-minifi-cpp/
> blob/MINIFI-350/docker/DockerVerify.sh?). This is intended to be
> consistent in naming and structure to the extant DockerBuild.sh.
>
>
> My question resides in a final bit of integration. I see that there is a
> custom CMake target in cmake/DockerConfig.cmake that calls DockerBuild.sh.
> It seems like the best thing to integrate DockerVerify.sh is to add a
> sibling CMake custom target, perhaps called something like 'docker-verify.'
>
>
> A few stylistic/structural questions come to mind. Do we want these
> integration tests to be invoked directly via a custom target, or do we want
> that custom target to be an intermediate target which is called by some
> existing 'make test' or 'make verify' target? The biggest risk, from what I
> can see, is that if we hooked into 'make test,' then it would fail if the
> user hadn't already run 'make docker.'
>
>
> Please advise on preferred style and structure with regard to integration
> of the new docker integration tests with CMake.
>
>
> -Andy I.C.
>


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: MINIFI-350 CMake target for docker integration tests

Aldrin Piri
Yeah, my vote would definitely be to have the separate targets.  I like
having a lot of options for testing but definitely like being able to
minimize needed dependencies appropriate to desired level of evaluation.

In terms of naming, would likely avoid calling these integration given
their special nature, but otherwise have no strong preferences.

On Wed, Aug 9, 2017 at 1:42 PM, Andy Christianson <
[hidden email]> wrote:

> These are good points. Having a separate target makes sense, plus it'll
> reduce risk of interfering with the existing native development workflow.
>
> What shall we call the new target? Some possibilities:
>
> - make sip (system integration tests)
> - make docker-verify
> - make verify-docker
> - make integration
>
> I'm open to ideas.
>
> -Andy I.C.
> ________________________________________
> From: Marc <[hidden email]>
> Sent: Wednesday, August 09, 2017 1:16 PM
> To: [hidden email]
> Subject: Re: MINIFI-350 CMake target for docker integration tests
>
> Andy,
>   This is great stuff. To facilitate use by developers we should probably
> come to agreement on terminology. In my opinion, it seems that you are
> facilitating more system integration testing (SIT) based on our
> discussions. While what you are doing doesn't preclude you from integrating
> one or two components it does cause confusion on what to use when I want to
> test something. I think the same exists for our unit and integration tests
> that we have now, but those are a function of design decisions made vice a
> focus on testability. I think most of the tests that use the docker
> framework will be system tests, and the importance of this is that we
> should probably isolate these to a separate target. I think better
> isolation of what these test facilities do will end users, and limiting
> what ctest executes to pure unit and modular integration tests will be very
> useful.
>    I think that also helps sell the story that one must install docker if
> they want to run system integration tests, but you can still integrate
> several components with ctest if you want to limit what you are testing. I
> personally look forward to these changes as I think having multi host tests
> will help find more bugs, but I wouldn't want this to run when I type make
> test.
>
> On Wed, Aug 9, 2017 at 12:39 PM, Andy Christianson <
> [hidden email]> wrote:
>
> > MiNiFi cpp team,
> >
> >
> > I am currently working on MINIFI-350. I have an integration test
> framework
> > plus a few integration tests which are invoked via a new script called
> > docker/DockerVerify.sh (https://github.com/achristian
> son/nifi-minifi-cpp/
> > blob/MINIFI-350/docker/DockerVerify.sh?). This is intended to be
> > consistent in naming and structure to the extant DockerBuild.sh.
> >
> >
> > My question resides in a final bit of integration. I see that there is a
> > custom CMake target in cmake/DockerConfig.cmake that calls
> DockerBuild.sh.
> > It seems like the best thing to integrate DockerVerify.sh is to add a
> > sibling CMake custom target, perhaps called something like
> 'docker-verify.'
> >
> >
> > A few stylistic/structural questions come to mind. Do we want these
> > integration tests to be invoked directly via a custom target, or do we
> want
> > that custom target to be an intermediate target which is called by some
> > existing 'make test' or 'make verify' target? The biggest risk, from
> what I
> > can see, is that if we hooked into 'make test,' then it would fail if the
> > user hadn't already run 'make docker.'
> >
> >
> > Please advise on preferred style and structure with regard to integration
> > of the new docker integration tests with CMake.
> >
> >
> > -Andy I.C.
> >
>
>
>
Loading...