S3 Cloudtrail log ingestion to Kafka via NiFi

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

S3 Cloudtrail log ingestion to Kafka via NiFi

Laurens Vets
Hello,

Has anyone been able to ingest S3 Cloudtrail logs into Kafka with NiFi?
I got as far ListS3 -> FetchS3Object -> Gunzip, but I'm stuck here. It
seems I'm not actually unzipping the logs, but references to the S3
objects?

Any help would be appreciated.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: S3 Cloudtrail log ingestion to Kafka via NiFi

James Wing
Are you seeing errors, or just unexpected results?  ListS3 only returns references to objects on S3, but FetchS3Object should return the object content.  I recommend looking at the output of FetchS3Object to make sure it is right (in size and content type) before trying to unzip it.

Thanks,

James

On Wed, Jun 7, 2017 at 9:56 AM, Laurens Vets <[hidden email]> wrote:
Hello,

Has anyone been able to ingest S3 Cloudtrail logs into Kafka with NiFi? I got as far ListS3 -> FetchS3Object -> Gunzip, but I'm stuck here. It seems I'm not actually unzipping the logs, but references to the S3 objects?

Any help would be appreciated.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: S3 Cloudtrail log ingestion to Kafka via NiFi

Jeff-2
Hello Laurens,

I set up a flow to test this as well, and also saw that the unzipped json looks like it contains references to other S3Objects.  I'm not familiar with the formats used by CloudTrail or how the actual logging data is stored.  I'll have to read up on it, but I think we can set up a flow to split the json, grab the referenced S3Object values, and route them to another FetchS3Object processor to pull back the actual logs and decompress them.

I'll be away from my computer for most of the night, but will hopefully get back to you tomorrow after doing some more research.

On Wed, Jun 7, 2017 at 1:30 PM James Wing <[hidden email]> wrote:
Are you seeing errors, or just unexpected results?  ListS3 only returns references to objects on S3, but FetchS3Object should return the object content.  I recommend looking at the output of FetchS3Object to make sure it is right (in size and content type) before trying to unzip it.

Thanks,

James

On Wed, Jun 7, 2017 at 9:56 AM, Laurens Vets <[hidden email]> wrote:
Hello,

Has anyone been able to ingest S3 Cloudtrail logs into Kafka with NiFi? I got as far ListS3 -> FetchS3Object -> Gunzip, but I'm stuck here. It seems I'm not actually unzipping the logs, but references to the S3 objects?

Any help would be appreciated.

Loading...