View Issue Details

IDProjectCategoryView StatusLast Update
0000015KafkaEsqueFeature Requestpublic2019-02-15 23:48
ReporterOliver G.Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0000015: Analytical Message Consuming
Description

Architects, analysts, developers or testers are sometimes interested in some statistical overview of the data consumed from a topic.
Consider introducing a simple analysis feature based on some predicates and calculating percentages.

For example:
1., there is topic orderCompleted
2., the following messages are published to the topic :
{
"orderNr": "abc/12",
"completedAt": "2018-05-01T10:00:00.000Z"
}
3., User defines predicates for e.g. quarters of the year based on the timestamp.
4., User starts consuming from a topic, either print the result live or gather data first and calculate result in a post-processing.
Then print something like (numbers are random in the example):
Q1 140.000 23,76%
Q2 150.000 26,24%
...
All 700.000 100%

Do NOT aim to implement an OLAP Engine / Cube based analysis tool, this is out of scope of KafkaEsque. Only provide possibility for simple analyis like above.
Instead of providing detailed technical descriptions or even patches to be apply, the goal of this feature request is to be a basis for further discussions and to gather ideas.
The following questions need to be answered:
1., Is this a useful feature or is there another simple way to gather this information? KSQL, "quickly" write a Streams App, other tools?
2., What would the syntax of the predicates be like?

TagsNo tags attached.

Activities

Oliver G.

Oliver G.

2018-12-05 23:19

reporter   ~0000015

I have created a (little weird) GIST to show an idea with Kotlin Scripting.
This is obviously not a ready-to-use solution, but maybe it helps to bring the original idea of the ticket to life.
However, the code in the gist is executable on my machine.

https://gist.github.com/grofoli/ac795868f24190d5698de8b33909dc90

If the solution will be really based on "JSR-000223 Scripting for the JavaTM Platform", we must check what they support in Java 9+.

Oliver G.

Oliver G.

2019-02-15 23:23

reporter   ~0000023

Old article, but maybe interesting for this tasks: https://bytes.com/topic/python/insights/949995-three-ways-run-python-programs-java

Oliver G.

Oliver G.

2019-02-15 23:48

reporter   ~0000024

Another thing to check is Groovy: http://docs.groovy-lang.org/latest/html/documentation/guide-integrating.html.
Groovy is definitely more alive, than Jython.

Issue History

Date Modified Username Field Change
2018-11-29 21:20 Oliver G. New Issue
2018-12-05 23:19 Oliver G. Note Added: 0000015
2019-02-15 23:23 Oliver G. Note Added: 0000023
2019-02-15 23:48 Oliver G. Note Added: 0000024