KafkaEsque Mantis - Issues

0000020: Show topics statistics

Thu, 02 Feb 2023 15:02:34 +0000

Add option to show statistics for a topic.
The attached code allows to sum up the offsets per day and calculate the differences.

Functionality would be useful per topic or as a report (csv-based or other), although it takes a while in the current form for a larger number of topics.

0000022: Separate Responsibilities of the Controller

Fri, 24 May 2019 16:56:41 +0000

The Controller class has too many responsibilities and it would be desirable to separate these responsibilities into separate classes:

1., Configuration Management (configHandler): inject the configHandler singleton whereever it is needed and try to remove/extract those parts of the code that have Feature Envy (e.g. when configHandler is used in connection with TopicMessageTypeConfig):
Map configs = configHandler.getTopicConfigForClusterIdentifier(selectedCluster().getIdentifier());
TopicMessageTypeConfig topicMessageTypeConfig = getTopicMessageTypeConfig(configs); //configHandler is also used inside this method
Map consumerConfig = configHandler.readConsumerConfigs(selectedCluster().getIdentifier());

2., Kafka Service Layer: e.g. Number of Runnables executed by runInDaemonThread need only singletons like consumerHandlers injected, but probably do not need to be part of the JavaFx Controller

3., EventHandlers passed as parameter in setOnAction can be very likely extracted. Be aware that these EventHandlers will also benefit from 1., and 2.,

I will give prio high to this issue, since it can dramatically increase the speed of future develepmoent tasks.

0000012: Improve Configuration Handling

Mon, 06 May 2019 06:22:17 +0000

Introduce a ConfigurationManager that is able to handle the following szenarios properly:
* Adding a new cluster
* Removing a Cluster
* Adding a schema-registry url to an existing cluster
* Adding Topic specific configuration (needed to enable avro consumers/producers)

0000019: In the TraceInputDialog the TextField for the key should be bigger

Mon, 06 May 2019 06:21:52 +0000

In order to improve the usability when tracing a key, make the TextField for the key larger.
Reason: the key can be a JSON like
{"transactionId":"8929caf8-de81-447f-823e-c9ef39522d9a", superSecretCode:"eventDrivenStream.v28"}
which is not easy to edit in the current TextField.

0000023: Integration Tests with test containers

Sat, 06 Apr 2019 22:17:29 +0000

As an alternative to EmbeddedKafka, you can let the testcontainers rule start the confluent docker containers.
https://www.testcontainers.org/modules/kafka/

Due to the fact that Docker is a requirement, integration tests should be a separate folder and/or category and must be excluded from default build.

0000008: NPE Cleanup

Mon, 25 Feb 2019 12:09:55 +0000

Fix NPEs when clicking on buttons without having made the required selections before.
Preferably disable control if the required selection has not been made.

0000015: Analytical Message Consuming

Fri, 15 Feb 2019 23:48:31 +0000

Architects, analysts, developers or testers are sometimes interested in some statistical overview of the data consumed from a topic.
Consider introducing a simple analysis feature based on some predicates and calculating percentages.

For example:
1., there is topic orderCompleted
2., the following messages are published to the topic :
{
"orderNr": "abc/12",
"completedAt": "2018-05-01T10:00:00.000Z"
}
3., User defines predicates for e.g. quarters of the year based on the timestamp.
4., User starts consuming from a topic, either print the result live or gather data first and calculate result in a post-processing.
Then print something like (numbers are random in the example):
Q1 140.000 23,76%
Q2 150.000 26,24%
...
All 700.000 100%

Do NOT aim to implement an OLAP Engine / Cube based analysis tool, this is out of scope of KafkaEsque. Only provide possibility for simple analyis like above.
Instead of providing detailed technical descriptions or even patches to be apply, the goal of this feature request is to be a basis for further discussions and to gather ideas.
The following questions need to be answered:
1., Is this a useful feature or is there another simple way to gather this information? KSQL, "quickly" write a Streams App, other tools?
2., What would the syntax of the predicates be like?

0000018: Open in Editor Patch

Fri, 18 Jan 2019 17:12:04 +0000

Patch to add open in txt/json editor functionality in message list context menu.

Currently throws exception if no default program is set for the file type you are trying to open (.txt or .json), should have popup or check

0000017: Json Tree View Patch

Sun, 16 Dec 2018 12:23:22 +0000

Patch for Json Tree View when using Format as Json.

Needs work to deal with nested ArrayLists/Maps and to clean up unsafe casts.

0000014: Add option for trace start position

Thu, 13 Dec 2018 18:53:23 +0000

Introduce an option to either trace from a specific offset or a specific timestamp.

Notes:
Using offsetsForTimes should work for this (could also make polling messages in a specified timeframe possible)

0000011: Fix consumer indefinite blocking behaviour

Thu, 13 Dec 2018 18:53:23 +0000

Use poll(Duration) instead of poll(long), but be aware to add some timeout instead of 0 (long).

https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior

0000016: Dependency Injection with Guice (or Spring)

Wed, 05 Dec 2018 19:01:18 +0000

(Spike :-) )
Analyse if you want to use a Dependency Injection Framework in KafkaEsque.
Consider checking Guice, maybe it is a good fit to JavaFX.

Simple sample containing a link to a short blog entry of the author
https://github.com/Pscheidl/JavaFX8DependencyInjection

Although German Wiki Page shows old version and some resources are from 2016, Guice seems to be alive and had a new version one month ago:
https://mvnrepository.com/artifact/com.google.inject/guice/4.2.2

When comparing to Spring, I recommend checking a number of aspects also listed in the following article, e.g. startup time, dependency size, compile time help:
https://dzone.com/articles/an-opinionless-comparison-of-spring-and-guice

0000010: WindowsEmbeddedKafka Patch

Tue, 27 Nov 2018 20:57:10 +0000

Patch ist im Anhang.

* Broker Port ist Fix und nicht mehr random.
* Hacks, damit Kafka Ordner unter Windows gelöscht werden.
* Tempus Fugit mit Test Scope.

0000013: Key Search für Partition

Mon, 26 Nov 2018 14:21:36 +0000

Limit searching for records by key to the partition the records belong to.
Basic code see below.

0000009: Improve Error Alert for JSON Validation Errors

Mon, 26 Nov 2018 12:56:57 +0000

Show Message instead of RuntimeException and StackTrace.
Patch with the a possible solution will be attached.

0000006: Header Support

Sun, 25 Nov 2018 16:05:11 +0000

Notizen
1., ProducerRecord hat Constructor Iterable

headers, partition muss null sein oder Integer >=0
2., CosumerRecord hat public Headers headers()
3., Konvertierung in KafkaMessage (convertAndAdd) muss angepasst werden
4., (i) Zeigen der Header in TableView fx:id="messageTable" nicht notwendig
5., Darstellung unten neben Key/Value in der TabPane gut möglich, Screenshot vom Prototyp wird angehängt
6., (!) Größte Herausforderung wird das Layout des Publish Message Dialogs. Dort muss eine Liste von Header Key - Header Value Paaren hinzugefügt werden können

0000002: Popup bei JSON Formatierungsfehler

Sun, 25 Nov 2018 15:08:26 +0000

IST:
Passiert ein Fehler bei der JSON Formatierung, kommt ein Popup (ErrorAlert).
Falls z.B. der Key ein String ist, bekommt man beim jeden Record eine Fehlermeldung.

VORSCHLAG:
Es reicht wahrscheinlich, wenn man im key oder value text area einen aussagekräftigen Text setzt (keyTextArea.setText).
Eventuell kann auch den original key oder value ins text area setzen, wenn die Formatierung nicht geklappt hat oder zumindest vor der Error Message angeben.

0000003: NullPointerException wenn man JSON formatieren will, aber kein Record ausgewählt hat

Sun, 25 Nov 2018 15:08:18 +0000

Wenn man sofort nach Start der Applikation auf "Format as JSON" klickt, bekommt man eine NullPointerException.

Ich habe es bei mir lokal mit folgendem Null Check gelöst:
formatJsonToggle.selectedProperty().addListener((observable, oldValue, newValue) -> {
KafkaMessage selectedItem = messageTable.getSelectionModel().getSelectedItem();
if (selectedItem == null) {
return;
}

0000004: Topic Management mit YAML Konfig

Sun, 25 Nov 2018 15:08:12 +0000

Man könnte ein YAML File mit allen Topics einlesen und anlegen. :-)
Für Entwickler oder "unknown Entwickler"

componentName: anyNameSinceItsTheConfigForAllComponents
topics:
- name: abc.abc
config:
cleanup.policy: compact

0000007: Get current topic Config

Sun, 25 Nov 2018 15:08:06 +0000

add an option to get and display the current topic configuration

0000005: Löschen der falschen / obsoleten Broker Configs über die GUI

Sun, 25 Nov 2018 15:08:00 +0000

IST: geht nicht
SOLL: soll gehen :-)
- Prio natürlich low