"Transaction Safety for Data Streaming to Kafka" is our newsroom entry from May 4, 2020.
Transaction safety should be a given - you would think. That transactional processing was not implemented in Kafka from the beginning is probably due to the original nature of data streaming. The transaction safety in Kafka was only implemented in the C-library librdkafka as of version 1.4.0.
What does transaction safety mean?
Let's deal briefly with the basic information.
tcVISION is based on the processing of completed transactions, so-called "Logical Units of Work" or a LUW.
According to Wikipedia a "database transaction symbolizes a unit of work performed within a database management system (or similar system) against a database, and treated in a coherent and reliable way independent of other transactions. A transaction generally represents any change in a database. “
For a better understanding, let us look at an example.
An online order in a shop leads to a reduction of the product stock, the data of the buyer is recorded and the delivery process is initiated (recording of the delivery data and the invoice data).
All changes that are made during this process may only be accepted and saved as data when everything has been processed completely.
If this is the case, all changes can be committed and the process is complete. If, however, a malfunction occurs during this process and not all necessary data can be entered or processed, the transaction is not completed and any changes made so far must be reversed. This is done by a rollback command.
tcVISION supports the transaction processing both during input (Change Data Capture) and output (APPLY).
Transaction processing is an integral part of database processing and is available on all platforms.
On the mainframe, transaction processing is available for Db2, IMS, Adabas, CA Datacom/DB and CA IDMS.
tcVISION supports transaction processing, no matter if the change data is captured in real time or based on database log files.
In the case of VSAM, tcVISION recognizes a LUW via the DBMS extensions or when processing a VSAM journal file. Only completed - i.e. committed - transactions are processed and replicated by tcVISION.
This is especially important for replications in z/VSE environments, because replication implementations from other vendors do not do this and replicate every change in the source, even if the processing ends – for example - with a program termination. The data sets involved in the synchronization are then no longer synchronous.
VSAM has a special position in z/OS in conjunction with CICS VR. Here, the entire CICS run is regarded as one LUW and any program terminations from CICS are reset internally.
In this case the processing is not based on LUWs. tcVISION supports this kind of processing.
When using Change Data Capture (CDC) and APPLY for databases under Unix/Linux/Windows, the changes are also processed on LUW level by tcVISION.
The transaction safety in Kafka came relatively late. The reasons might be found in the original goal of Kafka as pure data streaming. Data is created and streamed to Kafka immediately.
Only when Kafka became more and more popular in the area of classical operational IT (banks, insurance companies, manufacturing) did the demand for transaction processing arise - at least when data is entered into Kafka.
Kafka transaction processing is available from version 1.4.0 of the C-library librdkafka.
The transaction safety used in tcVISION ensures that either all or none of the messages sent in a transaction arrive. An error during processing will abort the complete transaction, so none of the messages will be transmitted.
This also works when messages are sent to multiple topics within one transaction.
tcVISION plays an important role in data streaming.
tcVISION is the supplier of the data that is created in an online processing on a mainframe system (CICS, IMS/DB, Adabas/NATURAL, CA IDMS) and is captured in real time by tcVISION and streamed into a Big Data environment.
Our tcVISION solution is perfectly suited to connect the traditional mainframe (no matter if the operating system is called z/OS or z/VSE) with a Big Data environment or Cloud.
Practical application experiences are available and the acceptance and demand is high. tcVISION already supports the strategically important Big Data systems and applications.
An overview of all supported input and output targets can be found here.