{"id":5506,"date":"2024-01-29T23:13:27","date_gmt":"2024-01-29T22:13:27","guid":{"rendered":"https:\/\/gpmfactory.com\/?p=5506"},"modified":"2024-02-13T10:42:01","modified_gmt":"2024-02-13T09:42:01","slug":"kafka-rest-proxy-and-oracle-apex","status":"publish","type":"post","link":"https:\/\/gpmfactory.com\/index.php\/2024\/01\/29\/kafka-rest-proxy-and-oracle-apex\/","title":{"rendered":"Kafka REST Proxy and Oracle APEX"},"content":{"rendered":"\n<p><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Goal<\/h3>\n\n\n\n<p>Use the Kafka REST Proxy through an Oracle APEX client application.<br>This Oracle APEX sample App, <a href=\"https:\/\/github.com\/patrickmonaco\/kafka\">available on Github<\/a>, includes a simple KAFKA producer and a KAFKA consumer. It can be used to get more familiar with the produce\/consume process and the <em>commit <\/em>features subtilities.<br>About the sample demo, let&rsquo;s assume a collection of devices spreaded in several cities and the ability given to any operator to insert a manual message in the stream or polling events from a given topic.<br><\/p>\n\n\n\n\n\n<h3 class=\"wp-block-heading\">Prerequisites<\/h3>\n\n\n\n<p>Either install an on-premise <a href=\"https:\/\/kafka.apache.org\/\">Apache KAFKA <\/a>cluster, or use a docker image or subscribe to <a href=\"https:\/\/www.confluent.io\/\" data-type=\"link\" data-id=\"https:\/\/www.confluent.io\/\">Confluent platform.<\/a><\/p>\n\n\n\n<p>If on-premise KAFKA installation, one must install, at least, the <a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/installing_cp\/zip-tar.html\">community version of Confluent REST Proxy<\/a> and setup TLS in order to make calls from a free tiers APEX instance. (In case of apex.oracle.com, it&rsquo;s possible to call a http endpoint instead https) cf <a href=\"https:\/\/gpmfactory.com\/index.php\/2024\/01\/29\/oracle-rules-for-calling-a-web-service\/\" data-type=\"link\" data-id=\"https:\/\/gpmfactory.com\/index.php\/2024\/01\/29\/oracle-rules-for-calling-a-web-service\/\">Oracle rules<\/a>.<\/p>\n\n\n\n<p>The <a href=\"https:\/\/github.com\/patrickmonaco\/kafka\">APEX application is available on github<\/a>. Export has been made with a version 23.2.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Installation of Kafka<\/h3>\n\n\n\n<p>For a single Broker, on a Linux server, follow the links:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/kafka.apache.org\/downloads\" data-type=\"link\" data-id=\"https:\/\/kafka.apache.org\/downloads\">Download Apache Kafka<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kafka.apache.org\/quickstart\">Install Kafka<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/installing_cp\/zip-tar.html\">Install REST Proxy<\/a><\/li>\n\n\n\n<li>Create start\/Stop scripts (cf appendices)<\/li>\n<\/ul>\n\n\n\n<p>About setup TLS for the REST Proxy, read for instance the very good <a href=\"https:\/\/blog.ordina-jworks.io\/security\/2019\/08\/14\/Using-Lets-Encrypt-Certificates-In-Java.html\">post from Ken Coenen<\/a> and get infos related to openssl and keystore and adapt the file etc\/kafka-rest\/kafka-rest.properties.<br>Read <a href=\"https:\/\/docs.confluent.io\/platform\/current\/kafka-rest\/production-deployment\/rest-proxy\/security.html\">REST Proxy Securit<\/a> and adapt  ssl.client.authentication.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Description of application<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"583\" src=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-1-1024x583.png\" alt=\"\" class=\"wp-image-5641\" srcset=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-1-1024x583.png 1024w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-1-300x171.png 300w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-1-768x438.png 768w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-1-1536x875.png 1536w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-1-2048x1167.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>A regular APEX app, named Kafka, relies on a package (KAFKA_PKG) which wraps calls to the REST Proxy. The material is <a href=\"https:\/\/github.com\/patrickmonaco\/kafka\">available from Github<\/a>.<br>Import it in a Oracle APEX instance >= 23.2.<br>During import process, set the REST Proxy endpoint and accept installation of supporting objects<br>Launch Kafka app, jump in <em>setup <\/em>option, check the <em>default consumer name<\/em> (ie: patrick) and the <em>Consumer Group name<\/em> (default: <em>my_json_consumer_group<\/em>).<br>The producer menu option proposes to add only one message or a batch of ten records based on the content of VILLES table. That can be changed in the package KAFKA_PKG.<\/p>\n\n\n\n<p>Sending Messages<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"563\" src=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-4-1024x563.png\" alt=\"\" class=\"wp-image-5645\" srcset=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-4-1024x563.png 1024w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-4-300x165.png 300w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-4-768x422.png 768w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-4-1536x845.png 1536w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-4-2048x1126.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Application offers following features: <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Listing existing topics<\/li>\n\n\n\n<li>Creating\/deleting a consumer instance and subscribing to one topic<\/li>\n\n\n\n<li>Consuming records from an offset.<\/li>\n<\/ul>\n\n\n\n<p>The records page relies on a data source and the other actions are implemented in a dedicated PLSQL package : KAFKA_PKG. This package is embeded as a supporting object in the APEX application.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"568\" src=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-2-1024x568.png\" alt=\"\" class=\"wp-image-5643\" srcset=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-2-1024x568.png 1024w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-2-300x166.png 300w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-2-768x426.png 768w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-2-1536x852.png 1536w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-2-2048x1135.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Notes about the consumer instance<\/h4>\n\n\n\n<p>When creating a new consumer instance in a consumer group, the max iddle session is set at the server side around 4 minutes. That means that we have to poll regularly, otherwise, we must re-create a new consumer instance. The sample application doesn&rsquo;t catch this situation, but there is a page which draws a chart on a regular basis and that prevents a too long idle time. <\/p>\n\n\n\n<p>List of existing Topics<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"470\" src=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-3-1024x470.png\" alt=\"\" class=\"wp-image-5644\" srcset=\"https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-3-1024x470.png 1024w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-3-300x138.png 300w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-3-768x353.png 768w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-3-1536x706.png 1536w, https:\/\/gpmfactory.com\/wp-content\/uploads\/2024\/02\/image-3-2048x941.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Clicking on a topic entry gives the <strong>lags <\/strong>between the last commit point and the last entry.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Appendices<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Scripts for starting and stopping Apache KAFKA<\/h4>\n\n\n\n<p>It&rsquo;s strangely tricky to start Kafka at boot, for obscure reasons of permissions, even as root &#8230;<br>I didnt&rsquo; want to dig in these details, not important in my context.<\/p>\n\n\n\n<p>So I just mention two scripts to launch manually the needed modules. <br>Because I used Kafka with Zookeeper, the first one starts <em>zookeeper<\/em>, then a Kafka server in background.<br>(Another option is to use Kafka with KRaft)<br>The second script launches REST Proxy. We can choose to let it in foreground or as a daemon.<\/p>\n\n\n\n<p>Copied from <a href=\"https:\/\/stackoverflow.com\/questions\/34512287\/how-to-automatically-start-kafka-upon-system-startup-in-ubuntu\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/stackoverflow.com\/questions\/34512287\/how-to-automatically-start-kafka-upon-system-startup-in-ubuntu<\/a><\/p>\n\n\n\n<p>These following scripts are available on the Github repository.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Start\/Stop\/Status Zookeeper and Kafka<\/li>\n\n\n\n<li>Start\/stop\/Status REST Proxy<br><br><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Goal Use the Kafka REST Proxy through an Oracle APEX client application.This Oracle APEX sample App, available on Github, includes a simple KAFKA producer&#8230;<\/p>\n","protected":false},"author":1,"featured_media":5590,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[146,147],"ppma_author":[150],"class_list":["post-5506","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-non-classe","tag-real-time","tag-streaming"],"authors":[{"term_id":150,"user_id":1,"is_guest":0,"slug":"admin8700","display_name":"Patrick","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/209d5ed69b74d288390621ab4c1d3773?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/posts\/5506","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/comments?post=5506"}],"version-history":[{"count":36,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/posts\/5506\/revisions"}],"predecessor-version":[{"id":5651,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/posts\/5506\/revisions\/5651"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/media\/5590"}],"wp:attachment":[{"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/media?parent=5506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/categories?post=5506"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/tags?post=5506"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/gpmfactory.com\/index.php\/wp-json\/wp\/v2\/ppma_author?post=5506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}