Posts on Ivan Yurchenko

Embedding JVM in Rust

Sun, 04 Jan 2026 07:02:00 +0100

Java ecosystem is rich and mature. I won’t be surprised if some application may need to call a piece of Java code. There are several options to do this, starting from the basic multi-processing. In this short post, I want to tell about embedding Java virtual machine (JVM) directly into Rust programs with the jni crate.

In this post:

Embedding JVM into a Rust application.
Calling Java code from Rust and Rust code from Java.

The post assumes you’re familiar with Java and Rust. You can find the code from this post in this repository on Github.

Proptest: property testing in Rust

Sun, 22 Sep 2024 17:35:12 +0300

You probably heard about property testing (AKA property-based testing), a testing technique where test inputs are randomly generated in great quantity and particular desired properties of the code under test are checked. It’s very practical and I’m a big proponent of using it where it makes sense.

In this post, I will tell you how I used property testing with the Proptest library in Rust to ensure the correctness of a bunch of generated serialization/deserialization code for the Apache Kafka protocol.

In this post:

A real-life example of using property testing in Rust with Proptest.
A step-by-step explanation.

The post assumes you’re familiar with Rust. However, to follow it you don’t need any knowledge about Kafka or its protocol. I will explain the problem I was working on briefly, but will not go into much detail.

Introduction

Apache Kafka has a sizeable custom binary protocol that is versioned, with various data types, optional fields, etc. Unfortunately, it doesn’t use a well-known serialization format like Protobuf. The protocol message schema is described in JSON. The actual Java code that does serialization and deserialization is generated from this description¹.

I created a library kafka_wire_protocol. It’s a generated de-/serialization code for the Kafka protocol, but for Rust (and in the future, for Go). The correctness of the generated code is paramount. I applied many testing techniques to it: quick unit tests, integration tests against a real Kafka instance, fuzzying. And also kafka_wire_protocol is a good example of software that can benefit from property testing. About this is going to be this post.

You don’t need to familiarize yourself with kafka_wire_protocol to understand what will be discussed. I will give examples from the code, but they will be self-contained. However, if you’re interested, feel free to check out this repo and follow the instruction in README to run the code generator, compile, and run the tests.

Kafka protocol practical guide

Sun, 08 Sep 2024 11:24:35 +0300

I worked with the Apache Kafka protocol on the low level quite a bit. It wasn’t easy to start doing this following the official guide only and I read the code a lot. With this post, I want to give you a head start by guiding you step by step from primitive values to meaningful requests.

In this post:

Explore the Kafka protocol code and the protocol in action with Wireshark.
Learn how to read and write primitive values.
Combine primitives to perform meaningful requests.

We will use Python as the programming language. However, the code will be zero-dependency and easily portable to the language of your choice.

Java agents, Javassist and Byte Buddy

Sat, 04 Nov 2017 16:57:00 +0300

Java Virtual Machine (JVM) is a really great platform, mature, and well-established. Apart from lots of normal features used by all developers, there are some that are more low-level, designed to serve more “system” or “tooling” purposes. One example is sun.misc.Unsafe, which gives e.g. low-level access to memory. Another such feature is agents. Agents are the JVM mechanism that enables external code to integrate with a running JVM, including access to the bytecode before it’s loaded and executed.

In this post:

The basics of Java agents and Instrumentation API.
An example agent for metrics collection.
Libraries for bytecode manipulation: Javassist and Byte Buddy.

Agents were introduced in Java 1.5, but programming them is rarely a part of JVM programmer’s everyday job. However, having JVM code in production means a high probability of using some agents. I’d like to mention several widely used classes of them:

JMX-HTTP bridges, e.g. Jolokia, which gives access to JXM MBeans over HTTP (very useful for monitoring);
profilers, e.g. YourKit or JProfiler;
debuggers, namely Java Debug Wire Protocol (JDWP) agent;
aspect-oriented programming toolkits, AspectJ in particular;
hot code reloading tools like JRebel, which are especially useful in Java EE environment.

There are two types of agents in JVM: Java agents and native agents. Java agents are in JVM bytecode (i.e. written in one of JVM languages, most commonly in Java) packed in JARs and follow special code organisation convention so as JVM could use them. Native agents are different: they’re in native code (most commonly compiled from C++) packed in dynamic libraries, use JVM Tool Interface, and operate on a more low level than Java agents. Particularly, they can affect the garbage collector, thread management, locking and synchronisation, etc. Profilers and debuggers use native agents.

In this post, we’re focusing solely on Java agents.

Publishing JARs to Bintray with Maven and SBT

Fri, 11 Aug 2017 15:03:00 +0300

Update 26.03.2023: In February 2021, Bintray was shut down. This post still may be useful from the point of view of artifact publishing in Maven and SBT.

Many developers are interested in making their JVM artifacts (e.g. libraries) available for others. Probably, the simplest way to do this is to publish an artifact to Bintray, which gives free hosting for publicly available artifacts. Also, some teams would like to have private repositories for their shared JARs. In this case, Bintray can help as well.

In this post:

Setting up a Bintray repository.
Publishing JARs of a Maven project to Bintray, including sources, Javadoc and tests.
Publishing JARs of an SBT project to Bintray + sbt-bintray plugin.

All the code used in this project is available in bintray-hello-world Github repository.

About Akka Streams

Mon, 12 Dec 2016 00:53:00 +0300

Excellent post about Akka Streams — Akka Team

Really nice introduction to #Akka Streams! FEEL THE POWER! ^^ — Viktor Klang

A must-read on #Akka #Streams!!! — Ellan Vannin CA

In many computer programs, the whole logic (or a vast part of it) is essentially step-by-step processing of data. Of course, this includes the situation when we iterate over the data and just execute the processing logic on every piece of it. However, there are a couple of complications here:

the processing logic may be quite complex, with various aggregations, merging, routing, error recoveries, etc.;
we might want to execute steps asynchronously, for instance, to take advantage of multi-processor machines or use I/O;
asynchronous execution of data processing steps inherently involves buffering, queues, congestion, and other matter, which are really difficult to handle properly (please, read “Handling Overload” by Fred Hébert).

Therefore, sometimes it is a good idea to express this logic on a high level using some kind of a framework, without a need to implement (possibly complex) mechanics of asynchronous pipeline processing. This was one of the rationales behind frameworks like Apache Camel or Apache Storm.

Actor systems like Erlang or Akka are fairly good for building robust asynchronous data pipelines. However, they are quite low-level by themselves, so writing such pipelines might be tiresome. Newer versions of Akka include the possibility for doing pipeline processing on a quite high level, which is called Akka Streams. Akka Streams grows from Reactive Streams initiative. It implements a streaming interface on top of Akka actor system. In this post I would like to give a short introduction to this library.

Time-based (version 1) UUIDs ordering in PostgreSQL

Mon, 28 Mar 2016 11:27:00 +0300

The problem

UUID standard has several versions. Version 4 is purely random numbers, version 1 relies on the identity of a machine and the timestamp with the counter, etc. Let’s consider version 1 UUIDs. They include timestamps with counters, so they naturally can be ordered by it. In other words, having two time-based UUIDs, I wanted to say if the first is lower, greater or equal to the second. No big deal, say, in Java: u1.timestamp().compare(u2.timestamp()). But I wanted to do this in PostgreSQL, inside SQL query. Postgresql does have uuid data type, but it provides no functions for comparing them by version 1 timestamps. That is why I decided to write such a function myself.

Version 1 UUID structure

You can find the full UUID specification in RFC 4122. Let’s consider here only the part of version 1 which we are interested in. UUIDs have length of 128 bits, i.e. 16 bytes, which have different meaning in different versions of the standard. Version 1 layout is shown on the picture:

Type-safe query builders in Scala revisited: shapeless

Mon, 11 Jan 2016 19:15:10 +0300

Not so long ago, I wrote a post about creating type-safe query builders in Scala from scratch. In it, I also suggested using shapeless library to do what was described. Now, I decided to write how it could be done. The code is in this repository.

Problem reminder

Without going into much details, the problem was to provide a type-safe way to build queries (to an abstract database, for instance) with parameters of different types. Something like

val query = beginQuery()
  .addParameter[String]("param1")
  .addParameter[Int]("param2")
  .addParameter[Boolean]("param3")
  .build()

// Compile:
query.execute("some string", 1, false)
// Won't compile:
query.execute(42, true, "another string")

Type-safe query builders in Scala

Sun, 18 Oct 2015 15:12:00 +0300

Update 11.01.2016: the next post about type-safe query builders using shapeless.

Recently, I was hacking on a Scala library for queries to Cassandra database, phantom. At the moment, it was not able to build prepared statements, which I needed, so I added this functionality. However, phantom developers also implemented prepared statements quickly :) Nevertheless, I decided to write this post about the core idea of my implementation.

Caution: Don’t do this at home, use shapeless :)

My plan was to be able to prepare queries like this:

val query = select
  .where(typedParamPlaceholder1)
  .and(typedParamPlaceholder2)
  .and(typedParamPlaceholder3)
  .build()

and after that to execute this query: query.execute("string", 1, false). Moreover, I wanted this execution to be type-safe, so it was not possible to execute something like query.execute(1, "string", 0).

I will abstract from phantom queries and will focus on the general idea. The code is in this repository.

The Bloom filter

Sun, 05 Apr 2015 11:54:00 +0300

In many software engineering problems, we have a set and need to determine if some value belongs to this set. If the possible maximum set cardinality (size; maximum size = total count of elements we consider) is small, the solution is straightforward: just store the set explicitly (for instance, in form of a RB-tree), update it when necessary and check if the set contains elements that we are interested in. But what if maximum set cardinality is large or we need many such sets to operate simultaneously? Or if the set membership test is an expensive operation?

Suppose we want to know if an element belongs to a set. We have decided that it is acceptable to get false positive answers (the answer is “yes”, but the element is not actually in the set) with probability p and not acceptable to get false negative (the answer is “no”, but the element in actually in the set). The data structure that could help us in this situation is called the Bloom filter.

A Bloom filter (proposed by Burton Howard Bloom in 1970) is a bit array of m bits (initially set to 0) and k different hash functions. Each hash function maps a value into a single integer number.

Look at this picture from Wikipedia:

Delayed message delivery in RabbitMQ

Mon, 16 Feb 2015 11:32:00 +0300

UPD June 01, 2015: there is a plugin for this now.

A lot of developers use RabbitMQ message broker. It is quite mature but still lacks for some features that one may need. One of them is delayed message delivery: there is no way to send a message that will be delivered after a specified delay (it’s a limitation of AMQP protocol). Hopefully, there is a hack for this.

Let’s start from dead letters. A message can become “dead” by several reasons, such as rejection or TTL (time to live) expiration. RabbitMQ can deal with such messages by redirecting them to a particular exchange and routing key. We can use this ability to implement delayed delivery. We will create a special queue for holding delayed messages. This queue will not have any subscribers in order for messages to expire. After the expiration, messages will be passed to a destination exchange and routing key, just as planned.

Introduction to Akka

Tue, 27 Jan 2015 01:27:00 +0300

Update 26.03.2023: In September 2022, Lightbend changed the license of Akka from Apache 2.0 to the source-available Business Source License (BSL) 1.1. Akka was forked as Pekko.

There are several models of concurrent computing, the actor model is one of them. I am going to give a glimpse of this model and one of its implementation - Akka toolkit.

The actor model

In the actor model, actors are objects that have state and behavior and communicate to each other by message passing. This sounds like good old objects from OOP, but the crucial difference is that message passing is one-way and asynchronous: an actor sends a message to another actor and continues its work. In fact, actors are totally reactive, all theirs activity is happening as reaction to incoming messages, which are processed one by one. However, it is not a limitation because messages can be of any sort including scheduled messages (by timer) and network messages.

Value Classes in Scala

Sun, 14 Dec 2014 20:51:00 +0300

Type systems and compile-time type checking are great things that can save you a couple of hours of debugging and also have documenting potential, could make the code more understandable. In my opinion, it’s wise to use them, and unfortunately, sometimes we don’t do this enough. Consider Integer/Int/int. A counter could be Integer, an entity identifier could be Integer, an integer number in arithmetic expression could be Integer. In most cases all this Integers have nothing to do with each other: in your domain it is a bad idea to compare them, do arithmetic operations on them, pass one instead of another as a function parameter etc.

In one of my projects (in C#) there are a dozen of domain entities that have integer identifiers that are passed all over the code. After a couple of bugs connected with mixed up identifiers of different entities I’ve solved this problem by replacing plain integer numbers with structs (in C#, structs are value types used for representing lightweight objects such as Point or Color) like Id<EntityName>T (T is to distinct type from property names). The key idea was to introduce a new level of types to let the type checker intently look at the code instead of me. It’s worked: I’ve gotten rid of some old bugs in rarely used parts of code and hope new bugs of such a type won’t bother me in the future. (Aside: I hope, this post will persuade you not only to consider using value classes but also to think about the role of types in code quality).

Why I like Scala

Mon, 24 Nov 2014 17:15:00 +0300

I am familiar (more or less) with a number of programming languages and have both emotional and rational thoughts of them. Scala is for certain in the group of languages I like. I have decided to summarize my judgments of Scala attractive parts in a blog post and here it is. Also, I have got some ideas of posts about Scala and its technology stack and an introduction is possibly needed.

Scala is a general purpose programming language created by Martin Odersky more than ten years ago. It compiles into JVM byte code and interoperable (both direction) with Java (including mixed compilation), which gives Scala an ability to use all this enormous amount of code created for JVM. The interesting property and also one of the strongest selling points of the language is fusion of object-oriented and functional programming paradigms.

Creating a simple parser with ANTLR

Sat, 13 Sep 2014 11:42:00 +0300

Recently, I’ve faced a task of developing a tool which allows the application to have base of (not very complex) logical rules. There were three demands:

The rules were to be written by non-programmers, so using of the languages which the program is written in (Java/Scala), wasn’t very good.
The rule base should be changeable without redeployment of the application, ideally, should be stored in a database.
We should have control on compilation and error emission.

The first and the second demand could be met by developing some kind of Scala- or Groovy-based DSL, extremely simple. But I’ve come with several arguments against:

The third requirement might be hard to meet.
The rules are quite simple, so embedding an interpreter of a general-purpose language might be overkill.
The language which rules consumer is written in might be changed (from Java/Scala to Python e.g.)

So, I’ve decided to write a very simple rule parser/compiler. After I’d created a prototype I decided to write this post, hope it’ll be useful. I say in advance that you can see the code adapted for this post in this repository.

GNU Parallel

Sun, 01 Dec 2013 18:43:00 +0300

How much CPU cores does your computer have? 2-8, I think. It’s very time to use them all, isn’t it? But there are plenty of Unix utils such as grep, find, wc etc., which have no idea about parallel data processing. They can’t split their input into 8 pieces and spawn the corresponding number of threads or processes to process it using all the power of your modern CPU.

Definitely, this problem is quite interesting and practical to rest unsolved. According to the Unix philosophy, 1) it is good for programs to do one thing well; 2) it is a good idea to combine simple programs to do complex things. grep do pattern matching well. How about parallelization?

There is an utility know as GNU Parallel which main purpose is to execute arbitrary jobs in parallel on one or even multiple machines. The program is quite complex and multifunctional, look at man and tutorial. Here, I want just to give a little flavor of it.

Finding a cycle in a linked list

Sun, 24 Nov 2013 18:34:00 +0300

There is a popular task at software developer job interviews: having a singly linked list, write a piece of code which tells if the list has a cycle.

In a linked list, each element is a structure which contains the value of an element and the link to the next element. The next-link of the last element has a special value which marks the end (usually, null). If a list has a cycle, the last element points to some element inside the list.

Z algorithm

Tue, 15 Oct 2013 20:35:00 +0300

Introduction

There are some algorithms of exact substring searching (e.g. Knuth-Morris-Pratt, Boyer-Moore etc.) I want to explain one of them which is called Z algorithm in some sources.

Z-boxes and Z-values

Let’s consider the concept of Z-box. Take the string S = “abcxxxabyyy”. We have an internal part “ab” in the string which repeats its prefix. The internal “ab” is a Z-box. It has the beginning at the position with index 6 and the end in 7 (0-based). Z-boxes are substrings which match string prefixes with the same length. For the Z-box, let’s call the corresponding prefix the prototype (it’s my term but I think it’s OK to use it.)