Java agents, Javassist and Byte Buddy

Java Virtual Machine (JVM) is a really great platform, mature, and well-established. Apart from lots of normal features used by all developers, there are some that are more low-level, designed to serve more “system” or “tooling” purposes. One example is sun.misc.Unsafe, which gives e.g. low-level access to memory. Another such feature is agents. Аgents are the JVM mechanism that enables external code to integrate with a running JVM, including access to the bytecode before it’s loaded and executed.

In this post:

  1. The basics of Java agents and Instrumentation API.
  2. An example agent for metrics collection.
  3. Libraries for bytecode manipulation: Javassist and Byte Buddy.

Agents were introduced in Java 1.5, but programming them is rarely a part of JVM programmer’s everyday job. However, having JVM code in production means a high probability of using some agents. I’d like to mention several widely used classes of them:

  • JMX-HTTP bridges, e.g. Jolokia, which gives access to JXM MBeans over HTTP (very useful for monitoring);
  • profilers, e.g. YourKit or JProfiler;
  • debuggers, namely Java Debug Wire Protocol (JDWP) agent;
  • aspect-oriented programming toolkits, AspectJ in particular;
  • hot code reloading tools like JRebel, which are especially useful in Java EE environment.

There are two types of agents in JVM: Java agents and native agents. Java agents are in JVM bytecode (i.e. written in one of JVM languages, most commonly in Java) packed in JARs and follow special code organisation convention so as JVM could use them. Native agents are different: they’re in native code (most commonly compiled from C++) packed in dynamic libraries, use JVM Tool Interface, and operate on a more low level than Java agents. Particularly, they can affect the garbage collector, thread management, locking and synchronisation, etc. Profilers and debuggers use native agents.

In this post, we’re focusing solely on Java agents.

Continue reading

Publishing JARs to Bintray with Maven and SBT

Many developers are interested in making their JVM artifacts (e.g. libraries) available for others. Probably, the simplest way to do this is to publish an artifact to Bintray, which gives free hosting for publicly available artifacts. Also, some teams would like to have private repositories for their shared JARs. In this case, Bintray can help as well.

In this post:

  1. Setting up a Bintray repository.
  2. Publishing JARs of a Maven project to Bintray, including sources, Javadoc and tests.
  3. Publishing JARs of an SBT project to Bintray + sbt-bintray plugin.

All the code used in this project is available in GitHub icon bintray-hello-world Github repository.

Continue reading

About Akka Streams

Excellent post about Akka Streams — Akka Team

Really nice introduction to #Akka Streams! FEEL THE POWER! ^^ — Viktor Klang

A must-read on #Akka #Streams!!! — Ellan Vannin CA

In many computer programs, the whole logic (or a vast part of it) is essentially step-by-step processing of data. Of course, this includes the situation when we iterate over the data and just execute the processing logic on every piece of it. However, there are a couple of complications here:

  • the processing logic may be quite complex, with various aggregations, merging, routing, error recoveries, etc.;
  • we might want to execute steps asynchronously, for instance, to take advantage of multi-processor machines or use I/O;
  • asynchronous execution of data processing steps inherently involves buffering, queues, congestion, and other matter, which are really difficult to handle properly (please, read “Handling Overload” by Fred Hébert).

Therefore, sometimes it is a good idea to express this logic on a high level using some kind of a framework, without a need to implement (possibly complex) mechanics of asynchronous pipeline processing. This was one of the rationales behind frameworks like Apache Camel or Apache Storm.

Actor systems like Erlang or Akka are fairly good for building robust asynchronous data pipelines. However, they are quite low-level by themselves, so writing such pipelines might be tiresome. Newer versions of Akka include the possibility for doing pipeline processing on a quite high level, which is called Akka Streams. Akka Streams grows from Reactive Streams initiative. It implements a streaming interface on top of Akka actor system. In this post I would like to give a short introduction to this library.

You might be interested in an introductory post about Akka in this blog.

We will need a Scala project with two dependencies:

Akka Streams basics

In Akka Streams, we represent data processing in a form of data flow through an arbitrary complex graph of processing stages. Stages have zero or more inputs and zero or more outputs. The basic building blocks are Sources (one output), Sinks (one input) and Flows (one input and one output). Using them, we can build arbitrary long linear pipelines. An example of a stream with just a Source, one Flow and a Sink:
Continue reading

Time-based (version 1) UUIDs ordering in PostgreSQL

The problem

Not so long ago I wrote about a little strange problem with time-based UUIDs I faced (Retrospective time-based UUID generation (with Spark)). This time I needed to do something more usual. As you surely know, UUID standard has several versions. Version 4 is purely random numbers, version 1 relies on the identity of a machine and the timestamp with the counter, etc.

Let’s consider version 1 UUIDs. They include timestamps with counters, so they naturally can be ordered by it. In other words, having two time-based UUIDs, I wanted to say if the first is lower, greater or equal to the second. No big deal, say, in Java: u1.timestamp().compare(u2.timestamp()). But I wanted to do this in PostgreSQL, inside SQL query. Postgresql does have uuid data type, but it provides no functions for comparing them by version 1 timestamps. That is why I decided to write such a function myself.

Version 1 UUID structure

You can find the full UUID specification in RFC 4122. Let’s consider here only the part of version 1 which we are interested in. UUIDs have length of 128 bits, i.e. 16 bytes, which have different meaning in different versions of the standard. Version 1 layout is shown on the picture:

UUID version 1 structure

Continue reading

Type-safe query builders in Scala revisited: shapeless

Not so long ago, I wrote a post about creating type-safe query builders in Scala from scratch. In it, I also suggested using shapeless library to do what was described. Now, I decided to write how it could be done. The code is in this repository.

Problem reminder

Without going into much details, the problem was to provide a type-safe way to build queries (to an abstract database, for instance) with parameters of different types. Something like

Continue reading

Retrospective time-based UUID generation (with Spark)

I have faced a problem: having about 50 Gb of data in one database, export records to another database with slight modifications, which include UUID generation based on timestamps of records (collisions were intolerable). We have Spark cluster, and with it the problem did not seem even a little tricky: create RDD, map it and send to the target DB.

(In this post I am telling about Spark. I have not told about it in this blog so far, but I will. Generally, it is a framework for large-scale data processing, like Hadoop. It operates on abstract distributed data collections called Resilient Distributed Datasets (RDDs). Their interface is quite similar to functional collections (map, flatMap, etc.), and also has some operations specific for Spark’s distributed and large-scale nature.)

However, I was too optimistic – there were difficulties.

Continue reading

Type-safe query builders in Scala

Update 11.01.2016: the next post about type-safe query builders using shapeless.

Recently, I was hacking on a Scala library for queries to Cassandra database, phantom. At the moment, it was not able to build prepared statements, which I needed, so I added this functionality. However, phantom developers also implemented prepared statements quickly :) Nevertheless, I decided to write this post about the core idea of my implementation.

Caution: Don’t do this at home, use shapeless :)

My plan was to be able to prepare queries like this:

and after that to execute this query: query.execute("string", 1, false). Moreover, I wanted this execution to be type-safe, so it was not possible to execute something like query.execute(1, "string", 0).

I will abstract from phantom queries and will focus on the general idea. The code is in this repository.

Continue reading

Linkblog #6

Articles

Projects

  • Futures for C++11 at Facebook – Facebook added futures to their Folly C++ library. Futures is a great abstraction for asynchronous programming (and in fact, it is not the first implementation of them in C++). And the article is interesting.
  • Realtime Webcam Sudoku Solver – this project is old (well, not very old – 2011), but very impressive. You point a webcam on a sudoku field, the program solves it and draws the result directly on the screen. The full process is explained in the article. A great thing to re-implement if you have a couple of weeks.

Podcasts

Sites

  • Papers We Love – a repository of academic computer science papers and a community who loves reading them. They even have meetups for discussion in different cities across the world.
  • DB-Engines – a knowledge base of relational and NoSQL database management systems.
  • How I Start – a collection of personal stories / tutorials of people who took up a new programming languages and done something with it.

Video

And a bit of humour for you →

Linkblog #5

Articles

Talks

  • Testing Distributed Systems w/ Deterministic Simulation by Will Wilson. I was impressed by this talk. The speaker is a Senior Engineer at FoundationDB (Apple now). He talks about how difficult it is to develop a distributed system, how it is more difficult to test it, to reproduce indeterministic bugs etc., and how they contended with this. Spoiler: before writing FoundationDB, they wrote a deterministic simulator of FoundationDB and developed end tested algorithms on it.

Projects

And a bit of humour for you →

Linkblog #4

Articles

Talks

  • One Hacker Way by Erik Meijer – an interesting and a bit controversial expressive talk about the organization of the software development process and software business.

Projects

  • Orleans. It was bound to happen: Microsoft released their actor framework for distributed systems that includes many interesting features such as cluster membership. If your want more meat, here is the paper.
  • Electron. You probably use Atom text editor and you probably know that it is in fact a Node.js + Webkit. Now, Github released Electron, a framework that Atom built with. Now, you can easily write almost-native application with JS.
  • Hubot – another project of Github (not as new as Electron). It is about creating a bot that can join your work chat (in HipChat, Slack, etc; a list of adapters), react on the messages, fulfil your commands and other things. In one of projects I take part, we use it for status checking, metrics monitoring and fun.
  • Motion sensing using the Doppler effect – an amazing idea of using the Doppler effect in order to detect user’s hand movements, in browser. However, I failed to run it :)

And a bit of humour for you →