Concurrency In Modern Programming Languages

Once this happens , we complete the futures causing the replies to be sent. An important component in Saft is the Timer, which schedules election & heartbeat events. When a follower or candidate node doesn’t receive any communication for a period of time, it should schedule an election.

As far as a single node is concerned, especially in our event-driven, actor-like implementation, concurrency is reduced—on purpose—to a minimum. Let’s look at some examples that show the power of virtual threads. Compare the below with Golang’s goroutines or Kotlin’s coroutines. It’s based on en EventLoop pattern, and has various flavors of “transport” mechanisms to open, close, accept, read from and write to sockets. To be more specific, it supports NIO transport (based on what you’ve seen above, but better), native epoll on Linux hosts and native kqueue on BSD hosts.

Virtual threads may be new to Java, but they aren’t new to the JVM. Those who know Clojure or Kotlin probably feel reminded of “coroutines” (and if you’ve heard of Flix, you might think of “processes”). Those are technically very similar and address the same problem. However, there’s at least one small but interesting difference from a developer’s perspective.

Project Loom includes an API for working with continuations, but it’s not meant for application development and is locked away in the jdk.internal.vm package. It’s the low-level construct that makes virtual threads possible. However, those who want to experiment with it have the option, see listing 3. So in a thread-per-request model, the throughput will be limited by the number of OS threads available, which depends on the number of physical cores/threads available on the hardware.

Benchmark Results

In traditional multi-threaded programming , if an application has to perform a complex task, it breaks the program into multiple smaller and independent units of sub-tasks. Then application submits all the tasks to ThreadPoolExecutor, generally with an ExecutorService that runs all tasks and sub-tasks. Seeing these results, the big question is of course whether this unfair scheduling of CPU-intensive threads in Loom is problematic in practice. Ron and Tim have debated this point extensively, and I suggest you check it out for yourself to form an opinion. According to Ron, support for conceding program execution points has been implemented in Loom, but this is not merged into the main thread with the initial drop of Loom.

Instead, you are probably going to decide to use any of the various flavors of Web servers available in the Java ecosystem. I’ve reduced the number of ScheduledThreadPoolExecutor from the last implementation from 3 to 1. The whole simulation lasted less than a minute, like the first implementation. Execute the callback and parsing (see #5) from a thread in the given handlerExecutor pool. Implementers not willing to stream the content to their callers should maintain an internal state aggregate of the data. In this implementation, both make HTTP calls using asyncNonBlockingRequest and must therefore provide aRequestHandler.

Learn More About Java, Multi

This solution resolves all the problems with unstructured concurrency as noted down in the first section. In the benchmarks using drill, the Asynchronous Java version outperformed Rust and was a surprise to me. Since recommendations based on benchmarks are hot topics, I’ll just share my observations, and you can make decisions yourself. The Go, Rust, and Java web server versions blow everything out of the water when it comes to req/second performance. The benchmark will be done ten times for each language with a warmup round, and the mean values will be used. First, I suggest you read the introduction post to understand this post better.

This may be a nice effect to show off, but is probably of little value for the programs we need to write. The attempt in listing 1 to start 10,000 threads will bring most computers to their knees . Attention – possibly the program reaches the thread limit of your operating system, and your computer might actually “freeze”. Or, more likely, the program will crash with an error message like the one below. A native thread in a 64-bit JVM with default settings reserves one megabyte alone for the call stack (the “thread stack size”, which can also be set explicitly with the -Xss option). And if the memory isn’t the limit, the operating system will stop at a few thousand.

From the CPU’s point of view, it would be perfect if exactly one thread ran permanently on each core and was never replaced. We won’t usually be able to achieve this state, since there are other processes running on the server besides the JVM. But “the more, the merrier” doesn’t apply for native threads – you can definitely overdo it. On my machine, the process hung after 14_625_956 virtual threads but didn’t crash, and as memory became available, it kept going slowly. It’s due to the parked virtual threads being garbage collected, and the JVM is able to create more virtual threads and assign them to the underlying platform thread. Starting from where we left in the previous entry, we can say that Asynchronous API are nice because they don’t block the calling thread.

Structured Concurrency

Execute the HTTP request from a thread in the boundedServiceExecutor pool. Use emoji reactions, time-stamped comments, and interactive features to respond to videos and keep your team connected. Another general observation is that Rust was quite consistent in terms of performance across runs while all other languages had some variance, especially when GC kicks in.

On a syntactic level, things are again quite similar, with the same almost mechanical process needed to translate between the two. The situation is a bit different with a method returning UIO vs just NodeRole. However, as far as Raft implementation was concerned, this did not really matter a lot.

Implementing Raft Using Project Loom

If application code encounters a blocking method, Loom will offload the virtual thread from the current carrier to make room for other virtual threads. Virtual threads are cheap and managed by the JVM, meaning that you can have many, if not millions. The beauty of this model is that developers can stick to the familiar per-request thread programming model without running into scaling problems due to the limited number of threads Java Loom available. I highly recommend you read Project Loom’s JEP, which is well written and provides more details and background. It will help in writing more complex and concurrent applications with excellent reliability and fewer thread leaks. But of course, if you want the best possible performance, then Rust clearly seems faster than other languages as it gives you the highest throughput, followed by Java and Golang.

Hence the path to stabilization of the features should be more precise. OS threads are at the core of Java’s concurrency model and have a very mature ecosystem around them, but they also come with some drawbacks and are expensive computationally. Let’s look at the two most common use cases for concurrency and the drawbacks of the current Java concurrency model in these cases. AsynchronousSocketChannel’s read and write operations work the same, but they are already asynchronous. Which means there’s no need to submit the operations to the executor manually. As soon as the request has been sent, we start listening for the answer, so we asynchronously read from the channel for incoming data.

The Unique Selling Point Of Project Loom

Project Loom introduces lightweight threads to the Java platform. Before, each thread created in a Java application corresponded 1-1 to an operating system thread. Loom introduces the notion of a VirtualThread, which is cheap to create and has low execution overhead. https://globalcloudteam.com/ Virtual threads are multiplexed onto a much smaller pool of system threads with efficient context switches. We’ll still use the Scala programming language so that we vary only one component of the implementation, which should make the comparison easier.

• The web server will just serve one endpoint, and it will add a sleep of two seconds on every tenth request.
• With virtual threads on the other hand it’s no problem to start a whole million threads.
• Execute the callback and parsing (see #5) from a thread in the given handlerExecutor pool.
• I kept it as simple as possible without using external dependencies as much as possible.
• Native threads are kicked off the CPU by the operating system, regardless of what they’re doing .
• But none of the differences is significant enough to justify suggesting one approach over another for this particular case.

When we are sure that no more response data remains, we can notify the caller that the call is finished. For each incoming data chunk, we send it to the caller so it can decide what to do (decode, aggregate, batch?, etc). When data comes in, we have to make sure that the asynchronous call made by the caller has not been cancelled. However, we may not be able to write the whole request , so we should continue until we are sure that the request has been sent entirely. We asynchronously establish a connection to the remote address.

Virtual threads are lightweight threads that are not tied to OS threads but are managed by the JVM. They are suitable for thread-per-request programming styles without having the limitations of OS threads. You can create millions of virtual threads without affecting throughput. This is quite similar to coroutines, like goroutines, made famous by the Go programming language . In the context of virtual threads, “channels” are particularly worth mentioning here.

From Zio To Loom

All the implementations used in this comparison can be found in the nosleep branch of this GitHub repository. As the environment in which the program description is fully controlled, in tests ZIO uses a TestClock along with a test interpreter. We can arbitrarily push the clock forward—time does not flow on its own in a test; only when we request it to.

Replying To New Entry Requests

Apache Benchmarks run on versions with and without thread.sleep doesn’t say much as the results are similar for all implementations, and it might be due to limitations of the ApacheBench tool. We will use promises, thread pools, and workers if required and if the language supports it. Both Loom and ZIO versions use the same immutable data structures to model the domain, represent server state, the events and node roles. They have the same interfaces for communications, persistence, and representing the state machine, to which entries are applied.

I.e. it’s the perfect time to get your hands onto virtual threads and explore the new feature. In this post I’m going to share an interesting aspect I learned about thread scheduling fairness for CPU-bound workloads running on Loom. Project Loom is Java’s answer to lightweight user-mode threads, and it will likely make Java very competitive with Go as a cloud language. Loom’s lightweight threading model is designed to be fully compatible with Java’s existing threading model, so you won’t have to learn anything to use it. This feat has been accomplished by integrating Loom’s virtual threads with java.lang.Thread and java.util.concurrent.Executor. The only different between kernel threads and Loom’s virtual threads are how threads are initially created.

Non-kernel-thread-blocking I/O calls are possible under the current JDK API. However, one has to write asynchronous handlers. Look how asyncNonBlockingRequest repetitively calls read, and write, even thought the number of bytes available to be read or written may be zero. The threads are not blocked by each call for sure, but this implementation also wastes a lot of CPU cycles issuing calls onto channel/file descriptors that may have nothing to offer.

It’s worth mentioning that virtual threads are a form of “cooperative multitasking”. Native threads are kicked off the CPU by the operating system, regardless of what they’re doing . Even an infinite loop will not block the CPU core this way, others will still get their turn. On the virtual thread level, however, there’s no such scheduler – the virtual thread itself must return control to the native thread. In the thread-per-request model with synchronous I/O, this results in the thread being “blocked” for the duration of the I/O operation.