NAV
java

Introduction

Yerbie is a job queue and scheduler. A system like Yerbie is used to distribute work across threads or machines, or to schedule code to run at a certain point in the future.

To initiate a job, a client sends a request to Yerbie indicating a queue, data and delay, and Yerbie then makes that data available on the queue after the specified delay.

A worker polls, and runs a job associated with the data when it is available on the queue.

Currently, Yerbie has a Java client library with more language support planned.

Quick Start

Prerequisites

  1. Pull the Yerbie image locally by running docker pull yerbie/yerbie-server.
  2. Start the Yerbie server with docker run yerbie/yerbie-server.

Yerbie assumes Redis is accessible on localhost port 6379 and will start listening for requests on port 5865.

Library

The client library exposes an easy to use interface to send requests to Yerbie and to run jobs from a queue when they become available.

Add the Yerbie client library as a dependency with the latest version:

// Yerbie is available on the Maven Central Repository
<dependencies>
    <dependency>
        <groupId>dev.yerbie</groupId>
        <artifactId>yerbie-java</artifactId>
        <version>1.1.0</version>
    </dependency>
</dependencies>

Job Data

In order to store relevant data that your job can access when it runs, you need to create a serializable representation of that data in JSON. The maximum data size is 512 megabytes.

Create your own job data:

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

public class TestJobData {
  private final String username;

  @JsonCreator
  public TestJobData(@JsonProperty("username") String username) {
    this.username = username;
  }

  public String getUsername() {
    return username;
  }
}

Jobs

A job is a simple interface that has a run method. When job data is pulled from a queue, a job instance is created and the run method is invoked with the data as a parameter.

Create a job:

import yerbie.job.Job;
import your.org.jobdata.TestJobData;

public class TestJob implements Job<TestJobData> {

  @Override
  public void run(TestJobData testJobData) {
    System.out.println("Hello " + testJobData.getUsername());
  }
}

Client Initialization

By default, a Yerbie server listens for requests on port 5865.

To initialize a client:

import yerbie.client.ClientProvider;
import yerbie.client.YerbieClient;

ClientProvider clientProvider =
  new ClientProvider(
    "localhost", 5865);

// The client takes in an ObjectMapper to perform serialization and deserialization of job data.
YerbieClient yerbieClient = clientProvider.initializeClient(new ObjectMapper());

Scheduling Jobs

At its simplest, Yerbie takes a delay in seconds, a queue name, and an instance of a job data.

The example tells Yerbie to make TestJobData available on high_priority_queue in 10 seconds.

Yerbie can easily handle delays of days or even weeks, but be cognizant that any changes to TestJobData will need to support old versions of the data.

Yerbie supports retrying on failure, and using this method will use the default retry mechanism of 30 seconds in between retries, with a total of 4 retries. This means that the job can run at most 5 times (the first run does not count as a retry), before no longer running.

To schedule a job in 10 seconds to queue high_priority_queue:

import your.org.jobdata.TestJobData;
import yerbie.serde.JSONJobData;

yerbieClient.scheduleJob(Duration.ofSeconds(10), "high_priority_queue", new TestJobData("naota_nandaba"));

Deleting Jobs

To delete a job that is going to be executed, Yerbie allows you to delete a job via its job token. A job that is already enqueued into a queue cannot be deleted. The client returns true if succesful, or false if it can't find the job (or it has already been enqueued).

To delete a job with token aa555552-c21c-4d0a-a7a0-713f2b02b169:

boolean result = yerbieClient.deleteJob("a555552-c21c-4d0a-a7a0-713f2b02b169");

Running Workers

This tells the library to start polling Yerbie for available work.

If you've been following along, with our test job data and job implementation, this job will print Hello naota_nandaba to the system output.

To run a worker:

import yerbie.client.JobRepositoryBuilder;
import yerbie.client.YerbieConsumer;
import yerbie.client.ClientProvider;
import your.org.jobdata.TestJobData;
import your.org.job.TestJob;

JobRepositoryBuilder jobRepositoryBuilder = new JobRepositoryBuilder();
jobRepositoryBuilder.withJobData(TestJobData.class, TestJob::new);

ClientProvider clientProvider =
  new ClientProvider(
    "localhost", 5865);

YerbieConsumer yerbieConsumer =
  clientProvider.initializeYerbieConsumer("high_priority_queue", jobRepositoryBuilder.build());

yerbieConsumer.start();

Your worker will now poll on the high_priority_queue. When it receives a TestJobData, it creates a new instance of TestJob and will invoke the run function with the job data.

Retrying

To initiate a job, the client library requests a job from a specific queue. Once popped, the library has 10 seconds to send an ack to Yerbie to acknowledge that it successfully ran the job. In the case where it doesn't run the job successfully, Yerbie will make the job available again.

In the case where an exception is thrown, the client library will re-queue the job according to the job's retry policy.

If you don't specify a retry policy when creating a job, Yerbie uses the default of a fixed interval 30 second retry policy with 4 total retries. This means that the job will run unsuccessfully at most 5 times (since the first run is not a retry).

Fixed Interval Retry Policy

This retry policy will make the job available on the queue in 30 seconds, and will retry 4 times, each waiting 30 seconds after the initial run if the job is still unsuccessful.

import yerbie.job.FixedRetryPolicy;

long initialDelaySeconds = 30;
int totalRetries = 4;
yerbieClient.scheduleJob(Duration.ofSeconds(10), "high_priority_queue", new TestJobData("naota_nandaba"), new FixedRetryPolicy(initialDelaySeconds, totalRetries));

Exponential Retry Policy

This retry policy will make the job available after an exception in 5 seconds, then 25, then 125, then 625 before stopping.

import yerbie.job.ExponentialRetryPolicy;

long initialDelaySeconds = 5;
int totalRetries = 4;
yerbieClient.scheduleJob(Duration.ofSeconds(10), "high_priority_queue", new TestJobData("naota_nandaba"), new ExponentialRetryPolicy(initialDelaySeconds, totalRetries));
java