Code Musings

Small Dog, Small Language Model: Training a Transformer for $5

Mon, 11 May 2026 11:44:30 +0000

“Once upon a time, there was a dog named Cookie. She loved to play fetch with her owner. One day, they went for a walk in the park and found a big ball. Cookie picked it up with her mouth and brought it back to her owner…”

That paragraph above? Generated by a 91-million-parameter transformer I trained from scratch over a Saturday afternoon.

The dog Cookie is real. She’s mine. She is currently asleep on the couch, blissfully unaware that there’s now a small language model wandering the internet that can write fan-fiction about her.

Total compute cost: about $5 of rented H100 time on runpod.io across three training runs. The headline is generous; the actual final run cost $1.50. I’m calling it $5 because “$1.50” makes a less catchy post.

This is a log of training a language model end-to-end.

The Roadmap:

What I built (and what it actually means at this scale)
The journey from gibberish to coherence — three training runs, what worked, what didn’t
The bugs

What I built

A 91M-parameter GPT-style transformer, trained from scratch on the TinyStories corpus — about 470 million tokens of synthetic children’s stories with a deliberately-constrained vocabulary (~3000 base words).

Architecture:

91 million parameters (including embeddings)
12 layers
12 attention heads with RoPE
Custom 4,096-word BPE tokenizer (since the corpus just had ~3000 words)
512 token context window

This is built on the Hugging Face stack — transformers, tokenizers, datasets, safetensors . This is the same ecosystem teams use in practice for running real-world training pipelines, just at the smallest possible scale.

Perspective on Scale

Model	Parameters	Training Tokens	Estimated Cost
GPT-2 base (2019)	124M	~few billions	undisclosed
Nanochat	~560M	~11.2B	~$100
This model	91M	~930M	$1.50
GPT-4-class (rumored)	~1.7T (est.)	~13T (est.)	$100M+ (est.)

The journey: from gibberish to coherence

Round 1: Smoke-test on my laptop

Data: WikiText-2, a 2M-token Wikipedia corpus.
Model: 28M-param transformer (8 layers × 512 embeddings, 8 heads).
Hardware: Apple Silicon MPS, ~3 minutes wall-clock.
Result: Final perplexity 194. Total gibberish.

To be fair, this wasn’t meant to be a real training run but meant to just be a validation exercise to confirm the pipeline ran end-to-end without crashing and that loss was progressively getting lower.

While doing this evaluation, it struck me that Wikipedia is the wrong corpus for a 28M model.

WikiText-2 is tiny (~2M tokens). Chinchilla rule of thumb states that I need to have ~560M tokens (20x) of the model param size (28M). So, the corpus clearly doesn’t have enough data to fill the model.
WikiText-2 data is complex. It has dates, technical jargons and a wide range of low-frequency vocabulary cutting across several domains. Even if it had 560M tokens, a 28M model would lack the capacity to model that distribution.

Time to move on to a different corpus. A bit of research revealed that TinyStories was a great fit: ~470M tokens of synthetic, simple-vocabulary (3000 words) children’s stories that a tiny model can actually saturate.

Round 2: The first real attempt

Switched the corpus to TinyStories (~470M tokens), kept the model at 28M params and trained for 15,000 steps with bf16 autocast and torch.compile.

Hardware: RunPod H100 SXM 80GB.
Wall-clock: ~6 minutes.
Cost: ~$0.30.
Result: Perplexity 4.0.

1
2
3
4


Step : 0/15000 | val: 8.40 | ppl: 4054
Step : 1000/15000 | val: 2.16 | ppl: 8.6
Step : 5000/15000 | val: 1.61 | ppl: 5.0
Step : 14999/15000 | val: 1.40 | ppl: 4.0

From literally random to decent prediction in 6 minutes of GPU time.

But when I sampled:

“Once upon a time, there was a dog named Cookie.. Igaintto-doo. He is like it was gone. It was a very sad…”

It spoke “English words” interleaved with baby talk (let’s call it that - “Igaintto-doo”, “Annabutterram”, “girlsmagulled”). It also suffered from mode collapse where every other prompt got stuck in loops like “Once upon a lotion. Once upon a lotion.”

It took a few minutes to figure out that there were two separate problems:

The model was undertrained. Look at the last third of the loss curve, the number of steps were clearly inadequate. The val loss kept dropping but I stopped the training.

The sampling was uncontrolled. I understood that the “Once upon a lotion” loops aren’t a model bug, they’re a decoder bug. Apparently, the model.generate() settings will happily emit the same high-probability sequence unless I tune 2 parameters:

repetition_penalty : lowers the probability of tokens that have already appeared and
top_p : keeps generation diverse without going into low-probability nonsense (aka nucleus sampling)

Both were fixed in Round 3.

Round 3: 50K steps (~$1.00)

Bumped NUM_STEPS = 15_000 to NUM_STEPS = 50_000. Also fixed the sampling: added repetition_penalty=1.3 and top_p=0.95 to workaround the mode collapse.

Hardware: RunPod H100 SXM 80GB.
Wall-clock: ~20 minutes.
Cost: ~$1.00.
Result: Perplexity 3.62.

The “lotion” loops were gone. Real names appeared. But the output was still largely incoherent with sentences being “alright” but the paragraphs were unrelated and lacked continuity.

A look at the training and validation loss curves told me this was the best I could do with a 28M model - both the training loss and the validation curves flattened. Learnt that the term for this was that I “hit a capacity ceiling” . Apparently, the model is no longer learning because it has run out of parameters to hold, not because it has run out of data to learn from.

The solution was not to add more steps but to add more data to the model. Then I thought I’ll add more capacity to the model to learn from larger amounts of data.

Round 4: 91M params (the final run)

Bumped the architecture: 12 layers (vs 8), 768 hidden dim (vs 512), 12 attention heads (vs 8). Total parameters: 28M → 91M. The idea behind using the same corpus but fewer training steps (30K vs 50K) was that each step at 91M params is ~3× more compute than at 28M, so 30K steps here is comparable total compute to 50K at the smaller size.

Hardware: RunPod H100 SXM 80GB.
Trained tokens: 30k steps * 64 batches * 512 context/seq len = 983M
Wall-clock: ~20 minutes.
Cost: ~$1.50.
Result: Perplexity ~2.7.

Output became a coherent narrative:

“Once upon a time, there was a dog named Cookie. She loved to play fetch with her owner… His owner threw the ball and Cookie ran after it as fast as she could. But when she came back, she had lost the ball!…”

This model produced stories. Character continuity was there and the sentences were related to each other in a coherent ashion. It even ended sentences with “The end.” and starts a new story.

The pronouns still drift — Cookie is sometimes “she” and sometimes “he” but I believe this would be fixed by scale.

This run was truly a qualitative jump. Same architecture family, same corpus, just more data and capacity of the model.

The bugs

The RoPE buffer NaN bug: I was excited to run inference only to realise that it crashed, every time.

1
2
3
4
5
6
7


 File ".../torch/utils/_contextlib.py", line 124, in decorate_context
 return func(*args, **kwargs)
 File ".../transformers/generation/utils.py", line 2560, in generate
 result = decoding_method(
 File ".../transformers/generation/utils.py", line 2808, in _sample
 next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) 
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

A bit of research said that the precomputed RoPE cos/sin buffers had NaN entries. The problem is that this NaN will propagate through every attention layer that touches that position. The root cause is that I had the cos and the sin buffers as persistent=False, which was the recommended approach for storing recomputable tables.

1
2


self.register_buffer('cos', None, persistent=False)
self.register_buffer('sin', None, persistent=False)

Learnt that Hugging Face’s from_pretrained uses init_empty_weights() during loading, which allocates memory without initializing it. Since RoPE’s cos/sin map was registered as non-persistent buffers, HF didn’t save them in the checkpoint (weights are persistent and stored in checkpoint). Since the cos/sin values of RoPE are deterministic, the fix was to manually call model._init_rope(seq_len, head_dim) right after from_pretrained.

MPS scaled_dot_product_attention: PyTorch’s F.scaled_dot_product_attention(..., is_causal=True) on Apple’s MPS produces NaN under certain conditions. Understood that it was a known PyTorch issue. The fix was to wrap inference in with sdpa_kernel(SDPBackend.MATH): to force the unoptimized math kernel. I didn’t particularly dig deep into this patch.

I am truly amazed at how easily accessible training an LLM has become now. Not so long ago, this kind of project would have meant a research budget.

The full code is available at github.com/arunma/learn-you-an-hf-llm. Cookie has had her dinner and remains unimpressed.

What is HyperLogLog and how to build yours in Rust

Mon, 01 May 2023 12:12:43 +0000

Typically, counting the number of unique values in a dataset is an easy problem. You maintain a BST or HashSet to remove duplicates as they come in while keeping track of the count of inserted unique values. It takes O(n) space, which is acceptable for smaller datasets. However, for large datasets, storage becomes quickly expensive, especially if you would like to keep track of counts in real-time.

Let’s say that you are working on a banking application that processes ~1 billion transactions every day. You would like to know the number of unique customer account numbers on which transactions were made but you have limited memory and computational resources. Let’s do a back-of-the-envelope calculation for the memory required if we were to go the route of a HashSet.

1
2
3
4
5
6
7


a. Assume 10% of the transactions originate from unique account numbers = 10 million 
b. Assume Customer account numbers are 10 digit numbers. 
c. Each account number could be represented by a long datatype.

 ---

10 million account numbers * 8 bytes = 80 MB in memory

Note that the earlier calculation does not include the storage required for the hashes in the set or the pointer to the next link in the chain (due to collisions).

One other alternative is to persist all the account numbers to a file or a database and run a map-reduce/distinct operation at a regular frequency. However, it requires a fair amount of work and also the output is not real-time.

This sounds like an impossible task but you’ll see shortly how we could leverage an algorithm called HyperLogLog to get a “close to actual” estimate of unique values in a dataset by using an incredibly small fraction of storage. For the case of account numbers that we discussed above, if you can live with a 1% error rate, you’ll just need 1 MB with HLL.

What is HyperLogLog?

HyperLogLog is an algorithm that allows for a highly accurate “estimation” of the number of distinct elements in a dataset. More formally, HyperLogLog is a probabilistic algorithm used for estimating the cardinality in a set.

Several popular databases and frameworks use HLL underneath to provide quick estimates of distinct values - Apache Spark, Cassandra, MongoDB, Redis, Druid, etc.

How does it work?

Let us consider the case where the account number just has one digit. There are 10 possible account numbers: 0 to 9 and each account number has an equal probability of occurrence. Therefore, the probability of seeing the account number 0 in one of the transactions is 1/10. Let’s now consider the case where the account number has 2 digits. Account numbers can be between 00 and 99. Again, the possibility of us seeing 00 is 1/100. To generalize this, the possibility of seeing k number of leading zeros in an account number of any size is :

Applying this, we can “kinda” conclude that if we saw an account number with 2 leading zeros (00) in our transactions, we have seen 100 (k=2) account numbers. (Hang in there, for now.)

HyperLogLog follows exactly the same approach of looking at leading zeros, except that, in order to represent all kinds of data and not just numbers, it uses the binary representation of a hashed value of an element instead of using the actual data. (Some implementations use trailing zeros but as far as the hashes are uniformly distributed, it doesn’t affect the accuracy.)

Since the data is binary, it is easy to see that the occurrence of 0 in a single-digit binary is 1/2, 00 in a two-digit binary is1/4 , and so on. So, our estimation formula changes as:

So, if we have seen an account number with 2-leading zero in its binary representation, we can “kinda” conclude that we have seen at least 4 unique account numbers.

Very incorrect estimates

You’ll quickly realize that we could estimate only in exponents of 2, which results in overestimation or underestimation. Also, imagine the situation when your first account number has 7 leading zeros. The HLL will immediately report that we have seen 127 unique accounts when we just saw 1.

Solution

In order to circumvent this problem Flajolet and Martin, the authors of the algorithm, proposed four major changes to their algorithm:

Use multiple hash functions and average the estimates from each bucket of the hash function.
Instead of using arithmetic mean to find the average, use harmonic mean.
Use different bias corrections based on the total number of buckets used and
Use correction for large and small estimates.

Through these changes, the error rate of the algorithm is brought down to

What this would also mean is that we could specify an error rate while instantiating an HLL and the buckets/registers (m) size would be adjusted to meet that error rate. For all practical purposes, the error rate is significantly small.

Now, let’s go over each of the improvements one by one:

Multiple buckets

We could use a handful of hash functions to hash the input element. We could then compute the count of leading zeros in the hash binary. This count is then stored in the buckets that the hash belongs to (which could be derived through a simple modulus operator).

Unfortunately, computing multiple hashes is expensive for large datasets and therefore, the authors decided to use the first few bits of the binary representation of the hash to decide the bucket. This was very cheap to compute.

In the visual above, the 8-bit binary representation of the hash has two parts,

bucket section (aka precision): the bits that decide which bucket does this hash output affect the estimates of.
estimate section: the set of bits for computing the leading zeros.

In this case, the bucket is 1 and the estimate is 1 due to the presence of 1 leading zero.

Use harmonic mean

The Harmonic mean is the reciprocal of the average of the reciprocals of the numbers. It is given by:

So, for our 8 buckets with estimations 1, 1, 2, 1 , we do:

Use correction factor based on bucket size

Ours is a contrived example and therefore the number of buckets that we have is 8. Typically, the bucket sizes are minimally 16. The bias correction (alpha) mentioned in the paper are :

During our final estimation, we multiply the raw estimates (in our case 2.2857) by this alpha correction factor.

Use correction factor based on estimate size

As a final step, one round of correction is also done based on how large or small the estimate is, in comparison to the bucket size.

There are three ranges:

(I have absolutely no idea how these numbers are derived. These are just lifted from Page 14 of the original paper)

Final estimate

The formula to derive the final estimate from the raw estimate is:

final_estimate = range_size_correction(alpha * raw_estimate)

That’s it! Now that we have all the theory out of the way, let’s implement the algorithm in Rust.

Implementation in Rust

Instantiation

There’s just one input parameter required for HLL, which is your expected error rate. As we saw before, the bucket size is adjusted based on the error rate.

The number of buckets is calculated as:

A recommended set of bits (bucket section/precision) used to identify the bucket is log2(m)

Putting them all together in our new function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


pub struct HyperLogLog<T: Hash + Eq> {
 alpha: f64,
 precision: usize,
 num_buckets_m: usize,
 buckets: Vec<u8>,
 hasher: SipHasher24,
 _p: PhantomData<T>,
}

impl<T: Hash + Eq> HyperLogLog<T> {
 pub fn new(error_rate: f64) -> Result<Self> {
 validate(error_rate)?;
 let precision = (1.04 / error_rate).powi(2).log2().ceil() as usize; // log2(m)
 let num_buckets_m = 1 << precision; // 2^precision
 let alpha = calculate_alpha(num_buckets_m)?;

 //Instantiate our single hashing function
 let random_key = generate_random_seed();
 let hasher = create_hasher_with_key(random_key);

 let hll = HyperLogLog {
 alpha,
 precision,
 num_buckets_m,
 buckets: vec![0; num_buckets_m],
 hasher,
 _p: PhantomData,
 };

 Ok(hll)
 }

The alpha function just encodes the table that we saw previously:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


fn calculate_alpha(num_buckets_m: usize) -> Result<f64> {
 if num_buckets_m < 16 {
 return Err(InputError(format!(
 "Invalid number of buckets calculated {num_buckets_m}"
 )));
 }
 let alpha = match num_buckets_m {
 16 => 0.673,
 32 => 0.697,
 64 => 0.709,
 _ => 0.7213 / (1.0 + 1.079 / num_buckets_m as f64),
 };

 Ok(alpha)
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


//Hash helper functions

/// Generates a random 64-bit key used for hashing
pub fn generate_random_seed() -> [u8; 16] {
 let mut seed = [0u8; 32];
 getrandom::getrandom(&mut seed).unwrap();
 seed[0..16].try_into().unwrap()
}

/// Creates a `SipHasher24` hasher with a given 64-bit key
pub fn create_hasher_with_key(key: [u8; 16]) -> SipHasher24 {
 SipHasher24::new_with_key(&key)
}

Building the APIs

Insert

In the insertion function, as we saw earlier,

The precision and the estimate bits as separated.
The precision bits are used to identify the bucket that we are going to update the new estimate.
The estimate bits are examined for the leading zeros.
The previous estimate in the bucket is examined and updated with the new estimate, if necessary.

Separate precision and estimation bits

Compare previous and current estimate and update bucket if necessary

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


pub fn insert<Q>(&mut self, item: &Q)
 where
 T: Borrow<Q>,
 Q: Hash + Eq + ?Sized,
 {
 let mut hasher = self.hasher;
 item.hash(&mut hasher);
 let hash = hasher.finish();

 let bucket_index = (hash >> (64 - self.precision)) as usize; // Take the first precision bits
 let estimate_bits_mask = (1 << (64 - self.precision)) - 1; // Create a mask for excluding the precision bits
 let estimate_bits = hash & estimate_bits_mask;
 let trailing_zeros = estimate_bits.leading_zeros() as u8 + 1; // Count the number of leading zeros
 let prev_estimate = self.buckets[bucket_index]; // Get the previous estimate
 self.buckets[bucket_index] = max(prev_estimate, trailing_zeros) // Update the estimate if necessary
 }

Estimate

For the estimate, we would

calculate the raw estimate by using harmonic mean.
multiply the estimate with the previously calculated alpha.
arrive at final estimate after correcting for size of estimates in comparison to bucket size.

The correct_for_estimate_size function just encodes our previous table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


fn correct_for_estimate_size(&self, raw_estimate: f64, m: f64, empty_registers: usize) -> f64 {
 match raw_estimate {
 //Small range correction
 sr if raw_estimate <= 2.5_f64 * m => {
 if empty_registers > 0 {
 m * (m / empty_registers as f64).ln()
 } else {
 sr
 }
 }
 //Medium range correction
 mr if raw_estimate <= (TWO_POW_32 / 30.0_f64) => mr,
 //Large range correction
 lr => -TWO_POW_32 * (1.0 - lr / TWO_POW_32).ln(),
 }
 }

The estimate function calculates the harmonic mean and factors in alpha and estimate size correction.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


pub fn estimate(&self) -> usize {
 let m = self.num_buckets_m as f64;

 let mut sum = 0.0_f64;
 let mut empty_registers = 0;
 for &v in self.buckets.iter() {
 sum += 2.0_f64.powf(-(v as f64));
 if v == 0 {
 empty_registers += 1;
 }
 }

 let raw_estimate = self.alpha * m.powi(2) / sum;
 let estimate = self.correct_for_estimate_size(raw_estimate, m, empty_registers);

 estimate as usize
}

Code

The complete code is here.

Build your own CountMinSketch in Rust

Sun, 02 Apr 2023 10:43:13 +0000

While discussing about Counting Bloom Filter, we came across this function called estimated_count which attempts to calculate the number of times an element was present in the bloom filter. During the discussion, I also said that the Counting Bloom Filter is not the best datastructure for calculating the count of an item.

CountMinSketch is a good data structure to do that. And we’ll see why.

Imagine you have created a Twitter-alternative and you would like to keep track of the frequency of all the hashtags so that you can derive the “heavy-hitters” or the top trending hashtags. Or you are building an e-commerce or ride-hailing app and would like to keep track of the top products or destinations within an hour or a period of time. How would we do it?

The solution actually has 2-parts and the regular way that one would go about is by :

building a frequency table - a hashtable with the hashtag as the key and a counter as its value.
Scan the hashtable to generate the heavy-hitters (top k) list. Alternatively, you could build a min-heap to hold k elements.

Now, in this post, you’ll see that while hashtables are great to maintain the frequency table, (part 1 of the problem) CountMinSketch could also be used in its place. In a subsequent post, we’ll build on this to solve the second part of the problem.

What is a CountMinSketch?

Crudely put, CountMinSketch is an approximate version of a frequency table. But why would you want to use an approximate data structure like CountMinSketch when you could use a Hashtable with counters? There are two reasons :

Space

One obvious reason is space. Like all probabilistic data structures, CMS uses considerably less space than a hashtable. Let’s do a back-of-the-envelope calculation:

Let’s assume we have 1 million hashtags and about 20% of them are unique. We’ll also assume that the hashtags are about 6 characters on average.

Hashtable:

Total number of hashtags: 200,000
Size of each hashtag (assuming ASCII): 6 bytes
Counter size (value of the hashmap) : 4 bytes

Total size : 200k * (7+4) = 2.2 MB

For the sake of simplicity and sloppiness, we are neither considering the size of the buckets in the hashmap nor the size of the list/tree nor their pointers to the next item, under each bucket.

In comparison, a CountMinSketch would take about 5.4 Kb provided we can live with a 1% variance in the estimate. For large datasets, this would mean a difference between 10s of GBs in comparison to 10s of MBs.

Count Min Sketch:

Size of each bucket : 272
Number of hash functions: 5
Size of each counter: 4 bytes

Total size: 272 * 5 * 4 bytes

(Assuming that each counter is an unsigned integer)

You’ll quickly be able to understand the calculation once you see the structure of the CMS in the next section.

Need

In all the use-cases mentioned above, do you really care about the accurate frequency of each element? I mean, do you really care if the hashtag was used 990,020 or 991,000 times? As far as you know a hashtag’s “approximate” frequency with an assurance that the frequency has a predictable error variance, you are fine.

CountMinSketches are very frequently (pun intended) used in streaming systems to estimate frequencies in real-time. They are also used to estimate frequencies in large datasets where keeping all the counts on a hashtable is not practical. Also, since they occupy very little memory, it is not uncommon to have multiple CMS for varying time windows.

Check out this FAQ on Twitter trends.

How do we build one?

The structure of CountMinSketch is incredibly close to that of the Counting Bloom Filter. In a Counting Bloom Filter, we had one set of buckets with counters (m). Based on the output of all hash functions, the counters in this one bucket was incremented.

With CountMinSketch, we create one bucket of size m per hash function:

I believe I am breaking all the suspense here. This structure answers one big question that we had in mind:

Why does CountMinSketch better than Counting Bloom Filter for estimating frequency/counts?

By using m counters per hashing function instead of m counters for ALL hashing functions put together, we are able to significantly reduce the number of collisions that would happen on the bucket. Of course, the downside is that we now need space for k * m counters instead of 1 *m counters but the estimates are very close to the actual counts.

Properties

Before we implement the data structure, let us get the properties summarized:

We saw that a CMS solves the “frequency” problem by enabling us to estimate frequencies of elements in streaming systems and/or in large datasets.
It uses multiple hash functions and each hash function uses its hash output to increment counters in their own dedicated buckets, for keeping track of frequencies.
They use very less memory in comparison to our traditional hashtable.
You can only insert elements and get their estimated count. Deletion, updation, or iteration of values is not possible.

Implementation

Input parameters

There are two important parameters that CMS expects from the users - epsilon and delta. Hehe. I got you, didn’t I? Let me elaborate it.

epsilon : Considering the estimated_count of an element is an approximation, you would want to control how closer to the actual count should the approximation be. The epsilon parameter is nothing but the maximum variance in the estimated count (or maximum allowable error rate). So, if you say the epsilon is 1%, and your estimated count is 1000, it means that the actual count could be between 990 and 1110. Now, why would someone not want the best possible estimate? The reason is that more accuracy comes at the price of more space (number of buckets).

delta : Since CMS is a probabilistic data structure, there could be false positives. In a Bloom Filter, the false positive rate of 1% means that approximately 1% of the time, the Bloom filter would say that an element is present in the set, while it was not. In order to minimize the false positive rate, in a bloom filter, we would typically increase the number of buckets and/or increase the number of hash functions.

In a CMS, a false positive rate of 1% would mean that 1% of the time, the count returned by CMS is incorrect - meaning 1% of the time the count is beyond the allowed error range specified by epsilon. Just like a Bloom Filter, a larger bucket and a higher number of hash functions are common approaches to reduce the false positive rate.

So, here’s our implementation. For the purposes of simplicity, the counters are 32-bit unsigned integers but it is very common to see other variations.

Declaring the struct and instantiation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


#[derive(Debug)]
pub struct CountMinSketch<K: Hash + Eq> {
 epsilon: f64,
 delta: f64,
 hasher: SipHasher24,
 counter: Vec<Vec<u32>>,
 m: usize,
 k: usize,
 len: usize,
 _p: PhantomData<K>,
}

impl<K: Hash + Eq> CountMinSketch<K> {
 pub fn new(max_variance_in_count: f64, fp_rate_of_count: f64) -> Result<Self> {
 let epsilon = max_variance_in_count;
 let delta = fp_rate_of_count;
 let m = optimal_m(delta);
 let k = optimal_k(fp_rate_of_count);
 let random_key = generate_random_key();
 let hasher = create_hasher_with_key(random_key);
 let counter = vec![vec![0_u32; m]; k];

 Ok(CountMinSketch {
 epsilon,
 delta,
 hasher,
 counter,
 m,
 k,
 len: 0,
 _p: PhantomData,
 })
 }
 ...
 ...

Hasher instantiation

We accept the epsilon and delta parameters and calculate the number of buckets (m) and the number of hash functions (k).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


/// Generates a random 128-bit key used for hashing
fn generate_random_key() -> [u8; 16] {
 let mut seed = [0u8; 32];
 getrandom::getrandom(&mut seed).unwrap();
 seed[0..16].try_into().unwrap()
}

/// Creates a `SipHasher24` hasher with a given 128-bit key
fn create_hasher_with_key(key: [u8; 16]) -> SipHasher24 {
 SipHasher24::new_with_key(&key)
}

Number of buckets and hash functions

The formulas for arriving at optimal_m and optimal_k from epsilon and delta are :

Their implementations being :

1
2
3
4
5
6
7


fn optimal_m(epsilon: f64) -> usize {
 (2.71828 / epsilon).ceil() as usize
}

fn optimal_k(delta: f64) -> usize {
 (1.0 / delta).ln().ceil() as usize
}

Building the APIs

Now that we have built the foundational functions, let’s get into the API. There are just two functions to be implemented and both are very intuitive.

Insert

For every item that was inserted, we need to calculate k hashes. The methodology is the same as Counting Bloom Filter (section Hashers) where:

we generate a 128-bit hash using SipHasher
split them into two 64-bit hashes
use Kirch and Mitzenmacher technique to create k-hash functions
scoped down the hash to bucket indices of size m
increment the counters corresponding to the bucket indices by 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


impl<T: ?Sized + Hash + Debug> CountingBloomFilter<T> {
 pub fn insert(&mut self, key: &K) {
 let bucket_indices = self.get_bucket_indices(key, self.hasher);
 bucket_indices
 .iter()
 .enumerate()
 .for_each(|(ki, &bi)| self.counter[ki][bi] = self.counter[ki][bi].saturating_add(1));
 self.len += 1
 }


 /// Returns the bucket indices of k hash functions for an item
 fn get_bucket_indices(&self, item: &K, hasher: SipHasher24) -> Vec<usize> {
 let (hash1, hash2) = self.get_hash_pair(item, hasher);
 let mut bucket_indices = Vec::with_capacity(self.k);
 if self.k == 1 {
 let bit = hash1 % self.m as u64;
 bucket_indices.push(bit as usize);
 return bucket_indices;
 } else {
 for ki in 0..self.k as u64 {
 let hash = hash1.wrapping_add(ki.wrapping_mul(hash2));
 let bit = hash % self.m as u64;
 bucket_indices.push(bit as usize)
 }
 }

 bucket_indices
 }

Estimated count

The estimated count is calculated by hashing the item and looking into the counter values for each hash function. The minimal counter value is considered to be the estimated count.

The reasoning behind choosing the minimum and not the maximum is that a counter could be incremented while inserting other elements due to hash collisions. Therefore, these counters overestimate the real frequency since the number of buckets is always much lesser than the number of items inserted into the sketch. Taking an average or a median is also not a fair estimate for items that have a lower frequency.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


pub fn estimated_count(&self, key: &K) -> u32 {
 let bucket_indices = self.get_bucket_indices(key, self.hasher);
 let mut estimated_count = u32::MAX;
 for (ki, &bi) in bucket_indices.iter().enumerate() {
 if self.counter[ki][bi] == 0 {
 return 0;
 } else {
 estimated_count = min(estimated_count, self.counter[ki][bi])
 }
 }
 estimated_count
}

Test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


#[test]
fn insert_and_check_several_items() -> Result<()> {
 let mut bf: CountMinSketch<&str> = CountMinSketch::new(0.2, 0.01)?;

 for _ in 0..1000000 {
 bf.insert(&"a1");
 bf.insert(&"a2");
 bf.insert(&"a3");
 bf.insert(&"a4");
 bf.insert(&"a5");
 bf.insert(&"a6");
 bf.insert(&"a7");
 }

 assert_eq!(bf.number_of_hashes(), 5);
 assert_eq!(bf.number_of_counters(), 272);
 assert_eq!(bf.estimated_count(&"a1"), 1000000);
 assert_eq!(bf.estimated_count(&"a2"), 1000000);
 assert_eq!(bf.estimated_count(&"b1"), 0);

 Ok(())
}

Code

The complete code is here.

Build your own Counting Bloom Filter in Rust

Sun, 19 Mar 2023 15:37:44 +0000

A Counting Bloom Filter is a probabilistic data structure that helps us quickly check if an element is present in a set or not. You might argue “Hey, can’t a simple Set do it?”.

Yes, indeed and to top it, Counting Bloom Filter is not even 100% accurate and expects us to provide the expected “false positive rate”. Where Bloom Filters shine is that for large volumes of data it would use much lesser memory than a regular HashSet.

This post attempts an implementation of the datastructure. At the end of the post, you’ll be able to build your own Counting Bloom Filter. The final code is available here.

Before we get into the implementation, let’s get some properties of CBF out of the way:

As with our traditional BloomFilter, Quotient Filter or CuckooFilter, Counting Bloom Filter is a probabilistic data structure that is used to test whether an element is a member of a set. Essentially, these filters aim to solve a class of problems called “Membership”. A key feature that CBF offers over traditional Bloom Filter is that CBF allows deletion of inserted items.
As with any probabilistic data structure, there is a chance that we might get some false positives - essentially, the response to the question “Is an element present in the set?” is not 100% accurate. There are times when the bloom filter would say that the element is present but it actually was not. This “false positive” rate is configurable through a parameter.
Unlike HashSet, you cannot iterate through the values stored in the set - we can only check whether the value is present in the CBF.

Bloom Filter vs Counting Bloom Filter

The underlying storage of a BloomFilter is just a fixed-size array of bits.

This is a fantastic write-up detailing how to implement your own Bloom Filter.

A Counting Bloom Filter uses a fixed-size array of counters.

To put it in crude terms, the underlying storage of Bloom Filter is a Vec<bool> (or a BitSet). With CBF, it is a Vec<u8> or Vec<u16> or Vec<u32> or a list of unsigned integers.

The counter size does not need to be 8, 16, 32 bits etc - rather, they don’t need to be aligned to the datatypes. Using arbitrary bit sizes is common and is achieved by partitioning a usize integer into arbitrary bit sizes.

For illustration, our implementation of a Counting Bloom Filter will use a u8 bit counter.

Setting the stage

Declaring our struct and instantiation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


#[derive(Debug)]
pub struct CountingBloomFilter<T: ?Sized + Hash> {
 counter: Vec<u8>,
 m: usize,
 k: usize,
 hasher: SipHasher24,
 expected_num_items: usize,
 false_positive_rate: f64,
 len: usize,
 _p: PhantomData<T>,
}

As expected, our struct encapsulates a list of counters. There are two parameters that we would be accepting from the user, which are expected_num_items and false_positive_rate.

The expected_num_items is an approximate number of items that the user would like to store in the CBF.
The false_positive_rate is an estimate of False Positives/inverse accuracy that the user can live with.

The len just stores the total number of items inserted into the data structure so far. We’ll discuss the m and k fields in the next section.

Number of buckets and number of hashes

Bloom Filter

In a BloomFilter, m is simply the size of the bit vector (or underlying bits representing the membership of the various inserted values). k is the number of hash functions that each inserted value has to be run against.

So, given a value x, it will be hashed by k number of hash functions and the corresponding bit in the m buckets will be set to True.

Let’s consider the illustration above. Each item ( x, y, z) are hashed by 2 hash functions (represented by arrows). The resulting output are expected to be the bucket indices. Considering the total number of buckets is just 8 in the above illustration, the output of the hash function is modulo-ed to reduce the range between 0 and 7.

Counting bloom filter.

The number of hash functions ( k ) is 2 as above and the m is 8. However, the underlying store is not a list of bits but a list of counters.

The formulas for calculating the size of m and k given the expected_num_of_items and false_positive_rate for Counting Bloom Filter are:

Let’s implement the optimal_m, optimal_k and the new functions

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33



fn optimal_m(num_items: usize, false_positive_rate: f64) -> usize {
 -(num_items as f64 * false_positive_rate.ln() / (2.0f64.ln().powi(2))).ceil() as usize
}

fn optimal_k(n: usize, m: usize) -> usize {
 let k = (m as f64 / n as f64 * 2.0f64.ln()).round() as usize;
 if k < 1 {
 1
 } else {
 k
 }
}

impl<T: ?Sized + Hash + Debug> CountingBloomFilter<T> {
 pub fn new(expected_num_items: usize, false_positive_rate: f64) -> Result<Self> {
 validate(expected_num_items, false_positive_rate)?;
 let m = optimal_m(expected_num_items, false_positive_rate);
 let counter = vec![0; m];
 let k = optimal_k(expected_num_items, m);
 let random_key = generate_random_key();
 let hasher = create_hasher_with_key(random_key);
 Ok(Self {
 counter,
 m,
 k,
 hasher,
 expected_num_items,
 false_positive_rate,
 len: 0,
 _p: PhantomData,
 })
 }

As you can see, only the “estimated number of items to be inserted” and the “false positive rate” is needed to construct our Counting Bloom Filter.

Hashers

Once we calculate the optimal number of hash functions (k), we need to create k independent hash functions. One approach is to have a repository of hash functions and initiate them with a random key. However, there is a simpler approach that is proposed by Adam Kirsch and Michael Mitzenmacher, where we could simulate additional hash functions with the help of two base hash functions.

Quoting the paper:

The idea is the following: two hash functions h1(x) and h2(x) can simulate more than two hash functions of the form gi(x) = h1(x) + ih2(x). In our context i will range from 0 up to some number k − 1 to give k hash functions, and the hash values are taken modulo the size of the relevant hash table.

Specifically, only two hash functions are necessary to effectively implement a Bloom filter without any increase in the asymptotic false positive probability. This leads to less computation and potentially less need for randomness in practice.

Essentially, we just need 2 independent hash functions to simulate k hash functions. Typically, Bloom Filters use 64-bit hash functions since they create enough bits for us to populate the buckets uniformly. In our implementation, we will use a 128-bit hash function and use the first and the second 64 bits to simulate two hash function outputs. We will then use these two 64-bit hashes to create more hashes.

Firstly, in order to provide a random seed to our 128-bit hasher, we’ll use the generate_random_key function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


use siphasher::sip128::SipHasher24;

/// Generates a random 128-bit key used for hashing
fn generate_random_key() -> [u8; 16] {
 let mut seed = [0u8; 32];
 getrandom::getrandom(&mut seed).unwrap();
 seed[0..16].try_into().unwrap()
}

/// Creates a `SipHasher24` hasher with a given 128-bit key
fn create_hasher_with_key(key: [u8; 16]) -> SipHasher24 {
 SipHasher24::new_with_key(&key)
}

Building the APIs

With all the foundational functions out of the way, let us get into implementing the API - the three functions insert, contains and delete

Insert

When an element is inserted into the filter, the counters corresponding to the element’s hash values are incremented by one. This is in contrast to BloomFilter where we just set the corresponding bit. While we do this, we will also use this opportunity to keep track of the number of items inserted into the bloom filter by incrementing the len field.

1
2
3
4
5
6


pub fn insert(&mut self, item: &T) {
 self.get_set_bits(item, self.k, self.m, self.hasher)
 .iter()
 .for_each(|&i| self.counter[i] = self.counter[i].saturating_add(1));
 self.len += 1;
}

The get_set_bits is a helper function that takes the item to be inserted and returns the indices of all buckets for whom the counters need to be incremented. Essentially, the get_set_bits function

Takes an item as an input
Creates a 128-bit hash (done in get_hash_pair)
Splits the 128-bit hash into 2 parts, thereby creating the 2 base hashes (done in get_hash_pair)
Creates k hashes out of the 2 base hashes using the Kirsch and Mitzenmacher method
Since we just have m buckets and the hash function output could be bigger, we do a modulus of m on each hash output. This returns k bucket indices on m . These are the indices of the counters in the bucket that needs to be incremented.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


fn get_set_bits(&self, item: &T, k: usize, m: usize, hasher: SipHasher24) -> Vec<usize> {
 let (hash1, hash2) = self.get_hash_pair(item, hasher);
 let mut set_bits = Vec::with_capacity(k);
 if k == 1 {
 let bit = hash1 % m as u64;
 set_bits.push(bit as usize);
 return set_bits;
 }
 for ki in 0..k as u64 {
 let hash = hash1.wrapping_add(ki.wrapping_mul(hash2));
 let bit = hash % m as u64;
 set_bits.push(bit as usize);
 }
 assert!(set_bits.len() == k);
 set_bits
}

 /// Computes the pair of 64-bit hashes for an item using the internal hasher
fn get_hash_pair(&self, item: &T, mut hasher: SipHasher24) -> (u64, u64) {
 item.hash(&mut hasher);
 let hash128 = hasher.finish128().as_u128();
 let hash1 = (hash128 & 0xffff_ffff_ffff_ffff) as u64;
 let hash2 = (hash128 >> 64) as u64;
 (hash1, hash2)
}

The wrapping_add, wrapping_mul and other wrapping functions allow for overflow of numbers without triggering a panic.

eg.

let a: u16 = 65534;
let b = a.wrapping_add(1); // 65535
let c = a.wrapping_add(2); // 0

Contains

As you might have guessed already, testing whether an item is present in the filter is just a matter of computing the hashes for the item and checking whether the counters in the bucket corresponding to the hashes have a non-zero value.

1
2
3
4
5
6



pub fn contains(&self, item: &T) -> bool {
 self.get_set_bits(item, self.k, self.m, self.hasher)
 .iter()
 .all(|&i| self.counter[i] > 0)
}

Delete

The deletion operation is just an inverse of the insertion operation with two additional nuances.

We can only delete an item that is already present in the filter. So, we check for it by calling the contains.
Our counter can never be less than 0. So, we use the saturating_sub function to ensure that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


	pub fn delete(&mut self, item: &T) {
 let is_present = self.contains(item);
 if is_present {
 self.get_set_bits(item, self.k, self.m, self.hasher)
 .iter()
 .for_each(|&i| {
 self.counter[i] = self.counter[i].saturating_sub(1);
 });
 self.len -= 1;
 }
 }

Estimated count

Although Counting Bloom Filter is not the best data structure to get an estimated count of an item, it is not uncommon to see instances where the function is implemented. Estimated count is arrived at by looking at all the counter values and picking the minimum of those values. The reasoning behind choosing minimum and not maximum or median is that a counter could be incremented by several items due to hash collisions. Minimum is chosen to avoid the skew created by high-frequency elements and to be fair to the low-frequency elements.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


pub fn estimated_count(&self, item: &T) -> u8 {
 let mut retu = u8::MAX;
 for each in self.get_set_bits(item, self.k, self.m, self.hasher) {
 if self.counter[each] == 0 {
 return 0;
 }
 if self.counter[each] < retu {
 retu = self.counter[each];
 }
 }
 retu
}

We will revisit this discussion when we build our own CountMinSketch later.

Overflow

Although not included in our implementation, I felt that a discussion on this topic is important.

In our example, we decided that our counter is represented by an unsigned 8-bit integer with a maximum value of 255. What if subsequent item insertions attempt to increment the value of that counter? This would result in an overflow and in Rust, a panic. To avoid overflowing, the counter could be capped/saturated at the maximum value. This essentially means that subsequent insertions into that bucket will have no effect on the filter’s state for that counter.

There are two common workarounds to solve overflows:

Freezing
Dynamic resizing

Freezing

Freezing is a process where

the buckets are scanned for counters that have already reached the maximum value.
reset the counter to a threshold value (around 80% of the maximum value). For u8 that’s approximately 200.

Dynamic resizing

The other simpler yet computationally expensive alternative is to resize the counter to accept larger values. Say, promoting the datatype from u8 to u16. This involves creating a new counter list, copying the existing counters and replacing the old list with the new one.

Code

Complete code is here.

The beautiful simplicity of Apache Ranger plugin

Mon, 13 May 2019 16:23:30 +0000

If you are here, you already know what Apache Ranger is. It is the most popular, if not the only, way to manage security in the Hadoop framework. It has integrations with Active Directory, Kerberos and various others for authentication but I believe the most interesting feature is its authorization support. Being part of the Hadoop ecosystem, one would not be surprised that it has inbuilt support (via plugins) for most frameworks in the Hadoop ecosystem - Hive, HBase, HDFS etc. However, I’ve found that it’s actually very easy to spin your own custom plugin for Ranger.

This post would focus on the simplicity of design in Ranger plugins and showcase how easy it is to build one for ourselves. As an example, we’ll build a Ranger plugin for managing access to a simple HTTP service written using Akka HTTP.

1

Note : You are not required to know about Akka HTTP to follow this post. All you needed to know is that Akka HTTP is just a way (albeit, a great way) to build HTTP services

The code behind this post is split into two repositories:

Writing a plugin

To reiterate what we are attempting to do here, we are going to write a REST service and let Ranger manage the authorization for it.

Writing a Ranger plugin is actually a two part problem - writing the server-side component and the application-side component.

Server-side component is the code/configuration that resides on the Ranger side.
Application-side component is the code that resides in our REST service that invokes the Ranger service and checks if the application’s end user has access to the resource that he is requesting.

We’ll look into these two things in detail. Let’s attempt to write the server-side components first.

Server-side components :

As an inspiration, if we open up the ranger code base, we can see some of the in-built plugins.

Pictorially, within the Ranger code base, we have bunch of plugins and we would like to add our own plugin.

Zooming in the previous picture, the server-side component on the plugin would mean writing a

servicedef configuration
A class that inherits RangerBaseService

So, there’s literally “one” configuration and “one” class that you need to implement for the server-side.

servicedef configuration

Let’s look at Hive’s servicedef configuration :

In my opinion, there are three important things that we are talking about here :

Resource:

In the Hive example, the “resource” that we are trying to protect are databases, tables and columns, for Kafka, the “resource” that we are trying to protect is the Kafka topic, for HDFS, it would be a file path. For our HTTP service, the resource that we are trying to protect is the REST slug. Let’s call it a “path”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


"resources": [
 {
 "itemId": 1,
 "name": "path",
 "type": "path",
 "level": 10,
 "parent": "",
 "mandatory": true,
 "lookupSupported": true,
 "recursiveSupported": true,
 "excludesSupported": true,
 "matcher": "org.apache.ranger.plugin.resourcematcher.RangerPathResourceMatcher",
 "matcherOptions": {
 "wildCard": true,
 "ignoreCase": true
 },
 "validationRegEx": "",
 "validationMessage": "",
 "uiHint": "",
 "label": "HTTP Path",
 "description": "HTTP Path"
 }

Access Type:

Access types simply means the kind of access that the user would require - say, for Hive, select, create, delete would be examples. For HDFS, read, write, execute would be examples. For Kafka, publish and consume. For our HTTP service, the access type would be the HTTP methods - GET, POST, DELETE.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


"accessTypes": [
 {
 "itemId": 1,
 "name": "get",
 "label": "get"
 },
 {
 "itemId": 2,
 "name": "post",
 "label": "post"
 },
 {
 "itemId": 3,
 "name": "delete",
 "label": "delete"
 }
 ]

Configs:

We know that Ranger can manage security for several Kakfa topics, HDFS and HBase clusters. Each of these services would be running in a different host and the way to authenticate into each of them would be different. The place to capture this information would be this configs part. For the sake of simplicity of this example, we don’t care about authentication for our HTTP service. So, we are just capturing a URL that we could ping to, to ensure that our service is up and running.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14



"configs": [
 {
 "itemId": 1,
 "name": "services_list_url",
 "type": "string",
 "subType": "",
 "mandatory": true,
 "validationRegEx": "",
 "validationMessage": "",
 "uiHint": "",
 "label": "HTTP URL for the services list eg. http://localhost:8080/services"
 }
 ]

A class that inherits RangerBaseService

The second and the last part of implementing our server-side component for the ranger plugin is to write a class that inherits the RangerBaseService.

The class expects two functions to be overridden:

validateConfig: Remember the configs section of the servicedef. Obviously, we would be accepting values for those parameters right? Now, this validateConfig is the place where we validate the values that are passed. For our HTTP service, all that we are accepting in the config is the services_list_url. Now, the implementation of this function would be to use a simple HTTP client to ping and check whether the service is up and running.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


class RangerServiceHTTP extends RangerBaseService {

 override def validateConfig(): util.Map[String, AnyRef] = {
 if (configs.containsKey("services_list_url")) {
 val serviceUp = HttpServiceClient.isServiceUp(configs.get("services_list_url"))
 if (serviceUp) retSuccessMap() else returnFailureMap()
 }
 else {
 returnFailureMap()
 }
 }

lookupResource: This is an interesting function. Consider the following screenshot.

Later, when we configure an access policy, we would be configuring the resources in it. Now, this function is used to lookup and autofill those resources. Say, if we are entering a HDFS resource or Hive table, the number of options are quite a lot and it’s easy to do a typo. In case of Hive, this function would connect to the metastore and populate the tables and databases for us.

In the case of HTTP service, remember the service_list_url? That URL would just return a comma separated list of REST resources. For implementing this function, I am just calling the service again and tokenizing the response.

1
2
3
4


override def lookupResource(resourceLookupContext: ResourceLookupContext): util.List[String] = {
 val serviceUrl = configs.get("services_list_url")
 HttpServiceClient.getServicePaths(serviceUrl).asJava
 }

Now, as a final step to the code, we’ll need to tie this class RangerServiceHTTP and the servicedef configuration together. The way we do this is by configuring the class in the implClass property. Also notice that we are configuring the name of this ranger plugin as httpservice:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{
 "name": "httpservice",
 "label": "HTTP Service",
 "description": "Rudimentary Ranger plugin to enforce security on top of a HTTP Service",
 "guid": "b8290b7f-6f69-44a9-89cc-06b6975ea676",
 "implClass": "com.arunma.ranger.http.RangerServiceHTTP",
* * "version": 1,
 "isEnabled": 1,
 "resources": [
 {
 "itemId": 1,
 "name": "path",
 ...
 ...

The full configuration looks like this.

There are two more minor administrative steps:

In order to ensure that our class is made available on the Ranger classpath, we’ll bundle it into a jar and drop it at <RANGER_HOME>/ews/webapp/WEB-INF/classes/ranger-plugins/httpservice. The name of the folder httpservice corresponds to the name that is declared in the servicedef configuration.

Upload our configuration into Ranger so that our service becomes visible in the Ranger UI.

1

curl -u admin:admin -X POST -H "Accept: application/json" -H "Content-Type: application/json" --data @http-ranger.json http://localhost:6080/service/plugins/definitions

Bounce the Ranger server.

Yaay! We now see HTTPSERVICE on our Ranger UI

Application-side components :

On the application side, things couldn’t get any simpler. In order to use the policies used in Ranger, all that an application would need to do is to call Ranger and check if the user has access to a resource. The function is literally called isAccessAllowed.

The following code is pretty much all that needs to be written on the application side:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


package com.arunma.ranger

import org.apache.ranger.plugin.audit.RangerDefaultAuditHandler
import org.apache.ranger.plugin.policyengine.{RangerAccessRequestImpl, RangerAccessResourceImpl}
import org.apache.ranger.plugin.service.RangerBasePlugin

import scala.collection.JavaConverters._

object RangerAuthorizer {
 lazy val plugin = {
 val plg = new RangerBasePlugin("httpservice", "httpservice")
 plg.setResultProcessor(new RangerDefaultAuditHandler)
 plg.init()
 plg
 }

 def authorize(path: String, accessType: String, userName: String, userGroups: Set[String] = Set("public")): Boolean = {
 val resource = new RangerAccessResourceImpl()
 resource.setValue("path", path)
 val request = new RangerAccessRequestImpl(resource, accessType, userName, userGroups.asJava)
 val result = plugin.isAccessAllowed(request)
 result != null && result.getIsAllowed
 }
}

The RangerBasePlugin("httpservice", "httpservice") and the init() function serves as our entry point into the Ranger service. Note the httpservice parameter inside the RangerBasePlugin. This must match the name that was given in the servicedef configuration.

The authorize function is the one that gets called by the interceptor just before the client is given access to a REST resource. The function simply constructs a AccessRequest - the RangerAccessRequestImpl and calls the plugin’s isAccessAllowed function, which returns a Boolean.

The interceptor directive authorize invokes the function isRangerAuthorized which then calls the authorize function in RangerAuthorizer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16



def isRangerAuthorized(path: String, httpMethod: String, userName: String): Boolean = RangerAuthorizer.authorize(path, httpMethod.toLowerCase, userName) 

lazy val userRoutes: Route =
 headerValueByName("username") { userName =>
 extractMethod { method =>
 pathPrefix("users") {
 extractMatchedPath { matchedPath =>
 authorize(isRangerAuthorized(matchedPath.toString(), method.name(), userName)) {
 concat(
 pathEnd {
 concat(
 get {
 val users: Future[Users] =
 (userRegistryActor ? GetUsers).mapTo[Users]
 complete(users)

One last thing that we are required to do is to copy an audit and security xml into our classpath. These are like the site xmls for Ranger. For this exercise, we’ll just place the xmls in our resources directory.

The audit xml and the security xml could be copied from the ranger codebase. If you are running a local ranger, the audit XML can remain as-is but security xml needs to be changed for our service. The easiest way to achieve this is to copy a sample xml from the ranger code base and start replacing the service as httpservice like so:

There’s also one property that needs special attention. That’s the property called ranger.plugin.httpservice.service.name. This property’s value must be the same as the Service Name that you use in your Ranger UI.

1
2
3
4
5
6
7


<property>
	<name>ranger.plugin.httpservice.service.name</name>
	<value>MyService</value>
	<description>
		Name of the Ranger service containing policies for this httpservice instance
	</description>
</property>

Test Ride

This would involve two steps

Configure a Ranger Policy
Verifying your HTTP Service

Configure a Ranger Policy

Verifying your HTTP Service

Let’s verify the policy by bringing up our HTTP Service - start the com.arunma.RangerManagedHttpServer

Policy-configured user

1

curl -X GET -H 'username:arunma' http://localhost:8080/users

Invalid user

1

curl -X GET -H 'username:nobody' http://localhost:8080/users

Summary

The Ranger plugin has two parts to it - a server-side component and a client-side component. For the server-side component, we created a servicedeef json and a class that inherited the RangerBaseService. For the client side-component, we just called an isAccessAllowed function of the plugin.

You now have a working Ranger authorized HTTP Service.

Thanks for reading. Happy Hacking !

Creating a YARN Application using Scala

Thu, 24 May 2018 15:27:43 +0000

I have been recently playing with Apache Amaterasu, which is an amazing project that helps to deploy data pipelines. It’s still incubating and has a super-friendly team of engineers working on it. Some exciting features are lined up. Don’t take my word for it. Please check it out yourself.

Amaterasu launches containers (on YARN/Mesos) all by itself for each of the stages in your data pipeline. All you need to provide is your repository and a YAML based configuration.

I was just curious about launching containers on YARN and how the API works and thought I should give it a try myself. It’s really intuitive if we understand a small set of constructs.

This post is an attempt on a minimal example of a YARN application on Scala.

Note : The complete code is available at github

There are three core classes in this App :

SampleYarnClient
ApplicationMaster
DummyApplication (the business logic)

SampleYarnClient

This is the entry-point of the program. Does the following :

Instantiates YarnClient.
Negotiates resources for the ApplicationMaster container with the help of the YarnClient. The way it does it is to initiate an ApplicationSubmissionContext which is just a wrapper around the Resource, Priority and ContainerLaunchContext (among others). Let’s quickly look at the code and we’ll go over in detail on these three components of the SubmissionContext.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


val yarnClient = YarnClient.createYarnClient()
...
val application = yarnClient.createApplication()
...
val context = application.getApplicationSubmissionContext
context.setAMContainerSpec(amContainer)
context.setApplicationName("Scala Yarn App")
context.setResource(resource)
context.setPriority(priority)

yarnClient.submitApplication(context)

Resource

Resources is a simple wrapper around CPU and Memory

1

val resource = Resource.newInstance(1024, 2) //1 GB memory and 2 cores

Priority

Priority is just an Integer - the higher the number, the higher the priority

1
2


val priority = Records.newRecord(classOf[Priority])
priority.setPriority(1)

ContainerLaunchContext

The ContainerLaunchRequest has three primary parameters in this simple example :

Commands (List[String]): The bootstrap command (ideally java <MainClass>)
LocalResources (Map[String,LocalResource]) : The jars and the other artifacts (properties, libraries etc) that’s essential for running your command
Environment (Map[String,String]): The environment variables essential for the program

(the other important one is the Security Tokens which is not used here because my local cluster isn’t kerberized)

1
2
3
4
5
6
7


def createContainerContext(commands: List[String], resources: Map[String, LocalResource], environment: Map[String, String]): ContainerLaunchContext = {
 val launchContext = Records.newRecord(classOf[ContainerLaunchContext])
 launchContext.setCommands(commands.asJava)
 launchContext.setLocalResources(resources.asJava)
 launchContext.setEnvironment(environment.asJava)
 launchContext
}

Commands

Like I said, the commands are just a sequence of instructions you would like to execute to run the ApplicationMaster from the shell.

1
2
3
4
5
6
7


val commands = List(
 "$JAVA_HOME/bin/java " +
 " -Xmx256m " +
 s" com.arunma.ApplicationMaster " +
 " 1> " + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" +
 " 2> " + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr"
 )

LocalResources

As for this program, you don’t need any properties or configuration files. All you need is just the jar binary alone.

Note :
The way that you make any binary or resource available for YARN is to place the binaries in a HDFS location. This particular step involving the setting up of local resources just means that we are telling YARN to download the binaries from the HDFS location and place in in the local path of the container when launched.

1
2
3


val localResources = Map(
 "SampleYarnApp-assembly-0.1.jar" -> setUpLocalResourceFromPath(yarnPath) 
 )

Environment

These are custom environment variables or just the classpath that needs to be set for your bundle to run.

ApplicationMaster

Now that we have discussed about the SampleYarnClient, let’s discuss the second. This class, as the name indicates, is your ApplicationMaster (Duh!). It’s responsible for launching the containers that is expected to run your “business logic”. The steps are :

The AppMaster uses the ResourceManager client (the AMRMClient) to raise a request for a container - ContainerRequest. This example uses the async version of the client - AMRMClientAsync that implements a series of callbacks - onContainersAllocated, onContainersCompleted, onError etc .
When the RM allocates a container for the application, the onContainersAllocated callback gets invoked.
Within the onContainersAllocated (now that we have the handle to the container), the AppMaster then uses the NMClientAsync to launch the “business” container (DummyApplication). This is achieved by constructing another ContainerLaunchContext (the one that wraps commands, local resources and environment variables).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


override def onContainersAllocated(containers: util.List[Container]): Unit = {

 val commands = List(
 "$JAVA_HOME/bin/java " +
 " -Xmx256m " +
 s" com.arunma.DummyApplication " +
 " 1> " + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" +
 " 2> " + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr"
 )

 val localResources = Map(
 "SampleYarnApp-assembly-0.1.jar" -> setUpLocalResourceFromPath(FileSystem.get(conf).makeQualified(new Path(sys.env("ARUN_JAR_PATH"))))
 )

 val containerLaunchContext = createContainerContext(commands, localResources, buildEnvironment(Map()))

 containers.asScala.foreach { container =>
 nmClient.startContainerAsync(container, containerLaunchContext)
 }
}

DummyApplication

This is the “business logic”. Not Scala in the purest sense but it helps us see the logs. Note that because this is an infinite loop, we’ll have to forcefully kill the application.

1
2
3
4
5
6


object DummyApplication extends App {
 while(true) {
 println("Niceeeeeeeeee !!! This is the core application that is running within the container that got negotiated by from Application Master !!!")
 Thread.sleep(1000)
 }
 }

Usage:

1

$ hadoop jar /Users/arun/IdeaProjects/SampleYarnApp/target/scala-2.11/SampleYarnApp-assembly-0.1.jar com.arunma.SampleYarnClient /Users/arun/IdeaProjects/SampleYarnApp/target/scala-2.11/SampleYarnApp-assembly-0.1.jar

Alternatively, you could just do an

1

sbt assembly

and run the SampleYarnClient from your IDE with the absolute path of the assembly jar as the first argument.

Scala notes - Futures - 3 (Combinators and Async)

Sun, 12 Jun 2016 14:26:28 +0000

In the previous parts of this post, we discussed about Futures and Promises. In this last part, we’ll compose Futures using its powerful combinators.

Composing Futures :

In the first post, we saw how to extract a value from Future using onComplete, foreach and in testcases using Await.result. Extracting a value from a single Future is good but many a time we spawn more than one asynchronous operation and wait on multiple Futures to arrive at the final result. Even better, sometimes the result of one Future would be fed into another or a chain of Futures.

Future is a Monad. (I am sorry to drop the M-bomb here and I will take a stab at explaining my understanding on what a Monoid, Functor, Monad and an Applicative later). But for now, let’s live with this crude explanation :

Future is a container of a value(s) of some type (i.e it accepts a type as an argument and it can’t exist without it). You can have a Future[Int] or Future[String] or Future[AwesomeClass] - you can’t just have a plain Future. A fancy term for this is type-constructor. To compare, a List is a type constructor (and a Monad as well). A list is a container of values that are of type Int, String or any of other types. A List/Future without a contained type does not exist.
Future has flatMap and unit functions (and consequentially a map function too).

The reason I brought this up is that instead of using the onComplete callback or the foreach, we could simply map or flatMap the result of the Future just like we would do it against an Option or a List.

Now, let’s look at the map and the flatMap combinators.

Mapping Futures that execute sequentially

Let’s consider this simple task, which adds three numbers which are asynchronously calculated after some interval.

Warning : The following code is messy and executes the Futures sequentially

Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23



class FutureCombinators {

 def sumOfThreeNumbersSequentialMap(): Future[Int] = {
 Future {
 Thread.sleep(1000)
 1
 }.flatMap { oneValue =>
 Future {
 Thread.sleep(2000)
 2
 }.flatMap { twoValue =>
 Future {
 Thread.sleep(3000)
 3
 }.map { thirdValue =>
 oneValue + twoValue + thirdValue
 }
 }
 }
 }
...
...

The first Future returns a 1 after 1 second, the second Future returns a 2 after 2 seconds and the third Future returns a 3 after 3 seconds. The nested block finally calculates the sum of three values and returns one single Future[Int].

Testcase

For the sake of calculating the time taken to compute the values, we have a small utility function (inside the ConcurrentUtils trait) called timed which calculates and prints the time taken by a block.

We Await.result to do a blocking wait on the result of futureCombinators.sumOfThreeNumbersSequentialMap. We also time the total execution and print it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


class FutureCombinatorsTest extends FunSpec with Matchers with ConcurrentUtils {

 describe("Futures") {
 it("could be composed using map") {
 val futureCombinators = new FutureCombinators
 val result = timed(Await.result(futureCombinators.sumOfThreeNumbersSequentialMap(), 7 seconds))
 result shouldBe 6
 }
 }
...
...
}

trait ConcurrentUtils {
 def timed[T](block: => T): T = {
 val start = System.currentTimeMillis()
 val result = block
 val duration = System.currentTimeMillis() - start
 println(s"Time taken : $duration")
 result
 }
}

Output

1

Time taken : 6049

The function took a little over 6 seconds to execute which indicates that the Futures are executed in sequence.

Using for-comprehension syntactic sugar instead of Map

Scala gives a great way to work with classes that has map and flatMap (Monads) - for comprehensions. For comprehensions are just syntactic sugar, which gets de-sugared into flatMap and map.

The following code means exactly the same as the above, except that the uglification is done by the Scala compiler.

Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


def sumOfThreeNumbersSequentialForComprehension(): Future[Int] = {
 for {
 localOne <- Future {
 Thread.sleep(1000)
 1
 }
 localTwo <- Future {
 Thread.sleep(2000)
 2
 }
 localThree <- Future {
 Thread.sleep(3000)
 3
 }
 } yield localOne + localTwo + localThree
}

Testcase

It’s the same as above.

1
2
3
4
5


it("could be composed using for comprehensions") {
 val futureCombinators = new FutureCombinators
 val result = timed(Await.result(futureCombinators.sumOfThreeNumbersSequentialForComprehension(), 7 seconds))
 result shouldBe 6
 }

Output

1

Time taken : 6012

Executing Futures in parallel

As we saw, the previous block of code runs the three Futures sequentially and therefore takes a total of 6 seconds to finish the computation. That isn’t good. Our Futures need to run in parallel. In order to achieve this, all we need to do is to extract the Future block out and declare them separately.

Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15



val oneFuture: Future[Int] = Future {
 Thread.sleep(1000)
 1
 }

 val twoFuture: Future[Int] = Future {
 Thread.sleep(2000)
 2
 }

 val threeFuture: Future[Int] = Future {
 Thread.sleep(3000)
 3
 }

Now, let’s use for-comprehension to calculate the value.

1
2
3
4
5


def sumOfThreeNumbersParallelMapForComprehension(): Future[Int] = for {
 oneValue <- oneFuture
 twoValue <- twoFuture
 threeValue <- threeFuture
} yield oneValue + twoValue + threeValue

Testcase

Let’s time the computation and assert the correct value using the following testcase.

1
2
3
4
5
6
7


describe("Futures that are executed in parallel") {
 it("could be composed using for comprehensions") {
 val futureCombinators = new FutureCombinators
 val result = timed(Await.result(futureCombinators.sumOfThreeNumbersParallel(), 4 seconds))
 result shouldBe 6
 }
 }

Output

1

Time taken : 3005

As we see, that sumOfThreeNumbersParallel takes almost the same time as the longest Future (threeFuture), which is 3 seconds.

Just for the sake of comparison, the above code could be written without using for-comprehension as :

1
2
3
4
5
6
7
8



def sumOfThreeNumbersParallelMap(): Future[Int] = oneFuture.flatMap { oneValue =>
 twoFuture.flatMap { twoValue =>
 threeFuture.map { threeValue =>
 oneValue + twoValue + threeValue
 }
 }
}

Guards in for-comprehensions

Just like we add a guarded if clause in for-comprehensions over List and other collections (aka other Monadic types), we could add guards against the generators of Future as well. The following if guard checks if the value returned by the twoFuture is more than 1, which it is.

1
2
3
4
5


def sumOfThreeNumbersParallelWithGuard(): Future[Int] = for {
 oneValue <- oneFuture
 twoValue <- twoFuture if twoValue > 1
 threeValue <- threeFuture
} yield oneValue + twoValue + threeValue

These guards get de-sugared as withFilter like so (I am 90% sure nobody wants to write this way):

1
2
3
4
5
6
7


def sumOfThreeNumbersMapAndFlatMapWithFilter(): Future[Int] = oneFuture.flatMap { oneValue =>
 twoFuture.withFilter(_ > 1).flatMap { twoValue =>
 threeFuture.map { threeValue =>
 oneValue + twoValue + threeValue
 }
 }
 }

Guards in for-comprehensions - Failure case

If the guard evaluates to false thereby the generator yielding a failure, a NoSuchElementException would be thrown. Let’s change the guard condition to evaluate to false.

1
2
3
4
5


def sumOfThreeNumbersParallelWithGuardAndFailure(): Future[Int] = for {
 oneValue <- oneFuture
 twoValue <- twoFuture if twoValue > 2
 threeValue <- threeFuture
} yield oneValue + twoValue + threeValue

Output

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


Future.filter predicate is not satisfied
java.util.NoSuchElementException: Future.filter predicate is not satisfied
	at scala.concurrent.Future$$anonfun$filter$1.apply(Future.scala:280)
	at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
	at scala.util.Try$.apply(Try.scala:192)
	at scala.util.Success.map(Try.scala:237)
	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
	at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Exception handling

Just like the NoSuchElementException thrown by the guard, code executing asynchronously inside a Future could throw a variety of exceptions. While one may argue that Exceptions are not very FP-like, chances are that with a distributed application or through usage of Java libraries inside your Future, exceptions do happen.

Code

Both the two functions below throw Exceptions - the first one throws a NoSuchElementException and the second one throws a LegacyException.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21



 //NoSuchElementException
 def throwsNoSuchElementIfGuardFails(): Future[Int] = for {
 oneValue <- oneFuture
 twoValue <- twoFuture if twoValue > 2
 threeValue <- threeFuture
 } yield oneValue + twoValue + threeValue

 //LegacyException

 val futureCallingLegacyCode: Future[Int] = Future {
 Thread.sleep(1000)
 throw new LegacyException("Danger! Danger!")
 }

 def throwsExceptionFromComputation(): Future[Int] = for {
 oneValue <- oneFuture
 futureThrowingException <- futureCallingLegacyCode
 } yield oneValue + futureThrowingException

case class LegacyException(msg: String) extends Exception(msg)

Testcases

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


 describe("Futures that throw exception") {
 it("could blow up on the caller code when guard fails") {
 val futureCombinators = new FutureCombinators
 intercept[NoSuchElementException] {
 val result = timed(Await.result(futureCombinators.throwsNoSuchElementIfGuardFails(), 4 seconds))
 }
 }

 it("could blow up on the caller code when exception comes from a computation executed inside the Future") {
 val futureCombinators = new FutureCombinators
 intercept[LegacyException] {
 val result = timed(Await.result(futureCombinators.throwsExceptionFromComputation(), 4 seconds))
 }
 }
...
...

Note that even if one of the Futures result in an exception, the entire result of the composed computation will result in propagating the exception.

Recovering from Exception :

Using `recover`

If a Future throws a scala.util.control.NonFatal and we would want to have a default fallback value instead of propagating the error to the caller, we could use the recover function. The recover is much like the catch block.

Let’s modify the above function throwsExceptionFromComputation which throws a LegacyException. The recover function accepts a PartialFunction that maps from Throwable to the type that the Future wraps.

Code

In the below code, if the futureCallingLegacyCode throws an Exception (which it does), the value that is the result of this computation is set to be 200. If it hadn’t thrown an Exception, the resulting value would be the result of that computation itself.

1
2
3
4
5
6
7
8


val futureCallingLegacyCodeWithRecover: Future[Int] = futureCallingLegacyCode.recover {
 case LegacyException(msg) => 200
}

def recoversFromExceptionUsingRecover(): Future[Int] = for {
 oneValue <- oneFuture
 futureThrowingException <- futureCallingLegacyCodeWithRecover
} yield oneValue + futureThrowingException

To reiterate, if the original Future yields a successful value, the recover block is never executed. Also, if the PartialFunction inside the recover function does not handle the exception that is originally thrown, the exception gets propagated to the caller.

Testcase

The testcase asserts that the result of the computation is the sum of values that are returned by oneFuture (which is 1) and the futureCallingLegacyCodeWithRecover (which is 200).

1
2
3
4
5


it("could be recovered with a recovery value") {
 val futureCombinators = new FutureCombinators
 val result = timed(Await.result(futureCombinators.recoversFromExceptionUsingRecover(), 2 seconds))
 result shouldBe 201
 }

Output

1

Time taken : 1004

Using `recoverWith`

Instead of recovering with a value when a Future results in an Exception, we might want to recover with a result of some other Future in some circumstances. Say, unavailability of a HTTP call to Server1 due to network failure could be recovered with a HTTP call to another service running on Server2.

Similar to recover, the recoverWith accepts a PartialFunction. However, the PartialFunction maps a Throwable to a Future of the same type as the original Future.

Just like recover, if the main Future on which the recoverWith is called fails, then the Future that is mapped to in the PartialFunction gets called. If the second future results in a successful value, then the new result is returned.

Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


val futureCallingLegacyCodeWithRecoverWith: Future[Int] = futureCallingLegacyCode.recoverWith {
 case LegacyException(msg) =>
 println("Exception occurred. Recovering with a Future that wraps 1000")
 Thread.sleep(2000)
 Future(1000)
 }

 def recoversFromExceptionUsingRecoverWith(): Future[Int] = for {
 oneValue <- oneFuture
 futureThrowingException <- futureCallingLegacyCodeWithRecoverWith
 } yield oneValue + futureThrowingException

Testcase

The oneFuture takes 1 second and the recovering Future takes 2 seconds. So, we set the Await.result timeout to 4 seconds. The final result 1001 is the sum of the result of oneFuture and futureCallingLegacyCodeWithRecoverWith.

1
2
3
4
5


it("could be recovered with a recovery Future") {
 val futureCombinators = new FutureCombinators
 val result = timed(Await.result(futureCombinators.recoversFromExceptionUsingRecoverWith(), 4 seconds))
 result shouldBe 1001
}

Output

1

Time taken : 3006

Note that just like the recover, if the second future also fails, then the error thrown by the second future gets propagated to the caller.

Code

In the following code, we create another Future that throws an Exception with message Dieded!! and we recover the first Future with this error-throwing-Future. The testcase would reveal that the exception from the second future (recovery one) gets thrown back to the caller.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


val anotherErrorThrowingFuture: Future[Int] = Future {
 Thread.sleep(1000)
 throw new LegacyException("Dieded!!")
 }

 val futureRecoveringWithAnotherErrorThrowingFuture: Future[Int] = futureCallingLegacyCode.recoverWith {
 case LegacyException(msg) =>
 anotherErrorThrowingFuture
 }

 def recoversFromExceptionUsingRecoverWithThatFails(): Future[Int] = for {
 oneValue <- oneFuture
 futureThrowingException <- futureRecoveringWithAnotherErrorThrowingFuture
 } yield oneValue + futureThrowingException

Testcase

1
2
3
4
5
6
7


it("when recovered with another Future that throws Exception would throw the error from the second Future") {
 val futureCombinators = new FutureCombinators
 val exception = intercept[LegacyException] {
 timed(Await.result(futureCombinators.recoversFromExceptionUsingRecoverWithThatFails(), 4 seconds))
 }
 exception.msg shouldBe "Dieded!!"
 }

Using fallbackTo :

fallbackTo works just like recoverWith when it comes to successful value. It uses the first Future’s value if it is successful or falls back to the second Future’s value. However, if both the first and the second Future fails, then the error that is propagated to the caller is that of the first Future and not the second Future.

Code

Let’s use the same Futures that we used in the recoverWith.

1
2
3
4
5
6
7



val futureFallingBackToAnotherErrorThrowingFuture: Future[Int] = futureCallingLegacyCode.fallbackTo (anotherErrorThrowingFuture)

def recoversFromExceptionUsingFallbackTo(): Future[Int] = for {
 oneValue <- oneFuture
 futureThrowingException <- futureFallingBackToAnotherErrorThrowingFuture
} yield oneValue + futureThrowingException

Notice that the fallbackTo function just accepts another Future and not a PartialFunction like recoverWith.

Testcase

1
2
3
4
5
6
7
8



it("when fallen back to another Future that throws Exception would throw the error from the first Future") {
 val futureCombinators = new FutureCombinators
 val exception = intercept[LegacyException] {
 timed(Await.result(futureCombinators.recoversFromExceptionUsingFallbackTo(), 4 seconds))
 }
 exception.msg shouldBe "Danger! Danger!"
 }

Other interesting and useful combinators

The following is a super-brief list of other Future combinators which I find very useful.

zip

zip works just like List.zip. It just merges two Futures and yields a Future of a Tuple.

1

def zipTwoFutures:Future[(Int,Int)]=oneFuture zip twoFuture

firstCompletedOf

Ah! The firstCompletedOf comes in really handy when you have two equivalent services and you want to proceed once the fastest service returns a value.

1
2


val listOfFutures=List(oneFuture,twoFuture,threeFuture)
def getFirstResult():Future[Int]=Future.firstCompletedOf(listOfFutures)

In the above case, the oneFuture returns the fastest.

sequence

The sequence is pure magic. Say, you have a List[Future[Int]] just like List(oneFuture,twoFuture,threeFuture) and you require that all the values are given back to you as a List[Int] instead of each Int wrapped inside a Future. The sequence takes your List[Future[Int]] and transforms into a Future[List[Int]]

1

def getResultsAsList():Future[List[Int]]=Future.sequence(listOfFutures)

The last time I used was for batching where I executed logic against chunks of data in parallel and combined them together with sequence.

Scala-async library

The Scala Async library is an external project and could be added to the project by adding the dependency into our build.sbt

1

"org.scala-lang.modules" %% "scala-async" % "0.9.6-RC2"

The Async library has just two powerful functions in its scala.async.Async class - async and await.

async

The async function is very similar to the Future.apply function. In fact their signatures are very much the same and we could comfortably replace the Future.apply with async wherever it is available.

Future.apply

1
2

def apply[T](body: =>T)(implicit executor: ExecutionContext): Future[T]

Async

1
2

def async[T](body: => T)(implicit execContext: ExecutionContext): Future[T]

The foremost advantage of using async over Future.apply, other than the general ease of readability, is that for each Future generator (when used with a for comprehension), the compiler yields a separate anonymous class while with async it is just one single anonymous class.

Therefore, we could re-write our oneFuture as,

1
2
3
4


val oneFuture: Future[Int] = async {
 Thread.sleep(1000)
 1
}

await

The await function accepts a Future and returns the result. But isn’t the same as Await.result which accepts a Future and returns the result as well? Nope. The key difference is that the Await.result is blocking and is strongly discouraged to be used in production code except for testcases. The await function, on the other hand is implemented using Scala macros and the implementation is that it returns the result of the Future using the onComplete callback.

Since the async function returns a Future, all other error handling and recovery mechanisms stays the same as before.

Code

Let’s rewrite the previous sum-of-three-numbers with async/await :

1
2
3


def sumOfThreeNumbersParallelWithAsyncAwait(): Future[Int] = async {
 await(oneFuture) + await(twoFuture) + await(threeFuture)
}

Testcase

1
2
3
4
5


it("could be composed using async/await") {
 val futureCombinators = new FutureCombinators
 val result = timed(Await.result(futureCombinators.sumOfThreeNumbersParallelWithAsyncAwait(), 4 seconds))
 result shouldBe 6
 }

As we see, the code written this way is not only asynchronous but also looks natural (in fact, it looks synchronous). We could argue that for-comprehensions are a huge leap from using map and flatMap but async/await goes one big step further.

Code

The code and its corresponding testcase are on github.

Scala notes - Futures - 2 (Promises)

Sun, 05 Jun 2016 14:29:24 +0000

In the last post, we saw how to extract values from the Future upon onComplete and their counterparts - onSuccess and onFailure. We also saw how to use Await.result in Testcases to block and get the value from Future. In this post, we’ll discuss briefly about the relationship between a Promise and a Future.

Promise

The concepts Promise and a Future go hand in hand. A scala.concurrent.Promise is the one which sets a value for the Future. In other words, the Promise is the brain behind executing the computation asynchronously and Future is just a handle for reading the result when it becomes available. Crudely put, the Promise is the setter and the Future is the getter.

Most often, we won’t need to explicitly create a Promise. However, we are required to understand what a Promise is, in order to truly understand how a Future works.

Let’s use the following examples to understand how to create a Promise.

Completing a Promise

In the following piece of code, we will see how a value is set in a promise and how it is read on the other side.

create a Promise
complete the Promise by setting a successful value
then return the read side of the Promise - the Future back to the caller by using the promise.future

There is no time consuming process happening behind the scenes. The value is set to the Promise immediately and therefore the value is immediately available via the Future.

Code

1
2
3
4
5
6
7
8
9


class PromiseInternals {
...
 def aCompletedPromiseUsingSuccess(num:Int): Future[Int] = {
 val promise=Promise[Int]()
 promise.success(num)
 promise.future
 }
...
...

Testcase

When we run the Testcase code, the onComplete callback gets called immediately after the promise.success(100) is called.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


class PromiseInternalsTest extends FunSpec with Matchers {
 describe("A Future") {
 it("gives out the correct value when a Promise is completed") {
 val promiseInternals = new PromiseInternals
 val aCompletedPromise=promiseInternals.aCompletedPromiseUsingSuccess(100)
 assertValue(aCompletedPromise, 100)
 }
...
...

 def assertValueUsingOnComplete(future: Future[Int], expectedValue: Int): Unit = {
 future.onComplete {
 case Success(result) => {
 println (s"Result is $result and expectedValue is $expectedValue")
 result shouldBe expectedValue
 }
 case Failure (msg) => fail(msg)
 }
 }

The promise.success is just a shortcut for using promise.complete which accepts a Try[T] as an argument. So, we could have actually written the above function as :

1
2
3
4
5


def aCompletedPromiseUsingComplete(num:Int): Future[Int] = {
 val promise=Promise[Int]()
 promise.complete(Success(num))
 promise.future
}

Alternatively, if we would like to indicate a failure in computation, we could either use a promise.complete(Failure(throwable)) or

1
2
3
4
5


def aCompletedPromiseUsingFailure(num:Int): Future[Int] = {
 val promise=Promise[Int]()
 promise.failure(new RuntimeException("Evil Exception"))
 promise.future
}

Let’s summarize the above in a picture :

Running a block asynchronously

Now that we saw how to complete a Promise by setting a successful value or an exception, we’ll see how to execute a block of code asynchronously.

In the following Testcase, we pass a block of code to the someExternalDelayedCalculation to be executed asynchronously.

Testcase

Let’s look at the testcase first.

We pass a block as argument. The block of code simply sleeps for 2 seconds and then returns a 100.
Assert the value after 3 seconds.

Simple enough.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


it("gives out the correct value when an asynchronous block is submitted and is completed through a Promise") {
 val promiseInternals = new PromiseInternals
 val longCalculationFuture = promiseInternals.someExternalDelayedCalculation{()=>
 Thread.sleep(2000)
 100
 }
 println (s"We have submitted a block to be executed asynchronously ${longCalculationFuture.isCompleted}") //false at this point
 assertValue(longCalculationFuture, 100)
 }

def assertValue(future: Future[Int], expectedValue: Int): Unit = {
 val resultVal=Await.result(future, 3000 seconds)
 resultVal shouldBe expectedValue
}

Code

The implementation of the someExternalDelayedCalculation is interesting :

create a FixedThreadPool to execute our asynchronous code.
create a Promise
create a Runnable and wrap the block to be run asynchronously in the run method
close the promise and complete the promise using the result of the run
execute the Runnable in the somePool threadpool.
return the promise.future from which the caller can read the value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19



val somePool=Executors.newFixedThreadPool(2)

def someExternalDelayedCalculation(f:()=>Int): Future[Int] = {
 val promise=Promise[Int]()
 val thisIsWhereWeCallSomeExternalComputation = new Runnable {
 override def run(): Unit ={
 promise.complete{
 try(Success(f()))
 catch {
 case NonFatal (msg)=> Failure(msg)
 }
 }
 }
 }

 somePool.execute(thisIsWhereWeCallSomeExternalComputation)
 promise.future
}

That’s it !!

How is the `Future.apply()` actually implemented?

Well, I cheated. The code in bullet 2 is actually stolen from the actual implementation of the Future.apply itself.

Remember in the previous post, we saw that when a block of code is passed into the Future’s apply function, it gets executed asynchronously.

Now, compare the code above in someExternalDelayedCalculation with the actual implementation of Future.apply and the Runnable that it wraps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


 def apply[T](body: =>T)(implicit executor: ExecutionContext): scala.concurrent.Future[T] = {
 val runnable = new PromiseCompletingRunnable(body)
 executor.prepare.execute(runnable)
 runnable.promise.future
 }

class PromiseCompletingRunnable[T](body: => T) extends Runnable {
 val promise = new Promise.DefaultPromise[T]()

 override def run() = {
 promise complete {
 try Success(body) catch { case NonFatal(e) => Failure(e) }
 }
 }
 }

To repeat the same steps as above, the apply function

holds the ThreadPool that we provide as the implicit ExecutionContext
creates a Promise by creating a PromiseCompletingRunnable that is a Runnable
wraps the block to be run asynchronously in the run method
closes the promise and completes the promise using the result of the run
executes the Runnable using the ExecutionContext
returns the promise.future from which the caller can read the value.

Once written, twice error

Once the promise gets completed either with a Success or a Failure, all we could do after that is to extract the value from its Future. Also, the onComplete callback of the Future gets called. The value wrapped inside the Future of the Promise is set in stone and cannot be changed.

If we attempt to set a new value by completing an already completed Promise, an IllegalStateException is thrown.

Code

Let’s look at this using a snippet. In the following code, we create a Promise and complete it with a value of 100. We then attempt to complete it with a failure.

1
2
3
4
5
6


def alreadyCompletedPromise(): Future[Int] = {
 val promise = Promise[Int]()
 promise.success(100) //completed
 promise.failure(new RuntimeException("Will never be set because an IllegalStateException will be thrown beforehand"))
 promise.future
}

Testcase

The testcase just asserts that the IllegalStateException gets thrown when an attempt to complete the Promise with a Failure.

1
2
3
4
5
6


it("should throw an error if a Promise is attempted to be completed more than once") {
 val promiseInternals = new PromiseInternals
 intercept[IllegalStateException] {
 promiseInternals.alreadyCompletedPromise()
 }
 }

Code

The complete code backing this blog is available in github

Scala Notes - Futures - 1

Sun, 29 May 2016 14:21:18 +0000

Almost all modern programming languages have a Future-Promise idiom for concurrent programming. I don’t intend to bore you with why we need higher level of concurrency abstractions. Instead, in this post, we’ll cut to the chase and discuss only about Scala’s approach to Futures.

A scala.concurrent.Future is a representation of a value that is yet to be realized. This value is generally a result of a longer and/or parallel computation.

In crude terms, a block of code which runs synchronously, when wrapped in a Future runs asynchronously. All implementations of Future gives us a handle through which we could can get retrieve the resulting value of the computation of the block (or what’s the point in it!).

In this post, we’ll look at the basics of how to construct a Future and extract the value out of it through blocking wait and callbacks. In the next part, we’ll talk about composing Futures and other advanced constructs such as recover and fallback for exception handling.

Creating an asynchronous computation

Creating a computation that runs asynchronously using Future is super easy. We just need to throw in our logic into the apply function of the Future

1
2
3


val aFuture: Future[Int] = Future {
 //Some massively huge super important computation
}

As an example, let’s create a oneFuture that returns a 1 after a delay of one second.

1
2
3
4


val oneFuture: Future[Int] = Future {
 Thread.sleep(1000)
 1
}

Let’s pause and explore the apply function

1

def apply[T](body: =>T)(implicit executor: ExecutionContext): scala.concurrent.Future[T]

Yup. You got me there. The method also accepts something called an ExecutionContext as an implicit argument. When a block of code is used to construct a Future, the computation runs on an ExecutionContext. The ExecutionContext is synonymous with a ThreadPool and when a Future is started, it runs on a separate Thread.

Scala provides a ready-made static global ExecutionContext in scala.concurrent.ExecutionContext.global which we plan to use now. (More on this in a moment)

So, the entire code actually looks like this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


class PromisingFutures{

 import scala.concurrent.ExecutionContext.Implicits.global

 val oneFuture: Future[Int] = Future {
 Thread.sleep(1000)
 1
 }
 ...
 ...

On using `scala.concurrent.ExecutionContext.global`

The global ExecutionContext is very convenient to use. However, the underlying ThreadPool is a ForkJoinPool. Nothing wrong with this and ForkJoinPool is amazing for short-lived computations but is strongly not recommended for Blocking IO such as database or webservice calls (or for that sake even long running computations or simply talking outside the JVM).

The good news is that we still can use Future but the workaround is to simply use a separate threadpool that is not ForkJoinPool - a fixed ThreadPool, for example, is a good option.

1
2
3
4


implicit lazy val fixedThreadPoolExecutionContext: ExecutionContext = {
 val fixedThreadPool: ExecutorService = Executors.newFixedThreadPool(Runtime.getRuntime.availableProcessors * 2) //or some fixed number
 ExecutionContext.fromExecutor(fixedThreadPool)
}

Future States and values

Before we go into the extracting the values out of Future, let’s see what are the various states of Future and when and what value is actually available during those states.

The Future just has two states - Not Completed and Completed.

Consider the following image

When given a computation, the resulting value that we could get from the Future is an Option[Try[T]].

In Not Complete state, the result of the computation is not yet realized. The value therefore would be a None.
Once Completed, the result is a Some(Try[T]) which means it could be one of the following :
1. A positive outcome of the computation or
2. An Exception

Let’s look at the Success case with an example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



def checkState(): Unit = {
 println("Before the job finishes")
 Thread.sleep(500)
 println(s"Completed : ${oneFuture.isCompleted}, Value : ${oneFuture.value}")

 println("After the job finishes")
 Thread.sleep(1100)
 println(s"Completed : ${oneFuture.isCompleted}, Value : ${oneFuture.value}")

}

The time taken for the delayedStringFuture Future to complete is 1 seconds. So, we first check after 500 milliseconds, whether the Future is complete and print its current value. The value function of the Future returns an Option[Try[T]]. Not surprisingly, we get a false for the isCompleted check and a None as the value itself.

Output for 500 ms:

1
2


Before the job finishes
Completed : false, Value : None

Let’s check back again after 1100 milliseconds giving some leeway after that 1000 millisecond sleep. The output now is the result of the computation itself and the completion status is true now..

Output for 1100 ms:

1
2


After the job finishes
Completed : true, Value : Some(Success(1))

Now that we’ve got that out of our way, let’s see how to extract a value out of a Future

Extracting value out of Future

Other than composing futures, which we’ll be seeing next week, there are two ways to extract just the value out of a Future - Blocking wait and callback

Blocking wait using Await.result :

The scala.concurrent.Await’s result function has the following syntax :

1
2
3


@throws(classOf[Exception])
def result[T](awaitable: Awaitable[T], atMost: Duration): T =
 blocking(awaitable.result(atMost)(AwaitPermission))

It accepts an implementation of the Awaitable trait, which a Future is. It also accepts a second parameter atMost which indicates the maximum time duration the caller thread has to block for the result. Upon expiry of the atMost duration, if the Future still didn’t complete, then a java.util.concurrent.TimeoutException would be thrown.

If the Future is complete, Await.result will extract us the actual value. If the Future is complete and if the Future’s result is a Throwable, the Exception gets propagated to the caller.

Using Await.result in production code is highly discouraged but this construct comes in handy for running testcases against Future.

Let’s consider two Futures, the original oneFuture and a oneDangerousFuture which throws an Exception.

Code

Here’s how they look :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


val oneFuture: Future[Int] = Future {
 Thread.sleep(1000)
 1
 }

 val oneDangerousFuture=Future{
 Thread.sleep(2000)
 throw new SomeComputationException("Welcome to the Dark side !")
 }

 case class SomeComputationException(msg: String) extends Exception(msg)

Testcases

We have three testcases here :

The first one is our happy day scenario - The computation takes 1 second, we wait for result for a maximum duration of 2 seconds. We obviously will have our result in hand. Our assertion that a value should be returned becomes true.
In the second testcase, we spawn a Future that throws a SomeComputationException and we assert that the exception gets propagated to the caller when we await for the result.
In the last testcase, we wait only for 500 milliseconds while the computation itself takes 1 second. As we saw from the implementation of the Await.result, this call throws a TimeoutException.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


class PromisingFutureTest extends FunSpec with Matchers {

 describe("A PromisingFuture") {

 it("should hold a Int value if the Await.result is called after the Future completes") {
 val promisingFuture = new PromisingFutures()
 val oneFuture = promisingFuture.oneFuture //Takes 1 second to compute
 val intValue = Await.result(oneFuture, 2 seconds)
 intValue should be(1)
 }

 it("should propagate the Exception to the callee if the computation threw an exception") {
 val promisingFuture = new PromisingFutures()
 val oneDangerousFuture = promisingFuture.oneDangerousFuture //throws exception
 intercept[SomeComputationException] {
 val intValue = Await.result(oneDangerousFuture, 2 seconds)
 }
 }

 it("should throw a TimeOutException exception when an Await.result's atMost parameter is lesser than the time taken for the Future to complete") {
 val promisingFuture = new PromisingFutures()
 val oneDelayedFuture = promisingFuture.oneFuture //Takes 1 second to compute
 intercept[TimeoutException] {
 Await.result(oneDelayedFuture, 500 millis)
 }
 }
 }

}

Callback :

The alternative and the clean way to extract a value from the Future (other than composing) is by way of callbacks. There are three different callbacks available on the Future - the onSuccess, the onFailure and the combined onComplete.

The onSuccess callback gets called only when the Future completes successfully with a result.
The onFailure callback gets called only when there is an Exception.
The onComplete is a combination of onSuccess and onFailure. It accepts a function that works on a Try[T] after getting the Option around the Future’s result unwrapped.

1

def onComplete[U](f: Try[T] => U)(implicit executor: ExecutionContext): Unit

Note that all callbacks return a Unit which means that they can’t be composed and are side-effecting.

Now let’s see how we could use the onComplete callback. I have this little method called printFuture which just writes to the console the contents of the Future once it is complete. Let’s try to pass in both the oneFuture and the oneDangerousFuture into it.

1
2
3
4
5
6
7
8


class PromisingFutures {
...
...
def printFuture[T](future: Future[T]): Unit = future.onComplete {
 case Success(result) => println(s"Success $result")
 case Failure(throwable) => println(s"Failure $throwable")
}
...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



object PromisingFutures{

 def main(args: Array[String]) {
 val promisingFutures=new PromisingFutures
 promisingFutures.printFuture(promisingFutures.oneFuture)
 promisingFutures.printFuture(promisingFutures.oneDangerousFuture)

 synchronized(wait(3000))
 }
}

Output :

1
2


Success 1
Failure SomeComputationException: Welcome to the Dark side !

As expected, the oneFuture goes into the Success case and yields 1 while the oneDangerousFuture goes into the Failure case and prints the Exception :

TimeUnit

Scala provides a very convenient DSL-like syntax to represent TimeUnit eg. 2 seconds, 5 minutes etc. Three implicit classes scala.concurrent.duration package - DurationInt, DurationLong and DurationDouble and the trait DurationConversions do this magic. All we need to do is to import scala.concurrent.duration._.

Also, the “500 millis” in our example could be represented as “500 milli”, “500 milliseconds” or “500 millisecond”. The entire list of aliases for various time units could be found in the sourcecode or Scaladoc of the trait DurationConversions.

Code

The complete code backing this blog is available in github

Addendum

@lightspeed7 pointed out that with Scala 2.12, the partial versions of onComplete, namely onSuccess and onFailure is deprecated. The deprecation is primarily to favor total versions such as onComplete and foreach in place of partials that handles either Success or Failure cases such as onSuccess and onFailure.

Let’s explore a little more on Future.foreach with the following printing function.

1

def printWithForEach[T](future: Future[T]): Unit = future.foreach(println)

With foreach, the Future.foreach delegates itself to the Try.foreach. So, if the Future returns a Success, a Success(value) would be printed but for a Failure, nothing would be printed simply because the implementation of Failure goes like this :

1

def foreach[U](f: T => U): Unit = ()

Akka Notes - Finite State Machines - 2

Sun, 22 May 2016 16:09:56 +0000

In the first part of notes on Akka FSM, we saw the basics of Akka FSM and the outline of the Coffee vending machine that we planned to build - the structure of the Actor and a list of messages we pass to the Actor. In this second and final part, we will go ahead and implement each of these States.

Recap

As a quick recap, let’s look at the structure of the FSM and the messages that can be sent to it.

States and Data

The three States of the FSM and the Data being sent across the States are :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



object CoffeeMachine {

 sealed trait MachineState
 case object Open extends MachineState
 case object ReadyToBuy extends MachineState
 case object PoweredOff extends MachineState

 case class MachineData(currentTxTotal: Int, costOfCoffee: Int, coffeesLeft: Int)

}

Messages

The Vendor and User Interaction messages that we send to the FSM are :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21



object CoffeeProtocol {

 trait UserInteraction
 trait VendorInteraction

 case class Deposit(value: Int) extends UserInteraction
 case class Balance(value: Int) extends UserInteraction
 case object Cancel extends UserInteraction
 case object BrewCoffee extends UserInteraction
 case object GetCostOfCoffee extends UserInteraction

 case object ShutDownMachine extends VendorInteraction
 case object StartUpMachine extends VendorInteraction
 case class SetNumberOfCoffee(quantity: Int) extends VendorInteraction
 case class SetCostOfCoffee(price: Int) extends VendorInteraction
 case object GetNumberOfCoffee extends VendorInteraction

 case class MachineError(errorMsg:String)

}

Structure of FSM Actor

Here’s the overall structure that we saw in Part 1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30



class CoffeeMachine extends FSM[MachineState, MachineData] {

 //What State and Data must this FSM start with (duh!)
 startWith(Open, MachineData(..))

 //Handlers of State
 when(Open) {
 ...
 ...

 when(ReadyToBuy) {
 ...
 ...

 when(PoweredOff) {
 ...
 ...

 //fallback handler when an Event is unhandled by none of the States.
 whenUnhandled {
 ...
 ...

 //Do we need to do something when there is a State change?
 onTransition {
 case Open -> ReadyToBuy => ...
 ...
 ...
}

Initial State

As with any State Machine, an FSM needs an initial state to start with. This can be declared inside Akka FSM in a very intuitive method named startWith. The startWith accepts two arguments - the initial state and the initial data.

1
2
3
4
5
6
7



class CoffeeMachine extends FSM[MachineState, MachineData] {

 startWith(Open, MachineData(currentTxTotal = 0, costOfCoffee = 5, coffeesLeft = 10))

...
...

The above code just says that the FSM’s initial state is Open and the initial data while the Coffee machine is opened is MachineData(currentTxTotal = 0, costOfCoffee = 5, coffeesLeft = 10).

Since the machine has just started, the vending machine starts with a clean slate. It had no interaction with any user yet and therefore the current showing balance for this transaction is 0. The price of the coffee is set at $5 and the total number of coffees that the machine could vend in total is 10. Once the coffees has vended 10 coffees and is left with 0, the machine shuts down.

Implementing the States

Ah, Finally !!

I felt that the easiest way to look at the interactions with the vending machine at different states is by grouping the interactions, write testcases around it and accompany that with the implementation in the FSM.

If you are referring to the github code, all tests are in the CoffeeSpec and the FSM is the CoffeeMachine

All the following tests are wrapped inside the CoffeeSpec test class whose declaration goes like :

1

class CoffeeSpec extends TestKit(ActorSystem("coffee-system")) with MustMatchers with FunSpecLike with ImplicitSender

Setting and Getting Price of coffee

As we saw above, the MachineData is initiated at $5 dollars per coffee and with a capacity of 10 coffees. This is just an initial state. The Vendor must have the capacity to set the price of the coffee and the capacity of the machine at any point of time.

Setting of price is achieved by sending the SetCostOfCoffee message to the Actor. We should also have the ability to get the price of coffee. This is done by using the GetCostOfCoffee message to which the Machine responds with the currently set price.

Testcase

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


describe("The Coffee Machine") {

 it("should allow setting and getting of price of coffee") {
 val coffeeMachine = TestActorRef(Props(new CoffeeMachine()))
 coffeeMachine ! SetCostOfCoffee(7)
 coffeeMachine ! GetCostOfCoffee
 expectMsg(7)
 }
...
...
...

Implementation

Like we discussed in Part 1, every message that is being sent to the FSM is received and wrapped in an Event class which also wraps around the MachineData:

1
2
3
4
5
6
7


 when(Open) {
 case Event(SetCostOfCoffee(price), _) => stay using stateData.copy(costOfCoffee = price)
 case Event(GetCostOfCoffee, _) => sender ! (stateData.costOfCoffee); stay()
 ...
 ...
 }
}

There are a few words which are new in the above code - stay, using and stateData. Let’s look at them in detail.

`stay` and `goto`

The idea is that each of the case blocks in a state must return a State. This could either be done using stay which just means that at the end of processing this message (SetCostOfCoffee or GetCostOfCoffee), the CoffeeMachine remains in the same State, which is Open in our case.

The goto, on the other hand transitions to a different State. We’ll see how it is done while discussing the Deposit message.

Not surprisingly, check out the implementation of the stay function

1

final def stay(): State = goto(currentState.stateName)

`using`

As you might already guessed, the using function allows us to pass modified data to the next state. In case of the SetCostOfCoffee message, we set the costOfCoffee field of the MachineData to the incoming price wrapped inside the SetCostOfCoffee. Since State is a case class (immutability is strongly advised unless you have fun debugging at odd hours), we do a copy.

`stateData`

The stateData is just a function that gives us a handle to the data of the FSM, which is the MachineData itself. So, the following blocks of code are equivalent

1

case Event(GetCostOfCoffee, _) => sender ! (stateData.costOfCoffee); stay()

1

case Event(GetCostOfCoffee, machineData) => sender ! (machineData.costOfCoffee); stay()

The implementation of the maximum number of coffees to be dispensed using GetNumberOfCoffee and SetNumberOfCoffee is almost the same as the setting and getting the price itself. Let’s skip that and go for the interesting part - Buying coffee.

Buying coffee

So, the coffee enthusiast deposits money for the coffee but we can’t allow the machine to dispense coffee until he has entered the cost of a coffee. Also, if he has given extra cash, we’ll have to give him the balance. So, the various cases goes like this :

Until the User deposits money to cover a coffee, we’ll keep track of the cumulative amount he has deposited and stay in the Open state.
Once the cumulative cash exceeds the price of a coffee, we’ll transition to the ReadyToBuy state and allow him to buy coffee.
At this ReadyToBuy state, he could change his mind and Cancel the transaction during which all his cumulative Balance is returned.
If he wishes to drink the coffee, he sends the machine a BrewCoffee message instead during which we dispense the coffee and give him back the Balance money as well. (Actually, in our code, we don’t dispense the coffee. We just subtract the price of the coffee from his deposit and give him the balance. Such a rip off !!)

Let’s take each of the above cases

Case 1 - User deposits cash but falls short of the price of a coffee

Testcase

The testcase starts by setting the cost of the coffee to the $5 and the total number of coffees in the machine to be 10. We then Deposit $2 which is less than the price of the coffee and check if the Machine is in the Open state and the total number of coffees in the machine remains at 10.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


it("should stay at Transacting when the Deposit is less then the price of the coffee") {
 val coffeeMachine = TestActorRef(Props(new CoffeeMachine()))
 coffeeMachine ! SetCostOfCoffee(5)
 coffeeMachine ! SetNumberOfCoffee(10)
 coffeeMachine ! SubscribeTransitionCallBack(testActor)

 expectMsg(CurrentState(coffeeMachine, Open))

 coffeeMachine ! Deposit(2)

 coffeeMachine ! GetNumberOfCoffee

 expectMsg(10)
 }

So, how exactly did we ensure that the machine is in the Open state?

Each FSM can handle a special message called FSM.SubscribeTransitionCallBack(callerActorRef) which enables the caller to be notified of any State transitions. The first notification that gets sent on subscription is the CurrentState, which tells us what State the FSM is currently in. This is followed by several Transition messages when that happens.

Implementation

So, we add up deposit to the cumulative transaction total and stay in Open state waiting for more Deposit

1
2
3
4
5
6
7


when(Open) {
...
...
 case Event(Deposit(value), MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) if (value + currentTxTotal) < stateData.costOfCoffee => {
 val cumulativeValue = currentTxTotal + value
 stay using stateData.copy(currentTxTotal = cumulativeValue)
 }

Case 2 and 4 - User deposits amount that covers the price of coffee

Testcase 1 - Deposit equal to the price of coffee

Our testcase bootstraps the machine, confirms if the current state is Open and then deposits $5 which is exactly the price of the coffee. We then assert that the machine has transitioned from Open to ReadyToBuy by way of expecting a Transition message which gives us information about the from and the to states of the coffee machine. In the first case, it is the transition from Open to ReadyToBuy.

We then go further and ask the machine to BrewCoffee during which we expect a transition again from ReadyToBuy to Open upon dispensing the coffee. Finally an assertion is made against the remaining number of coffees in the machine (which is 9 now).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20



it("should transition to ReadyToBuy and then Open when the Deposit is equal to the price of the coffee") {
 val coffeeMachine = TestActorRef(Props(new CoffeeMachine()))
 coffeeMachine ! SetCostOfCoffee(5)
 coffeeMachine ! SetNumberOfCoffee(10)
 coffeeMachine ! SubscribeTransitionCallBack(testActor)

 expectMsg(CurrentState(coffeeMachine, Open))

 coffeeMachine ! Deposit(5)

 expectMsg(Transition(coffeeMachine, Open, ReadyToBuy))

 coffeeMachine ! BrewCoffee
 expectMsg(Transition(coffeeMachine, ReadyToBuy, Open))

 coffeeMachine ! GetNumberOfCoffee

 expectMsg(9)
 }

Testcase 2 - Deposit greater than the price of the coffee

The second test case is 90% similar to the first testcase except that we deposit cash in increments that is greater than the price of the coffee ($6). Since we set the price of coffee to be $5, we now expect a Balance message with value $1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


it("should transition to ReadyToBuy and then Open when the Deposit is greater than the price of the coffee") {
 val coffeeMachine = TestActorRef(Props(new CoffeeMachine()))
 coffeeMachine ! SetCostOfCoffee(5)
 coffeeMachine ! SetNumberOfCoffee(10)
 coffeeMachine ! SubscribeTransitionCallBack(testActor)

 expectMsg(CurrentState(coffeeMachine, Open))

 coffeeMachine ! Deposit(2)
 coffeeMachine ! Deposit(2)
 coffeeMachine ! Deposit(2)

 expectMsg(Transition(coffeeMachine, Open, ReadyToBuy))

 coffeeMachine ! BrewCoffee

 expectMsgPF(){
 case Balance(value)=>value==1
 }

 expectMsg(Transition(coffeeMachine, ReadyToBuy, Open))

 coffeeMachine ! GetNumberOfCoffee

 expectMsg(9)
 }

Implementation

The implementation is much simpler than the test cases itself. If the deposit amount is greater than or equal to the cost of coffee, then we goto ReadyToBuy state using the accumulated amount.

1
2
3
4
5
6


when(Open){
...
...
 case Event(Deposit(value), MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) if (value + currentTxTotal) >= stateData.costOfCoffee => {
 goto(ReadyToBuy) using stateData.copy(currentTxTotal = currentTxTotal + value)
 }

Once transitioned to the ReadyToBuy state, when the user sends a BrewCoffee, we check if there is a balance to be dispensed. If not, we just transition to Open state after subtracting one coffee from the total number of coffees. Else, we disburse the balance and transition to Open state after subtracting the number of coffees. (Like I said earlier, we don’t actually dispense coffee in this example)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


when(ReadyToBuy) {
 case Event(BrewCoffee, MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) => {
 val balanceToBeDispensed = currentTxTotal - costOfCoffee
 logger.debug(s"Balance is $balanceToBeDispensed")
 if (balanceToBeDispensed > 0) {
 sender ! Balance(value = balanceToBeDispensed)
 goto(Open) using stateData.copy(currentTxTotal = 0, coffeesLeft = coffeesLeft - 1)
 }
 else goto(Open) using stateData.copy(currentTxTotal = 0, coffeesLeft = coffeesLeft - 1)
 }
}

That’s it !! We’ve covered the juice of the program.

Case 3 - User wishes to Cancel the transaction

Actually, the User should be in a position to Cancel the transaction at any point, whatever state he is in. Like we discussed in Part 1, the perfect place to hold these kind of generic messages is in the whenUnhandled block. We should also make sure that if the user has deposited some cash before canceling, we should give it back to them.

Implementation

1
2
3
4
5
6
7
8


whenUnhandled {
...
...
 case Event(Cancel, MachineData(currentTxTotal, _, _)) => {
 sender ! Balance(value = currentTxTotal)
 goto(Open) using stateData.copy(currentTxTotal = 0)
 }
}

Testcase

The testcase is simply the same as the one we saw above except that the balance issued upon cancellation is the cumulative deposit.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


it("should transition to Open after flushing out all the deposit when the coffee is canceled") {
 val coffeeMachine = TestActorRef(Props(new CoffeeMachine()))
 coffeeMachine ! SetCostOfCoffee(5)
 coffeeMachine ! SetNumberOfCoffee(10)
 coffeeMachine ! SubscribeTransitionCallBack(testActor)

 expectMsg(CurrentState(coffeeMachine, Open))

 coffeeMachine ! Deposit(2)
 coffeeMachine ! Deposit(2)
 coffeeMachine ! Deposit(2)

 expectMsg(Transition(coffeeMachine, Open, ReadyToBuy))

 coffeeMachine ! Cancel

 expectMsgPF(){
 case Balance(value)=>value==6
 }

 expectMsg(Transition(coffeeMachine, ReadyToBuy, Open))

 coffeeMachine ! GetNumberOfCoffee

 expectMsg(10)
 }

Code

I didn’t want to bore you to death and took the liberty of skipping the explanation for ShutDownMachine message and the PoweredOff state but if you are looking forward to an explanation for them, please leave a comment.

As always, the code is available on github.

Akka Notes - Finite State Machines - 1

Sat, 21 May 2016 17:40:33 +0000

I recently had the opportunity to play with Akka FSM at work for some really interesting use-case. The API (in fact, the DSL) is pretty awesome and the entire experience was amazing. Here’s my attempt to log my notes on building a Finite State Machine using Akka FSM. As an example, we’ll walk through the steps of building an (limited) Coffee vending machine.

Why not `become` and `unbecome`?

We know that the plain vanilla Akka Actors can switch its behavior by using become/unbecome. Then, why do we need Akka FSM? Can’t a plain Actor just switch between the States and behave differently? Yes, it could. But while Akka’s become and unbecome is most often enough to switch the behavior of Actors with a couple of states involved, building a State Machine with more than a few states quickly makes the code hard to reason with (and even harder to debug).

Not surprisingly, the popular recommendation is to switch to Akka FSM if there are more than 2 states in our Actor.

What’s Akka FSM

To expand on it further, Akka FSM is Akka’s approach to building Finite State Machines that simplifies management of behavior of an Actor in various states and transitions between those states.

Under the hood, Akka FSM is just a trait that extends Actor.

1
2

trait FSM[S, D] extends Actor with Listeners with ActorLogging

What this FSM trait provides is pure magic - it provides a DSL that wraps a regular Actor enabling us to focus on building the state machine that we have in hand, faster.

In other words, our regular Actor has just one receive function and the FSM trait wraps a sophisticated implementation of the receive method which delegates calls to the block of code that handles the data in a particular state.

One other good thing I personally noticed is that after writing, the complete FSM Actor still looks clean and easily readable.

Alright, let’s get to the code. Like I said, we will be building a Coffee Vending Machine using Akka FSM. The State Machine looks like this :

State and Data

With any FSM, there are two things involved at any moment in a FSM - the State of the machine at any point in time and the Data that is shared among the states. In Akka FSM, in order to check which is our Data and which are the States, all we need to do is to check its declaration.

1
2

class CoffeeMachine extends FSM[MachineState, MachineData]

This simply means that all the States of the FSM extend from the MachineState and the data that is shared between these various States is just MachineData.

As a matter of style, just like with normal Actor where we declare all our messages in a companion object, we declare our States and Data in a companion object:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



object CoffeeMachine {

 sealed trait MachineState
 case object Open extends MachineState
 case object ReadyToBuy extends MachineState
 case object PoweredOff extends MachineState

 case class MachineData(currentTxTotal: Int, costOfCoffee: Int, coffeesLeft: Int)

}

So, as we captured in the State Machine diagram, we have three States - Open, ReadyToBuy and PoweredOff. Our data, the MachineData holds (in reverse order) the numbers of coffees that the vending machine could dispense before it shuts itself down (coffeesLeft), the price of each cup of coffee (costOfCoffee) and finally, the amount deposited by the vending machine user (currentTxTotal) - if it is less than the cost of the coffee, the machine doesn’t dispense coffee, if it is more, then we ought to give back the balance cash.

That’s it. We are done with the States and the Data.

Before we go through the implementation of each of the States that the vending machine can be in and the various interactions that the user can have with the machine at a particular State, we’ll have a 50,000 feet view of the FSM Actor itself.

Structure of FSM Actor

The structure of the FSM Actor looks very similar to the State Machine diagram itself and it looks like this :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30



class CoffeeMachine extends FSM[MachineState, MachineData] {

 //What State and Data must this FSM start with (duh!)
 startWith(Open, MachineData(..))

 //Handlers of State
 when(Open) {
 ...
 ...

 when(ReadyToBuy) {
 ...
 ...

 when(PoweredOff) {
 ...
 ...

 //fallback handler when an Event is unhandled by none of the States.
 whenUnhandled {
 ...
 ...

 //Do we need to do something when there is a State change?
 onTransition {
 case Open -> ReadyToBuy => ...
 ...
 ...
}

What we understand from the structure:

We have an initial State (which is Open) and any messages that is being sent to the Machine during Open State is handled in the when(Open) block, ReadyToBuy state is handled in when(ReadyToBuy) block and so on. The messages that I am referring to here are just like regular messages that we tell a plain Actor except that in case of FSMs, the message is wrapped along with the Data as well. The wrapper is called an Event (akka.actor.FSM.Event) and an example would look like Event(deposit: Deposit, MachineData(currentTxTotal, costOfCoffee, coffeesLeft))

From the Akka documentation :

1
2
3
4
5
6



/**
 * All messages sent to the [[akka.actor.FSM]] will be wrapped inside an
 * `Event`, which allows pattern matching to extract both state and data.
 */
 case class Event[D](event: Any, stateData: D) extends NoSerializationVerificationNeeded

We also notice that the when function accepts two mandatory parameters - the first being being the name of the State itself, eg. Open, ReadyToBuy etc and the second argument is a PartialFunction, just like an Actor’s receive where we do pattern matching. The most important thing to note here is that each of these pattern matching case blocks must return a State (more on this in next post). So, the code block would look something like

1
2
3
4
5



when(Open) {
 case Event(deposit: Deposit, MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) => {
 ...
 ...

Generally, only those messages that match the patterns declared inside the when’s second argument gets handled at a particular State. If there is no matching pattern, then the FSM Actor tries to match our message to a pattern declared in the whenUnhandled block. Ideally, all the messages that are common across all the States is coded away in the whenUnhandled. (I am not the one to suggest style but alternatively, you could declare smaller PartialFunctions and compose them using andThen if you would like to reuse pattern matching across selected States)
Finally, there is an onTransition function which allows you to react or get notified of changes in States.

Interactions/Messages

There are two kinds of people who interact with this Vending machine - Coffee drinkers, who need coffee and Vendors, who do the machine’s administrative tasks.

For sake of organization, I have introduced two traits for all the interactions with the Machine. (Just to refresh, an Interaction/Message is the first element wrapped inside the Event along with the MachineData). In plain old Actor terms, this is equivalent to the message that we send to the Actors.

1
2
3
4
5
6
7



object CoffeeProtocol {

 trait UserInteraction
 trait VendorInteraction
...
...

VendorInteraction

Let’s also declare the various interactions that a Vendor can make with the machine.

1
2
3
4
5
6
7



case object ShutDownMachine extends VendorInteraction
case object StartUpMachine extends VendorInteraction
case class SetCostOfCoffee(price: Int) extends VendorInteraction
//Sets Maximum number of coffees that the vending machine could dispense
case class SetNumberOfCoffee(quantity: Int) extends VendorInteraction
case object GetNumberOfCoffee extends VendorInteraction

So, the Vendor can

start and shutdown the machine
set the price of the coffee and
set and get the number of coffee remaining in the machine.

UserInteraction

1
2
3
4
5
6



case class Deposit(value: Int) extends UserInteraction
case class Balance(value: Int) extends UserInteraction
case object Cancel extends UserInteraction
case object BrewCoffee extends UserInteraction
case object GetCostOfCoffee extends UserInteraction

Now, for the UserInteraction, the User can

deposit money to buy a coffee
get dispensed the extra cash if the deposited money is more than the cost of the coffee
ask the machine to brew coffee if the deposit money is equal or more than the cost of the coffee
cancel the transaction before brewing the coffee and get back all the money deposited
query the machine for the cost of the coffee.

In the next post, we’ll go through each of the States and explore on their interactions (along with testcases) in detail.

Code

For the benefit of the impatient, the entire code is available on github.

Akka Notes - Actor Supervision - 8

Mon, 10 Nov 2014 09:26:35 +0000

Failures are more like a feature among distributed systems. And with Akka’s let it crash fault tolerance model, you could achieve a clear separation between your business logic and your failure handling logic (supervision logic). All with very little effort. It’s pretty amazing. This is the topic of our discussion now.

Actor Supervision

Imagine a method call stack and the top most method in your stack throws an Exception. What could be done by the methods down the stack?

The exception could be caught and handled in order to recover
The exception could be caught, may be logged and kept quiet.
The methods down the stack could also choose to duck the exception completely (or may be caught and rethrown)

Imagine if all the methods until the main method doesn’t handle the exception. In that case, the program exits after writing an essay for an exception to the console.

You could also compare the same scenario with spawning Threads. If a child thread throws an exception and if the run or the call method doesn’t handle it, then the exception is expected to be handled by the parent thread or the main thread, whatever be the case. If the main thread doesn’t handle it, then the system exits.

Let’s do it one more time - if the Child Actor which was created using the context.actorOf fails with an Exception, the parent actor (aka supervisor) could prefer to handle any failures of the child actor. If it does, it could prefer to handle it and recover (Restart/Resume). Else, duck the exception (Escalate) to its parent. Alternatively, it could just Stop the child actor - that’s the end of story for that child. Why did I say parent (aka supervisor)? Simply because Akka’s approach towards supervision is Parental supervision - which means that only the creators of the Actors could supervise over them.

That’s it !! We have pretty much covered all the supervision Directives (what could be done about the failures).

Strategies

Ah, I forgot to mention this one : You already know that an Akka Actor could create children and that they could create as many children as they want.

Now, consider two scenarios :

1. OneForOneStrategy

Your Actor spawns multiple child actors and each one of these child actors connect to different datasources. Say you are running an app which translates an english word into multiple languages.

Suppose, one child actor fails and you are fine to skip that result in the final list, what would you want to do? Shut down the service? Nope, you might want to just restart/stop only that child actor. Isn’t it? Now that’s called OneForOneStrategy in Akka supervision strategy terms - If one actor goes down, just handle one alone.

Depending on your business exceptions, you would want to react differently (Stop, Restart, Escalate, Resume) to different exceptions. To configure your own strategy, you just override the supervisorStrategy in your Actor class.

An example declaration of OneForOneStrategy would be

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19



import akka.actor.Actor
import akka.actor.ActorLogging
import akka.actor.OneForOneStrategy
import akka.actor.SupervisorStrategy.Stop


class TeacherActorOneForOne extends Actor with ActorLogging {

	...
 ...
 override val supervisorStrategy=OneForOneStrategy() {

 case _: MinorRecoverableException 	=> Restart
 case _: Exception => Stop

 }
 ...
 ... 

2. AllForOneStrategy

Assume that you are doing an External Sort (One more example to prove that my creativity sucks!!), and each of your chunk is handled by a different Actor. Suddenly, one Actor fails throwing an exception. It doesn’t make any sense to continue processing the rest of the chunks because the final result wouldn’t be correct. So, it is logical to Stop ~~ALL~~ the actors.

Why did I say Stop instead of Restart in the previous line? Because Restarting would also not make any sense for this use-case considering the mailbox for each of these Actors would not be cleared on Restart. So, if we restart, the rest of the chunks would still be processed. That’s not what we want. Recreating the Actors with shiny new mailboxes would be the right approach here.

Again, just like the OneForOneStrategy, you just override the supervisorStrategy with an implementation of AllForOneStrategy

And example would be

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


import akka.actor.{Actor, ActorLogging}
import akka.actor.AllForOneStrategy
import akka.actor.SupervisorStrategy.Escalate
import akka.actor.SupervisorStrategy.Stop


class TeacherActorAllForOne extends Actor with ActorLogging {

 ...

 override val supervisorStrategy = AllForOneStrategy() {

 case _: MajorUnRecoverableException => Stop
 case _: Exception => Escalate

 }
 ...
 ...

Directives

The constructor of both AllForOneStrategy and the OneForOneStrategy accepts a PartialFunction[Throwable,Directive] called Decider which maps a Throwable to a Directive as you may see here :

1

case _: MajorUnRecoverableException => Stop

There are simply just four kinds of directives - Stop, Resume, Escalate and Restart

Stop

The child actor is stopped in case of exception and any messages to the stopped actor would obviously go to the deadLetters queue.

Resume

The child actor just ignores the message that threw the exception and proceeds with processing the rest of the messages in the queue.

Restart

The child actor is stopped and a brand new actor is initialized. Processing of the rest of the messages in the mailbox continue. The rest of the world is unaware that this happened since the same ActorRef is attached to the new Actor.

Escalate
The supervisor ducks the failure and lets its supervisor handle the exception.

Default Strategy

What if our Actor doesn’t specify any Strategy but has created child Actors. How are they handled? There is a default strategy declared in the Actor trait which (if condensed) looks like below :

1
2
3
4
5
6
7
8


override val supervisorStrategy=OneForOneStrategy() {

 case _: ActorInitializationException=> Stop
 case _: ActorKilledException		=> Stop
 case _: DeathPactException 			=> Stop
 case _: Exception => Restart

}

So, in essence, the default strategy handles four cases :

ActorInitializationException => Stop

When the Actor could not be initialized, it would throw an ActorInitializationException. The Actor would be stopped then. Let’s simulate it by throwing an exception in the preStart callback :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


package me.rerun.akkanotes.supervision

import akka.actor.{ActorSystem, Props}
import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest
import akka.actor.Actor
import akka.actor.ActorLogging

object ActorInitializationExceptionApp extends App{

 val actorSystem=ActorSystem("ActorInitializationException")
 val actor=actorSystem.actorOf(Props[ActorInitializationExceptionActor], "initializationExceptionActor")
 actor!"someMessageThatWillGoToDeadLetter"
}

class ActorInitializationExceptionActor extends Actor with ActorLogging{
 override def preStart={
 throw new Exception("Some random exception")
 }
 def receive={
 case _=>
 }
}

Running the ActorInitializationExceptionApp would generate a ActorInitializationException (duh!!) and then move all the messages into the message queue of the deadLetters Actor:

Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[ERROR] [11/10/2014 16:08:46.569] [ActorInitializationException-akka.actor.default-dispatcher-2] [akka://ActorInitializationException/user/initializationExceptionActor] Some random exception
akka.actor.ActorInitializationException: exception during creation
	at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
...
...
Caused by: java.lang.Exception: Some random exception
	at me.rerun.akkanotes.supervision.ActorInitializationExceptionActor.preStart(ActorInitializationExceptionApp.scala:17)
...
...

[INFO] [11/10/2014 16:08:46.581] [ActorInitializationException-akka.actor.default-dispatcher-4] [akka://ActorInitializationException/user/initializationExceptionActor] Message [java.lang.String] from Actor[akka://ActorInitializationException/deadLetters] to Actor[akka://ActorInitializationException/user/initializationExceptionActor#-1290470495] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

ActorKilledException => Stop

When the Actor was killed using the Kill message, then it would throw an ActorKilledException. The default strategy would stop the child Actor if it throws the exception. At first, it seems that there’s no point in stopping an already killed Actor. However, consider this :

ActorKilledException would just be propagated to the supervisor. What about the lifecycle watchers or deathwatchers of this Actor that we saw during DeathWatch. The watchers won’t know anything until the Actor is Stopped.
Sending a Kill on an Actor would just affect that particular actor which the supervisor knows. However, handling that with Stop would suspend the mailbox of that Actor, suspends the mailboxes of child actors, stops the child actors, sends a Terminated to all the child actor watchers, send a Terminated to all the immediate failed Actor’s watchers and finally stop the Actor itself. (Wow, that’s pretty awesome !!)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


package me.rerun.akkanotes.supervision

import akka.actor.{ActorSystem, Props}
import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest
import akka.actor.Actor
import akka.actor.ActorLogging
import akka.actor.Kill

object ActorKilledExceptionApp extends App{

 val actorSystem=ActorSystem("ActorKilledExceptionSystem")
 val actor=actorSystem.actorOf(Props[ActorKilledExceptionActor])
 actor!"something"
 actor!Kill
 actor!"something else that falls into dead letter queue"
}

class ActorKilledExceptionActor extends Actor with ActorLogging{
 def receive={
 case message:String=> log.info (message)
 }
}

Log

The logs just say that once the ActorKilledException comes in, the supervisor stops that actor and then the messages go into the queue of deadLetters

1
2
3
4
5
6


INFO m.r.a.s.ActorKilledExceptionActor - something

ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill

INFO akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://ActorKilledExceptionSystem/deadLetters] to Actor[akka://ActorKilledExceptionSystem/user/$a#-1569063462] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

DeathPactException => Stop

From DeathWatch, you know that when an Actor watches over a child Actor, it is expected to handle the Terminated message in its receive. What if it doesn’t? You get the DeathPactException

The code shows that the Supervisor watches the child actor after creation but doesn’t handle the Terminated message from the child.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


package me.rerun.akkanotes.supervision

import akka.actor.{ActorSystem, Props}
import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest
import akka.actor.Actor
import akka.actor.ActorLogging
import akka.actor.Kill
import akka.actor.PoisonPill
import akka.actor.Terminated

object DeathPactExceptionApp extends App{

 val actorSystem=ActorSystem("DeathPactExceptionSystem")
 val actor=actorSystem.actorOf(Props[DeathPactExceptionParentActor])
 actor!"create_child" //Throws DeathPactException
 Thread.sleep(2000) //Wait until Stopped
 actor!"someMessage" //Message goes to DeadLetters

}

class DeathPactExceptionParentActor extends Actor with ActorLogging{

 def receive={
 case "create_child"=> {
 log.info ("creating child")
 val child=context.actorOf(Props[DeathPactExceptionChildActor])
 context.watch(child) //Watches but doesnt handle terminated message. Throwing DeathPactException here.
 child!"stop"
 }
 case "someMessage" => log.info ("some message")
 //Doesnt handle terminated message
 //case Terminated(_) =>
 }
}

class DeathPactExceptionChildActor extends Actor with ActorLogging{
 def receive={
 case "stop"=> {
 log.info ("Actor going to stop and announce that it's terminated")
 self!PoisonPill
 }
 }
}

Log

The logs tell us that the DeathPactException comes in, the supervisor stops that actor and then the messages go into the queue of deadLetters

1
2
3
4
5
6
7
8


INFO m.r.a.s.DeathPactExceptionParentActor - creating child

INFO m.r.a.s.DeathPactExceptionChildActor - Actor going to stop and announce that it's terminated

ERROR akka.actor.OneForOneStrategy - Monitored actor [Actor[akka://DeathPactExceptionSystem/user/$a/$a#-695506341]] terminated
akka.actor.DeathPactException: Monitored actor [Actor[akka://DeathPactExceptionSystem/user/$a/$a#-695506341]] terminated

INFO akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://DeathPactExceptionSystem/deadLetters] to Actor[akka://DeathPactExceptionSystem/user/$a#-1452955980] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Exception => Restart

For all other Exceptions, the default Directive is to Restart the Actor. Check the following app. Just to prove that the Actor is restarted, OtherExceptionParentActor makes the child throw an exception and immediately sends a message. The message reaches the mailbox and when the the child actor restarts, the message gets processed. Nice !!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


package me.rerun.akkanotes.supervision

import akka.actor.Actor
import akka.actor.ActorLogging
import akka.actor.ActorSystem
import akka.actor.OneForOneStrategy
import akka.actor.Props
import akka.actor.SupervisorStrategy.Stop

object OtherExceptionApp extends App{

 val actorSystem=ActorSystem("OtherExceptionSystem")
 val actor=actorSystem.actorOf(Props[OtherExceptionParentActor])
 actor!"create_child"

}

class OtherExceptionParentActor extends Actor with ActorLogging{

 def receive={
 case "create_child"=> {
 log.info ("creating child")
 val child=context.actorOf(Props[OtherExceptionChildActor])

 child!"throwSomeException"
 child!"someMessage"
 }
 }
}

class OtherExceptionChildActor extends akka.actor.Actor with ActorLogging{

 override def preStart={
 log.info ("Starting Child Actor")
 }

 def receive={
 case "throwSomeException"=> {
 throw new Exception ("I'm getting thrown for no reason") 
 }
 case "someMessage" => log.info ("Restarted and printing some Message")
 }

 override def postStop={
 log.info ("Stopping Child Actor")
 }

}

Log

The logs of this program is pretty neat.

The exception gets thrown. We see the trace
The child restarts - Stop and Start gets called (we’ll see about the preRestart and postRestart callbacks soon)
The message that was send to the Child actor before restart is processed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


INFO m.r.a.s.OtherExceptionParentActor - creating child

INFO m.r.a.s.OtherExceptionChildActor - Starting Child Actor

ERROR akka.actor.OneForOneStrategy - I'm getting thrown for no reason

java.lang.Exception: I'm getting thrown for no reason
	at me.rerun.akkanotes.supervision.OtherExceptionChildActor$$anonfun$receive$2.applyOrElse(OtherExceptionApp.scala:39) ~[classes/:na]
	at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.11-2.3.4.jar:na]
...
...	

INFO m.r.a.s.OtherExceptionChildActor - Stopping Child Actor

INFO m.r.a.s.OtherExceptionChildActor - Starting Child Actor

INFO m.r.a.s.OtherExceptionChildActor - Restarted and printing some Message

Escalate and Resume

We saw examples of Stop and Restart via the defaultStrategy. Now, let’s have a quick look at the Escalate.

Resume just ignores the exception and proceeds processing the next message in the mailbox. It’s more like catching the exception and doing nothing about it. Awesome stuff but not a lot to talk about there.

Escalating generally means that the exception is something critical and the immediate supervisor would not be able to handle it. So, it asks help from its supervisor. Let’s take an example.

Consider three Actors - EscalateExceptionTopLevelActor, EscalateExceptionParentActor and EscalateExceptionChildActor. If the child actor throws an exception and if the parent level actor could not handle it, it could Escalate it to the Top level Actor. The Top level actor could choose to react with any of the Directives. In our example, we just Stop. Stop would stop the immediate child (which is the EscalateExceptionParentActor). As you know, when a Stop is executed on an Actor, all its children would also be stopped before the Actor itself is stopped.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


package me.rerun.akkanotes.supervision

import akka.actor.Actor
import akka.actor.ActorLogging
import akka.actor.ActorSystem
import akka.actor.OneForOneStrategy
import akka.actor.Props
import akka.actor.SupervisorStrategy.Escalate
import akka.actor.SupervisorStrategy.Stop
import akka.actor.actorRef2Scala

object EscalateExceptionApp extends App {

 val actorSystem = ActorSystem("EscalateExceptionSystem")
 val actor = actorSystem.actorOf(Props[EscalateExceptionTopLevelActor], "topLevelActor")
 actor ! "create_parent"
}

class EscalateExceptionTopLevelActor extends Actor with ActorLogging {

 override val supervisorStrategy = OneForOneStrategy() {
 case _: Exception => {
 log.info("The exception from the Child is now handled by the Top level Actor. Stopping Parent Actor and its children.")
 Stop //Stop will stop the Actor that threw this Exception and all its children
 }
 }

 def receive = {
 case "create_parent" => {
 log.info("creating parent")
 val parent = context.actorOf(Props[EscalateExceptionParentActor], "parentActor")
 parent ! "create_child" //Sending message to next level
 }
 }
}

class EscalateExceptionParentActor extends Actor with ActorLogging {

 override def preStart={
 log.info ("Parent Actor started")
 }

 override val supervisorStrategy = OneForOneStrategy() {
 case _: Exception => {
 log.info("The exception is ducked by the Parent Actor. Escalating to TopLevel Actor")
 Escalate
 }
 }

 def receive = {
 case "create_child" => {
 log.info("creating child")
 val child = context.actorOf(Props[EscalateExceptionChildActor], "childActor")
 child ! "throwSomeException"
 }
 }

 override def postStop = {
 log.info("Stopping parent Actor")
 }
}

class EscalateExceptionChildActor extends akka.actor.Actor with ActorLogging {

 override def preStart={
 log.info ("Child Actor started")
 }

 def receive = {
 case "throwSomeException" => {
 throw new Exception("I'm getting thrown for no reason.")
 }
 }
 override def postStop = {
 log.info("Stopping child Actor")
 }
}

Log

As you could see from the logs,

The child actor throws exception.
The immediate supervisor (EscalateExceptionParentActor) escalates that exception to its supervisor (EscalateExceptionTopLevelActor)
The resultant directive from EscalateExceptionTopLevelActor is to Stop the Actor. As a sequence, the child actors gets stopped first.
The parent actor gets stopped next (only after the watchers have been notified)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


INFO m.r.a.s.EscalateExceptionTopLevelActor - creating parent

INFO m.r.a.s.EscalateExceptionParentActor - Parent Actor started

INFO m.r.a.s.EscalateExceptionParentActor - creating child

INFO m.r.a.s.EscalateExceptionChildActor - Child Actor started

INFO m.r.a.s.EscalateExceptionParentActor - The exception is ducked by the Parent Actor. Escalating to TopLevel Actor

INFO m.r.a.s.EscalateExceptionTopLevelActor - The exception from the Child is now handled by the Top level Actor. Stopping Parent Actor and its children.

ERROR akka.actor.OneForOneStrategy - I'm getting thrown for no reason.
java.lang.Exception: I'm getting thrown for no reason.
	at me.rerun.akkanotes.supervision.EscalateExceptionChildActor$$anonfun$receive$3.applyOrElse(EscalateExceptionApp.scala:71) ~[classes/:na]
	...
	...

INFO m.r.a.s.EscalateExceptionChildActor - Stopping child Actor

INFO m.r.a.s.EscalateExceptionParentActor - Stopping parent Actor

Please note that whatever directive that was issued would only apply to the immediate child that escalated. Say, if a Restart is issued at the TopLevel, only the Parent would be restarted and anything in its constructor/preStart would be executed. If the children of the Parent actor was created in the constructor, they would also be created. However, children that were created through messages to the Parent Actor would still be in the Terminated state.

TRIVIA

Actually, you could control whether the preStart gets called at all. We’ll see about this in the next minor write-up. If you are curious, just have a look at the postRestart method of the Actor

1
2
3


def postRestart(reason: Throwable): Unit = {
 preStart()
}

Code

As always, code is on github

(my .gitignore wasn’t setup right for this project. will fix it today. sorry)

Akka Notes - DeathWatch - 7

Fri, 31 Oct 2014 08:09:54 +0000

When we talked about Actor lifecycle, we saw that Actors could be stopped by various means (using ActorSystem.stop or ActorContext.stop or sending a PoisonPill - there’s also the Kill and the gracefulStop).

Whatever reason an Actor dies, there are cases when a few other actors in the system would like to know about it. Let’s take a trivial example of an Actor who talks to a database - let’s call it a RepositoryActor. For obvious reasons, there would be few other actors in the system who would be sending message to this RepositoryActor. These “interested” Actors would like to keep an eye on or watch this Actor if it goes down. Now, that in Actor terms is called DeathWatch. And the methods to watch and unwatch over this is intuitively ActorContext.watch and ActorContext.unwatch. If watched, the watchers would receive a Terminated message from the stopped Actor which they could comfortably add in to their receive function.

Unlike Supervision, where there is a strict enforcement of parent-child hierarchy, any Actor could watch any other Actor in the ActorSystem.

Let’s have a look at the code.

Code

QuoteRepositoryActor

Our QueryRepositoryActor holds a bunch of quotes as a List and serves a random one upon receiving a QuoteRepositoryRequest.
It keeps track of the number of messages received and if it receives more than 3 messages, it kills itself with a PoisonPill

Nothing fancy here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


package me.rerun.akkanotes.deathwatch

import akka.actor.{PoisonPill, Actor, ActorLogging, actorRef2Scala}
import me.rerun.akkanotes.protocols.QuoteRepositoryProtocol._
import scala.util.Random

class QuoteRepositoryActor() extends Actor with ActorLogging {

 val quotes = List(
 "Moderation is for cowards",
 "Anything worth doing is worth overdoing",
 "The trouble is you think you have time",
 "You never gonna know if you never even try")

 var repoRequestCount:Int=1

 def receive = {

 case QuoteRepositoryRequest => {

 if (repoRequestCount>3){
 self!PoisonPill
 }
 else {
 //Get a random Quote from the list and construct a response
 val quoteResponse = QuoteRepositoryResponse(quotes(Random.nextInt(quotes.size)))

 log.info(s"QuoteRequest received in QuoteRepositoryActor. Sending response to Teacher Actor $quoteResponse")
 repoRequestCount=repoRequestCount+1
 sender ! quoteResponse
 }

 }

 }

}

TeacherActorWatcher

Again, nothing fancy with TeacherActorWatcher except that it creates the QuoteRepositoryActor and watches over it using a context.watch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


package me.rerun.akkanotes.deathwatch

import akka.actor.{Terminated, Props, Actor, ActorLogging}
import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest
import me.rerun.akkanotes.protocols.QuoteRepositoryProtocol.QuoteRepositoryRequest

class TeacherActorWatcher extends Actor with ActorLogging {

 val quoteRepositoryActor=context.actorOf(Props[QuoteRepositoryActor], "quoteRepositoryActor")
 context.watch(quoteRepositoryActor)


 def receive = {
 case QuoteRequest => {
 quoteRepositoryActor ! QuoteRepositoryRequest
 }
 case Terminated(terminatedActorRef)=>{
 log.error(s"Child Actor {$terminatedActorRef} Terminated")
 }
 }
}

TestCases

This is the interesting bit. Frankly, I never thought that these could be tested. akka-testkit FTW. We will analyze three testcases here :

1. Assert receipt of Terminated message if watched

The QuoteRepositoryActor should send the testcase a Terminated message on receipt of the 4th message. The first three messages should go in fine.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


"A QuoteRepositoryActor" must {
 ...
 ...
 ...

 "send back a termination message to the watcher on 4th message" in {
 val quoteRepository=TestActorRef[QuoteRepositoryActor]

 val testProbe=TestProbe()
 testProbe.watch(quoteRepository) //Let's watch the Actor

 within (1000 millis) {
 var receivedQuotes = List[String]()
 (1 to 3).foreach(_ => quoteRepository ! QuoteRepositoryRequest)
 receiveWhile() {
 case QuoteRepositoryResponse(quoteString) => {
 receivedQuotes = receivedQuotes :+ quoteString
 }
 }

 receivedQuotes.size must be (3)
 println(s"receiveCount ${receivedQuotes.size}")

 //4th message
 quoteRepository!QuoteRepositoryRequest
 testProbe.expectTerminated(quoteRepository) //Expect a Terminated Message
 }
 }

2. Assert non-receipt of Terminated message if not watched/unwatched

Actually, we are over-doing things just to showcase the context.unwatch. The testcase would work just fine if we remove the testProbe.watch and testProbe.unwatch lines.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25



"not send back a termination message on 4th message if not watched" in {
 val quoteRepository=TestActorRef[QuoteRepositoryActor]

 val testProbe=TestProbe()
 testProbe.watch(quoteRepository) //watching

 within (1000 millis) {
 var receivedQuotes = List[String]()
 (1 to 3).foreach(_ => quoteRepository ! QuoteRepositoryRequest)
 receiveWhile() {
 case QuoteRepositoryResponse(quoteString) => {
 receivedQuotes = receivedQuotes :+ quoteString
 }
 }

 testProbe.unwatch(quoteRepository) //not watching anymore
 receivedQuotes.size must be (3)
 println(s"receiveCount ${receivedQuotes.size}")

 //4th message
 quoteRepository!QuoteRepositoryRequest
 testProbe.expectNoMsg() //Not Watching. No Terminated Message
 }
}

3. Assert receipt of Terminated message in the TeacherActorWatcher

We subscribe to the EventStream and check for a specific log message to assert termination.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14



"end back a termination message to the watcher on 4th message to the TeacherActor" in {

 //This just subscribes to the EventFilter for messages. We have asserted all that we need against the QuoteRepositoryActor in the previous testcase
 val teacherActor=TestActorRef[TeacherActorWatcher]

 within (1000 millis) {
 (1 to 3).foreach (_=>teacherActor!QuoteRequest) //this sends a message to the QuoteRepositoryActor

 EventFilter.error (pattern="""Child Actor .* Terminated""", occurrences = 1).intercept{
 teacherActor!QuoteRequest //Send the dangerous 4th message
 }
 }
 }

The pattern property of the EventFilter, not surprisingly, expects a regex pattern. The pattern="""Child Actor .* Terminated""" is expected to match a log message which is of the format Child Actor {Actor[akka://TestUniversityMessageSystem/user/$$d/quoteRepositoryActor#-1905987636]} Terminated

Github

As always, the code is available at github. Watch for the deathwatch package.

Akka Notes - Child Actors and ActorPath - 6

Tue, 21 Oct 2014 17:51:09 +0000

Actors are completely hierarchical. Whatever Actors that you create HAS to be a child of some other Actor.

Let’s analyze that a bit :

Path

Say, we create an ActorRef using ActorSystem.actorOf and try to print its path.

1
2
3


val actorSystem=ActorSystem("SupervisionActorSystem")
val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor])
println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/$a

As you see, a path looks very similar to a file path in a file system.

akka here is fixed because all these are addresses of Akka Actors - more like file:// or http:// prefix (nothing to do with protocol though).
The SupervisionActorSystem is just the name of your ActorSystem that you created.
We’ll talk about the user in the next section.
The $a is the name of your Actor that the system generated for you. How would you like if your operating system generated random file names for your files? You’d obviously hate it because you would want to refer to that name in future. So, let’s give it a proper meaningful name to it :

1
2
3



val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor], "teacherActor")
println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/teacherActor

That’s it. Now, the path actually makes sense.

Child Actors

Similar to top level actors which we create out of ActorSystem, we could also create child actors out of the ActorContext. In fact, the power of Actors’s fault tolerance primarily lie within leveraging the Actor hierarchy and the ability of a parent to manage the life of child actors.

Assume you have a TeacherSupervisor and you would like to create a TeacherActor to be a child of the Supervisor, you do a ActorContext.actorOf instead of a ActorSystem.actorOf:

1
2
3
4


class TeacherSupervisor extends Actor with ActorLogging {
 val teacherActor=context.actorOf(Props[TeacherActor], "teacherActor")
...
...

Frankly, in any application, you will be creating a whole lot of child actors than top level actors - which means that you’ll be calling a lot more actorContext.actorOf than actorSystem.actorOf.

You’ll notice that the path of the child actor is akka://SupervisionActorSystem/user/teacherSupervisor/teacherActor, which is very similar to the path you get when you create a child folder within a parent folder.

When do you create child Actors?

You generally create child actors when a particular task is composed of a subtask or multiple subtasks. You also create a child actor when a particular task to be executed by the parent is error-prone and you would want to isolate it (so that if it fails, you could recover). When there is no parent child relationship between tasks, then you DON’T create child actors.

Also, there’s nothing which is stopping a child actor from creating children to delegate its subtasks. Actors and their creation are really cheap but the power that comes with it is amazing (we’ll see about this while we talk about supervision).

Now what is that `user` in the path?

For lack of a creativity, let me compare the ActorSystem to a Unix file system - with a / root folder and all those /etc, /usr, /bin and various other folders.

ActorSystem are much like that. It creates a few top level Actors - the most important being the root Actor with the path /, the user Actor with the path /user and a system Actor with the path /system. (there’s also a /deadLetters that represent the DeadLetterActorRef. We saw this in our previous post)

Codewise, the ActorSystem composes three Actors inside it (via ActorRefProvider). Those are the root for ALL the Actors that gets created under the ActorSystem.

systemGuardian actor - root of all actors under /system
guardian actor - root of all actors under /user and
rootGuardian Actor - root of both the systemGuardian and the userGuardian actors.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


/**
 * Reference to the supervisor of guardian and systemGuardian; ....
 */
def rootGuardian: InternalActorRef

/**
 * Reference to the supervisor used for all top-level user actors.
 */
def guardian: LocalActorRef

/**
 * Reference to the supervisor used for all top-level system actors.
 */
def systemGuardian: LocalActorRef

/user (aka) user guardian

Any Actor that you create in your program like the StudentActor or the TeacherActor using the ActorSystem’s actorOf method would directly fall under /user. That’s the reason your teacherActor in the first part of this write-up had the path slug /user/teacherActor.

/system (aka) system guardian

The system guardian shuts itself down when it notices that the userGuardian is dead. Makes sense considering if the userGuardian is down, then all the business actors under it is also down and therefore all administrative actors needs to be gone too.

We could see two distinct places where System Actors are being created - I mean actors under the /system hierarchy.

Like we saw earlier, any message that you send to an Actor that is terminated gets forwarded to the mailbox of an internal Actor called DeadLetterActor. The DeadLetter Actor wraps each message as a DeadLetter and publishes it to the EventStream. One other Actor called DeadLetterListener consumes all DeadLetters and publishes that as a log message. Now, the DeadLetterListener is a system Actor with path /system/deadLetterListener.
Remember the TestEventListener that we created in our previous write-up to subscribe to log messages in the EventStream? They are System actors too. In fact, all akka.loggers are created as System actors.

1
2
3


class TeacherTest extends TestKit(ActorSystem("UniversityMessageSystem", ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]""")))
...
...

The documentation here says that any Actor that is configured in the configuration files and created and deployed into the ActorSystem while it starts, falls under the /system umbrella. Let me update this post when I find something interesting around this

/ (aka) root guardian

As we saw earlier, the / Actor is the parent of the user and the system guardians.

TRIVIA

Technically, there’s a superfluous parent for the root actor too. This Actor’s only job is to shut down the entire ActorSystem if the root actor fails. Since it’s strictly not considered within the Actor hierarchy, the Akka team calls it :

1
2


 private[akka] val theOneWhoWalksTheBubblesOfSpaceTime: InternalActorRef = new MinimalActorRef {
...

Akka Notes - Actor Lifecycle - Basic - 5

Tue, 21 Oct 2014 07:16:18 +0000

(Please note that this lifecycle write-up does not cover the preRestart or the postRestart methods. We’ll talk about them when we discuss supervision)

The basic Actor lifecycle is very much intuitive. You could actually compare the basic Actor lifecycle with a Java servlet lifecycle with one special difference.

Just like any other regular class, we have a Constructor
The preStart method gets called back next. Here, you could initialize resources that you would like to clean-up in postStop
The “servicing” or the message handling by the receive method occupies the major chunk of time and that happens in between.

Let’s look at a simple actor which prints the lifecycle.

Dumb Lifecycle Actor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


package me.rerun.akkanotes.lifecycle

import akka.actor.{ActorLogging, Actor}
import akka.event.LoggingReceive

class BasicLifecycleLoggingActor extends Actor with ActorLogging{

 log.info ("Inside BasicLifecycleLoggingActor Constructor")
 log.info (context.self.toString())
 override def preStart() ={
 log.info("Inside the preStart method of BasicLifecycleLoggingActor")
 }

 def receive = LoggingReceive{
 case "hello" => log.info ("hello")
 }

 override def postStop()={
 log.info ("Inside postStop method of BasicLifecycleLoggingActor")
 }

}

App

The LifecycleApp just initiates, sends a message to the Actor and shuts down the ActorSystem.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


import akka.actor.{ActorSystem, Props}

object LifecycleApp extends App{

 val actorSystem=ActorSystem("LifecycleActorSystem")
 val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")

 lifecycleActor!"hello"

 //wait for a couple of seconds before shutdown
 Thread.sleep(2000)
 actorSystem.shutdown()

 }

Output

 1
 2
 3
 4
 5
 6
 7
 8
 9
10



Inside BasicLifecycleLoggingActor Constructor

Actor[akka://LifecycleActorSystem/user/lifecycleActor#-2018741361]

Inside the preStart method of BasicLifecycleLoggingActor

hello

Inside postStop method of BasicLifecycleLoggingActor

What’s that special difference between Servlets and the basic Actor lifecycle?

That there is no difference between constructor and preStart in Actor lifecycle - more or less.

The reason why I printed the context.self in the constructor is this - unlike Servlets, Actors have access to the ActorContext even inside the constructor. The difference between the preStart and the constructor then becomes very subtle. We’ll revisit the difference while we talk about supervision but if you are curious - calling the preStart when the Actor restarts (in case of failure) could be controlled. With constructor, that isn’t possible.

When is postStop called?

As we saw from the program, the postStop gets called when the ActorSystem shuts down. There are a couple of other times when the callback gets invoked too.

ActorSystem.stop()

We could stop an Actor using the stop method of the ActorSystem and the ActorContext

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


object LifecycleApp extends App{

 val actorSystem=ActorSystem("LifecycleActorSystem")
 val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")

 actorSystem.stop(lifecycleActor);

 ...
 ...

} 

ActorContext.stop

1) Either by way of a message (externally or passing a message to itself)

1
2
3
4
5
6
7


class BasicLifecycleLoggingActor extends Actor with ActorLogging{
...
...
 def receive = LoggingReceive{
 case "hello" => log.info ("hello")
 case "stop" => context.stop(self)
 }

and

1
2
3
4
5
6
7
8
9


object LifecycleApp extends App{

 val actorSystem=ActorSystem("LifecycleActorSystem")
 val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")


 lifecycleActor!"stop"
...
...

2) OR kill itself for no reason (this is just for fun. No Actor with an ambition would do this)

1
2
3
4
5
6
7


class BasicLifecycleLoggingActor extends Actor with ActorLogging{

 log.info ("Inside BasicLifecycleLoggingActor Constructor")
 log.info (context.self.toString())
 context.stop(self)
 ...
 ...

PoisonPill

In the previous example, we passed a message called stop from the LifecycleApp to the Actor. The Actor received that message and killed itself using a context.stop. We could achieve the same thing by passing a PoisonPill message to the target actor. Please note that the PoisonPill message, just like the previous stop message gets enqueued in the regular mailbox and will be processed when it turn comes up.

1
2
3
4
5
6
7
8


object LifecycleApp extends App{

 val actorSystem=ActorSystem("LifecycleActorSystem")
 val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")

 lifecycleActor!PoisonPill
 ...
 ...

Kill

Instead of sending a PoisonPill, you could also send a Kill message to the target Actor.

1

lifecycleActor ! Kill

Between sending PoisonPill or Kill messages, the difference is subtle but important.

With PoisonPill, a Terminated message is sent to all watchers (we’ll see that shortly in our DeathWatch part).
With a Kill message, an ActorKilledException is thrown by the host Actor which gets propagated to its Supervisor (we’ll see this shortly in our Supervision part)

TRIVIA

What do I mean regular mailbox? Is there a “special” mailbox too? Yup. There is. And we’ll talk about it when we talk about it when we discuss supervision and system messages.

Termination

Once the Actor is stopped, it is said to enter into a Terminated state. The immediate question that would come up to your mind is what would happen to the messages that is sent to an Actor which is already terminated?

Let’s see that :

App

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


object LifecycleApp extends App{

 val actorSystem=ActorSystem("LifecycleActorSystem")
 val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")

 lifecycleActor!"hello"
 lifecycleActor!"stop"
 lifecycleActor!"hello" //Sending message to an Actor which is already stopped

}

Actor - Just as before

1
2
3
4
5
6
7
8


class BasicLifecycleLoggingActor extends Actor with ActorLogging{

 def receive = LoggingReceive{
 case "hello" => log.info ("hello")
 case "stop" => context.stop(self)

 }
}

Output

1
2
3


BasicLifecycleLoggingActor - hello 

akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://LifecycleActorSystem/deadLetters] to Actor[akka://LifecycleActorSystem/user/lifecycleActor#-569230546] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

From the logs, you see that there are some references of deadletters. Any message that you send to an Actor that is terminated gets forwarded to the mailbox of an internal Actor called DeadLetterActor.

Curious as to what happens next?

The DeadLetter Actor processes the messages in its mailbox, wraps each message as a DeadLetter and publishes it to the EventStream.

One other Actor called DeadLetterListener consumes all DeadLetters and publishes that as a log message. Check this out.

Remember, when we talked about logging, we saw that all log messages gets published to the EventStream and that we are free to subscribe to that EventStream - just that the subscriber also needs to be an Actor. Let’s try that now.

For our example, we’ll subscribe to the EventStream and watch out for all DeadLetter messages and will print to the console (so much for creativity??!!). Frankly, we are free to do anything from generating an alert, storing it into a database or even feeding into analytics.

Subscribing to DeadLetters in EventStream

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26



import akka.actor.ActorSystem
import akka.actor.Props
import akka.actor.PoisonPill
import akka.actor.DeadLetter
import akka.actor.Actor

object LifecycleApp extends App {

 val actorSystem = ActorSystem("LifecycleActorSystem")
 val lifecycleActor = actorSystem.actorOf(Props[BasicLifecycleLoggingActor], "lifecycleActor")

 val deadLetterListener = actorSystem.actorOf(Props[MyCustomDeadLetterListener])
 actorSystem.eventStream.subscribe(deadLetterListener, classOf[DeadLetter])

 lifecycleActor ! "hello"
 lifecycleActor ! "stop"
 lifecycleActor ! "hello"

}

class MyCustomDeadLetterListener extends Actor {
 def receive = {
 case deadLetter: DeadLetter => println(s"FROM CUSTOM LISTENER $deadLetter")
 }
}

Output

1
2
3
4
5


164 [LifecycleActorSystem-akka.actor.default-dispatcher-4] INFO BasicLifecycleLoggingActor - hello 

167 [LifecycleActorSystem-akka.actor.default-dispatcher-4] INFO akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://LifecycleActorSystem/deadLetters] to Actor[akka://LifecycleActorSystem/user/lifecycleActor#-782937925] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 

FROM CUSTOM LISTENER DeadLetter(hello,Actor[akka://LifecycleActorSystem/deadLetters],Actor[akka://LifecycleActorSystem/user/lifecycleActor#-782937925])

Akka Notes - ActorSystem (Configuration and Scheduling) - 4

Mon, 06 Oct 2014 14:31:09 +0000

As we saw from our previous posts, we could create an Actor using the actorOf method of the ActorSystem. There’s actually much more you could do with ActorSystem. We’ll touch upon just the Configuration and the Scheduling bit in this write-up

Let’s look at the subsets of methods available in the ActorSystem.

Configuration Management

Remember the application.conf file we used for configuring our log level in the previous write-up? This configuration file is just like those .properties files in Java applications and much more. We’ll be soon seeing how we could use this configuration file to customize our dispatchers, mailboxes etc. (I am not even closely doing justice to the power of the typesafe config. Please go through some examples to really appreciate its awesomeness)

So, when we create the ActorSystem using the ActorSystem object’s apply method without specifying any configuration, it looks out for application.conf, application.json and application.properties in the root of the classpath and loads them automatically.

So,

1

val system=ActorSystem("UniversityMessagingSystem")

is the same as

1

val system=ActorSystem("UniversityMessagingSystem", ConfigFactory.load())

To provide evidence to that argument, check out the apply method in ActorSystem.scala

1
2
3
4
5


def apply(name: String, config: Option[Config] = None, classLoader: Option[ClassLoader] = None, defaultExecutionContext: Option[ExecutionContext] = None): ActorSystem = {
 val cl = classLoader.getOrElse(findClassLoader())
 val appConfig = config.getOrElse(ConfigFactory.load(cl))
 new ActorSystemImpl(name, appConfig, cl, defaultExecutionContext).start()
}

Overriding default configuration

If you are not keen on using the application.conf (as in testcases) or would like to have your own custom configuration file (as in testing againt different configuration or deploying to different environments), you are free to override this by passing in your own configuration instead of wanting the one from the classpath.

ConfigFactory.parseString is one option

1

val actorSystem=ActorSystem("UniversityMessageSystem", ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]"""))

simply in your Testcase as

1
2
3
4


class TeacherTestLogListener extends TestKit(ActorSystem("UniversityMessageSystem", ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]""")))
 with WordSpecLike
 with MustMatchers
 with BeforeAndAfterAll {

There’s also a ConfigFactory.load

1

val system = ActorSystem("UniversityMessageSystem", ConfigFactory.load("uat-application.conf"))

If you need access to your own config parameters in runtime, you could do it via its API like so :

1
2
3



val system=ActorSystem("UniversityMessageSystem", ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]"""))
println (system.settings.config.getValue("akka.loggers")) // Results in > SimpleConfigList(["akka.testkit.TestEventListener"])

Extending default configuration

Other than overriding, you could also extend the default configuration with your custom configuration using the withFallback method of the Config.

Let’s say your application.conf looks like :

1
2
3
4
5


akka{
 loggers = ["akka.event.slf4j.Slf4jLogger"]
 loglevel = DEBUG
 arun="hello"
}

and you decide to override the akka.loggers property like :

1
2


val config=ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]""")
val system=ActorSystem("UniversityMessageSystem", config.withFallback(ConfigFactory.load()))

You end up with a merged configuration of both :

1
2
3



 println (system.settings.config.getValue("akka.arun")) //> ConfigString("hello")
println (system.settings.config.getValue("akka.loggers")) //> SimpleConfigList(["akka.testkit.TestEventListener"])

So, why did I tell this whole story on configuration? Because our ActorSystem is the one which loads and provides access to all the configuration information.

IMPORTANT NOTE :

Watch out the order of falling back here - which is the default and which is the extension configuration. Remember, you have to fall back to the default configuration. So,

1

config.withFallback(ConfigFactory.load())

would work

but

1

ConfigFactory.load().withFallback(config)

would not get the results that you may need.

Scheduler

As you can see from the API of ActorSystem, there is a powerful little method in ActorSystem called scheduler which returns a Scheduler. The Scheduler has a variety of schedule methods with which we could do some fun stuff inside the Actor environment.

Schedule something to execute once

Taking our Student-Teacher example, assume our StudentActor would want to send message to the teacher only after 5 seconds of it receiving the InitSignal from our Testcase and not immediately, our code looks like :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13



class StudentDelayedActor (teacherActorRef:ActorRef) extends Actor with ActorLogging {

 def receive = {
 case InitSignal=> {
 import context.dispatcher
 context.system.scheduler.scheduleOnce(5 seconds, teacherActorRef, QuoteRequest)
 //teacherActorRef!QuoteRequest
 }
 ...
 ...
 }
}

Testcase

Let’s cook up a testcase to verify this :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


"A delayed student" must {

 "fire the QuoteRequest after 5 seconds when an InitSignal is sent to it" in {

 import me.rerun.akkanotes.messaging.protocols.StudentProtocol._

 val teacherRef = system.actorOf(Props[TeacherActor], "teacherActorDelayed")
 val studentRef = system.actorOf(Props(new StudentDelayedActor(teacherRef)), "studentDelayedActor")

 EventFilter.info (start="Printing from Student Actor", occurrences=1).intercept{
 studentRef!InitSignal
 }
 }

 }

Increasing the timeout for Eventfilter interception

Ouch. The default timeout for the EventFilter to wait for the message to appear in the EventStream is 3 seconds. Let’s increase that to 7 seconds now to verify our testcase. The filter-leeway configuration property helps us achieve that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


class RequestResponseTest extends TestKit(ActorSystem("TestUniversityMessageSystem", ConfigFactory.parseString("""
 akka{
 loggers = ["akka.testkit.TestEventListener"]
 test{
 filter-leeway = 7s
 }
 }
 """)))
 with WordSpecLike
 with MustMatchers
 with BeforeAndAfterAll 
 with ImplicitSender {
 ...
 ...

Schedule something to execute repeatedly

In order to execute something repeatedly, you use the schedule method of the Scheduler.

One of the frequently used overload of the schedule method is the one which sends a message to the Actor on a regular basis. It acccepts 4 parameters :

How long should be initial delay be before the first execution begins
Frequency of subsequent executions
The target ActorRef that we are going to send a message to
The Message

1
2
3
4
5


case InitSignal=> {
 import context.dispatcher
 context.system.scheduler.schedule(0 seconds, 5 seconds, teacherActorRef, QuoteRequest)
 //teacherActorRef!QuoteRequest
 }

TRIVIA

The import import context.dispatcher is very important here.

The schedule methods requires a very important implicit parameter - ExecutionContext, the reason for which would be pretty obvious once we see the implementation of the schedule method :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


final def schedule(
 initialDelay: FiniteDuration,
 interval: FiniteDuration,
 receiver: ActorRef,
 message: Any)(implicit executor: ExecutionContext,
 sender: ActorRef = Actor.noSender): Cancellable =
 schedule(initialDelay, interval, new Runnable {
 def run = {
 receiver ! message
 if (receiver.isTerminated)
 throw new SchedulerException("timer active for terminated actor")
 }
 })

The schedule method just wraps the tell in a Runnable which eventually is executed by the ExecutionContext that we pass in.

In order to make an ExecutionContext available in scope as an implicit, we leverage upon the implicit dispatcher available on the context.

From ActorCell.scala (Context)

1
2
3
4
5


/**
 * Returns the dispatcher (MessageDispatcher) that is used for this Actor.
 * Importing this member will place an implicit ExecutionContext in scope.
 */
 implicit def dispatcher: ExecutionContextExecutor

Code

As always, the entire project could be downloaded from github here.

Akka Notes - Actor Messaging - Request and Response - 3

Mon, 06 Oct 2014 02:01:43 +0000

Last time when we saw Actor messaging, we saw how fire-n-forget messages are sent (Meaning, we just send a message to the Actor but don’t expect a response from the Actor).

Technically, we fire messages to Actors for its side-effects ALL THE TIME. It is by design. Other than not responding, the target Actor could ALSO do the following with that message -

Send a response back to the sender (in our case, the TeacherActor would respond with a quote back to the StudentActor OR
Forward a response back to some other Actor who might be the intended audience which in turn might respond/forward/have a side-effect. Routers and Supervisors are examples of those cases. (we’ll look at them very soon)

Request & Response

In this write-up, we’ll be focussing only on Point 1 - the request-response cycle.

The picture conveys what we are trying to achieve this time. For sake of brevity, I didn’t represent the ActorSystem, Dispatcher or Mailboxes in the picture.

The DriverApp sends an InitSignal message to the StudentActor.
The StudentActor reacts to the InitSignal message and sends a QuoteRequest message to the TeacherActor.
The TeacherActor, like we saw in the first discussion, responds with a QuoteResponse.
The StudentActor just logs the QuoteResponse to the console/logger.

We’ll also cook up a testcase to verify it.

Let’s look at these 4 points in detail now :

The `DriverApp` sends an `InitSignal` message to the `StudentActor`

By now, you would have guessed what would the DriverApp do. Just 4 things :

Initialize the ActorSystem

1
2


//Initialize the ActorSystem
 val system = ActorSystem("UniversityMessageSystem")

Create the TeacherActor

1
2


//create the teacher actor
 val teacherRef = system.actorOf(Props[TeacherActor], "teacherActor")

Create the StudentActor

1
2


//create the Student Actor - pass the teacher actorref as a constructor parameter to StudentActor
 val studentRef = system.actorOf(Props(new StudentActor(teacherRef)), "studentActor")

You’ll notice that I am passing in the ActorRef of the TeacherActor to the constructor of the StudentActor so that the StudentActor could use the ActorRef for sending messages to the TeacherActor. There are other ways to achieve this (like passing in the Props) but this method would come in handy when we look at Supervisors and Routers in the following write-ups. We’ll also be looking at child actors pretty soon but that wouldn’t semantically be the right approach here - Student creating Teacher doesn’t sound nice. Does it?

Lastly,

The DriverApp would then send an InitSignal to the StudentActor, so that the StudentActor could start sending the QuoteRequest message to the TeacherActor.

1
2


//send a message to the Student Actor
 studentRef ! InitSignal

That’s pretty much the DriverClass. The Thread.sleep and the ActorSystem.shutdown are just to wait for a couple of seconds for the message sending to finish before we finally shut down the ActorSystem.

DriverApp.scala

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29



package me.rerun.akkanotes.messaging.requestresponse

import akka.actor.ActorSystem
import akka.actor.Props
import me.rerun.akkanotes.messaging.protocols.StudentProtocol._
import akka.actor.ActorRef

object DriverApp extends App {

 //Initialize the ActorSystem
 val system = ActorSystem("UniversityMessageSystem")

 //construct the teacher actor
 val teacherRef = system.actorOf(Props[TeacherActor], "teacherActor")

 //construct the Student Actor - pass the teacher actorref as a constructor parameter to StudentActor
 val studentRef = system.actorOf(Props(new StudentActor(teacherRef)), "studentActor")

 //send a message to the Student Actor
 studentRef ! InitSignal

 //Let's wait for a couple of seconds before we shut down the system
 Thread.sleep(2000)

 //Shut down the ActorSystem. 
 system.shutdown()

}

The `StudentActor` reacts to the `InitSignal` message and sends a `QuoteRequest` message to the `TeacherActor`

AND

The `StudentActor` receives the `QuoteResponse` from `TeacherActor` and just logs to the console/logger

Why did I combine Points 2 and 4? Because it is so simple you’ll hate me if I separate them.

So, Point 2 - the StudentActor receives the InitSignal message from the DriverApp and sends QuoteRequest to the TeacherActor.

1
2
3
4
5
6


def receive = {
 case InitSignal=> {
 teacherActorRef!QuoteRequest
 }
 ...
 ...

That’s it !!!

Point 4 - The StudentActor logs the message that it receives from the TeacherActor.

Just, as promised :

1
2
3
4


case QuoteResponse(quoteString) => {
 log.info ("Received QuoteResponse from Teacher")
 log.info(s"Printing from Student Actor $quoteString")
}

I am sure you’d agree that it almost looks like pseudocode now.

So, the entire StudentActor class looks like :

StudentActor.scala

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


package me.rerun.akkanotes.messaging.requestresponse

import akka.actor.Actor
import akka.actor.ActorLogging
import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._
import me.rerun.akkanotes.messaging.protocols.StudentProtocol._
import akka.actor.Props
import akka.actor.ActorRef

class StudentActor (teacherActorRef:ActorRef) extends Actor with ActorLogging {

 def receive = {
 case InitSignal=> {
 teacherActorRef!QuoteRequest
 }

 case QuoteResponse(quoteString) => {
 log.info ("Received QuoteResponse from Teacher")
 log.info(s"Printing from Student Actor $quoteString")
 }
 }
}

The `TeacherActor` responds with a `QuoteResponse`.

This is the exact same code as we saw in the fire-n-forget write-up.

The TeacherActor receives a QuoteRequest message and sends QuoteResponse back.

TeacherActor.scala

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34



package me.rerun.akkanotes.messaging.requestresponse

import scala.util.Random

import akka.actor.Actor
import akka.actor.ActorLogging
import akka.actor.actorRef2Scala
import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._


class TeacherActor extends Actor with ActorLogging {

 val quotes = List(
 "Moderation is for cowards",
 "Anything worth doing is worth overdoing",
 "The trouble is you think you have time",
 "You never gonna know if you never even try")

 def receive = {

 case QuoteRequest => {

 import util.Random

 //Get a random Quote from the list and construct a response
 val quoteResponse = QuoteResponse(quotes(Random.nextInt(quotes.size)))

 //respond back to the Student who is the original sender of QuoteRequest
 sender ! quoteResponse

 }
 }
}

Testcases

Now, our testcase would simulate the DriverApp. Since, the StudentActor just logs the message and we won’t be able to assert on the QuoteResponse itself, we’ll just assert the presence of the log message in the EventStream (just like we talked last time)

So, our testcase looks like :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15



"A student" must {

 "log a QuoteResponse eventually when an InitSignal is sent to it" in {

 import me.rerun.akkanotes.messaging.protocols.StudentProtocol._

 val teacherRef = system.actorOf(Props[TeacherActor], "teacherActor")
 val studentRef = system.actorOf(Props(new StudentActor(teacherRef)), "studentActor")

 EventFilter.info (start="Printing from Student Actor", occurrences=1).intercept{
 studentRef!InitSignal
 }
 }
 }

Code

The entire project could be downloaded from github here.

Next up, we’ll see how to use schedulers in Akka and monitoring your Akka app using Kamon

Akka Notes - Logging and Testing Actors - 2

Mon, 29 Sep 2014 07:36:01 +0000

In the first two parts (one, two), we briefly talked about Actors and how messaging works. In this part, let’s look at fixing up Logging and Testing our TeacherActor.

Recap

This is how our Actor from the previous part looked like :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23



class TeacherActor extends Actor {

 val quotes = List(
 "Moderation is for cowards",
 "Anything worth doing is worth overdoing",
 "The trouble is you think you have time",
 "You never gonna know if you never even try")

 def receive = {

 case QuoteRequest => {

 import util.Random

 //Get a random Quote from the list and construct a response
 val quoteResponse=QuoteResponse(quotes(Random.nextInt(quotes.size)))

 println (quoteResponse)

 }
 }
}

Logging Akka with SLF4J

You notice that in the code we are printing the quoteResponse to the standard output which you would obviously agree is a bad idea. Let’s fix that up by enabling logging via the SLF4J Facade.

Fix the Class to use Logging

Akka provides a nice little trait called ActorLogging to achieve it. Let’s mix that in :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25



class TeacherLogActor extends Actor with ActorLogging {

 val quotes = List(
 "Moderation is for cowards",
 "Anything worth doing is worth overdoing",
 "The trouble is you think you have time",
 "You never gonna know if you never even try")

 def receive = {

 case QuoteRequest => {

 import util.Random

 //get a random element (for now)
 val quoteResponse=QuoteResponse(quotes(Random.nextInt(quotes.size)))
 log.info(quoteResponse.toString())
 }
 }

 //We'll cover the purpose of this method in the Testing section
 def quoteList=quotes

}

A small detour here :

Internally, when we log a message, the the logging methods in the ActorLogging (eventually) publishes the log message to an EventStream. Yes, I did say publish. So, what actually is an EventStream?

EventStream and Logging

EventStream behaves just like a message broker to which we could publish and receive messages. One subtle distinction from a regular MOM is that the subscribers of the EventStream could only be an Actor.

In case of logging messages, all log messages would be published to the EventStream. By default, the Actor that subscribes to these messages is the DefaultLogger which simply prints the message to the standard output.

1
2
3
4
5
6


class DefaultLogger extends Actor with StdOutLogger {
 override def receive: Receive = {
 ...
 case event: LogEvent ⇒ print(event)
 }
}

So, that’s the reason when we try to kick off the StudentSimulatorApp, we see the log message written to the console.

That said, EventStream isn’t suited only for logging. It is a general purpose publish-subscribe mechanism available inside the ActorWorld inside a VM (more on that later).

Back to SLF4J setup :

Configure Akka to use SLF4J

1
2
3
4
5
6



akka{
 loggers = ["akka.event.slf4j.Slf4jLogger"]
 loglevel = "DEBUG"
 logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
}

We store this information in a file called application.conf which should be in your classpath. In our sbt folder structure, we would throw this in your main/resources directory.

From the configuration, we could derived that

the loggers property indicates the Actor that is going to subscribe to the log events. What Slf4jLogger does is to simply consume the log messages and delegate that to the SLF4J Logger facade.
the loglevel property simply indicates the minimum level that should be considered for logging.
the logging-filter compares the currently configured loglevel and incoming log message level and chucks out any log message below the configured loglevel before publishing to the EventStream.

But Why didn’t we have an application.conf for the previous example?

Simply because Akka provides some sane defaults so that we needn’t build a configuration file before we start playing with it. We’ll revisit this file too often here on for customizing various things. There are a whole bunch of awesome parameters that you could use inside the application.conf for logging alone. They are explained in detail here.

Throw in a logback.xml

We’ll be configuring an SLF4J logger backed by logback now.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


<?xml version="1.0" encoding="UTF-8"?>
<configuration>
 <appender name="FILE"
 class="ch.qos.logback.core.rolling.RollingFileAppender">
 <encoder>
 <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
 </encoder>

 <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
 <fileNamePattern>logs\akka.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
 <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
 <maxFileSize>50MB</maxFileSize>
 </timeBasedFileNamingAndTriggeringPolicy>
 </rollingPolicy>
 </appender>

 <root level="DEBUG">
 <appender-ref ref="FILE" />
 </root>
</configuration>

I threw this inside the main/resources folder too along with application.conf. Please ensure that the main/resources is now in your eclipse or other IDE’s classpath. Also include logback and slf4j-api to your build.sbt.

And when we kick off our StudentSimulatorApp and send a message to our new TeacherLogActor, the akkaxxxxx.log file that we configured looks like this.

Testing Akka

Please note that this is by no means an exhaustive coverage of Testing Akka. We would be building our tests on more features of Testing in the following parts under their respective topic headers. These testcases are aimed to cover the Actors we wrote earlier.

While the StudentSimulatorApp does what we need, you would agree that it should be driven out of testcases.

To ease the testing pain, Akka came up with an amazing testing toolkit with which we could do some magical stuff like probing directly into the Actor implementation’s internals.

Enough talk, let’s see the testcases.

Let’s first try to map the StudentSimulatorApp to a Testcase.

Let’s look at the declaration alone now.

1
2
3
4


class TeacherPreTest extends TestKit(ActorSystem("UniversityMessageSystem"))
 with WordSpecLike
 with MustMatchers
 with BeforeAndAfterAll {

So, from the definition of the TestCase class we see that :

The TestKit trait accepts an ActorSystem through which we would be creating Actors. Internally, the TestKit decorates the ActorSystem and replaces the default dispatcher too.
We use WordSpec which is one of the many fun ways to write testcases with ScalaTest.
The MustMatchers provide convenient methods to make the testcase look like natural language
We mixin the BeforeAndAfterAll to shutdown the ActorSystem after the testcases are complete. The afterAll method that the trait provides is more like our tearDown in JUnit

1, 2 - Sending message to Actors

The first testcase just sends a message to the PrintActor. It doesn’t assert anything :-(
The second case sends message to the Log actor which uses the log field of the ActorLogging to publish the message to the EventStream. This doesn’t assert anything too :-(

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19



//1. Sends message to the Print Actor. Not even a testcase actually
"A teacher" must {

 "print a quote when a QuoteRequest message is sent" in {

 val teacherRef = TestActorRef[TeacherActor]
 teacherRef ! QuoteRequest
 }
}

//2. Sends message to the Log Actor. Again, not a testcase per se
"A teacher with ActorLogging" must {

 "log a quote when a QuoteRequest message is sent" in {

 val teacherRef = TestActorRef[TeacherLogActor]
 teacherRef ! QuoteRequest
 }

- Asserting internal state of Actors

The third case uses the underlyingActor method of the TestActorRef and calls upon the quoteList method of the TeacherActor. The quoteList method returns the list of quotes back. We use this list to assert its size.

If reference to quoteList throws you back, refer to the TeacherLogActor code listed above and look for

1
2
3


//From TeacherLogActor
//We'll cover the purpose of this method in the Testing section
 def quoteList=quotes

1
2
3
4
5
6
7


//3. Asserts the internal State of the Log Actor. 
"have a quote list of size 4" in {

 val teacherRef = TestActorRef[TeacherLogActor]
 teacherRef.underlyingActor.quoteList must have size (4)
 teacherRef.underlyingActor.quoteList must have size (4)
}

- Asserting log messages

As we discussed earlier in the EventStream and Logging section (above), all log messages go to the EventStream and the SLF4JLogger subscribes to it and uses its appenders to write to the log file/console etc. Wouldn’t it be nice to subscribe to the EventStream directly in our testcase and assert the presence of the log message itself? Looks like we can do that too.

This involves two steps :

You need to add an extra configuration to your TestKit like so :

1
2
3
4
5



class TeacherTest extends TestKit(ActorSystem("UniversityMessageSystem", ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]""")))
 with WordSpecLike
 with MustMatchers
 with BeforeAndAfterAll {

Now that we have a subscription to the EventStream, we could assert it from our testcase as :

1
2
3
4
5
6
7
8
9



//4. Verifying log messages from eventStream
 "be verifiable via EventFilter in response to a QuoteRequest that is sent" in {

 val teacherRef = TestActorRef[TeacherLogActor]
 EventFilter.info(pattern = "QuoteResponse*", occurrences = 1) intercept {
 teacherRef ! QuoteRequest
 }
 }

The EventFilter.info block just intercepts for 1 log message which starts with QuoteResponse (pattern='QuoteResponse*). (You could also achieve it by using a start='QuoteResponse'. If there is no log message as a result of sending a message to the TeacherLogActor, the testcase would fail.

- Testing Actors with constructor parameters

Please note that the way we create Actors in the testcase is via the TestActorRef[TeacherLogActor] and not via system.actorOf. This is just so that we could get access to the Actor’s internals through the underlyingActor method in the TeacherActorRef. We wouldn’t be able to achieve this via the ActorRef that we have access during the regular runtime. (That doesn’t give us any excuse to use TestActorRef in production. You’ll be hunted down).

If the Actor accepts parameters, then the way we create TestActorRef would be like :

1
2

val teacherRef = TestActorRef(new TeacherLogParameterActor(quotes))

The entire testcase would then look something like :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


//5. have a quote list of the same size as the input parameter
 " have a quote list of the same size as the input parameter" in {

 val quotes = List(
 "Moderation is for cowards",
 "Anything worth doing is worth overdoing",
 "The trouble is you think you have time",
 "You never gonna know if you never even try")

 val teacherRef = TestActorRef(new TeacherLogParameterActor(quotes))
 //val teacherRef = TestActorRef(Props(new TeacherLogParameterActor(quotes)))

 teacherRef.underlyingActor.quoteList must have size (4)
 EventFilter.info(pattern = "QuoteResponse*", occurrences = 1) intercept {
 teacherRef ! QuoteRequest
 }
 }

Shutting down ActorSystem

And finally, the afterAll lifecycle method

1
2
3
4


override def afterAll() {
 super.afterAll()
 system.shutdown()
 }

CODE

As always, the entire project could be downloaded from github here.

Akka Notes - Actor Messaging - 1

Fri, 19 Sep 2014 10:30:31 +0000

From the introductory first part of the Akka Notes, we saw a bird’s eye view of Actors in the Akka Toolkit. In this second part of the Akka Notes, we’ll look at the messaging part of Actors. As for the example, we would use the same Student-Teacher example that we discussed earlier.

In this first part of Actor Messaging, we’ll create the Teacher Actor and instead of the Student Actor, we’ll use a main program called StudentSimulatorApp.

Revisiting Student-Teacher in Detail

Let’s for now consider the message sent by the StudentSimulatorApp to the TeacherActor alone. When I say StudentSimulatorApp, I just mean a normal main program.

The picture conveys this :

(if the terms are overwhelming, don’t worry, we’ll go through them in detail)

Student creates something called an ActorSystem
It uses the ActorSystem to create something called as ActorRef. The QuoteRequest message is sent to the ActorRef (a proxy to TeacherActor)
Actor ref passes the message along to a Dispatcher
The Dispatcher enqueues the message in the target Actor’s MailBox.
The Dispatcher then puts the Mailbox on a Thread (more on that in the next section).
The MailBox dequeues a message and eventually delegates that to the actual Teacher Actor’s receive method.

Like I said, don’t worry about it. Let’s look at each step in detail now. You can come back and revisit these five steps once we are done.

The `StudentSimulatorApp` program

We would use this StudentSimulatorApp to bring up the JVM and initialize the ActorSystem.

As we understand from the picture, the StudentSimulatorApp

Creates an ActorSystem
Uses the ActorSystem to create a proxy to the Teacher Actor (ActorRef)
Sends the QuoteRequest message to the proxy.

Let’s focus on these three points alone now.

1. Creating an ActorSystem

ActorSystem is the entry point into the ActorWorld. ActorSystems are through which you could create and stop Actors. Or even shutdown the entire Actor environment.

On the other end of the spectrum, Actors are hierarchical and the ActorSystem is also similar to the java.lang.Object or scala.Any for all Actors - meaning, it is the root for all Actors. When you create an Actor using the ActorSystem’s actorOf method, you create an Actor just below the ActorSystem.

The code for initializing the ActorSystem looks like

1
2

val system=ActorSystem("UniversityMessageSystem")

The UniversityMessageSystem is simply a cute name you give to your ActorSystem.

2. Creating a Proxy for TeacherActor?

Let’s consider the following snippet :

1
2

val teacherActorRef:ActorRef=actorSystem.actorOf(Props[TeacherActor])

The actorOf is the Actor creation method in ActorSystem. But, as you can see, it doesn’t return a TeacherActor which we need. It returns something of type ActorRef.

The ActorRef acts as a Proxy for the actual Actors. The clients do not talk directly with the Actor. This is Actor Model’s way of avoiding direct access to any custom/private methods or variables in the TeacherActor or any Actor for that sake.

To repeat, you send messages only to the ActorRef and it eventually reaches your actual Actor. You can NEVER talk to your Actor directly. People will hate you to death if you find some mean ways to do that.

3. Send a QuoteRequest to the Proxy

It’s an one-liner again. You just tell the QuoteRequest message to the ActorRef. The tell method in Actor is actually !. (there’s also a tell method in ActorRef which just delegates the call back to !)

1
2
3



//send a message to the Teacher Actor
teacherActorRef!QuoteRequest

That’s it !!!

If you think I am lying, check the entire code of the StudentSimulatorApp below :

StudentSimulatorApp.scala

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27



package me.rerun.akkanotes.messaging.actormsg1

import akka.actor.ActorSystem
import akka.actor.Props
import akka.actor.actorRef2Scala
import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._


object StudentSimulatorApp extends App{

 //Initialize the ActorSystem
 val actorSystem=ActorSystem("UniversityMessageSystem")

 //construct the Teacher Actor Ref
 val teacherActorRef=actorSystem.actorOf(Props[TeacherActor])

 //send a message to the Teacher Actor
 teacherActorRef!QuoteRequest

 //Let's wait for a couple of seconds before we shut down the system
 Thread.sleep (2000) 

 //Shut down the ActorSystem.
 actorSystem.shutdown()

} 

Well, I cheated a little. You’ll have to shutdown the ActorSystem or otherwise, the JVM keeps running forever. And I am making the main thread sleep for a little while just to give the TeacherActor to finish off its task. I know this sounds stupid. Don’t worry about it. We’ll write some neat testcases in the next part in order to avoid this hack.

The Message

We just told a QuoteRequest to the ActorRef but we didn’t see the message class at all !!

Here it comes :

(It is a recommended practice to wrap your messages in a nice object for easier organization)

TeacherProtocol

1
2
3
4
5
6
7
8
9



package me.rerun.akkanotes.messaging.protocols

object TeacherProtocol{

 case class QuoteRequest()
 case class QuoteResponse(quoteString:String)

}

As you know, the QuoteRequest is for the requests that come to the TeacherActor. The Actor would respond back with a QuoteResponse.

Dispatcher and a MailBox

The ActorRef delegates the message handling functionality to the Dispatcher. Under the hood, while we created the ActorSystem and the ActorRef, a Dispatcher and a MailBox was created. Let’s see what they are about.

MailBox

Every Actor has one MailBox (we’ll see one special case later). Per our analogy, every Teacher has one mailbox too. The Teacher has to check the mailbox and process the message. In Actor world, it’s the other way round - the mailbox, when it gets a chance uses the Actor to accomplish its work.

Also the mailbox has a queue to store and process the messages in a FIFO fashion - a little different from our regular inbox where the most latest is the one at the top.

Now, the dispatcher

Dispatcher does some really cool stuff. From the looks of it, the Dispatcher just gets the message from the ActorRef and passes it on to the MailBox. But there’s one amazing thing happening behind the scenes :

The Dispatcher wraps an ExecutorService (ForkJoinPool or ThreadPoolExecutor). It executes the MailBox against this ExecutorService.

Check out this snippet from the Dispatcher

1
2
3
4
5
6
7



protected[akka] override def registerForExecution(mbox: Mailbox, ...): Boolean = {
 ...
 try {
 executorService execute mbox
 ...
}

What? Did you just say you execute the MailBox?

Yup. We already saw that the MailBox holds all the messages in a Queue. Also since the Executor runs the MailBox, the MailBox must be a Thread. You’re right. That’s pretty much MailBox’s declaration and constructor.

Here’s the signature of the Mailbox

1
2

private[akka] abstract class Mailbox(val messageQueue: MessageQueue) extends SystemMessageQueue with Runnable

Teacher Actor

The MailBox, when it gets its run method fired, dequeues a message from the message queue and passes it to the Actor for processing.

The method that eventually gets called when you tell a message to an ActorRef is the receive method of the target Actor.

The TeacherActor is a rudimentary class which has a List of quotes and obviously the receive method which handles the messages.

Check this out :

TeacherActor.scala

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41



package me.rerun.akkanotes.messaging.actormsg1

import scala.util.Random

import akka.actor.Actor
import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._

/*
 * Your Teacher Actor class. 
 * 
 * The class could use refinement by way of 
 * using ActorLogging which uses the EventBus of the Actor framework
 * instead of the plain old System out
 * 
 */

class TeacherActor extends Actor {

 val quotes = List(
 "Moderation is for cowards",
 "Anything worth doing is worth overdoing",
 "The trouble is you think you have time",
 "You never gonna know if you never even try")

 def receive = {

 case QuoteRequest => {

 import util.Random

 //Get a random Quote from the list and construct a response
 val quoteResponse=QuoteResponse(quotes(Random.nextInt(quotes.size)))

 println (quoteResponse)

 }

 }

}

The TeacherActor’s receive method pattern matches for just one Message - the QuoteRequest (actually, it is a good practice to pattern match the default case but there’s an interesting story to tell there)

All that the receive method does is to

pattern match for QuoteRequest
pick a random quote from the static list of quotes
construct a QuoteResponse
print the QuoteResponse to the console

Code

The entire project could be downloaded from github here.

We expand this example to Logging and Testing in here.

Akka Notes - Introducing Actors

Thu, 11 Sep 2014 09:04:06 +0000

Anyone who has done multithreading in the past won’t deny how hard and painful it is to manage multithreaded applications. I said manage because it starts out simple and it became a whole lot of fun once you start seeing performance improvements. However, it aches when you see that you don’t have a easier way to recover from errors in your sub-tasks OR those zombie bugs that you find hard to reproduce OR when your profiler shows that your threads are spending a lot of time blocking wastefully before writing to a shared state.

I prefer not to talk about how Java concurrency API and their collections made it better and easier because I am sure if you are here, you probably needed more control over the sub-tasks or simply because you don’t like to write locks and synchronized blocks and would prefer a higher level of abstraction.

In this series of Akka Notes, we would go through simple Akka examples to explore the various features that we have in the toolkit.

What are Actors?

~~Akka’s Actors follow the Actor Model (duh!).~~

Treat Actors like People. People who don’t talk to each other in person. They just talk through mails.

Let’s expand on that a bit.

Messaging

Consider two persons - A wise Teacher and Student. The Student sends a mail every morning to the Teacher and the wise Teacher sends a wise quote back.

Points to note :

The student sends a mail. Once sent, the mail couldn’t be edited. Talk about natural immutability.
The Teacher checks his mailbox when he wishes to do so.
The Teacher also sends a mail back (immutable again).
The student checks the mailbox at his own time.
The student doesn’t wait for the reply. (no blocking)

That pretty much sums up the basic block of the Actor Model - passing messages.

Concurrency

Now, imagine there are 3 wise teachers and 3 students - every student sends notes to every other teacher. What happens then? Nothing changes actually. Everybody has their own mailbox. One subtle point to note here is this :

By default, Mails in the mailbox are read/processed in the order they arrived.

Internally, by default it is a ConcurrentLinkedQueue. And since nobody waits for the mail to be picked up, it is simply a non-blocking message. (There are a variety of built-in mailboxes including bounded and priority based. In fact, we could build one ourself too)

Failover

Imagine these 3 teachers are from three different departments - History, Geography and Philosophy.

History teachers replies with a note on an Event in the past, Geography teachers sends an Interesting Place and Philosophy teachers, a quote. Each student sends message to each teacher and gets responses. The student doesnt care which teacher in the department sends the reply back. What if one day, a teacher falls sick? There has to be at least one teacher handling the mails from the department. In this case, another teacher in the department steps up and does the job.

Points to note :

There could be a pool of Actors who does different things.
An Actor could do something that causes an exception. It wouldn’t be able to recover by itself. In which case a new Actor could be created in place of the old one. Alternatively, the Actor could just ignore that one particular message and proceed with the rest of the messages. These are called Directives and we’ll discuss them later.

Multitasking

For a twist, let’s assume that each of these teachers also send the exam score through mail too, if the student asks for it. Similarly, an the Actor could handle more than one type of message comfortably.

Chaining

What if the student would like to get only one final consolidated trivia mail instead of three?

We could do that too with Actors too. We could chain the teachers as a hierarchy. We’ll come back to that later when we talk about Supervisors and revisit the same thought when we talk about Futures.

As requested by Mohan, let’s just try to map the analogy components with the the components in the Actor Model.

Students and the Teachers becomes our Actors. The Email Inbox becomes the Mailbox component. The request and the response can’t be modified. They are immutable objects. Finally, the MessageDispatcher component in Actor manages the mailbox and routes the messages to the respective Mailbox.

Enough talk, let’s cook up some code….

The Knapsack problem

Tue, 27 May 2014 13:25:00 +0000

I found the Knapsack problem tricky and interesting at the same time. I am sure if you are visiting this page, you already know the problem statement but just for the sake of completion :

Problem :

Given a Knapsack of a maximum capacity of W and N items each with its own value and weight, throw in items inside the Knapsack such that the final contents has the maximum value. Yikes !!!

Link to the problem page in wiki

Here’s the general way the problem is explained - Consider a thief gets into a home to rob and he carries a knapsack. There are fixed number of items in the home - each with its own weight and value - Jewellery, with less weight and highest value vs tables, with less value but a lot heavy. To add fuel to the fire, the thief has an old knapsack which has limited capacity. Obviously, he can’t split the table into half or jewellery into 3/4ths. He either takes it or leaves it.

Example :

1
2
3
4
5
6
7


Knapsack Max weight 	: 		W = 10 (units)

Total items 			: 		N = 4

Values of items 		: 		v[] = {10, 40, 30, 50}

Weight of items 		: 		w[] = {5, 4, 6, 3}

A cursory look at the example data tells us that the max value that we could accommodate with the limit of max weight of 10 is 50 + 40 = 90 with a weight of 7.

Approach :

The way this is optimally solved is using dynamic programming - solving for smaller sets of knapsack problems and then expanding them for the bigger problem.

Let’s build an Item x Weight array called V (Value array)

1

V[N][W] = 4 rows * 10 columns

Each of the values in this matrix represent a smaller Knapsack problem.

Base case 1 : Let’s take the case of 0th column. It just means that the knapsack has 0 capacity. What can you hold in them? Nothing. So, let’s fill them up all with 0s

Base case 2 : Let’s take the case of 0 row. It just means that there are no items in the house. What do you do hold in your knapsack if there are no items. Nothing again !!! All zeroes.

Solution :

Now, let’s start filling in the array row-wise. What does row 1 and column 1 mean? That given the first item (row), can you accommodate it in the knapsack with capacity 1 (column). Nope. The weight of the first item is 5. So, let’s fill in 0. In fact, we wouldn’t be able to fill in anything until we reach the column 5 (weight 5).
Once we reach column 5 (which represents weight 5) on the first row, it means that we could accommodate item 1. Let’s fill in 10 there (remember, this is a Value array)

Moving on, for weight 6 (column 6), can we accommodate anything else with the remaining weight of 1 (weight - weight of this item => 6 - 5). Hey, remember, we are on the first item. So, it is kind of intuitive that the rest of the row will just be the same value too since we are unable to add in any other item for that extra weight that we have got.

So, the next interesting thing happens when we reach the column 4 in third row. The current running weight is 4.

We should check for the following cases.

Can we accommodate Item 2 - Yes, we can. Item 2’s weight is 4.
Is the value for the current weight is higher without Item 2? - Check the previous row for the same weight. Nope. the previous row* has 0 in it, since we were not able able accommodate Item 1 in weight 4.
Can we accommodate two items in the same weight so that we could maximize the value? - Nope. The remaining weight after deducting the Item2’s weight is 0.

Why previous row ?

Simply because the previous row at weight 4 itself is a smaller knapsack solution which gives the max value that could be accumulated for that weight until that point (traversing through the items).

Exemplifying,

The value of the current item = 40
The weight of the current item = 4
The weight that is left over = 4 - 4 = 0
Check the row above (the Item above in case of Item 1 or the cumulative Max value in case of the rest of the rows). For the remaining weight 0, are we able to accommodate Item 1? Simply put, is there any value at all in the row above for the given weight)

The calculation goes like so :

Take the max value for the same weight without this item

previous row, same weight = 0

=> V[item-1][weight]
Take the value of the current item + value that we could accommodate with the remaining weight

Value of current item
- value in previous row with weight 4 (total weight until now (4) - weight of the current item (4))
=> val[item-1] + V[item-1][weight-wt[item-1]]

Max among the two is 40 (0 and 40)

The next and the most important event happens at column 9 and row 2. Meaning we have a weight of 9 and we have two items. Looking at the example data we could accommodate the first two items. Here, we consider few things
1. The value of the current item = 40
2. The weight of the current item = 4
3. The weight that is left over = 9 - 4 = 5
4. Check the row above. At the remaining weight 5, are we able to accommodate Item 1.

So, the calculation is :

Take the max value for the same weight without this item

previous row, same weight = 10
Take the value of the current item + value that we could accumulate with the remaining weight

Value of current item (40)
- value in previous row with weight 5 (total weight until now (9) - weight of the current item (4))
= 10

10 vs 50 = 50

At the end of solving all these smaller problems, we just need to return the value at V[N][W] - Item 4 at Weight 10

Complexity

Analyzing the complexity of the solution is pretty straight-forward. We just have a loop for W within a loop of N => O (NW)

Implementation :

Here comes the obligatory implementation code in Java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73



class Knapsack {

 public static void main(String[] args) throws Exception {
 int val[] = {10, 40, 30, 50};
 int wt[] = {5, 4, 6, 3};
 int W = 10;

 System.out.println(knapsack(val, wt, W));
 }

 public static int knapsack(int val[], int wt[], int W) {

		//Get the total number of items. 
 	//Could be wt.length or val.length. Doesn't matter
 int N = wt.length; 

		//Create a matrix. 
 //Items are in rows and weight at in columns +1 on each side
 int[][] V = new int[N + 1][W + 1]; 


		//What if the knapsack's capacity is 0 - Set
 //all columns at row 0 to be 0
 for (int col = 0; col <= W; col++) {
 V[0][col] = 0;
 }

		//What if there are no items at home. 
 //Fill the first row with 0
 for (int row = 0; row <= N; row++) {
 V[row][0] = 0;
 }

 for (int item=1;item<=N;item++){

 //Let's fill the values row by row
 for (int weight=1;weight<=W;weight++){

 //Is the current items weight less
 //than or equal to running weight
 if (wt[item-1]<=weight){

//Given a weight, check if the value of the current 
//item + value of the item that we could afford 
//with the remaining weight is greater than the value
//without the current item itself
 V[item][weight]=Math.max (val[item-1]+V[item-1][weight-wt[item-1]], V[item-1][weight]);
 }
 else {
//If the current item's weight is more than the
//running weight, just carry forward the value
//without the current item
 V[item][weight]=V[item-1][weight];
 }
 }

 }

 //Printing the matrix
 for (int[] rows : V) {
 for (int col : rows) {

 System.out.format("%5d", col);
 }
 System.out.println();
 }

 return V[N][W];

 }

}

Camel CXF Service with Multiple Query Parameters

Thu, 14 Nov 2013 19:35:00 +0000

While the awesome Apache Camel team is busy fixing the handling of the multiple parameters in the query, here’s a workaround. Hopefully, this post will become obsolete with the next versions of Camel. (Currently, I use 2.7.5)

Problem

Query parameters more than 1 is passed as a null value into a Camel-CXF service. Say, if the URL has four query parameters as in

1

name=arun&email=arun@arunma.com&age=10&phone=123456

only the first one gets populated when you do a

1
2
3
4
5
6
7
8
9



@GET
@Path("search")
@Produces(MediaType.APPLICATION_JSON)
public String sourceResultsFromTwoSources(@QueryParam("name") String name,
					@QueryParam("age") String age,
		 @QueryParam("phone") String phone,
 @QueryParam("email") String email
);

All other parameters are null.

Final Output

For url

http://localhost:8181/cxf/karafcxfcamel/search?name=arun&email=arun@arunma.com&age=31&phone=232323

the result expected is :

Workaround

Interestingly, we could get the entire query string in the header.

1

String queryString = exchange.getIn().getHeader(Exchange.HTTP_QUERY, String.class);

We could then do a

1

MultivaluedMap<String, String> queryMap = JAXRSUtils.getStructuredParams(queryString, "&", false, false);

to get the query parameters as a multi valued Map.

The query parameters could then be set as a property to the Exchange and used across the exchange.

Code

The entire code could be downloaded from github

Please note that I am running Camel as part of OSGi inside the Karaf container. While the workaround does not differ because of the environment in which you are using Camel-CXF, please be wary of this fact when you download the code from github. Watch out for the blueprint xmls for Camel configuration.

The most important piece here is the Router

Router

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51



package me.rerun.karafcxfcamel.camel.beans;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.dataformat.JsonLibrary;
import org.apache.cxf.jaxrs.utils.JAXRSUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.ws.rs.core.MultivaluedMap;
import java.util.List;
import java.util.Map;

public class RestToBeanRouter extends RouteBuilder {

 private static Logger logger= LoggerFactory.getLogger(RouteBuilder.class);

 @Override
 public void configure() throws Exception {

 from ("cxfrs://bean://rsServer")
 .process(new ParamProcessor())
 .multicast()
 .parallelProcessing()
 .aggregationStrategy(new ResultAggregator())
 .beanRef("restServiceImpl", "getNameEmailResult")
 .beanRef("restServiceImpl", "getAgePhoneResult")
 .end()
 .marshal().json(JsonLibrary.Jackson)
 .to("log://camelLogger?level=DEBUG");
 }

 private class ParamProcessor implements Processor {

 @Override
 public void process(Exchange exchange) throws Exception {
 String queryString = exchange.getIn().getHeader(Exchange.HTTP_QUERY, String.class);

 MultivaluedMap<String, String> queryMap = JAXRSUtils.getStructuredParams(queryString, "&", false, false);

 for (Map.Entry<String, List<String>> eachQueryParam : queryMap.entrySet()) {
 exchange.setProperty(eachQueryParam.getKey(), eachQueryParam.getValue());
 }


 }

 }
}

Interface

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16



package me.rerun.karafcxfcamel.rest;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

public interface RestService {

 @GET
 @Path("search")
 @Produces(MediaType.APPLICATION_JSON)
 public String sourceResultsFromTwoSources();

}

Implementation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73



package me.rerun.karafcxfcamel.rest;

import me.rerun.karafcxfcamel.model.AgePhoneResult;
import me.rerun.karafcxfcamel.model.NameEmailResult;
import me.rerun.karafcxfcamel.service.base.AgePhoneService;
import me.rerun.karafcxfcamel.service.base.NameEmailService;
import me.rerun.karafcxfcamel.service.impl.AgePhoneServiceImpl;
import org.apache.camel.Exchange;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.List;

public class RestServiceImpl implements RestService {

 private static Logger logger= LoggerFactory.getLogger(AgePhoneServiceImpl.class);

 private NameEmailService nameEmailService;
 private AgePhoneService agePhoneService;

 public RestServiceImpl(){
 }

 //Do nothing. Camel intercepts and routes the requests
 public String sourceResultsFromTwoSources() {
 return null;
 }


 public NameEmailResult getNameEmailResult(Exchange exchange){
 logger.info("Invoking getNameEmailResult from RestServiceImpl");

 String name=getFirstEntrySafelyFromList(exchange.getProperty("name", List.class));
 String email=getFirstEntrySafelyFromList(exchange.getProperty("email", List.class));

 return nameEmailService.getNameAndEmail(name, email);
 }


 public AgePhoneResult getAgePhoneResult(Exchange exchange){
 logger.info("Invoking getAgePhoneResult from RestServiceImpl");

 String age=getFirstEntrySafelyFromList(exchange.getProperty("age", List.class));
 String phone=getFirstEntrySafelyFromList(exchange.getProperty("phone", List.class));

 return agePhoneService.getAgePhoneResult(age, phone);
 }

 public NameEmailService getNameEmailService() {
 return nameEmailService;
 }

 public AgePhoneService getAgePhoneService() {
 return agePhoneService;
 }

 public void setNameEmailService(NameEmailService nameEmailService) {
 this.nameEmailService = nameEmailService;
 }

 public void setAgePhoneService(AgePhoneService agePhoneService) {
 this.agePhoneService = agePhoneService;
 }

 private String getFirstEntrySafelyFromList(List<String> list){

 if (list!=null && !list.isEmpty()){
 return list.get(0);
 }
 return null;
 }
}

Reference

Camel Mailing List Question

SLF4J binding for ADFLogger - the missing piece

Sun, 03 Nov 2013 22:48:00 +0000

For reasons best left untold, in my day job, I was expected to provide an SLF4J Adapter for ADF Logger Oracle ADF. Not surprisingly, slf4j does not have an adapter for ADFLogger but since ADFLogger was just a gentle wrapper over Java Util Logging, it took a little over an hour to fill that gap.

The testcases (more like main programs) in the repository will confirm that the adapter framework plays well with the Oracle Diagnostic Logging –without breaking the format of the log messages. (which is more or less the only advantage which the ADFLogger provides).

You could download the entire codebase from the repository at github. Optionally, if you are interested only in the binding jar, please download it here.

Log Levels :

Considering there are many log levels on the ADF Logger (7) and fewer levels in SLF4J (5), a compromise was made on certain levels on the ADF Logger. The mapping is made as follows :

NOTE : Support for Log levels FINER and CONFIG were dropped during this adaptation.

Usage :

Just go ahead and replace your ADFLogger with SLF4J instantiations.

1
2
3

private static final Logger slfLogger = LoggerFactory.getLogger(LoggingChecker.class);

instead of

1
2

private static final ADFLogger adfLogger= ADFLogger.createADFLogger(LoggingChecker.class);

Logging methods

Logging to various levels

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20




@Test
	public void testLoggerLevels(){

 adfLogger.finest("finest Message from ADF Logger");
 slfLogger.trace("finest Message from SLF Logger");

 adfLogger.fine("fine Message from ADF Logger");
 slfLogger.debug("fine Message from SLF Logger");

 adfLogger.info("info Message from ADF Logger");
 slfLogger.info("info Message from SLF Logger");

 adfLogger.warning("warning Message from ADF Logger");
 slfLogger.warn("warning Message from SLF Logger");

 adfLogger.severe("severe Message from ADF Logger");
 slfLogger.error("severe Message from SLF Logger");
 }

Logging exceptions

where dummyException inherits a Throwable

Logging parameters

Development Notes :

The ADF Logger Adapter framework composes of two core classes (the ADFLoggerFactory.java and the ADFLoggerAdapter.java) and three other helper classes.
The Adapter can support multiple loggers including the ROOT logger of the ADFLogger (which is the default and has an empty string as its name).
It is no surprise that SLF4J uses the ADFLoggerFactory.java to instantiate the ADFLoggerAdapter.
For each Logger instance, the ADFLoggerAdapter.java composes an instance of ADFLogger (bound to the logger name) and delegates the call to the ADFLogger itself.
The fillCallerData method of the ADFLoggerAdapter.java filters the stackframes of the ADFLoggerAdapter from the entire logging stack so that the caller class and method name is retained as the one from the host application and not of the Adapter framework classes.

Testcase (well, sort of)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


public class TestClass {


 private static final ADFLogger adfLogger= ADFLogger.createADFLogger(TestClass.class);
 private static final Logger slfLogger = LoggerFactory.getLogger(TestClass.class);

 private List<String> dummyList1=null;
 private List<Integer> dummyList2=null;

 private Exception dummyException=null;

 @Before
 public void setUp(){

 	dummyList1=new ArrayList<String>();
 	dummyList1.add("Rock");
 	dummyList1.add("Paper");
 	dummyList1.add("Scissors");

 	dummyList2=new ArrayList<Integer>();
 	dummyList2.add(21);
 	dummyList2.add(22);
 	dummyList2.add(23);

 	dummyException=new Exception("Mind blowing Exception");

 }

 @Test
	public void testLoggerLevels(){

 adfLogger.finest("finest Message from ADF Logger");
 slfLogger.trace("finest Message from SLF Logger");

 adfLogger.fine("fine Message from ADF Logger");
 slfLogger.debug("fine Message from SLF Logger");

 adfLogger.info("info Message from ADF Logger");
 slfLogger.info("info Message from SLF Logger");

 adfLogger.warning("warning Message from ADF Logger");
 slfLogger.warn("warning Message from SLF Logger");

 adfLogger.severe("severe Message from ADF Logger");
 slfLogger.error("severe Message from SLF Logger");
	}


 @Test
 	public void testException(){

 adfLogger.severe("severe Message from ADF Logger", dummyException);
 slfLogger.error("severe Message from SLF Logger", dummyException);

 	}

 @Test
 	public void testParameters(){

 adfLogger.severe("severe Message from ADF Logger Param :{0}", dummyList1 );
 slfLogger.error("severe Message from SLF Logger Param :{}", dummyList1);

 adfLogger.severe("severe Message from ADF Logger Param 1 :[{0}] \n [{1}]", new Object[]{dummyList1, dummyList2} );
 slfLogger.error("severe Message from SLF Logger Param :{} \n {} ", dummyList1, dummyList2);
 slfLogger.error("severe Message from SLF Logger Param :[{}] \n [{}] ", new Object[]{dummyList1, dummyList2});

 	}

Output :

**Logger levels : **

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels
INFO: info Message from ADF Logger
Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels
INFO: info Message from SLF Logger
Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels
WARNING: warning Message from ADF Logger
Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels
WARNING: warning Message from SLF Logger
Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels
SEVERE: severe Message from ADF Logger
Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels
SEVERE: severe Message from SLF Logger

Testing Parameters :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14



Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters
SEVERE: severe Message from ADF Logger Param :[Rock, Paper, Scissors]
Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters
SEVERE: severe Message from SLF Logger Param :[Rock, Paper, Scissors]
Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters
SEVERE: severe Message from ADF Logger Param 1 :[[Rock, Paper, Scissors]]
 [[21, 22, 23]]
Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters
SEVERE: severe Message from SLF Logger Param :[Rock, Paper, Scissors]
 [21, 22, 23]
Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters
SEVERE: severe Message from SLF Logger Param :[[Rock, Paper, Scissors]]
 [[21, 22, 23]]

Testing Exception :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56



Jul 01, 2013 4:32:28 PM org.slf4j.test.TestClass testException
SEVERE: severe Message from ADF Logger
java.lang.Exception: Mind blowing Exception
	at org.slf4j.test.TestClass.setUp(TestClass.java:38)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Jul 01, 2013 4:32:28 PM org.slf4j.test.TestClass testException
SEVERE: severe Message from SLF Logger
java.lang.Exception: Mind blowing Exception
	at org.slf4j.test.TestClass.setUp(TestClass.java:38)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Building Camel-CXF REST Service in OSGi for Karaf - Multicasting and Aggregation

Sun, 20 Oct 2013 16:02:00 +0000

Please check out my other post on building plain CXF services (without Camel) in OSGi on Karaf.

This is a basic tutorial on how to

create a CXF REST service
multicast (and parallelize) the incoming request using Camel
source data from two different services
aggregate the response and
finally return the consolidated result as JSON to the the end user.

You could download the entire codebase from github.

What this application does, in simple terms

The result expected from this service is a hardcoded response which looks like

As you could see from the image, the top portion of the response is sourced from a service called NameEmailService and the second portion of the response is sourced from a service called AgePhoneService. The calls to enrich both the data are done concurrently and the consolidated result entity - ConsolidatedSearchResult is populated.

The Project structure looks like this :

There are two baby steps for Step 1.

Step 1.a - Create a CXF REST Service

As you might have guessed, there’s nothing complicated in this step. Just an interface and an implementation.

Interface

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



@Path("rest")
public interface RestService {


 @GET
 @Path("query/{queryString}")
 @Produces(MediaType.APPLICATION_JSON)
 public String sourceResultsFromTwoSources(@PathParam("queryString") String queryString);

}

Implementation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44



public class RestServiceImpl implements RestService {

 private static Logger logger= LoggerFactory.getLogger(AgePhoneServiceImpl.class);

 private NameEmailService nameEmailService;
 private AgePhoneService agePhoneService;

 public RestServiceImpl(){
 }

 //Do nothing. Camel intercepts and routes the requests
 public String sourceResultsFromTwoSources(String queryString) {
 return null;
 }


 public NameEmailResult getNameEmailResult(String queryString){
 logger.info("Invoking getNameEmailResult from RestServiceImpl");
 return nameEmailService.getNameAndEmail(queryString);
 }


 public AgePhoneResult getAgePhoneResult(String queryString){
 logger.info("Invoking getAgePhoneResult from RestServiceImpl");
 return agePhoneService.getAgePhoneResult(queryString);
 }

 public NameEmailService getNameEmailService() {
 return nameEmailService;
 }

 public AgePhoneService getAgePhoneService() {
 return agePhoneService;
 }

 public void setNameEmailService(NameEmailService nameEmailService) {
 this.nameEmailService = nameEmailService;
 }

 public void setAgePhoneService(AgePhoneService agePhoneService) {
 this.agePhoneService = agePhoneService;
 }
}

Note that the method implementation sourceResultsFromTwoSources returns a null. The truth is that this method doesn’t even get called when making a REST call. Camel intercepts all requests to the URL and routes it to various endpoints (calls two methods - getNameEmailResult() and getAgePhoneResult(), in our case).

Step 1.b - Create the Service Implementation

Kiddish implementations of the NameEmailService and the AgePhoneService are below :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10



public class NameEmailServiceImpl implements NameEmailService {

 public NameEmailResult getNameAndEmail(String queryString){

 return new NameEmailResult("Arun", "arun@arunma.com");

 }

}

1
2
3
4
5
6
7



public class AgePhoneServiceImpl implements AgePhoneService {

 public AgePhoneResult getAgePhoneResult(String queryString){
 return new AgePhoneResult(32, "111-222-333");
 }
}

Step 2, 3, 4 & 5

Well, I lied when I said 2,3,4 and 5 were 4 steps. They are all done as a single step using Camel routing and its Enterprise Integration Pattern implementations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18



public class RestToBeanRouter extends RouteBuilder {


 @Override
 public void configure() throws Exception {

 from ("cxfrs://bean://rsServer")
 .multicast()
 .parallelProcessing()
 .aggregationStrategy(new ResultAggregator())
 .beanRef("restServiceImpl", "getNameEmailResult")
 .beanRef("restServiceImpl", "getAgePhoneResult")
 .end()
 .marshal().json(JsonLibrary.Jackson)
 .to("log://camelLogger?level=DEBUG");
 }
}

Our Routing explained

Simply put, what our routerbuilder does is that it

from ("cxfrs://bean://rsServer") intercepts all requests to a JAX-RS server endpoint defined in the rest-blueprint.xml as

1
2
3
4
5




<cxf:rsServer id="rsServer" address="/karafcxfcamel"
 serviceClass="me.rerun.karafcxfcamel.rest.RestServiceImpl"
 loggingFeatureEnabled="true" />

The .multicast() forwards the original request untouched to
1. getNameEmailResult &
2. getAgePhoneResult methods in RestServiceImpl
The .parallelProcessing() places concurrent calls to the methods.
The .aggregationStrategy(new ResultAggregator()) specifies how the results from various multicasted sources should be, well, aggregated.

Our aggregator looks like :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33



public class ResultAggregator implements AggregationStrategy {

 @Override
 public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {


 ConsolidatedSearchResult consolidatedSearchResult=null;

 if (oldExchange==null){
 consolidatedSearchResult=new ConsolidatedSearchResult();
 }
 else{
 consolidatedSearchResult=oldExchange.getIn().getBody(ConsolidatedSearchResult.class);
 }


 NameEmailResult nameEmailResult=newExchange.getIn().getBody(NameEmailResult.class);
 AgePhoneResult agePhoneResult=newExchange.getIn().getBody(AgePhoneResult.class);

 if (nameEmailResult!=null){
 consolidatedSearchResult.setNameEmailResult(nameEmailResult);
 }

 if (agePhoneResult!=null){
 consolidatedSearchResult.setAgePhoneResult(agePhoneResult);
 }

 newExchange.getIn().setBody(consolidatedSearchResult);

 return newExchange;
 }
}

Our Aggregator explained

The aggregate method in our ResultAggregator is a little crude but does the job.

The aggregate method gets called for all multicasted endpoints whenever they finish.
So, the first time, the oldExchange will be null. We take that as an opportunity to construct the final consolidated result entity that we wanted to respond to the user.
We check whether the newExchange that comes in is the result of a call to the NameEmailService or AgePhoneService and populate the consolidated entity accordingly.
Finally, we return the consolidated entity - the returning does two jobs.
1. The consolidated entity comes in as oldExchange for the next call to the aggregate method. (more like chaining - the last returned object from the entity is the one which comes in as the incoming exchange for the next call)
2. Gets returned back to the user if it is the last call of aggregate (all multicast endpoints calls are complete).

Building CXF REST Service in OSGi for Karaf

Sun, 01 Sep 2013 15:50:00 +0000

I’ll leave it to the experts to tell how awesome OSGi is. Among the many benefits, I could tell you why we picked up OSGi for a pet project - Modularity, avoiding JAR hell and dynamic updates (hey, why not?)

We chose Apache Felix (an OSGi framework specification implementation) and Apache Karaf (ummm, how do I put this - something like an app server for OSGi applications). Besides serving as an OSGi container, Karaf has a lot of awesome features (pun intended). And we like the idea of managing multiple Karaf instances managed through Zookeeper.

Enough talk, let’s see some code. This is a rudimentary tutorial on how to run a basic OSGi CXF-JAX Rest Service on Karaf. Given a REST parameter (a name), the service would just return Hello, (name). That’s it !! The entire project could be downloaded here

So, if the request is http://localhost:8181/cxf/karafsimple/say/hello/arun, the response would be Hello, arun

Please note that this project does not have any domain models and therefore has only two projects - one for the REST (Controller) and the other for the actual service implementation.

The project structure looks like this :

Step 1 - Service Implementation

This project just has two ‘useful’ things - the HelloService interface and the HelloServiceImpl class.

1
2
3
4
5
6
7



package me.rerun.karafcxf.service.impl;

public interface HelloService {

 public String sayHello(String name);
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



package me.rerun.karafcxf.service.impl;

public class HelloServiceImpl implements HelloService {

 @Override
 public String sayHello(String name) {
 return "Hello, "+name;
 }

}

Stupid right?

Step 2 - REST project

Similar to the Service project, this project also has just two notable things - the HelloRestService interface and the HelloRestServiceImpl class.

HelloRestService

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17



package me.rerun.karafcxf.rest;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;

//Maps for the `say` in the URL

@Path("say")
public interface HelloRestService {

 @GET
 @Path("hello/{name}") //Maps for the `hello/John` in the URL
 public String handleGet(@PathParam("name") String name);

}

HelloRestServiceImpl

The HelloRestServiceImpl does nothing but calls the HelloService which gets injected through Blueprint Dependency Injection. Hey, does the inject look very familiar to Spring DI? Exactly !! The Blueprint DI is heavily influenced by Spring DI. In fact, the original work for blueprint is done by Spring.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


package me.rerun.karafcxf.rest;

import me.rerun.karafcxf.service.impl.HelloService;

public class HelloRestServiceImpl implements HelloRestService{


 //Just like Spring. Please add Getters/Setters. Blueprint annotations are still work in progress
 private HelloService helloService;


 public String handleGet(String name){

 return helloService.sayHello(name);

 }


 /*
 Constructor
 */

 public HelloRestServiceImpl(){
 }

 /*
 Getters and Setters
 */

 public HelloService getHelloService() {
 return helloService;
 }

 public void setHelloService(HelloService helloService) {
 this.helloService = helloService;
 }

}

Step 3 - Injections

XMLs (of any name) inside OSGI-INF/blueprint folder will get picked up for DI scanning.

serviceimpl.xml

Does two things in one tag :

Registers the HelloService into the service registry for lookup
Says that its implementation is HelloServiceImpl

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11



<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">


 <service ref="helloServiceBean" interface="me.rerun.karafcxf.service.impl.HelloService">
 <bean class="me.rerun.karafcxf.service.impl.HelloServiceImpl"/>
 </service>

</blueprint>

In fact, you could do this in two separate steps. More on that here.

1
2
3
4
5



<service id="helloServiceBean" ref="helloServiceImpl"
 interface="me.rerun.karafcxf.service.impl.HelloService" />

<bean id="helloServiceImpl" class="me.rerun.karafcxf.service.impl.HelloServiceImpl" />

rest.xml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40



<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
 xmlns:jaxws="http://cxf.apache.org/blueprint/jaxws"
 xmlns:jaxrs="http://cxf.apache.org/blueprint/jaxrs"
 xmlns:cxf="http://cxf.apache.org/blueprint/core"
 xsi:schemaLocation="
 http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
 http://cxf.apache.org/blueprint/jaxws http://cxf.apache.org/schemas/blueprint/jaxws.xsd
 http://cxf.apache.org/blueprint/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd
 http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd">
<!-- 1 -->
 <cxf:bus id="cxfBus1">
 <cxf:features>
 <cxf:logging/>
 </cxf:features>
 </cxf:bus>

<!-- 2 -->
 <jaxrs:server address="/karafsimple" id="someRestService">
 <jaxrs:serviceBeans>
 <ref component-id="restServiceImpl"/>
 </jaxrs:serviceBeans>
 </jaxrs:server>

<!-- 3 -->
 <!-- Implementation of the rest service -->
 <bean id="restServiceImpl" class="me.rerun.karafcxf.rest.HelloRestServiceImpl">
 <property name="helloService" ref="helloServiceBean"/> <!--Points to the reference below -->
 </bean>

<!-- 4 -->
 <!-- This has to point to the service registered through serviceimpl.xml in the service.impl project -->

 <reference id="helloServiceBean"
 interface="me.rerun.karafcxf.service.impl.HelloService">
 </reference>

</blueprint>

cxf-bus is the bus configuration for CXF. It is like the manager for all CXF services. The most common use as far as I know is to configure custom interceptors (for auditing, request/response manipulation, headers manipulation etc)
the jaxrs:server initiates a server which would start listening to the URLs that we mapped for. Of course, we would want to map the handlers for the URLs and those go under the serviceBeans.
The third note in the XML is the just the restServiceImpl configuration and the injection of the helloService as a property inside the HelloRestServiceImpl
The reference tag will lookup the service registry for a bean with the same id.

Step 4 - KAR - Karaf Archive (Optional but easier this way)

Technically, you could just start throwing the bundles generated through the Maven into to the deploy directory of karaf. However, as the project goes big and your dependencies are becoming many, it is advisable to use the .kar archive. Picture .kar as your .war or .ear bundle. A .kar archive, internally, looks something like this :

Building .kar

Building the .kar is composed of two steps :

Step 1

feature.xml

Similar to your web descriptor or application descriptor, we have the features descriptor in Karaf to represent the entire repository that we are bundling together to compose our application.

In our case, we don’t have any external .jar dependencies except for the cxf and the http service for which we are using the in-built features available inside karaf.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31



<?xml version="1.0" encoding="UTF-8"?>
<features xmlns="http://karaf.apache.org/xmlns/features/v1.0.0"
 name="${project.artifactId}-${project.version}">
 <feature name="karafcxf" description="karaf cxf hello" version="1.0-SNAPSHOT" resolver="(obr)">
 <details>${project.description}</details>
 <feature>http</feature>
 <feature>cxf</feature>
 <feature>this-project-dependants</feature>

 <bundle>mvn:karafcxf/karafcxf.service.impl/1.0-SNAPSHOT</bundle>
 <bundle>mvn:karafcxf/karafcxf.rest/1.0-SNAPSHOT</bundle>
 </feature>


 <feature name="this-project-dependants">
 <!-- Dependent bundles if you have any. This project doesn't have one
 <bundle>mvn:org.apache.commons/commons-lang3/3.1</bundle>
 <bundle>mvn:org.codehaus.jackson/jackson-core-asl/1.9.5</bundle>
 <bundle>mvn:org.codehaus.jackson/jackson-mapper-asl/1.9.5</bundle>
 <bundle>mvn:org.codehaus.jackson/jackson-jaxrs/1.9.5</bundle>
 <bundle>mvn:com.google.guava/guava/14.0.1</bundle>
 <bundle>wrap:mvn:org.apache.httpcomponents/httpcore/4.2.4</bundle>
 <bundle>wrap:mvn:org.apache.httpcomponents/httpmime/4.2.5</bundle>
 <bundle>wrap:mvn:org.noggit/noggit/0.5</bundle>
 <bundle>wrap:mvn:org.apache.solr/solr-solrj/4.4.0</bundle>
 -->
 </feature>


</features>

Step 2

Maven plugin configuration to create .kar, which indicates where your feature.xml is located (Note that this file is located inside your karaf project)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21



<build>
 <plugins>
 <plugin>
 <groupId>org.apache.karaf.tooling</groupId>
 <artifactId>features-maven-plugin</artifactId>
 <version>2.3.2</version>
 <executions>
 <execution>
 <id>create-kar</id>
 <goals>
 <goal>create-kar</goal>
 </goals>
 <configuration>
 <featuresFile>${project.basedir}/src/main/resources/feature.xml</featuresFile>
 </configuration>
 </execution>
 </executions>
 </plugin>
 </plugins>
</build>

Wraps

Notice the wrap protocol in front of the mvn protocol in few bundles as in

<bundle>wrap:mvn:org.apache.httpcomponents/httpmime/4.2.5</bundle>

Not all Jars are OSGi ready but they would obviously be used as a dependency in our project. In those cases, Karaf notices the wrap protocol and bundles the JAR into an OSGi bundle. In case of doubt, just drop off the wrap and Karaf would complain that the jar is not OSGi compatible. (Alternatively, you could open up each of the dependant jars and check their manifest files)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


013-08-28 01:38:48,669 | WARN | raf-2.3.2/deploy | KarArtifactInstaller | eployer.kar.KarArtifactInstaller 192 | 24 - org.apache.karaf.deployer.kar - 2.3.2 | Unable to install Kar feature xx-xxx-xxxxxx/0.0.0
org.osgi.framework.BundleException: Jar is not a bundle, no Bundle-SymbolicName mvn:org.apache.httpcomponents/httpcore/4.2.4
	at org.apache.karaf.features.internal.FeaturesServiceImpl.installBundleIfNeeded(FeaturesServiceImpl.java:836)[26:org.apache.karaf.features.core:2.3.2]
	at org.apache.karaf.features.internal.FeaturesServiceImpl.doInstallFeature(FeaturesServiceImpl.java:618)[26:org.apache.karaf.features.core:2.3.2]
	at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:414)[26:org.apache.karaf.features.core:2.3.2]
	at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:402)[26:org.apache.karaf.features.core:2.3.2]
	at Proxy508d2419_d21e_4a93_b7fb_26e28d2f03a6.installFeature(Unknown Source)[:]
	at org.apache.karaf.deployer.kar.KarArtifactInstaller.installFeatures(KarArtifactInstaller.java:189)[24:org.apache.karaf.deployer.kar:2.3.2]
	at org.apache.karaf.deployer.kar.KarArtifactInstaller.install(KarArtifactInstaller.java:134)[24:org.apache.karaf.deployer.kar:2.3.2]
	at org.apache.karaf.deployer.kar.KarArtifactInstaller.update(KarArtifactInstaller.java:348)[24:org.apache.karaf.deployer.kar:2.3.2]
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.update(DirectoryWatcher.java:1103)[6:org.apache.felix.fileinstall:3.2.6]
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.update(DirectoryWatcher.java:898)[6:org.apache.felix.fileinstall:3.2.6]
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:482)[6:org.apache.felix.fileinstall:3.2.6]
	at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:291)[6:org.apache.felix.fileinstall:3.2.6]

Changing Log Levels

For all our development environment, we would want to increase our log level to get more feedback from Karaf. This could be achieved by modifying the org.ops4j.pax.logging.cfg file located in your <karaf-installation-directory>/etc

Step 5 - Other Maven configuration

Not technically a step because we would have covered this from the beginning anyway. And nothing fancy here.

Parent pom.xml

Has the rest of the sub-modules configured.
The various library dependencies configured

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27



<project xmlns="http://maven.apache.org/POM/4.0.0"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

 <modelVersion>4.0.0</modelVersion>

 <groupId>karafcxf</groupId>
 <artifactId>karafcxf</artifactId>
 <packaging>pom</packaging>
 <version>1.0-SNAPSHOT</version>


 <properties>
 <cxf.version>2.7.5</cxf.version>
 </properties>

 <modules>
 <module>karafcxf.rest</module>
 <module>karafcxf.service.impl</module>
 <module>karafcxf.kar</module>
 </modules>


 <dependencies>
		…
		…

Rest and ServiceImpl pom.xml

The other poms.xmls aren’t interesting

They just affiliate themselves to the parent pom with the parent tag as in

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20



<build>
 <plugins>
 <plugin>
 <groupId>org.apache.felix</groupId>
 <artifactId>maven-bundle-plugin</artifactId>
 <version>2.3.7</version>
 <extensions>true</extensions>
 <configuration>
 <instructions>
 <Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
 <Bundle-Version>${project.version}</Bundle-Version>
 <Bundle-Activator>com.nutraspace.coreservices.search.rest.Activator</Bundle-Activator>
 <Export-Package>com.nutraspace.coreservices.search.rest*;version=${project.version}</Export-Package>
 <Import-Package>*</Import-Package>
 </instructions>
 </configuration>
 </plugin>
 </plugins>
 </build>

Step 6 - Bring up Karaf

My Karaf installation is located at : /Users/Gabriel/apps/apache-karaf-2.3.2

**Start Karaf : **

<installation-dir>/bin/./karaf

Installing CXF and HTTP services

features:chooseurl cxf 2.7.5

features:install http cxf

Checking whether your bundle is installed and your service running

osgi:list

dxf:list-endpoints

Stop Karaf

Ctrl +D (Please note that Ctrl +C closes the connection abruptly instead of stopping Karaf)

In case of accidental Ctrl+C and if Karaf isn’t starting properly, do a

rm -rf <installation-dir>/data/cache/*

URL Mapping

Like I mentioned earlier, the target request URL will be something like http://localhost:8181/cxf/karafsimple/say/hello/arun and the response would be Hello, arun

The HelloRestService interface has all the JAX RS annotations for the URL mapping. Well, technically, the interfaces just maps for say/hello/(name)
The karafsimple in the URL is derived from the the JAX RS address in blueprint.xml (about that later in Step 3 - rest.xml).
The cxf is a default if you deploy a CXF service on Karaf which obviously you could change.

Quicksorting - 3-way and Dual Pivot

Thu, 13 Jun 2013 12:35:00 +0000

It’s no news that Quicksort is considered one of the most important algorithms of the century and that it is the defacto system sort for many languages, including the Arrays.sort in Java.

So, what’s new about quicksort?

Well, nothing except that I figured just now (after 2 damn years of release of Java 7) that the Quicksort implementation of the Arrays.sort has been replaced with a variant called Dual-Pivot QuickSort. This thread is not only awesome for this reason but also how humble Jon Bentley and Joshua Bloch really are.

What did I do next?

Just like everybody else, I wanted to implement it and do some benchmarking - against some 10 million integers (random and duplicate).

Oddly enough, I found the following results :

Random Data :

Quick Sort Basic : 1222 millis
Quick Sort 3 Way : 1295 millis (seriously !!)
Quick Sort Dual Pivot : 1066 millis

Duplicate Data :

Quick Sort Basic : 378 millis
Quick Sort 3 Way : 15 millis
Quick Sort Dual Pivot : 6 millis

Stupid Question 1 :

I am afraid that I am missing something in the implementation of 3-way partition. Across several runs against random inputs (of 10 million) numbers, I could see that the single pivot always performs better (although the difference is less than 100 milliseconds for 10 million numbers).

I understand that the whole purpose of making the 3-way Quicksort as the default Quicksort until now is that it does not give 0(n^2) performance on duplicate keys - which is very evident when I run it against duplicate input. But is it true that for the sake of handling duplicate data, a small penalty is taken by 3-way? Or is my implementation bad?

Stupid Question 2

My Dual Pivot implementation (link below) does not handle duplicates well. It takes a sweet forever (0(n^2)) to execute. Is there a good way to avoid this? Referring to the Arrays.sort implementation, I figured out that ascending sequence and duplicates are eliminated well before the actual sorting is done. So, as a dirty fix, if the pivots are equal I fast-forward the lowerIndex until it is different than pivot2. Is this a fair implementation?

1
2
3
4
5
6
7



	else if (pivot1==pivot2){
 while (pivot1==pivot2 && lowIndex<highIndex){
 lowIndex++;
 pivot1=input[lowIndex];
 }
 }

That’s it. That is all I did?

I always find the tracing of the algorithm interesting but with the number of variables in Dual Pivot quicksort, my eyes found it overwhelming while debugging. So, I also went ahead and created trace-enabled implementations (for all 3) so that I could see where the variable pointers currently are.

These trace-enabled classes just overlay where the pointers are below the array values. I hope you find these classes useful.

eg. for a Dual Pivot iteration

Where can you download the code?

The entire project (along with a few lame implementations of DSA) is available on github here. The quicksort classes alone could be found here.

Here’s my implementation of the SinglePivot (Hoare), 3-way (Sedgewick) and the new Dual-Pivot (Yaroslavskiy)

Single Pivot

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


package basics.sorting.quick;

import static basics.sorting.utils.SortUtils.exchange;
import static basics.sorting.utils.SortUtils.less;
import basics.shuffle.KnuthShuffle;

public class QuickSortBasic {

	public void sort (int[] input){

		//KnuthShuffle.shuffle(input);
		sort (input, 0, input.length-1);
	}

	private void sort(int[] input, int lowIndex, int highIndex) {

		if (highIndex<=lowIndex){
			return;
		}

		int partIndex=partition (input, lowIndex, highIndex);

		sort (input, lowIndex, partIndex-1);
		sort (input, partIndex+1, highIndex);
	}

	private int partition(int[] input, int lowIndex, int highIndex) {

		int i=lowIndex;
		int pivotIndex=lowIndex;
		int j=highIndex+1;


		while (true){

			while (less(input[++i], input[pivotIndex])){
				if (i==highIndex) break;
			}

			while (less (input[pivotIndex], input[--j])){
				if (j==lowIndex) break;
			}

			if (i>=j) break;

			exchange(input, i, j);

		}

		exchange(input, pivotIndex, j);

		return j;
	}

}

3-way

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


package basics.sorting.quick;

import static basics.shuffle.KnuthShuffle.shuffle;
import static basics.sorting.utils.SortUtils.exchange;
import static basics.sorting.utils.SortUtils.less;

public class QuickSort3Way {


	public void sort (int[] input){
		//input=shuffle(input);
		sort (input, 0, input.length-1);
	}

	public void sort(int[] input, int lowIndex, int highIndex) {


		if (highIndex<=lowIndex) return;

		int lt=lowIndex;
		int gt=highIndex;
		int i=lowIndex+1;

		int pivotIndex=lowIndex;
		int pivotValue=input[pivotIndex];


		while (i<=gt){


			if (less(input[i],pivotValue)){
				exchange(input, i++, lt++);
			}
			else if (less (pivotValue, input[i])){
				exchange(input, i, gt--);
			}
			else{
				i++;
			}


		}

		sort (input, lowIndex, lt-1);
		sort (input, gt+1, highIndex);


	}

}

Dual Pivot

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65



package basics.sorting.quick;

import static basics.shuffle.KnuthShuffle.shuffle;
import static basics.sorting.utils.SortUtils.exchange;
import static basics.sorting.utils.SortUtils.less;

public class QuickSortDualPivot {

	public void sort (int[] input){
		//input=shuffle(input);
		sort (input, 0, input.length-1);
	}

	private void sort(int[] input, int lowIndex, int highIndex) {

		if (highIndex<=lowIndex) return;

		int pivot1=input[lowIndex];
		int pivot2=input[highIndex];


		if (pivot1>pivot2){
			exchange(input, lowIndex, highIndex);
			pivot1=input[lowIndex];
			pivot2=input[highIndex];
			//sort(input, lowIndex, highIndex);
		}
		else if (pivot1==pivot2){
			while (pivot1==pivot2 && lowIndex<highIndex){
				lowIndex++;
				pivot1=input[lowIndex];
			}
		}


		int i=lowIndex+1;
		int lt=lowIndex+1;
		int gt=highIndex-1;

		while (i<=gt){

			if (less(input[i], pivot1)){
				exchange(input, i++, lt++);
			}
			else if (less(pivot2, input[i])){
				exchange(input, i, gt--);
			}
			else{
				i++;
			}

		}


		exchange(input, lowIndex, --lt);
		exchange(input, highIndex, ++gt);

		sort(input, lowIndex, lt-1);
		sort (input, lt+1, gt-1);
		sort(input, gt+1, highIndex);

	}

}

Your's deeply - Why Arrays.deepEquals when we have Arrays.equals

Sun, 04 Nov 2012 18:04:00 +0000

While everybody would naturally accept the following lines of code on grounds of reference equality and value equality and that String and wrappers override the equals method, it takes some effort at first to accept the behavior of Arrays.equals and Arrays.deepEquals

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12



Object obj1=new Object();
Object obj2=new Object();

String hello1=new String("hello");
String hello2=new String("hello");

System.out.println(hello1.equals(hello2)); //returns true
System.out.println(hello1==hello2); //returns false

System.out.println(obj1.equals(obj2)); //returns false
System.out.println(obj1==obj2); //returns false

Boring Stuff :-(

First, the similarities

Both Arrays.equals and Arrays.deepEquals are similar in certain behaviors as

If they are the same object (reference equality), they return true
If either of the compared objects are null, then return false
If the array lengths are not equal, then return false
They care about order (position)

Next, the differences

Arrays.equals is really just skin deep

Opening up the souce, we could see that the lousy Array.equals just does this

1
2
3
4
5
6
7
8



for (int i=0; i<length; i++) {
 Object o1 = a[i];
 Object o2 = a2[i];

 if (!(o1==null ? o2==null : o1.equals(o2)))
 return false;
}

So, it just loops through the given arrays, does a equals on each of the pairs. This means that if you are passing in a String/Wrapper array or any other arrays whose equals method is overridden, then they are equal. Non-equals-overridden classes (derived from Object) will return false.

Arrays.deepEquals looks really deep

From the source, we could understand that Arrays.deepEquals

Loops through the input arrays, gets each pair
Analyses the type of each pair
Delegates the equal deciding logic to one of the overloaded Arrays.equals if they are one of the primitive arrays
Delegates recursively to Arrays.deepEquals if it is an Object array
Calls the respective object’s equals, for any other object

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25



for (int i = 0; i < length; i++) {
 Object e1 = a1[i];
 Object e2 = a2[i];

 if (e1 == e2)
 continue;
 if (e1 == null)
 return false;

 // Figure out whether the two elements are equal
 boolean eq;
 if (e1 instanceof Object[] && e2 instanceof Object[])
 eq = deepEquals ((Object[]) e1, (Object[]) e2);
 else if (e1 instanceof byte[] && e2 instanceof byte[])
 eq = equals((byte[]) e1, (byte[]) e2);
 …
 …
 else
 eq = e1.equals(e2);

 if (!eq)
 return false;
 }
 return true;

Now, the Awesome stuff !!

Equals not overridden (nested and non-nested)

While doing an Arrays.equals for nested or non-nested ’non-overridden equals’ objects, it is safe to assume that if Arrays.equals return false, then Arrays.deepEquals also return false.

Non-Nested

So, given two arrays

1
2
3



private YourClass[] equalsNotOverriddenArrayNonNested1={new YourClass(), new YourClass()};
private YourClass[] equalsNotOverriddenArrayNonNested2={new YourClass(), new YourClass()};

where YourClass is simply

1
2
3



public class YourClass {
}

then the following assertions are true

1
2
3



assertFalse(Arrays.equals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2));
assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2));

Nested

Also given two arrays,

1
2
3



private Object[] equalsNotOverriddenArrayNested1={new YourClass(), new YourClass[]{new YourClass()}};
private Object[] equalsNotOverriddenArrayNested2={new YourClass(), new YourClass()};

the following assertions are true

1
2
3



assertFalse(Arrays.equals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2));
assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2));

Equals overridden

Non-Nested

While doing an Arrays.equals for non-nested ‘overridden equals’ objects, it can be said that if Arrays.equals is true, then Arrays.deepEquals also return true.

Given two String arrays

1
2
3



private Object[] equalsOverriddenArrayNonNested1={"101","201"};
private Object[] equalsOverriddenArrayNonNested2={"101","201"};

the following assertions are true

1
2
3



assertTrue(Arrays.equals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2));
assertTrue(Arrays.deepEquals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2));

since they are just one-to-one equals call on each pair.

Nested

Interesting scenario : While doing an Arrays.equals for nested ‘overridden equals’ objects, if Arrays.equals is false, then Arrays.deepEquals need not be false.

Consider two Object arrays which has two values - a String and a String array

1
2
3



private Object[] equalsOverriddenArrayNested1={new String("hello"), new String[]{new String("hello")}};
private Object[] equalsOverriddenArrayNested2={new String("hello"), new String[]{new String("hello")}};

you can notice that the Arrays.equals return false while Arrays.deepEquals return true

1
2
3



assertFalse(Arrays.equals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2));
assertTrue(Arrays.deepEquals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2));

The result for Arrays.deepEquals is logical since each (from the source), the method loops through each pair of elements, checks whether it is an array type and calls deepEquals on each of the pair. If it is an non-array type, then it just calls the equals on the object.

However, the result of Arrays.equals is tricky but at the same time obvious. The Arrays.equals method blindly calls equals on each pair and since the second arguments String[] are of Object type (whose equals is not overridden), it checks for reference equality and fails !!

The entire testcase can be found below :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;

import java.util.Arrays;

import org.testng.annotations.Test;


public class DeepEquals {

	private Object[] equalsOverriddenArrayNonNested1={"101","201"};
	private Object[] equalsOverriddenArrayNonNested2={"101","201"};

	private YourClass[] equalsNotOverriddenArrayNonNested1={new YourClass(), new YourClass()};
	private YourClass[] equalsNotOverriddenArrayNonNested2={new YourClass(), new YourClass()};

	private Object[] equalsNotOverriddenArrayNested1={new YourClass(), new YourClass[]{new YourClass()}};
	private Object[] equalsNotOverriddenArrayNested2={new YourClass(), new YourClass()};

	private Object[] equalsOverriddenArrayNested1={new String("hello"), new String[]{new String("hello")}};
	private Object[] equalsOverriddenArrayNested2={new String("hello"), new String[]{new String("hello")}};

	@Test
	public void stringArrayTest(){

		assertFalse (equalsOverriddenArrayNonNested1==equalsOverriddenArrayNonNested2);
		assertFalse(equalsOverriddenArrayNonNested1.equals(equalsOverriddenArrayNonNested2));

		assertTrue(Arrays.equals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2));

		assertTrue(Arrays.deepEquals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2));

	}


	@Test
	public void objectArrayTestNonNested(){

		assertFalse (equalsNotOverriddenArrayNonNested1==equalsNotOverriddenArrayNonNested2);
		assertFalse(equalsNotOverriddenArrayNonNested1.equals(equalsNotOverriddenArrayNonNested2));

		assertFalse(Arrays.equals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2));

		assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2));

	}



	@Test
	public void objectArrayTestNested(){

		assertFalse (equalsNotOverriddenArrayNested1==equalsNotOverriddenArrayNested2);
		assertFalse(equalsNotOverriddenArrayNested1.equals(equalsNotOverriddenArrayNested2));

		assertFalse(Arrays.equals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2));

		assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2));

	}



	@Test
	public void objectArrayTest2(){

		assertFalse (equalsOverriddenArrayNested1==equalsOverriddenArrayNested2);
		assertFalse(equalsOverriddenArrayNested1.equals(equalsOverriddenArrayNested2));

		assertFalse(Arrays.equals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2));

		assertTrue(Arrays.deepEquals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2));

	}


}

Evaluating Infix expression - multiple digits

Sun, 14 Oct 2012 16:06:00 +0000

If you are looking for evaluating an infix expression with parantheses, don’t waste your time here. Visit my other fresh write up here

These images have been running around Facebook for a while now. Though it is eye-damaging primary level arithmetic, it is kind of sweet to write it as a program. Just for the fun of it, I created an html page driven by a javascript implementation. Here is the link.

So, essentially the idea is to evaluate the infix notation in a natural way - respecting the operator precedence. (If you consider this as a puzzle, then there are other variations which involves evaluating the expression in such a way that the result is a maximum value/minimum value)

Please note that the algorithm (and therefore, the program), as it stands does not handle parentheses. This is just a two-stack algorithm with a small variation to accommodate numbers more than a digit.

The algorithm goes like this :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


1. Initialize two stacks - one for holding values and one for holding operators
2. Read the input expression from left to right - one character at a time.
3. If the character is a space, ignore
4. If the character is a number, then append it to an accumulating variable
 (don't push it to the stack yet)
5. If the character is an operator, push the accumulated number to the value stack and
 reinitialize the accumulator.

	5a. If the current operator is of higher precedence than the first item in the operator stack,
	 then safely ignore and read further.

	5b. If the current operator is of lower precedence than the first item in the operator
	 stack (peek view), then evaluate the previous set (two from value stack and one
	 from operator stack) and insert the result into the value stack

	5c. Finally, insert the current operator to the operator stack

A Groovy implementation is here :

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174


/**
 * Only basic arithmetic operators are supported. However, could easily be
 * extended to support any binary operators
 *
 * This program is not an infix to postfix converter. However, this program does the following
 *
 * 1) Evaluates a non-parenthesized infix expression and drives it to a final value
 * 2) Does not support parentheses
 * 3) Supports multiple digits as operands
 * 4) Empty space allowed
 * 5) Floating point numbers not supported
 * 6) Supports only the 4 basic arithmetic operators but could easily be extended to support any other
 * binary operation
 *
 * @author Arun Manivannan
 */



public class InfixToValue {


 public static final String OPERATORS="+-/*)" //ignore the "(" as an operator
 public static final Map OPERATOR_LEVELS= ["*":1, "/":1, "+":2,"-":2]

 public static void main(String[] args) {

 String inputExpression="9 - 5+5*0+3" //610
 InfixToValue ifixToPfix=new InfixToValue()
 ifixToPfix.infixToValue(inputExpression);
 }

 def infixToValue(String inputExpression) {
 Stack<String> expressionStack = new Stack<String>()
 Stack<Long> valueStack = new ArrayDeque<Long>() //to hold only non-decimal values

 String eachNumber="" //to hold multiple digit values. lame idea, i know, but can't think of anything else at 2 AM

 int totalCharInInput=inputExpression.length()

 inputExpression.each { eachChar->
 totalCharInInput--

 println ("each char : "+eachChar)

 if (eachChar.trim()==""){
 //ignore
 }
 else if (isValidOperator(eachChar)){


 valueStack.push(Long.parseLong(eachNumber)) //push the till now accumulated number to the value stack
 eachNumber=""


 println ("Iteration Value Stack 1: "+valueStack)
 println ("Iteration Expression Stack 1: "+expressionStack)

 if (expressionStack.isEmpty()){ //first item 
 expressionStack.push(eachChar)
 }
 //if the current operator has higher precedence than the first item in the stack, then it is fine.
 //Just insert the incoming
 else if (getHigherPrecedenceOperator(eachChar, expressionStack.peek())==eachChar){
 expressionStack.push(eachChar)
 }

 else if (isRightParen(eachChar)){
 evaluateAndPushValueToValueStack()

 }
 //if the current operator is lesser, then the previous higher precedence expression have to be
 //evaluated. Else, we would be making wrong calculations while popping off for final evaluation
 else{

 println ("Value Stack 1: "+valueStack)
 println ("Expression Stack 1: "+expressionStack)

 //the current operator has higher precedence. Evaluate expression
 evaluateAndPushValueToValueStack(valueStack, expressionStack)

 //after evaluation of the higher pair, don't forget to insert the current character into the expression stack
 expressionStack.push(eachChar)


 }



 }
 else if (eachChar.isNumber()){
 eachNumber+=eachChar

 if (totalCharInInput==0){
 valueStack.push(Long.parseLong(eachNumber)) //the last element
 }
 }

 /*else {
 //other input (alphabets and special chars) are treated as garbage
 }*/



 }

 println ("Expression stack " +expressionStack);
 println ("Value stack : "+valueStack)

 while (!expressionStack.empty){
 evaluateAndPushValueToValueStack(valueStack,expressionStack)
 }
 println ("Final value : "+valueStack)
 }

 boolean isRightParen(String operator) {
 operator==")"?true:false
 }

 boolean isLeftParen(String operator) {
 operator=="("?true:false
 }


 private void evaluateAndPushValueToValueStack(Stack<Long> valueStack, Stack<String> expressionStack) {

 Long firstOperand=valueStack.pop()
 Long secondOperand=valueStack.pop()
 println ("Value stack : "+firstOperand+ ":"+ secondOperand )
 Long evaluatedValue = this.evaluate(secondOperand, firstOperand, expressionStack.pop()) //intermediate result
 valueStack.push(evaluatedValue)
 }


 Long evaluate(Long firstOperand, Long secondOperand, String operator) {
 Long returnValue
 switch (operator) {
 case "+" :
 returnValue=firstOperand+secondOperand
 break
 case "-" :
 returnValue=firstOperand-secondOperand
 break;
 case "*":
 returnValue=firstOperand*secondOperand
 break
 case "/":

 if (secondOperand==0){ //cheating death by zero
 returnValue=0
 }
 else{
 returnValue=firstOperand/secondOperand
 }
 break;

 }

 }

 boolean isValidOperator(String input) {

 OPERATORS.contains(input)?true:false

 }

 String getHigherPrecedenceOperator(String firstOperator, String secondOperator){

 OPERATOR_LEVELS[firstOperator]<OPERATOR_LEVELS[secondOperator]?firstOperator:secondOperator

 }


}

Grokking Geb - Prerequisites

Mon, 01 Oct 2012 19:10:00 +0000

Extremely sorry about the delay on Part 2 of this series. Graduate exams are just round the corner and I am unable to find time for quality research. Exams get over by end of November.

I thoroughly enjoy Geb and I think it is the most stylish way to write functional tests.

**So, the Prerequisites : **

Some Groovy Magic. Most of all that you need to learn Groovy is covered in this manual but for obvious reasons if you get obsessed with the language you might want to consider Groovy in Action. If you are from a Java (except for closures) or a Python background, you could probably skim through the tutorial for 15 minutes and you are there already).
A little Selenium. The more, the better but fear not this single page tells you all that you need to know about the Selenium Webdriver that you would generally use.
jQuery selectors (everybody says that it is easy but frankly, I refer to the manual at least twice per hour. I am dumb, so…). If you are new to jQuery, you would want to start from basic selectors and click on the left navigation menu for more.
PageObjects is actually not a prerequisite for Geb but PageObjects are so awesome that you never wanted to think about Geb outside of it. Interestingly, PageObjects is not a technology that you need to pick. It is cute little pattern which says that you wrap the structure of your HTML page into an Object so that the actual test does not have to deal with it. Hah. Got you. Let me put that in plain English.

Say, you have a registration form with input textbox which has an id of “nametext”. How would you get the handle of the textbox? In DOM terms, in javascript, you would just do a

1

document.getElementById("nametext")

In Selenium, you would do a very similar thing

1

driver.findElement(By.id("nametext"))

So, if you would want to populate Jason in your text box in Selenium, you would do a

1

driver.findElement(By.id("nametext")).sendKeys("Jason");

If you do that for all your input fields, very soon your testcases become ugly and hateful. Instead of that, in OO terms, you encapsulate. You create a new class, say RegistrationPage and wrap your findElement and sendKeys as in :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public class RegistrationPage{

	…

	public RegistrationPage fillRegistrationForm(String name, String email){

		driver.findElement(By.id("nametext")).sendKeys(name);
		driver.findElement(By.id("emailtext")).sendKeys(email);

	}

}

and from your testcase, you would say

1
2
3



RegistrationPage regPage=new RegistrationPage();
regPage.fillRegistrationForm("Jason","jason@bourne.com");

(Even better idea is to wrap your input values into a class and pass it to the fillRegistrationForm)

In fact, Geb leverages PageObjects in a much better way - jQuery selectors to the rescue

 1
 2
 3
 4
 5
 6
 7
 8
 9
10



	class InputFormPage extends Page{

		…

 		static content={
 	name {$("input", id:"entry_0")}
	 emailAddress {$("input", id:"entry_1")}
 	}
 }

and in your testcase, you would just say

1
2


name.value ("Jason")
emailAddress.value ("jason@bourne.com")

(you could do even better. Stay tuned)

Architexa - A fine code reading tool

Mon, 24 Sep 2012 19:51:00 +0000

I love reading code.

For two reasons :

If the code is bad, it is an awesome ego boost PLUS you get to foul mouth someone who has a good reputation for designing amazing things.
If the code is good, then you get to learn some new tricks and some cool patterns yet to be published anywhere

I am sure you read a lot of code and I am sure you’ll love this awesome code reading tool which I came across a few days ago - Architexa. Architexa also has collaborative documentation support, as I could clearly see with their codemaps project, but I’ve been using primarily for code reading.

Hey, forgot to tell you - Architexa is absolutely free for individual users.

Before going any further, a WOW video is here

A UML diagram is worth a thousand lines of code :

Architexa does not bring in a lot of diagrams but only the most relevant.

Class diagram - for Static view
Sequence diagram - for Dynamic view
Layered diagram (package diagram + awesomeness) - for overall project static view

Class first - Class diagrams :

Imagine you are assigned to a new project or you are simply opening up an open source library to check its implementation.

If you are like me, you would probably

go directly to the class that you are interested and start from there
check all the methods of the class in the Outline window
go to the method of choice and start drawing a Class dependency graph on your mind using the caller and the callee hierarchy.

With Architexa, all you need to do is to throw all your classes into a diagram. Inheritence hierarchy gets automatically mapped by default.

If you would like to map the associations (which I always do), then all you need to do is a right click.

You get to choose what associates/members you would want to display in your diagram

The best part I like about Architexa is that you could add/remove items that are not of interest to you. No more noisy A3 print outs.

The big picture - Layered diagrams

If you dont have any Class of interest, then you would probably do a package scan - you simply open up all packages and check how the files in the project are arranged. Again, you are trying to construct a Package Dependency graph and identify commonly used design patterns or frameworks. From my personal experience, I always found that idenfifying package dependency is more difficult than individual classes. However, Architexa does it naturally.

The following is a layered representation of a Google web toolkit project where the client and the server folder are dependent on the shared package (for models, of course). The white components indicate that they are siblings.

Say 1, 2, 3 - Sequence diagrams

I love the way that the messages are marked with not only its name but also the complete parameter list. And I absolutely love the way that the classes are marked with either a “C” (for class with source), “I” and “E” for interfaces and enums and a “jar icon” if the class is sourced from a bundle. Generics support in messages. FTW !!

No more lonely notes

The best part is that the diagrams does not live in isolation. You could just double click on your class component in your diagram and go to the respective class, or double click on your sequence message and go to that method in the code directly. This helps in switching back and forth between code and diagram (Ctrl+F6). Cool huh?

Optionally, you could export your diagram and have a hard copy printed out for reference - in case you hate switching windows.

Notes :

A little disappointment when you are writing in language other than Java. Architexa currently supports only Java. And it is a plugin only for Eclipse. You might want to check with them for support for your other favourite IDEs. Mine is eclipse. I am good.
Referenced constants being a part of the sequence diagrams when you select the “Add All” references is strange but probably would come in handy. You can always remove the Constants files off the sequences, if you dont like them.
I see that the time taken to draw the diagram is stunning and the UI too looks light-weight. It takes no lag to drag and drop a class or remove a component off the diagram. I am running a 16GB + quad core though (show off time !!). YMMV
You could export the diagram as an image in your local file or in their community server (publicly available). However, I understand that if you need to have your own private secured server setup in your intranet you might want to go for an upgraded edition.
No support of Generics on the class diagram could be painful for some but I am not deeply concerned there.

Links :

Architexa Project site

CodeMaps Beta site (Interesting)

A lazy developers introduction to Java Concurrency Executors

Sat, 15 Sep 2012 20:31:00 +0000

I would make a fool out of myself if I tell you that util.concurrent APIs kicks cheetah’s ass when the classes are available since 2004. However, there are some cool features which I would like to revisit. Concurrency experts, now is the time for you to close this window. All others, stay tight for the fun ride.

Thou shall not forget your roots

Executor is the root interface with a single execute method. Anything that implements a Runnable interface can passed as a parameter. Silly Executor, however, has no support for Callable though.

Good news : ExecutorService interface, which extends Executor, adds support for Callable. Its implementation class is ThreadPoolExecutor.

I am going to pretend that the ScheduledExecutorService interface and its implementation class ScheduledThreadPoolExecutor does not exist as they just add scheduling capabilities on top of the ExecutorService and ThreadPoolExecutor. But remember this class when the powerful but boring java.util.Timer is not enough and a full blown external scheduler is just too much.

If you are new to concurrency or forgot the difference between Callable and Runnable, you might want to read up a little before reading further. A dummy guide is here

The ExecutorService.submit Facts :

The three submit variants :

1
2
3
4



Future submit(Callable task) <br>
Future submit(Runnable task) <br>
Future submit(Runnable task, T result) 

The submit method of the ExecutorService is overloaded and accepts either a Callable or Runnable.
Since the run method of the Runnable returns void, it is no surprise that Future.get always returns a null when the task is complete.

1

Future<?> 	submit(Runnable task)

The other overloaded submit method that accepts a Runnable and a generic returns whatever you passed in as the second parameter as the result.

1

<T> Future<T>	submit(Runnable task, T result)

In fact, opening up the code (FutureTask), you’ll notice that the RunnableAdapter top level nested class of Executors simply holds the result and returns the same result after the run method is complete.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


static final class RunnableAdapter<T> implements Callable<T> {

 	final Runnable task;
		final T result;

		RunnableAdapter(Runnable task, T result) {
 	this.task = task;
 	this.result = result;
 }

 public T [More ...] call() {
 task.run();
 return result;
 }

}

RunnableAdapter source

In both the cases, if you would like to (you should !) terminate the program instead of your executor thread blocking the program and entering a busy loop, you should call the shutdown method as in

1

executorService.shutdown()

shutDown facts

You could imagine shutdown as a half-closed door of a mall. No new customers will be let in but the existing customers can leave the mall once they are done.

To reiterate,

shutdown is a polite method. It does not actually shut down the tasks in the pool immediately. It just says that no new tasks will be accepted.
Unless you are executing your tasks using invokeAll, you would need to wait for all tasks in progress to complete. This is achieved by calling the awaitTermination method.
(invokeAll and submit examples at the bottom of the post)
Once all the current tasks are done, the executor service shuts down.

If you are in need of an impolite, intruding method which doesn’t care whether the current threads are done with their tasks, then shutdownNow is your guy. However, there’s no guarantee that the method will shutdown the service on the dot but it is the closest you have to immediate shutdown.

On awaitTermination, you could specify the timeout period until which the main thread should wait for the pool threads to complete its tasks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10



ExecutorService executorService=Executors.newFixedThreadPool(10);
…
future = executorService.submit(getInstanceOfCallable(count,sum));
…
executorService.shutdown();

if (executorService.awaitTermination(10, TimeUnit.SECONDS)){
System.out.println("All threads done with their jobs");
}

Executors - the factory guy

The classes above are all awesome. But, say, you wanted to create a single thread executor, you would write something like

1

new ThreadPoolExecutor(1, 1,0L, TimeUnit.MILLISECONDS,new LinkedBlockingQueue<Runnable>());

Compare that to

1

Executors.newSingleThreadExecutor()

So, here you go. Executors is a class with just factory methods for creating various forms of executor service with some commonly used defaults. Note that, other than the awesome factory methods, it doesn’t bring any new features to the table.

It is recommended that you have a quick look at the implementation of the factory methods and check if it suits your needs.

The invokeAll and the submit

The All part of invokeAll method of the ExecutorService gives no surprise. It just says that you need to pass in a Collection of Callables. Again, as expected, the method does not return until all the Threads are done with their tasks. So, for cases when you are interested in the result only after all the jobs are complete, invokeAll is your guy.

On the other hand, the submit method returns immediately after the callable is submitted to the executor service. Unless you are doing nothing at all in your call method of your Callable, your worker threads should ideally be running when the submit method returns.

The following samples might be useful for your reference. The programs just tries to find the sum of all the natural numbers till 100 (Brute force, of course)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


package me.rerun.incubator;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicInteger;

public class ExecutorInvokeAll {

	public void runApp() throws InterruptedException, ExecutionException{

 //variable to store the sum
		AtomicInteger sum=new AtomicInteger();

 //Use our friendly neighbourhood factory method of the Executors.
		ExecutorService executorService=Executors.newFixedThreadPool(10);

		List<Callable<AtomicInteger>> callableList=new ArrayList<Callable<AtomicInteger>>();

		for (int count = 0; count <= 100;count++) {
			callableList.add(getInstanceOfCallable(count,sum));

		}

 //returns only after all tasks are complete
		List<Future<AtomicInteger>> resultFuture = executorService.invokeAll(callableList);

		//Prints 5050 all through
		for (Future<AtomicInteger> future : resultFuture) {
			//Didn't deliberately put a timeout here for the get method. Remember, the invoke All does not return until the task is done.
			System.out.println("Status of future : " + future.isDone() +". Result of future : "+future.get().get());
		}


		executorService.shutdown();

		// You might as well call a resultFuture.get(0).get().get() and that would give you the same 
		//result since all your worker threads hold reference to the same atomicinteger sum.
		System.out.println("Final Sum : "+sum); 

	}
	//Adds count to the sum and returns the reference of the sum as the result
	private Callable<AtomicInteger> getInstanceOfCallable(final int count, final AtomicInteger sum) {

		Callable<AtomicInteger> clientPlanCall=new Callable<AtomicInteger>(){
			public AtomicInteger call() {
				sum.addAndGet(count);
				System.out.println("Intermediate sum :"+sum);
				return sum;
			}
		};

		return clientPlanCall;
	}

	public static void main(String[] args) throws ExecutionException {

		try {
			new ExecutorInvokeAll().runApp();
		} catch (InterruptedException e) {
			e.printStackTrace();
		}
	} 

}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72


package me.rerun.incubator;

import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;

public class ExecutorSubmit {

	public void runApp() throws InterruptedException, ExecutionException{

 //holder for the total sum
		AtomicInteger sum=new AtomicInteger();

 //Use the factory method of Executors
		ExecutorService executorService=Executors.newFixedThreadPool(10);

		Future<AtomicInteger> future = null;

		for (int count = 0; count <= 100; count++) {
			future = executorService.submit(getInstanceOfCallable(count,sum));
			//prints intermediate sum
			try {
				System.out.println("Status of future : " + future.isDone() +". Result of future : "+future.get(1000, TimeUnit.MILLISECONDS).get());
			} catch (TimeoutException e) {
				System.out.println("<IGNORE> Timeout exception for count : "+count);
				//e.printStackTrace();
			}
			//System.out.println("Result of future : "+future.get().get() +".Status of future : " + future.isDone());
		}


		executorService.shutdown();

		if (executorService.awaitTermination(10, TimeUnit.SECONDS)){
			System.out.println("All threads done with their jobs");
		}
		//exec
		System.out.println("Final Sum : "+sum);

	}

 //Adds count to the sum and returns the reference of the sum as the result
	private Callable<AtomicInteger> getInstanceOfCallable(final int count, final AtomicInteger sum) {



		Callable<AtomicInteger> clientPlanCall=new Callable<AtomicInteger>(){
			public AtomicInteger call() {
				sum.addAndGet(count);
				//System.out.println("Intermediate sum :"+sum);
				return sum;
			}
		};

		return clientPlanCall;
	}

	public static void main(String[] args) throws ExecutionException {

		try {
			new ExecutorSubmit().runApp();
		} catch (InterruptedException e) {
			e.printStackTrace();
		}
	} 

}

Further reading :

What does java.util.concurrent.Future hold?

Sat, 15 Sep 2012 20:18:00 +0000

Let’s be sure of what the Future holds

Future, which is a part of the Java concurrency Task execution framework, is the result of your computation as the javadoc claims. And more.

When you are executing a task in a different thread, there is a lot of information you would need from it soon after you submit it to the pool.

What can the Future give?

Result : If you are spawning multiple threads and you are interested in the consolidated/incremental result, obviously you wanted a way to get it. So, here you go. Future.get()
Cancel : Just in case if you spawned 10 threads and you are happy with just one result and wanted to kill the others (or) you wanted to kill that rogue thread (or) you simply loved to power to cancel any thread at will, you have a nice Future.cancel().
Status : Future.isDone() is your way of asking Are you done with your work?

Interesting fact :

The Future.get() method waits for the result forever until the result gets returned. For some cases, you might need this. However, if you don’t want to wait forever, you might choose the overloaded get method

1

V get(long timeout, TimeUnit unit)

Fun Fact :

If you are just spawning a single thread in your Executor and calling the Future.get() (the one which waits forever), you are actually mimicking a Thread.join() call on a spawned single thread by the main thread.

Further reading :

Callable vs Runnable - The brawl of the runners

Sat, 15 Sep 2012 20:15:00 +0000

Runnable : I am the Yoda of multithreading. For generations, I was the only way (other than lang.Thread) that people could do parallel activities in Java. Remember the cool method that I have - run. Too bad, it returns a void though.

Callable : I am a Runnable, just better and cooler. I have this amazing method called call which returns the result of my work. The return type of call is a cool generic and the result could be got using Future. Now, top that Runnable !!

Runnable : Yeah, Yeah, Whatever. Probably, you just came into existence to fill the gaps left in my design. Like, you know, my run method does not throw any Exception which would just mean that if the task wants to throw an exception, it can throw only a Runtime exception.

Callable : I agree. I do stand on your shoulders. And it’s no wonder the call method throws an Exception which enables the task to throw a checked exception back to the caller.

Runnable API

1

void 	run()

Callable API

1

V 	call() throws Exception

More details on the Future is here

Unity in Diversity

FutureTask is an interesting class which can accept both Runnable and Callable in its constructor. Since FutureTask implements Runnable, it can safely pass by the Executor.execute door. Remember, the Executor.execute accepts only a Runnable. I still need to figure out why somebody would want that considering the execute method has a void return type and cannot return results

Shamelessly copied and modified from javadoc

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


Executor executor= …

FutureTask<String> future =

 new FutureTask<String>(new Callable<String>() {
 public String call() {
 … do your thing…

 }});

executor.execute(future);

Further reading :

Filter lines in log file with ERROR

Thu, 30 Aug 2012 18:50:00 +0000

A couple of days ago, a friend of mine was interested to know how to filter a log file (in my case a log4j log file) for ERRORs alone.

An hour of tricks sharing followed and here is the gist of the conversation that you will be interested in.

Find me wherever I am

Listing all the lines in the log file which has occurrences of ERROR is as simple as executing the following command

1

grep "ERROR" <YOUR LOG FILE NAME>

When there is a beginning

And if you are interested in the lines which start with ERROR, then it just involves the usage of the regular expression for “begin”

1

grep "^ERROR" <YOUR LOG FILE NAME>

…there’s an end

A similar trick applies if you are interested in searching lines which has occurrences of “Exception” at the end of the line (specially in case of Java stack traces), then you would use

1

grep "Exception$" <YOUR LOG FILE NAME>

Don’t be so sensitive

Good news is that you need not be case sensitive to grep with regard to search string. You could just use

1
2
3


grep -i "exception$" <YOUR LOG FILE NAME>

(i as in insensitive)

Second thoughts

That said, imagine you are printing the username or a unique trace id of your request in your logs and you would want to filter your log based on that. And suppose this username/trace id is located as the 3rd word in every line. To print the lines which has the third word as say “12345” and your log file delimiter is a space (which is generally the case unless you are logging as xml or json), then you could use

1
2
3
4
5


cut -d " " -f3 <YOUR LOG FILE NAME> | grep "12345"

where
	-d " " says that your delimiter is a space
	-f3	 says that you interested in the 3rd delimiter tokenized string

Moving target

What if you wanted to filter lines on a live log file. Just add some “tail” to it and you are good to go

1

tail -f <YOUR LOG FILE NAME> | grep "ERROR"

Too bad, this does not take care of rolling files.

Less is more

Your production box threw some exception and you are expected to check it out. So, you know that the exception logs exist somewhere towards the end of the file. How do you search for the exception?

Open the file

1

less <YOUR LOG FILE NAME>

(Don’t get me wrong, I love the vi editor but vi tries to load the entire file in its buffer and for large files like our typical log files, performance suffers)

So, yeah less <YOUR FILENAME> will open the file and keep you at the first page. If you need to go to the last page, press

1

Shift + G

Now, you are on the last line.

Sift though the last pages by pressing

1

b (as in back)

and

1

f (as in forward)

Elementary, Dr.Watson

So, if you would like to search for “Exception”, just type

1

/Exception

You would get a

1

"Pattern not found" error

because the default search direction is forward.

To search backward, press

1

Shift + N

Keep pressing Shift + N to repeat search upwards

If you went past the search string upwards and would like to continue search downwards, then just press

1

n

and keep pressing n to repeat search downwards

Case-insensitive less search

If you like to make your search keywords insensitive, just type

1

-i

(remember -i toggles the case insensitivity)

Better to be a head of a dog

tail -f <YOUR LOG FILE> is awesome but if you are analysing the log files inside the less buffer and would like to do a tail of your file, all you need to do is to press

1

Shift + F

Well, that’s all it is for now. I am sure there are many more awesome tricks that you can do in a shell which I don’t know or didn’t mention here. Please share your favourite trick so that I can do some serious show off to my friends.

PS : There is a way to open the file at the last line with less instead of opening the file at the first page and pressing Shift+G. Can’t remember it and can’t find it too.

Find Continuous subarray with maximum sum problem - Kadane's algorithm

Wed, 29 Aug 2012 17:47:00 +0000

Update : Previous version of the code failed for some inputs, as pointed out by @Thinker in the first comment. Changes to the program and the presentation were made. Tested to satisfy most conditions that I could google for.

In a desperate attempt to increase my sad looking stackoverflow reputation, I replied to an old but interesting problem.

The problem goes like this :

Given Random integers in array. Find the greatest sum of a continuous subset

The problem is otherwise known as the Maximum subarray problem and is beaten to death numerous times.

So, instead of just translating a Python program into Java, I decided to pull together some slides which explains how this algorithm works.

And to bore you to death, here is the Java version of the modified algorithm :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


package me.rerun;

public class Kadane {

 public static void main(String[] args) {
 	int[] intArr={3, -1, -1, -1, -1, -1, 2, 0, 0, 0 };
 	//int[] intArr = {-1, 3, -5, 4, 6, -1, 2, -7, 13, -3};
 	//int[] intArr={-6,-2,-3,-4,-1,-5,-5};
 findMaxSubArray(intArr);
 }

 public static void findMaxSubArray(int[] inputArray){

 int maxStartIndex=0;
 int maxEndIndex=0;
 int maxSum = Integer.MIN_VALUE; 

 int cumulativeSum= 0;
 int maxStartIndexUntilNow=0;

 for (int currentIndex = 0; currentIndex < inputArray.length; currentIndex++) {

 int eachArrayItem = inputArray[currentIndex];

 cumulativeSum+=eachArrayItem;

 if(cumulativeSum>maxSum){
 maxSum = cumulativeSum;
 maxStartIndex=maxStartIndexUntilNow;
 maxEndIndex = currentIndex;
 }
 else if (cumulativeSum<0){
 	maxStartIndexUntilNow=currentIndex+1;
 	cumulativeSum=0;
 }
 }

 System.out.println("Max sum : "+maxSum);
 System.out.println("Max start index : "+maxStartIndex);
 System.out.println("Max end index : "+maxEndIndex);
 }

}

Rajesh was kind enough to write the implementation in Scala. Sweet huh?

Heap datastructure in pictures

Sat, 25 Aug 2012 02:58:00 +0000

A binary heap (generally referred as heap) is a rooted left-complete binary tree which has two properties

1
2
3


1)	heap property

2)	shape property

Wow. Now, that is a lot of jargon. Let’s see what each word in the definition means.

What is a tree?

A tree is just a set of nodes connected by edges. Yikes !

Let’s put this in a picture

What is a binary tree?

A binary tree is simply a tree with utmost 2 nodes.

What is a rooted binary tree?

A root is just a node which is specially qualified as root. For clarity, a root is generally drawn as the topmost node when graphically representing the datastructure.

What is a left-complete binary tree?

When all the levels of the binary tree has nodes except the last level, then it is a complete binary tree.

Even if the last one does not have all the nodes, then it should be filled in such a way that the left most nodes are filled before the right ones. ie. There should not be any gaps in the middle. Now, that makes a left-complete binary tree

A legal heap

Now that clears up “A binary heap (generally referred as heap) is a rooted complete binary tree”.

Let’s see what heap and the shape properties mean.

Shape-property :

Well, I cheated. The shape property of a heap simply says that the heap must be a left-complete binary tree. So, dangling nodes can appear only on the left-side. No gaps in between.

Heap property :

The children/child of each node should be lesser than the node itself (in case of max heap)

The children/child of each node should be greater than the node itself (in case of min heap)

Representing Heap in an array :

The easiest and the general way of representing a heap is using an array. You’ll see how the complete/shape property of the heap makes it easier to represent in an array.

Just read the tree left to right (level-order traversal, if you like it that way)

Calculating location of children and parent in array :

The root is the first item of the array

Indices of Children can be calculated, known a parent index, using the simple calculation :

1
2
3
4


Child 1 = 2i +1
Child 2 = 2i +2

where i is the index of the parent

Again, the index of the parent can be calculated by reversing the formula

1
2
3


Index of parent given a child index is

Parent = i/2 (round the result to the lower bound)

So, considering the image above, the children of 5 at index 2 are items located in indices 5 and 6 which are 2 and 3

Fun facts :

1) A sorted array is a min heap

2) Unlike a Stack, a JVM heap has nothing to do with the heap data structure. Though rumor has it that the usage of name “heap” for its memory area comes from one of the early languages using Heap datastructure to store objects in memory at runtime.

1 million nodes = 20 levels :

Since Heap is a complete binary tree (remember : no spaces in between), the maximum number of levels is easy to calculate.

1
2
3
4
5
6
7


First level 	=	1

Second level 	=	2 	=	2^1

Third level 	=	4 	= 	2^2

Fourth level 	= 	8	=	2^3

…

eg. until 4th level, there are 15 nodes.

1

total number of nodes until 4th level = 2^4 -1.

In other words, the height of the heap = log base 2 (n) (round to the upper bound)

Applications of Heaps:

A heap could be used to implement a priority queue, which is a queue that orders entities not a on first-come first-serve basis, but on a priority basis. So, since the heap has the minimum or the maximum key at the top, the next “priority” item is at the top with a removal time of 0(1).

Also, Heaps are used in the Heap Sorting algorithm which has the worst case performance of 0 (n log n). No wonder Heap sort is the preferred sorting algorithm for all the mission critical applications.

Quicksort - the easy way

Mon, 20 Aug 2012 19:06:00 +0000

Note : For 3-way partition and Dual Pivot Quicksort (with programs to trace the sort process), please refer to this recent post.

Quick sort is the fastest known non-hybrid comparision sort for arrays which has no knowledge of the data beforehand. To top it, it could be done in-place for arrays. For Linked Lists, Merge Sort might be a better option. Also since Quicksort improves its performance by using random numbers, it also becomes an example of a breed of algorithms named “Randomized Algorithms”.

The naive version of the Algorithm goes like this :

1
2
3
4
5
6
7
8
9


1)	Start with list I of n items
2)	Choose pivot v from I
3)	Partition I into 2 unsorted lists I1 and I2 such that
		I1 : all keys smaller than pivot
		I2 : all keys larger than pivot
		Items same as pivot goes to either list
4)	Sort I1 recursively, yielding sorted list S1
5)	Sort I2 recursively, yielding sorted list S2
6)	Concatenate S1,v,S2 yielding sorted list S

Pseudocode in a Video

How Long does Quick sort take ?

Interestingly, Quicksort has a worst case running time of 0(n^2). However, with careful implementations, n^2 never happens and it almost always runs in 0(n log n). As compared to Heapsort, whose best and worst case is guaranteed to be n log n, Quicksort beats Heapsort in clock time for two reasons :

The constant factor surrounding the 0(nlogn) is lesser than Heapsort due to the very thin core logic. Heap sort’s detailed comparison and exchange mechanism inside loops increases the constant factor.
Quicksort could leverage on the locality of reference (and therefore, memory access) advantages provided by the hardware and the operating system

How n log n?

If you choose the pivot right, then, on an average, you get a 1/4, 3/4 split which gives an average running time of 0(n log n)

Source : Goodrich and Tamassia

Pivot Choosing strategy :

There are many methods to choose a pivot

1
2
3
4
5


1)	First Item
2)	Last item
3)	Median
4)	Median of k (generally of 3)
5)	Random item

Choosing a random key as the pivot is the preferred method because of the guaranteed worst case possibility with the other selection methods with following kinds of sequences :

1
2
3
4
5


1)	Presorted list
		when chosing first item as your pivot in case of natural order
		 		 last item as your pivot in case of reverse order
2)	Repeated elements
3)	Unimodal sequences* (when median is chosen as pivot)

Bottom line : When you choose a pivot in such a way that one portion of your split (either I1 or I2) becomes empty AND if you are doing it all through the list, then you are essentially doing the split N times.

Quicksort in Java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92


package me.rerun;

import java.util.Arrays;

public class QuickSort {

	public static void quickSort (Comparable comparableArray[], int lowIndex, int highIndex){

		//at least one item must exist in the array 
		if (lowIndex>=highIndex){
			return;
		}

		int pivotIndex=getMedianIndexAsPivotIndex(lowIndex, highIndex);
		//1) Choose pivot from the sublist
		Comparable pivot=comparableArray[pivotIndex];
		System.out.println("Pivot : "+pivot);
		//2) Swap the pivot to the last item in the array
		swapItemsWithIndices(comparableArray, pivotIndex, highIndex); 	

		/*	
		 	Get the border indices sandwiching the unsorted items alone (ignore pivot (now, in the highIndex))
			set 'i' to point to the item before the first Index
			set 'j' to point to the item before pivot

			Notice that this way, the following invariant gets maintained 
			all through the sorting procedure

			a. All items left of Index 'i' have a value <=pivot
			b. All items right of Index 'j' have a value >=pivot
		*/

		int i=lowIndex-1;
		int j=highIndex;

		do{ //Notice the <j (pivot item is ignored). We stop when both the counters cross

			//compareTo will return 0 when it reaches the pivot - will exit loop
			do {i++;} while (comparableArray[i].compareTo(pivot)<0);
			//we dont have the protection as the previous loop. 
			//So, add extra condition to prevent 'j' from overflowing outside the current sub array
			do {j--;} while (comparableArray[j].compareTo(pivot)>0 &&(j>lowIndex));

			if (i<j){
				swapItemsWithIndices(comparableArray, i, j);
			}
			System.out.println("I :"+i + " J :"+j);
		} while (i<j);

		swapItemsWithIndices(comparableArray, highIndex, i);//bring pivot to i's position	
		System.out.println("Comparable array : "+Arrays.asList(comparableArray));

		//the big subarray is partially sorted (agrees to invariant). Let's recurse and bring in more hands

		quickSort(comparableArray, lowIndex, i-1); //sort subarray between low index and one before the pivot
		quickSort(comparableArray, i+1, highIndex); //sort subarray between low index and one before the pivot
	}


	//... since swapping with array is the easiest way to swap two objects
	private static void swapItemsWithIndices(Comparable[] comparableArray, int firstItem, int secondItem) {
		System.out.println("Swapping "+comparableArray[firstItem] +" and "+comparableArray[secondItem]);
		final Comparable tempItem=comparableArray[firstItem];
		comparableArray[firstItem]=comparableArray[secondItem];
		comparableArray[secondItem]=tempItem;
		System.out.println("After swap array : "+Arrays.asList(comparableArray));
	}


	//Variation 1 - chose median as pivot 
	private static int getMedianIndexAsPivotIndex(int lowIndex, int highIndex) {
		return lowIndex+((highIndex-lowIndex)/2);
	}




	public static void main(String[] args) {

		//Integer[] unsortedArray=new Integer[]{1,32,121,1424,2,1214,121214,3535,754,343};
		//Integer[] unsortedArray=new Integer[]{4,4,8,0,8,9,7,3,7,6}; 
		Integer[] unsortedArray=new Integer[]{5,5,5,5,5,5,5,5,5,5};

		long startTime = System.nanoTime();
		System.out.println("Original array : "+Arrays.asList(unsortedArray));
		quickSort(unsortedArray, 0, unsortedArray.length-1);

		System.out.println("Sorted array : "+Arrays.asList(unsortedArray));
		System.out.println(System.nanoTime()-startTime);
	}

}

References

Testing automation with Selenium

Mon, 30 Jul 2012 06:31:00 +0000

Never send a human to do a machine’s job

Testing a CRM application which I am part of became very difficult over the recent months. There were simply too many usecases. Despite unit testing on the backend services (SOA), bugs crept in on a regular basis considering a lot of logic is on the javascript and other layers of the application.

This is Part 1 of this series explaining how the following problems that I faced during testing were addressed :

There were too many combinations of test scenarios
Application logic was spread over javascript, front-end, service and database tiers
Existing testcases for services/portals are not exhaustive
Our Unit testcases verified against static data
Enabling/disabling/visibility/invisibility of fields on the front end could not be verified by plain Unit tests
Incremental failure built inside JUnit testcases is not always useful. A report of all the issues would be nice.

The itch and the scratch

The Tool was built upon Selenium, Spring and Velocity.

The following features were built into the tool

Selenium

As part of the web automation, a series of utility methods was written/stolen from various sources. I am sure you’ll find the SeleniumUtils class interesting and useful.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238


import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.commons.collections.ListUtils;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.StringUtils;
import org.apache.log4j.Logger;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.TimeoutException;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.Select;
import org.openqa.selenium.support.ui.Wait;
import org.openqa.selenium.support.ui.WebDriverWait;

/**
 * @author Arun Manivannan
 *
 */
public class SeleniumUtils {

	private static Logger logger=Logger.getLogger(SeleniumUtils.class);

	public static void populateDropDown(WebDriver driver, By by, String value) {

		Select select = new Select(driver.findElement(by));
		select.selectByValue(value);

	}

	public static ExpectedCondition<WebElement> visibilityOfElementLocated(final By locator) {
		return new ExpectedCondition<WebElement>() {
			public WebElement apply(WebDriver driver) {
				WebElement toReturn = driver.findElement(locator);
				if (toReturn.isDisplayed()) {
					return toReturn;
				}
				return null;
			}
		};
	}

	public static void populateTextBox(WebDriver driver, By by, String value) {
		WebElement inputElement = driver.findElement(by);
		if ("".equals(value)) {
			inputElement.clear();
		} else {
			inputElement.sendKeys(value);
		}
	}

	public static void checkRadio(WebDriver driver, By by) {
		WebElement inputElement = driver.findElement(by);
		inputElement.click();
	}


	public static void goToTab(WebDriver driver, By by) {
		waitUntilClickable(driver, by);
		driver.findElement(by).click();
	}

	public static WebElement waitForVisibility(WebDriver driver, By by) {
		return waitForVisibility(driver, by, 45);
	}

	public static WebElement waitUntilClickable(WebDriver driver, By by){
		WebDriverWait wait = new WebDriverWait(driver, 45);
		WebElement element = wait.until(ExpectedConditions.elementToBeClickable(by));
		return element;
	}

	public static WebElement waitForVisibility(WebDriver driver, By by, int waitTime) {
		Wait<WebDriver> wait = new WebDriverWait(driver, waitTime);
		WebElement divElement = wait.until(visibilityOfElementLocated(by));
		return divElement;
	}


	public static WebElement switchToNewWindow(WebDriver driver, String iframeId) {
		driver.switchTo().frame(iframeId);
		WebElement window = driver.switchTo().activeElement();
		return window;
	}

	public static String getTextFromId(WebDriver driver, WebElement navigator, String id) {

		//String text = navigator.findElement(By.id(id)).getText();
		System.out.print(id); 
		final WebElement element = findElement(driver, By.id(id),3);

		ElementType currentElementType=getCurrentElement(element);
		String text=StringUtils.EMPTY;
		switch (currentElementType) {
		case INPUT:
			text=element.getAttribute("value");
			break;
		case DIV:
			text=element.getText().trim();
			break;
		case TEXTAREA:
			//text=element.getText();
			text = (String)((JavascriptExecutor) driver).executeScript("return document.getElementById('"+id+"').value","");
			break;
		case SELECT:
			//Select select=new Select(element);
			//text=select.getFirstSelectedOption().getAttribute("value");
			text = (String)((JavascriptExecutor) driver).executeScript("return document.getElementById('"+id+"').value","");
			break;	
		default:
			break;
		}

		System.out.println(" : Text value : "+text);
		return text;

	}

	private static ElementType getCurrentElement(WebElement element) {
		String tagName=element.getTagName();
		ElementType elementType=null;
		if (StringUtils.equalsIgnoreCase(tagName, Constants.INPUT)){
			elementType=ElementType.INPUT;
		}
		else if (StringUtils.equalsIgnoreCase(tagName, Constants.SELECT)){
			elementType=ElementType.SELECT;
		} 
		else if (StringUtils.equalsIgnoreCase(tagName, Constants.DIV)){
			elementType=ElementType.DIV;
		} 
		else if (StringUtils.equalsIgnoreCase(tagName, Constants.TEXTAREA)){
			elementType=ElementType.TEXTAREA;
		} 
		else{
			logger.error("Unhandled element type : "+element.getTagName());
		}

		return elementType;
	}

	public static WebElement findElement(WebDriver driver, By by, int timeoutInSeconds){
	 try {
			WebDriverWait wait = new WebDriverWait(driver, timeoutInSeconds);
			wait.until(ExpectedConditions.presenceOfElementLocated(by));
		} catch (TimeoutException e) {
			logger.error(e.getMessage(),e);
			e.printStackTrace();
			return null;
		} 
	 return driver.findElement(by);
	}

	public static boolean isEnabled(WebDriver driver, String eachField) {
		return (driver.findElement(By.id(eachField)).isEnabled());
	}

	public static boolean isDisabled(WebDriver driver, String eachField) {
		return (!driver.findElement(By.id(eachField)).isEnabled());
	}

	public static boolean isVisible(WebDriver driver, String eachField) {
		return (driver.findElement(By.id(eachField)).isDisplayed());
	}

	public static boolean isInvisible(WebDriver driver, String eachField) {
		return (!driver.findElement(By.id(eachField)).isDisplayed());
	}


	public static List<Message> checkEnabledFields(WebDriver driver, List<String> enabledFields) {
		System.out.println("Check enabled fields");
		if (enabledFields==null) return ListUtils.EMPTY_LIST;
		List<Message> messages=new ArrayList<Message>();
		boolean result=false;
		for (String eachField: enabledFields) {
			result=isEnabled(driver, eachField);
			messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.ENABLED, Constants.DISABLED, eachField));
		}
		return messages;
	}


	public static List<Message> checkDisabledFields(WebDriver driver, List<String> fields) {
		if (fields==null) return ListUtils.EMPTY_LIST;
		List<Message> messages=new ArrayList<Message>();
		boolean result=false;
		for (String eachField: fields) {
			result=isDisabled(driver, eachField);
			messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.DISABLED, Constants.ENABLED, eachField));
		}
		return messages;

	}

	public static List<Message> checkVisibleFields(WebDriver driver, List<String> fields) {
		if (fields==null) return ListUtils.EMPTY_LIST;
		List<Message> messages=new ArrayList<Message>();
		boolean result=false;
		for (String eachField: fields) {
			result=isVisible(driver, eachField);
			messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.VISIBLE, Constants.INVISIBLE, eachField));
		}
		return messages;

	}

	public static List<Message> checkInvisibleFields(WebDriver driver, List<String> fields) {
		if (fields==null) return ListUtils.EMPTY_LIST;
		List<Message> messages=new ArrayList<Message>();
		boolean result=false;
		for (String eachField: fields) {
			result=isInvisible(driver, eachField);
			messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.INVISIBLE, Constants.VISIBLE, eachField));
		}
		return messages;

	}

	public static String captureScreenshot(WebDriver driver, String folder, String fileName) {

		File screenshotFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
		File targetFile=new File(folder, fileName+".png");
		try {
			FileUtils.copyFile(screenshotFile,targetFile );
		} catch (IOException e) {
			logger.error ("Error while writing file ",e);
		}

		return targetFile.getAbsolutePath();
	}
}