[{"content":" “Once upon a time, there was a dog named Cookie. She loved to play fetch with her owner. One day, they went for a walk in the park and found a big ball. Cookie picked it up with her mouth and brought it back to her owner\u0026hellip;”\nThat paragraph above? Generated by a 91-million-parameter transformer I trained from scratch over a Saturday afternoon.\nThe dog Cookie is real. She\u0026rsquo;s mine. She is currently asleep on the couch, blissfully unaware that there\u0026rsquo;s now a small language model wandering the internet that can write fan-fiction about her.\nTotal compute cost: about $5 of rented H100 time on runpod.io across three training runs. The headline is generous; the actual final run cost $1.50. I\u0026rsquo;m calling it $5 because \u0026ldquo;$1.50\u0026rdquo; makes a less catchy post.\nThis is a log of training a language model end-to-end.\nThe Roadmap: What I built (and what it actually means at this scale) The journey from gibberish to coherence — three training runs, what worked, what didn\u0026rsquo;t The bugs What I built A 91M-parameter GPT-style transformer, trained from scratch on the TinyStories corpus — about 470 million tokens of synthetic children\u0026rsquo;s stories with a deliberately-constrained vocabulary (~3000 base words).\nArchitecture:\n91 million parameters (including embeddings) 12 layers 12 attention heads with RoPE Custom 4,096-word BPE tokenizer (since the corpus just had ~3000 words) 512 token context window This is built on the Hugging Face stack — transformers, tokenizers, datasets, safetensors . This is the same ecosystem teams use in practice for running real-world training pipelines, just at the smallest possible scale.\nPerspective on Scale Model Parameters Training Tokens Estimated Cost GPT-2 base (2019) 124M ~few billions undisclosed Nanochat ~560M ~11.2B ~$100 This model 91M ~930M $1.50 GPT-4-class (rumored) ~1.7T (est.) ~13T (est.) $100M+ (est.) The journey: from gibberish to coherence Round 1: Smoke-test on my laptop Data: WikiText-2, a 2M-token Wikipedia corpus. Model: 28M-param transformer (8 layers × 512 embeddings, 8 heads). Hardware: Apple Silicon MPS, ~3 minutes wall-clock. Result: Final perplexity 194. Total gibberish. To be fair, this wasn\u0026rsquo;t meant to be a real training run but meant to just be a validation exercise to confirm the pipeline ran end-to-end without crashing and that loss was progressively getting lower.\nWhile doing this evaluation, it struck me that Wikipedia is the wrong corpus for a 28M model.\nWikiText-2 is tiny (~2M tokens). Chinchilla rule of thumb states that I need to have ~560M tokens (20x) of the model param size (28M). So, the corpus clearly doesn\u0026rsquo;t have enough data to fill the model. WikiText-2 data is complex. It has dates, technical jargons and a wide range of low-frequency vocabulary cutting across several domains. Even if it had 560M tokens, a 28M model would lack the capacity to model that distribution. Time to move on to a different corpus. A bit of research revealed that TinyStories was a great fit: ~470M tokens of synthetic, simple-vocabulary (3000 words) children\u0026rsquo;s stories that a tiny model can actually saturate.\nRound 2: The first real attempt Switched the corpus to TinyStories (~470M tokens), kept the model at 28M params and trained for 15,000 steps with bf16 autocast and torch.compile.\nHardware: RunPod H100 SXM 80GB. Wall-clock: ~6 minutes. Cost: ~$0.30. Result: Perplexity 4.0. 1 2 3 4 Step : 0/15000 | val: 8.40 | ppl: 4054 Step : 1000/15000 | val: 2.16 | ppl: 8.6 Step : 5000/15000 | val: 1.61 | ppl: 5.0 Step : 14999/15000 | val: 1.40 | ppl: 4.0 From literally random to decent prediction in 6 minutes of GPU time.\nBut when I sampled:\n\u0026ldquo;Once upon a time, there was a dog named Cookie.. Igaintto-doo. He is like it was gone. It was a very sad\u0026hellip;\u0026rdquo;\nIt spoke \u0026ldquo;English words\u0026rdquo; interleaved with baby talk (let\u0026rsquo;s call it that - \u0026ldquo;Igaintto-doo\u0026rdquo;, \u0026ldquo;Annabutterram\u0026rdquo;, \u0026ldquo;girlsmagulled\u0026rdquo;). It also suffered from mode collapse where every other prompt got stuck in loops like \u0026ldquo;Once upon a lotion. Once upon a lotion.\u0026rdquo;\nIt took a few minutes to figure out that there were two separate problems:\nThe model was undertrained. Look at the last third of the loss curve, the number of steps were clearly inadequate. The val loss kept dropping but I stopped the training.\nThe sampling was uncontrolled. I understood that the \u0026ldquo;Once upon a lotion\u0026rdquo; loops aren\u0026rsquo;t a model bug, they\u0026rsquo;re a decoder bug. Apparently, the model.generate() settings will happily emit the same high-probability sequence unless I tune 2 parameters:\nrepetition_penalty : lowers the probability of tokens that have already appeared and top_p : keeps generation diverse without going into low-probability nonsense (aka nucleus sampling) Both were fixed in Round 3.\nRound 3: 50K steps (~$1.00) Bumped NUM_STEPS = 15_000 to NUM_STEPS = 50_000. Also fixed the sampling: added repetition_penalty=1.3 and top_p=0.95 to workaround the mode collapse.\nHardware: RunPod H100 SXM 80GB. Wall-clock: ~20 minutes. Cost: ~$1.00. Result: Perplexity 3.62. The \u0026ldquo;lotion\u0026rdquo; loops were gone. Real names appeared. But the output was still largely incoherent with sentences being \u0026ldquo;alright\u0026rdquo; but the paragraphs were unrelated and lacked continuity.\nA look at the training and validation loss curves told me this was the best I could do with a 28M model - both the training loss and the validation curves flattened. Learnt that the term for this was that I \u0026ldquo;hit a capacity ceiling\u0026rdquo; . Apparently, the model is no longer learning because it has run out of parameters to hold, not because it has run out of data to learn from.\nThe solution was not to add more steps but to add more data to the model. Then I thought I\u0026rsquo;ll add more capacity to the model to learn from larger amounts of data.\nRound 4: 91M params (the final run) Bumped the architecture: 12 layers (vs 8), 768 hidden dim (vs 512), 12 attention heads (vs 8). Total parameters: 28M → 91M. The idea behind using the same corpus but fewer training steps (30K vs 50K) was that each step at 91M params is ~3× more compute than at 28M, so 30K steps here is comparable total compute to 50K at the smaller size.\nHardware: RunPod H100 SXM 80GB. Trained tokens: 30k steps * 64 batches * 512 context/seq len = 983M Wall-clock: ~20 minutes. Cost: ~$1.50. Result: Perplexity ~2.7. Output became a coherent narrative:\n\u0026ldquo;Once upon a time, there was a dog named Cookie. She loved to play fetch with her owner\u0026hellip; His owner threw the ball and Cookie ran after it as fast as she could. But when she came back, she had lost the ball!\u0026hellip;\u0026rdquo;\nThis model produced stories. Character continuity was there and the sentences were related to each other in a coherent ashion. It even ended sentences with \u0026ldquo;The end.\u0026rdquo; and starts a new story.\nThe pronouns still drift — Cookie is sometimes \u0026ldquo;she\u0026rdquo; and sometimes \u0026ldquo;he\u0026rdquo; but I believe this would be fixed by scale.\nThis run was truly a qualitative jump. Same architecture family, same corpus, just more data and capacity of the model.\nThe bugs The RoPE buffer NaN bug: I was excited to run inference only to realise that it crashed, every time. 1 2 3 4 5 6 7 File \u0026#34;.../torch/utils/_contextlib.py\u0026#34;, line 124, in decorate_context return func(*args, **kwargs) File \u0026#34;.../transformers/generation/utils.py\u0026#34;, line 2560, in generate result = decoding_method( File \u0026#34;.../transformers/generation/utils.py\u0026#34;, line 2808, in _sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either `inf`, `nan` or element \u0026lt; 0 A bit of research said that the precomputed RoPE cos/sin buffers had NaN entries. The problem is that this NaN will propagate through every attention layer that touches that position. The root cause is that I had the cos and the sin buffers as persistent=False, which was the recommended approach for storing recomputable tables.\n1 2 self.register_buffer(\u0026#39;cos\u0026#39;, None, persistent=False) self.register_buffer(\u0026#39;sin\u0026#39;, None, persistent=False) Learnt that Hugging Face\u0026rsquo;s from_pretrained uses init_empty_weights() during loading, which allocates memory without initializing it. Since RoPE\u0026rsquo;s cos/sin map was registered as non-persistent buffers, HF didn\u0026rsquo;t save them in the checkpoint (weights are persistent and stored in checkpoint). Since the cos/sin values of RoPE are deterministic, the fix was to manually call model._init_rope(seq_len, head_dim) right after from_pretrained.\nMPS scaled_dot_product_attention: PyTorch\u0026rsquo;s F.scaled_dot_product_attention(..., is_causal=True) on Apple\u0026rsquo;s MPS produces NaN under certain conditions. Understood that it was a known PyTorch issue. The fix was to wrap inference in with sdpa_kernel(SDPBackend.MATH): to force the unoptimized math kernel. I didn\u0026rsquo;t particularly dig deep into this patch. I am truly amazed at how easily accessible training an LLM has become now. Not so long ago, this kind of project would have meant a research budget.\nThe full code is available at github.com/arunma/learn-you-an-hf-llm. Cookie has had her dinner and remains unimpressed.\n","date":"2026-05-11T11:44:30Z","permalink":"/small-dog-small-language-model-training-a-transformer-for-5/","title":"Small Dog, Small Language Model: Training a Transformer for $5"},{"content":"Typically, counting the number of unique values in a dataset is an easy problem. You maintain a BST or HashSet to remove duplicates as they come in while keeping track of the count of inserted unique values. It takes O(n) space, which is acceptable for smaller datasets. However, for large datasets, storage becomes quickly expensive, especially if you would like to keep track of counts in real-time.\nLet\u0026rsquo;s say that you are working on a banking application that processes ~1 billion transactions every day. You would like to know the number of unique customer account numbers on which transactions were made but you have limited memory and computational resources. Let\u0026rsquo;s do a back-of-the-envelope calculation for the memory required if we were to go the route of a HashSet.\n1 2 3 4 5 6 7 a. Assume 10% of the transactions originate from unique account numbers = 10 million b. Assume Customer account numbers are 10 digit numbers. c. Each account number could be represented by a long datatype. --- 10 million account numbers * 8 bytes = 80 MB in memory Note that the earlier calculation does not include the storage required for the hashes in the set or the pointer to the next link in the chain (due to collisions).\nOne other alternative is to persist all the account numbers to a file or a database and run a map-reduce/distinct operation at a regular frequency. However, it requires a fair amount of work and also the output is not real-time.\nThis sounds like an impossible task but you\u0026rsquo;ll see shortly how we could leverage an algorithm called HyperLogLog to get a \u0026ldquo;close to actual\u0026rdquo; estimate of unique values in a dataset by using an incredibly small fraction of storage. For the case of account numbers that we discussed above, if you can live with a 1% error rate, you\u0026rsquo;ll just need 1 MB with HLL.\nWhat is HyperLogLog? HyperLogLog is an algorithm that allows for a highly accurate \u0026ldquo;estimation\u0026rdquo; of the number of distinct elements in a dataset. More formally, HyperLogLog is a probabilistic algorithm used for estimating the cardinality in a set.\nSeveral popular databases and frameworks use HLL underneath to provide quick estimates of distinct values - Apache Spark, Cassandra, MongoDB, Redis, Druid, etc.\nHow does it work? Let us consider the case where the account number just has one digit. There are 10 possible account numbers: 0 to 9 and each account number has an equal probability of occurrence. Therefore, the probability of seeing the account number 0 in one of the transactions is 1/10. Let\u0026rsquo;s now consider the case where the account number has 2 digits. Account numbers can be between 00 and 99. Again, the possibility of us seeing 00 is 1/100. To generalize this, the possibility of seeing k number of leading zeros in an account number of any size is :\nApplying this, we can \u0026ldquo;kinda\u0026rdquo; conclude that if we saw an account number with 2 leading zeros (00) in our transactions, we have seen 100 (k=2) account numbers. (Hang in there, for now.)\nHyperLogLog follows exactly the same approach of looking at leading zeros, except that, in order to represent all kinds of data and not just numbers, it uses the binary representation of a hashed value of an element instead of using the actual data. (Some implementations use trailing zeros but as far as the hashes are uniformly distributed, it doesn\u0026rsquo;t affect the accuracy.)\nSince the data is binary, it is easy to see that the occurrence of 0 in a single-digit binary is 1/2, 00 in a two-digit binary is1/4 , and so on. So, our estimation formula changes as:\nSo, if we have seen an account number with 2-leading zero in its binary representation, we can \u0026ldquo;kinda\u0026rdquo; conclude that we have seen at least 4 unique account numbers.\nVery incorrect estimates You\u0026rsquo;ll quickly realize that we could estimate only in exponents of 2, which results in overestimation or underestimation. Also, imagine the situation when your first account number has 7 leading zeros. The HLL will immediately report that we have seen 127 unique accounts when we just saw 1.\nSolution In order to circumvent this problem Flajolet and Martin, the authors of the algorithm, proposed four major changes to their algorithm:\nUse multiple hash functions and average the estimates from each bucket of the hash function. Instead of using arithmetic mean to find the average, use harmonic mean. Use different bias corrections based on the total number of buckets used and Use correction for large and small estimates. Through these changes, the error rate of the algorithm is brought down to\nWhat this would also mean is that we could specify an error rate while instantiating an HLL and the buckets/registers (m) size would be adjusted to meet that error rate. For all practical purposes, the error rate is significantly small.\nNow, let\u0026rsquo;s go over each of the improvements one by one:\nMultiple buckets We could use a handful of hash functions to hash the input element. We could then compute the count of leading zeros in the hash binary. This count is then stored in the buckets that the hash belongs to (which could be derived through a simple modulus operator).\nUnfortunately, computing multiple hashes is expensive for large datasets and therefore, the authors decided to use the first few bits of the binary representation of the hash to decide the bucket. This was very cheap to compute.\nIn the visual above, the 8-bit binary representation of the hash has two parts,\nbucket section (aka precision): the bits that decide which bucket does this hash output affect the estimates of. estimate section: the set of bits for computing the leading zeros. In this case, the bucket is 1 and the estimate is 1 due to the presence of 1 leading zero.\nUse harmonic mean The Harmonic mean is the reciprocal of the average of the reciprocals of the numbers. It is given by:\nSo, for our 8 buckets with estimations 1, 1, 2, 1 , we do:\nUse correction factor based on bucket size Ours is a contrived example and therefore the number of buckets that we have is 8. Typically, the bucket sizes are minimally 16. The bias correction (alpha) mentioned in the paper are :\nDuring our final estimation, we multiply the raw estimates (in our case 2.2857) by this alpha correction factor.\nUse correction factor based on estimate size As a final step, one round of correction is also done based on how large or small the estimate is, in comparison to the bucket size.\nThere are three ranges:\n(I have absolutely no idea how these numbers are derived. These are just lifted from Page 14 of the original paper)\nFinal estimate The formula to derive the final estimate from the raw estimate is:\nfinal_estimate = range_size_correction(alpha * raw_estimate)\nThat\u0026rsquo;s it! Now that we have all the theory out of the way, let\u0026rsquo;s implement the algorithm in Rust.\nImplementation in Rust Instantiation There\u0026rsquo;s just one input parameter required for HLL, which is your expected error rate. As we saw before, the bucket size is adjusted based on the error rate.\nThe number of buckets is calculated as:\nA recommended set of bits (bucket section/precision) used to identify the bucket is log2(m)\nPutting them all together in our new function:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 pub struct HyperLogLog\u0026lt;T: Hash + Eq\u0026gt; { alpha: f64, precision: usize, num_buckets_m: usize, buckets: Vec\u0026lt;u8\u0026gt;, hasher: SipHasher24, _p: PhantomData\u0026lt;T\u0026gt;, } impl\u0026lt;T: Hash + Eq\u0026gt; HyperLogLog\u0026lt;T\u0026gt; { pub fn new(error_rate: f64) -\u0026gt; Result\u0026lt;Self\u0026gt; { validate(error_rate)?; let precision = (1.04 / error_rate).powi(2).log2().ceil() as usize; // log2(m) let num_buckets_m = 1 \u0026lt;\u0026lt; precision; // 2^precision let alpha = calculate_alpha(num_buckets_m)?; //Instantiate our single hashing function let random_key = generate_random_seed(); let hasher = create_hasher_with_key(random_key); let hll = HyperLogLog { alpha, precision, num_buckets_m, buckets: vec![0; num_buckets_m], hasher, _p: PhantomData, }; Ok(hll) } The alpha function just encodes the table that we saw previously:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 fn calculate_alpha(num_buckets_m: usize) -\u0026gt; Result\u0026lt;f64\u0026gt; { if num_buckets_m \u0026lt; 16 { return Err(InputError(format!( \u0026#34;Invalid number of buckets calculated {num_buckets_m}\u0026#34; ))); } let alpha = match num_buckets_m { 16 =\u0026gt; 0.673, 32 =\u0026gt; 0.697, 64 =\u0026gt; 0.709, _ =\u0026gt; 0.7213 / (1.0 + 1.079 / num_buckets_m as f64), }; Ok(alpha) } 1 2 3 4 5 6 7 8 9 10 11 12 13 //Hash helper functions /// Generates a random 64-bit key used for hashing pub fn generate_random_seed() -\u0026gt; [u8; 16] { let mut seed = [0u8; 32]; getrandom::getrandom(\u0026amp;mut seed).unwrap(); seed[0..16].try_into().unwrap() } /// Creates a `SipHasher24` hasher with a given 64-bit key pub fn create_hasher_with_key(key: [u8; 16]) -\u0026gt; SipHasher24 { SipHasher24::new_with_key(\u0026amp;key) } Building the APIs Insert In the insertion function, as we saw earlier,\nThe precision and the estimate bits as separated. The precision bits are used to identify the bucket that we are going to update the new estimate. The estimate bits are examined for the leading zeros. The previous estimate in the bucket is examined and updated with the new estimate, if necessary. Separate precision and estimation bits\nCompare previous and current estimate and update bucket if necessary\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 pub fn insert\u0026lt;Q\u0026gt;(\u0026amp;mut self, item: \u0026amp;Q) where T: Borrow\u0026lt;Q\u0026gt;, Q: Hash + Eq + ?Sized, { let mut hasher = self.hasher; item.hash(\u0026amp;mut hasher); let hash = hasher.finish(); let bucket_index = (hash \u0026gt;\u0026gt; (64 - self.precision)) as usize; // Take the first precision bits let estimate_bits_mask = (1 \u0026lt;\u0026lt; (64 - self.precision)) - 1; // Create a mask for excluding the precision bits let estimate_bits = hash \u0026amp; estimate_bits_mask; let trailing_zeros = estimate_bits.leading_zeros() as u8 + 1; // Count the number of leading zeros let prev_estimate = self.buckets[bucket_index]; // Get the previous estimate self.buckets[bucket_index] = max(prev_estimate, trailing_zeros) // Update the estimate if necessary } Estimate For the estimate, we would\ncalculate the raw estimate by using harmonic mean. multiply the estimate with the previously calculated alpha. arrive at final estimate after correcting for size of estimates in comparison to bucket size. The correct_for_estimate_size function just encodes our previous table:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 fn correct_for_estimate_size(\u0026amp;self, raw_estimate: f64, m: f64, empty_registers: usize) -\u0026gt; f64 { match raw_estimate { //Small range correction sr if raw_estimate \u0026lt;= 2.5_f64 * m =\u0026gt; { if empty_registers \u0026gt; 0 { m * (m / empty_registers as f64).ln() } else { sr } } //Medium range correction mr if raw_estimate \u0026lt;= (TWO_POW_32 / 30.0_f64) =\u0026gt; mr, //Large range correction lr =\u0026gt; -TWO_POW_32 * (1.0 - lr / TWO_POW_32).ln(), } } The estimate function calculates the harmonic mean and factors in alpha and estimate size correction.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 pub fn estimate(\u0026amp;self) -\u0026gt; usize { let m = self.num_buckets_m as f64; let mut sum = 0.0_f64; let mut empty_registers = 0; for \u0026amp;v in self.buckets.iter() { sum += 2.0_f64.powf(-(v as f64)); if v == 0 { empty_registers += 1; } } let raw_estimate = self.alpha * m.powi(2) / sum; let estimate = self.correct_for_estimate_size(raw_estimate, m, empty_registers); estimate as usize } Code The complete code is here.\n","date":"2023-05-01T12:12:43Z","permalink":"/build-your-own-hyperloglog/","title":"What is HyperLogLog and how to build yours in Rust"},{"content":"While discussing about Counting Bloom Filter, we came across this function called estimated_count which attempts to calculate the number of times an element was present in the bloom filter. During the discussion, I also said that the Counting Bloom Filter is not the best datastructure for calculating the count of an item.\nCountMinSketch is a good data structure to do that. And we\u0026rsquo;ll see why.\nImagine you have created a Twitter-alternative and you would like to keep track of the frequency of all the hashtags so that you can derive the \u0026ldquo;heavy-hitters\u0026rdquo; or the top trending hashtags. Or you are building an e-commerce or ride-hailing app and would like to keep track of the top products or destinations within an hour or a period of time. How would we do it?\nThe solution actually has 2-parts and the regular way that one would go about is by :\nbuilding a frequency table - a hashtable with the hashtag as the key and a counter as its value. Scan the hashtable to generate the heavy-hitters (top k) list. Alternatively, you could build a min-heap to hold k elements. Now, in this post, you\u0026rsquo;ll see that while hashtables are great to maintain the frequency table, (part 1 of the problem) CountMinSketch could also be used in its place. In a subsequent post, we\u0026rsquo;ll build on this to solve the second part of the problem.\nWhat is a CountMinSketch? Crudely put, CountMinSketch is an approximate version of a frequency table. But why would you want to use an approximate data structure like CountMinSketch when you could use a Hashtable with counters? There are two reasons :\nSpace One obvious reason is space. Like all probabilistic data structures, CMS uses considerably less space than a hashtable. Let\u0026rsquo;s do a back-of-the-envelope calculation:\nLet\u0026rsquo;s assume we have 1 million hashtags and about 20% of them are unique. We\u0026rsquo;ll also assume that the hashtags are about 6 characters on average.\nHashtable:\nTotal number of hashtags: 200,000\nSize of each hashtag (assuming ASCII): 6 bytes\nCounter size (value of the hashmap) : 4 bytes\nTotal size : 200k * (7+4) = 2.2 MB\nFor the sake of simplicity and sloppiness, we are neither considering the size of the buckets in the hashmap nor the size of the list/tree nor their pointers to the next item, under each bucket.\nIn comparison, a CountMinSketch would take about 5.4 Kb provided we can live with a 1% variance in the estimate. For large datasets, this would mean a difference between 10s of GBs in comparison to 10s of MBs.\nCount Min Sketch:\nSize of each bucket : 272\nNumber of hash functions: 5\nSize of each counter: 4 bytes\nTotal size: 272 * 5 * 4 bytes\n(Assuming that each counter is an unsigned integer)\nYou\u0026rsquo;ll quickly be able to understand the calculation once you see the structure of the CMS in the next section.\nNeed In all the use-cases mentioned above, do you really care about the accurate frequency of each element? I mean, do you really care if the hashtag was used 990,020 or 991,000 times? As far as you know a hashtag\u0026rsquo;s \u0026ldquo;approximate\u0026rdquo; frequency with an assurance that the frequency has a predictable error variance, you are fine.\nCountMinSketches are very frequently (pun intended) used in streaming systems to estimate frequencies in real-time. They are also used to estimate frequencies in large datasets where keeping all the counts on a hashtable is not practical. Also, since they occupy very little memory, it is not uncommon to have multiple CMS for varying time windows.\nCheck out this FAQ on Twitter trends.\nHow do we build one? The structure of CountMinSketch is incredibly close to that of the Counting Bloom Filter. In a Counting Bloom Filter, we had one set of buckets with counters (m). Based on the output of all hash functions, the counters in this one bucket was incremented.\nWith CountMinSketch, we create one bucket of size m per hash function:\nI believe I am breaking all the suspense here. This structure answers one big question that we had in mind:\nWhy does CountMinSketch better than Counting Bloom Filter for estimating frequency/counts?\nBy using m counters per hashing function instead of m counters for ALL hashing functions put together, we are able to significantly reduce the number of collisions that would happen on the bucket. Of course, the downside is that we now need space for k * m counters instead of 1 *m counters but the estimates are very close to the actual counts.\nProperties Before we implement the data structure, let us get the properties summarized:\nWe saw that a CMS solves the \u0026ldquo;frequency\u0026rdquo; problem by enabling us to estimate frequencies of elements in streaming systems and/or in large datasets. It uses multiple hash functions and each hash function uses its hash output to increment counters in their own dedicated buckets, for keeping track of frequencies. They use very less memory in comparison to our traditional hashtable. You can only insert elements and get their estimated count. Deletion, updation, or iteration of values is not possible. Implementation Input parameters There are two important parameters that CMS expects from the users - epsilon and delta. Hehe. I got you, didn\u0026rsquo;t I? Let me elaborate it.\nepsilon : Considering the estimated_count of an element is an approximation, you would want to control how closer to the actual count should the approximation be. The epsilon parameter is nothing but the maximum variance in the estimated count (or maximum allowable error rate). So, if you say the epsilon is 1%, and your estimated count is 1000, it means that the actual count could be between 990 and 1110. Now, why would someone not want the best possible estimate? The reason is that more accuracy comes at the price of more space (number of buckets).\ndelta : Since CMS is a probabilistic data structure, there could be false positives. In a Bloom Filter, the false positive rate of 1% means that approximately 1% of the time, the Bloom filter would say that an element is present in the set, while it was not. In order to minimize the false positive rate, in a bloom filter, we would typically increase the number of buckets and/or increase the number of hash functions.\nIn a CMS, a false positive rate of 1% would mean that 1% of the time, the count returned by CMS is incorrect - meaning 1% of the time the count is beyond the allowed error range specified by epsilon. Just like a Bloom Filter, a larger bucket and a higher number of hash functions are common approaches to reduce the false positive rate.\nSo, here\u0026rsquo;s our implementation. For the purposes of simplicity, the counters are 32-bit unsigned integers but it is very common to see other variations.\nDeclaring the struct and instantiation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 #[derive(Debug)] pub struct CountMinSketch\u0026lt;K: Hash + Eq\u0026gt; { epsilon: f64, delta: f64, hasher: SipHasher24, counter: Vec\u0026lt;Vec\u0026lt;u32\u0026gt;\u0026gt;, m: usize, k: usize, len: usize, _p: PhantomData\u0026lt;K\u0026gt;, } impl\u0026lt;K: Hash + Eq\u0026gt; CountMinSketch\u0026lt;K\u0026gt; { pub fn new(max_variance_in_count: f64, fp_rate_of_count: f64) -\u0026gt; Result\u0026lt;Self\u0026gt; { let epsilon = max_variance_in_count; let delta = fp_rate_of_count; let m = optimal_m(delta); let k = optimal_k(fp_rate_of_count); let random_key = generate_random_key(); let hasher = create_hasher_with_key(random_key); let counter = vec![vec![0_u32; m]; k]; Ok(CountMinSketch { epsilon, delta, hasher, counter, m, k, len: 0, _p: PhantomData, }) } ... ... Hasher instantiation We accept the epsilon and delta parameters and calculate the number of buckets (m) and the number of hash functions (k).\n1 2 3 4 5 6 7 8 9 10 11 /// Generates a random 128-bit key used for hashing fn generate_random_key() -\u0026gt; [u8; 16] { let mut seed = [0u8; 32]; getrandom::getrandom(\u0026amp;mut seed).unwrap(); seed[0..16].try_into().unwrap() } /// Creates a `SipHasher24` hasher with a given 128-bit key fn create_hasher_with_key(key: [u8; 16]) -\u0026gt; SipHasher24 { SipHasher24::new_with_key(\u0026amp;key) } Number of buckets and hash functions The formulas for arriving at optimal_m and optimal_k from epsilon and delta are :\nTheir implementations being :\n1 2 3 4 5 6 7 fn optimal_m(epsilon: f64) -\u0026gt; usize { (2.71828 / epsilon).ceil() as usize } fn optimal_k(delta: f64) -\u0026gt; usize { (1.0 / delta).ln().ceil() as usize } Building the APIs Now that we have built the foundational functions, let\u0026rsquo;s get into the API. There are just two functions to be implemented and both are very intuitive.\nInsert For every item that was inserted, we need to calculate k hashes. The methodology is the same as Counting Bloom Filter (section Hashers) where:\nwe generate a 128-bit hash using SipHasher split them into two 64-bit hashes use Kirch and Mitzenmacher technique to create k-hash functions scoped down the hash to bucket indices of size m increment the counters corresponding to the bucket indices by 1. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 impl\u0026lt;T: ?Sized + Hash + Debug\u0026gt; CountingBloomFilter\u0026lt;T\u0026gt; { pub fn insert(\u0026amp;mut self, key: \u0026amp;K) { let bucket_indices = self.get_bucket_indices(key, self.hasher); bucket_indices .iter() .enumerate() .for_each(|(ki, \u0026amp;bi)| self.counter[ki][bi] = self.counter[ki][bi].saturating_add(1)); self.len += 1 } /// Returns the bucket indices of k hash functions for an item fn get_bucket_indices(\u0026amp;self, item: \u0026amp;K, hasher: SipHasher24) -\u0026gt; Vec\u0026lt;usize\u0026gt; { let (hash1, hash2) = self.get_hash_pair(item, hasher); let mut bucket_indices = Vec::with_capacity(self.k); if self.k == 1 { let bit = hash1 % self.m as u64; bucket_indices.push(bit as usize); return bucket_indices; } else { for ki in 0..self.k as u64 { let hash = hash1.wrapping_add(ki.wrapping_mul(hash2)); let bit = hash % self.m as u64; bucket_indices.push(bit as usize) } } bucket_indices } Estimated count The estimated count is calculated by hashing the item and looking into the counter values for each hash function. The minimal counter value is considered to be the estimated count.\nThe reasoning behind choosing the minimum and not the maximum is that a counter could be incremented while inserting other elements due to hash collisions. Therefore, these counters overestimate the real frequency since the number of buckets is always much lesser than the number of items inserted into the sketch. Taking an average or a median is also not a fair estimate for items that have a lower frequency.\n1 2 3 4 5 6 7 8 9 10 11 12 pub fn estimated_count(\u0026amp;self, key: \u0026amp;K) -\u0026gt; u32 { let bucket_indices = self.get_bucket_indices(key, self.hasher); let mut estimated_count = u32::MAX; for (ki, \u0026amp;bi) in bucket_indices.iter().enumerate() { if self.counter[ki][bi] == 0 { return 0; } else { estimated_count = min(estimated_count, self.counter[ki][bi]) } } estimated_count } Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 #[test] fn insert_and_check_several_items() -\u0026gt; Result\u0026lt;()\u0026gt; { let mut bf: CountMinSketch\u0026lt;\u0026amp;str\u0026gt; = CountMinSketch::new(0.2, 0.01)?; for _ in 0..1000000 { bf.insert(\u0026amp;\u0026#34;a1\u0026#34;); bf.insert(\u0026amp;\u0026#34;a2\u0026#34;); bf.insert(\u0026amp;\u0026#34;a3\u0026#34;); bf.insert(\u0026amp;\u0026#34;a4\u0026#34;); bf.insert(\u0026amp;\u0026#34;a5\u0026#34;); bf.insert(\u0026amp;\u0026#34;a6\u0026#34;); bf.insert(\u0026amp;\u0026#34;a7\u0026#34;); } assert_eq!(bf.number_of_hashes(), 5); assert_eq!(bf.number_of_counters(), 272); assert_eq!(bf.estimated_count(\u0026amp;\u0026#34;a1\u0026#34;), 1000000); assert_eq!(bf.estimated_count(\u0026amp;\u0026#34;a2\u0026#34;), 1000000); assert_eq!(bf.estimated_count(\u0026amp;\u0026#34;b1\u0026#34;), 0); Ok(()) } Code The complete code is here.\n","date":"2023-04-02T10:43:13Z","permalink":"/build-your-own-countminsketch-in-rust/","title":"Build your own CountMinSketch in Rust"},{"content":"A Counting Bloom Filter is a probabilistic data structure that helps us quickly check if an element is present in a set or not. You might argue \u0026ldquo;Hey, can\u0026rsquo;t a simple Set do it?\u0026rdquo;.\nYes, indeed and to top it, Counting Bloom Filter is not even 100% accurate and expects us to provide the expected \u0026ldquo;false positive rate\u0026rdquo;. Where Bloom Filters shine is that for large volumes of data it would use much lesser memory than a regular HashSet.\nThis post attempts an implementation of the datastructure. At the end of the post, you\u0026rsquo;ll be able to build your own Counting Bloom Filter. The final code is available here.\nBefore we get into the implementation, let\u0026rsquo;s get some properties of CBF out of the way:\nAs with our traditional BloomFilter, Quotient Filter or CuckooFilter, Counting Bloom Filter is a probabilistic data structure that is used to test whether an element is a member of a set. Essentially, these filters aim to solve a class of problems called \u0026ldquo;Membership\u0026rdquo;. A key feature that CBF offers over traditional Bloom Filter is that CBF allows deletion of inserted items. As with any probabilistic data structure, there is a chance that we might get some false positives - essentially, the response to the question \u0026ldquo;Is an element present in the set?\u0026rdquo; is not 100% accurate. There are times when the bloom filter would say that the element is present but it actually was not. This \u0026ldquo;false positive\u0026rdquo; rate is configurable through a parameter. Unlike HashSet, you cannot iterate through the values stored in the set - we can only check whether the value is present in the CBF. Bloom Filter vs Counting Bloom Filter The underlying storage of a BloomFilter is just a fixed-size array of bits.\nThis is a fantastic write-up detailing how to implement your own Bloom Filter.\nA Counting Bloom Filter uses a fixed-size array of counters.\nTo put it in crude terms, the underlying storage of Bloom Filter is a Vec\u0026lt;bool\u0026gt; (or a BitSet). With CBF, it is a Vec\u0026lt;u8\u0026gt; or Vec\u0026lt;u16\u0026gt; or Vec\u0026lt;u32\u0026gt; or a list of unsigned integers.\nThe counter size does not need to be 8, 16, 32 bits etc - rather, they don\u0026rsquo;t need to be aligned to the datatypes. Using arbitrary bit sizes is common and is achieved by partitioning a usize integer into arbitrary bit sizes.\nFor illustration, our implementation of a Counting Bloom Filter will use a u8 bit counter.\nSetting the stage Declaring our struct and instantiation 1 2 3 4 5 6 7 8 9 10 11 #[derive(Debug)] pub struct CountingBloomFilter\u0026lt;T: ?Sized + Hash\u0026gt; { counter: Vec\u0026lt;u8\u0026gt;, m: usize, k: usize, hasher: SipHasher24, expected_num_items: usize, false_positive_rate: f64, len: usize, _p: PhantomData\u0026lt;T\u0026gt;, } As expected, our struct encapsulates a list of counters. There are two parameters that we would be accepting from the user, which are expected_num_items and false_positive_rate.\nThe expected_num_items is an approximate number of items that the user would like to store in the CBF. The false_positive_rate is an estimate of False Positives/inverse accuracy that the user can live with. The len just stores the total number of items inserted into the data structure so far. We\u0026rsquo;ll discuss the m and k fields in the next section.\nNumber of buckets and number of hashes Bloom Filter In a BloomFilter, m is simply the size of the bit vector (or underlying bits representing the membership of the various inserted values). k is the number of hash functions that each inserted value has to be run against.\nSo, given a value x, it will be hashed by k number of hash functions and the corresponding bit in the m buckets will be set to True.\nLet\u0026rsquo;s consider the illustration above. Each item ( x, y, z) are hashed by 2 hash functions (represented by arrows). The resulting output are expected to be the bucket indices. Considering the total number of buckets is just 8 in the above illustration, the output of the hash function is modulo-ed to reduce the range between 0 and 7.\nCounting bloom filter. The number of hash functions ( k ) is 2 as above and the m is 8. However, the underlying store is not a list of bits but a list of counters.\nThe formulas for calculating the size of m and k given the expected_num_of_items and false_positive_rate for Counting Bloom Filter are:\nLet\u0026rsquo;s implement the optimal_m, optimal_k and the new functions\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 fn optimal_m(num_items: usize, false_positive_rate: f64) -\u0026gt; usize { -(num_items as f64 * false_positive_rate.ln() / (2.0f64.ln().powi(2))).ceil() as usize } fn optimal_k(n: usize, m: usize) -\u0026gt; usize { let k = (m as f64 / n as f64 * 2.0f64.ln()).round() as usize; if k \u0026lt; 1 { 1 } else { k } } impl\u0026lt;T: ?Sized + Hash + Debug\u0026gt; CountingBloomFilter\u0026lt;T\u0026gt; { pub fn new(expected_num_items: usize, false_positive_rate: f64) -\u0026gt; Result\u0026lt;Self\u0026gt; { validate(expected_num_items, false_positive_rate)?; let m = optimal_m(expected_num_items, false_positive_rate); let counter = vec![0; m]; let k = optimal_k(expected_num_items, m); let random_key = generate_random_key(); let hasher = create_hasher_with_key(random_key); Ok(Self { counter, m, k, hasher, expected_num_items, false_positive_rate, len: 0, _p: PhantomData, }) } As you can see, only the \u0026ldquo;estimated number of items to be inserted\u0026rdquo; and the \u0026ldquo;false positive rate\u0026rdquo; is needed to construct our Counting Bloom Filter.\nHashers Once we calculate the optimal number of hash functions (k), we need to create k independent hash functions. One approach is to have a repository of hash functions and initiate them with a random key. However, there is a simpler approach that is proposed by Adam Kirsch and Michael Mitzenmacher, where we could simulate additional hash functions with the help of two base hash functions.\nQuoting the paper:\nThe idea is the following: two hash functions h1(x) and h2(x) can simulate more than two hash functions of the form gi(x) = h1(x) + ih2(x). In our context i will range from 0 up to some number k − 1 to give k hash functions, and the hash values are taken modulo the size of the relevant hash table.\nSpecifically, only two hash functions are necessary to effectively implement a Bloom filter without any increase in the asymptotic false positive probability. This leads to less computation and potentially less need for randomness in practice.\nEssentially, we just need 2 independent hash functions to simulate k hash functions. Typically, Bloom Filters use 64-bit hash functions since they create enough bits for us to populate the buckets uniformly. In our implementation, we will use a 128-bit hash function and use the first and the second 64 bits to simulate two hash function outputs. We will then use these two 64-bit hashes to create more hashes.\nFirstly, in order to provide a random seed to our 128-bit hasher, we\u0026rsquo;ll use the generate_random_key function.\n1 2 3 4 5 6 7 8 9 10 11 12 13 use siphasher::sip128::SipHasher24; /// Generates a random 128-bit key used for hashing fn generate_random_key() -\u0026gt; [u8; 16] { let mut seed = [0u8; 32]; getrandom::getrandom(\u0026amp;mut seed).unwrap(); seed[0..16].try_into().unwrap() } /// Creates a `SipHasher24` hasher with a given 128-bit key fn create_hasher_with_key(key: [u8; 16]) -\u0026gt; SipHasher24 { SipHasher24::new_with_key(\u0026amp;key) } Building the APIs With all the foundational functions out of the way, let us get into implementing the API - the three functions insert, contains and delete\nInsert When an element is inserted into the filter, the counters corresponding to the element\u0026rsquo;s hash values are incremented by one. This is in contrast to BloomFilter where we just set the corresponding bit. While we do this, we will also use this opportunity to keep track of the number of items inserted into the bloom filter by incrementing the len field.\n1 2 3 4 5 6 pub fn insert(\u0026amp;mut self, item: \u0026amp;T) { self.get_set_bits(item, self.k, self.m, self.hasher) .iter() .for_each(|\u0026amp;i| self.counter[i] = self.counter[i].saturating_add(1)); self.len += 1; } The get_set_bits is a helper function that takes the item to be inserted and returns the indices of all buckets for whom the counters need to be incremented. Essentially, the get_set_bits function\nTakes an item as an input Creates a 128-bit hash (done in get_hash_pair) Splits the 128-bit hash into 2 parts, thereby creating the 2 base hashes (done in get_hash_pair) Creates k hashes out of the 2 base hashes using the Kirsch and Mitzenmacher method Since we just have m buckets and the hash function output could be bigger, we do a modulus of m on each hash output. This returns k bucket indices on m . These are the indices of the counters in the bucket that needs to be incremented. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 fn get_set_bits(\u0026amp;self, item: \u0026amp;T, k: usize, m: usize, hasher: SipHasher24) -\u0026gt; Vec\u0026lt;usize\u0026gt; { let (hash1, hash2) = self.get_hash_pair(item, hasher); let mut set_bits = Vec::with_capacity(k); if k == 1 { let bit = hash1 % m as u64; set_bits.push(bit as usize); return set_bits; } for ki in 0..k as u64 { let hash = hash1.wrapping_add(ki.wrapping_mul(hash2)); let bit = hash % m as u64; set_bits.push(bit as usize); } assert!(set_bits.len() == k); set_bits } /// Computes the pair of 64-bit hashes for an item using the internal hasher fn get_hash_pair(\u0026amp;self, item: \u0026amp;T, mut hasher: SipHasher24) -\u0026gt; (u64, u64) { item.hash(\u0026amp;mut hasher); let hash128 = hasher.finish128().as_u128(); let hash1 = (hash128 \u0026amp; 0xffff_ffff_ffff_ffff) as u64; let hash2 = (hash128 \u0026gt;\u0026gt; 64) as u64; (hash1, hash2) } The wrapping_add, wrapping_mul and other wrapping functions allow for overflow of numbers without triggering a panic.\neg.\nlet a: u16 = 65534;\nlet b = a.wrapping_add(1); // 65535\nlet c = a.wrapping_add(2); // 0\nContains As you might have guessed already, testing whether an item is present in the filter is just a matter of computing the hashes for the item and checking whether the counters in the bucket corresponding to the hashes have a non-zero value.\n1 2 3 4 5 6 pub fn contains(\u0026amp;self, item: \u0026amp;T) -\u0026gt; bool { self.get_set_bits(item, self.k, self.m, self.hasher) .iter() .all(|\u0026amp;i| self.counter[i] \u0026gt; 0) } Delete The deletion operation is just an inverse of the insertion operation with two additional nuances.\nWe can only delete an item that is already present in the filter. So, we check for it by calling the contains. Our counter can never be less than 0. So, we use the saturating_sub function to ensure that. 1 2 3 4 5 6 7 8 9 10 11 pub fn delete(\u0026amp;mut self, item: \u0026amp;T) { let is_present = self.contains(item); if is_present { self.get_set_bits(item, self.k, self.m, self.hasher) .iter() .for_each(|\u0026amp;i| { self.counter[i] = self.counter[i].saturating_sub(1); }); self.len -= 1; } } Estimated count Although Counting Bloom Filter is not the best data structure to get an estimated count of an item, it is not uncommon to see instances where the function is implemented. Estimated count is arrived at by looking at all the counter values and picking the minimum of those values. The reasoning behind choosing minimum and not maximum or median is that a counter could be incremented by several items due to hash collisions. Minimum is chosen to avoid the skew created by high-frequency elements and to be fair to the low-frequency elements.\n1 2 3 4 5 6 7 8 9 10 11 12 pub fn estimated_count(\u0026amp;self, item: \u0026amp;T) -\u0026gt; u8 { let mut retu = u8::MAX; for each in self.get_set_bits(item, self.k, self.m, self.hasher) { if self.counter[each] == 0 { return 0; } if self.counter[each] \u0026lt; retu { retu = self.counter[each]; } } retu } We will revisit this discussion when we build our own CountMinSketch later.\nOverflow Although not included in our implementation, I felt that a discussion on this topic is important.\nIn our example, we decided that our counter is represented by an unsigned 8-bit integer with a maximum value of 255. What if subsequent item insertions attempt to increment the value of that counter? This would result in an overflow and in Rust, a panic. To avoid overflowing, the counter could be capped/saturated at the maximum value. This essentially means that subsequent insertions into that bucket will have no effect on the filter\u0026rsquo;s state for that counter.\nThere are two common workarounds to solve overflows:\nFreezing Dynamic resizing Freezing Freezing is a process where\nthe buckets are scanned for counters that have already reached the maximum value. reset the counter to a threshold value (around 80% of the maximum value). For u8 that\u0026rsquo;s approximately 200. Dynamic resizing The other simpler yet computationally expensive alternative is to resize the counter to accept larger values. Say, promoting the datatype from u8 to u16. This involves creating a new counter list, copying the existing counters and replacing the old list with the new one.\nCode Complete code is here.\n","date":"2023-03-19T15:37:44Z","permalink":"/build-your-own-counting-bloom-filter-in-rust/","title":"Build your own Counting Bloom Filter in Rust"},{"content":"If you are here, you already know what Apache Ranger is. It is the most popular, if not the only, way to manage security in the Hadoop framework. It has integrations with Active Directory, Kerberos and various others for authentication but I believe the most interesting feature is its authorization support. Being part of the Hadoop ecosystem, one would not be surprised that it has inbuilt support (via plugins) for most frameworks in the Hadoop ecosystem - Hive, HBase, HDFS etc. However, I\u0026rsquo;ve found that it\u0026rsquo;s actually very easy to spin your own custom plugin for Ranger.\nThis post would focus on the simplicity of design in Ranger plugins and showcase how easy it is to build one for ourselves. As an example, we\u0026rsquo;ll build a Ranger plugin for managing access to a simple HTTP service written using Akka HTTP.\n1 Note : You are not required to know about Akka HTTP to follow this post. All you needed to know is that Akka HTTP is just a way (albeit, a great way) to build HTTP services The code behind this post is split into two repositories:\nRanger HTTP plugin Ranger Managed Akka HTTP Service Writing a plugin To reiterate what we are attempting to do here, we are going to write a REST service and let Ranger manage the authorization for it.\nWriting a Ranger plugin is actually a two part problem - writing the server-side component and the application-side component.\nServer-side component is the code/configuration that resides on the Ranger side. Application-side component is the code that resides in our REST service that invokes the Ranger service and checks if the application\u0026rsquo;s end user has access to the resource that he is requesting. We\u0026rsquo;ll look into these two things in detail. Let\u0026rsquo;s attempt to write the server-side components first.\nServer-side components : As an inspiration, if we open up the ranger code base, we can see some of the in-built plugins.\nPictorially, within the Ranger code base, we have bunch of plugins and we would like to add our own plugin.\nZooming in the previous picture, the server-side component on the plugin would mean writing a\nservicedef configuration A class that inherits RangerBaseService So, there\u0026rsquo;s literally \u0026ldquo;one\u0026rdquo; configuration and \u0026ldquo;one\u0026rdquo; class that you need to implement for the server-side.\nservicedef configuration Let\u0026rsquo;s look at Hive\u0026rsquo;s servicedef configuration :\nIn my opinion, there are three important things that we are talking about here :\nResource: In the Hive example, the \u0026ldquo;resource\u0026rdquo; that we are trying to protect are databases, tables and columns, for Kafka, the \u0026ldquo;resource\u0026rdquo; that we are trying to protect is the Kafka topic, for HDFS, it would be a file path. For our HTTP service, the resource that we are trying to protect is the REST slug. Let\u0026rsquo;s call it a \u0026ldquo;path\u0026rdquo;.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 \u0026#34;resources\u0026#34;: [ { \u0026#34;itemId\u0026#34;: 1, \u0026#34;name\u0026#34;: \u0026#34;path\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;path\u0026#34;, \u0026#34;level\u0026#34;: 10, \u0026#34;parent\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;mandatory\u0026#34;: true, \u0026#34;lookupSupported\u0026#34;: true, \u0026#34;recursiveSupported\u0026#34;: true, \u0026#34;excludesSupported\u0026#34;: true, \u0026#34;matcher\u0026#34;: \u0026#34;org.apache.ranger.plugin.resourcematcher.RangerPathResourceMatcher\u0026#34;, \u0026#34;matcherOptions\u0026#34;: { \u0026#34;wildCard\u0026#34;: true, \u0026#34;ignoreCase\u0026#34;: true }, \u0026#34;validationRegEx\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;validationMessage\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;uiHint\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;label\u0026#34;: \u0026#34;HTTP Path\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;HTTP Path\u0026#34; } Access Type: Access types simply means the kind of access that the user would require - say, for Hive, select, create, delete would be examples. For HDFS, read, write, execute would be examples. For Kafka, publish and consume. For our HTTP service, the access type would be the HTTP methods - GET, POST, DELETE.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 \u0026#34;accessTypes\u0026#34;: [ { \u0026#34;itemId\u0026#34;: 1, \u0026#34;name\u0026#34;: \u0026#34;get\u0026#34;, \u0026#34;label\u0026#34;: \u0026#34;get\u0026#34; }, { \u0026#34;itemId\u0026#34;: 2, \u0026#34;name\u0026#34;: \u0026#34;post\u0026#34;, \u0026#34;label\u0026#34;: \u0026#34;post\u0026#34; }, { \u0026#34;itemId\u0026#34;: 3, \u0026#34;name\u0026#34;: \u0026#34;delete\u0026#34;, \u0026#34;label\u0026#34;: \u0026#34;delete\u0026#34; } ] Configs: We know that Ranger can manage security for several Kakfa topics, HDFS and HBase clusters. Each of these services would be running in a different host and the way to authenticate into each of them would be different. The place to capture this information would be this configs part. For the sake of simplicity of this example, we don\u0026rsquo;t care about authentication for our HTTP service. So, we are just capturing a URL that we could ping to, to ensure that our service is up and running.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 \u0026#34;configs\u0026#34;: [ { \u0026#34;itemId\u0026#34;: 1, \u0026#34;name\u0026#34;: \u0026#34;services_list_url\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;subType\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;mandatory\u0026#34;: true, \u0026#34;validationRegEx\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;validationMessage\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;uiHint\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;label\u0026#34;: \u0026#34;HTTP URL for the services list eg. http://localhost:8080/services\u0026#34; } ] A class that inherits RangerBaseService The second and the last part of implementing our server-side component for the ranger plugin is to write a class that inherits the RangerBaseService.\nThe class expects two functions to be overridden:\nvalidateConfig: Remember the configs section of the servicedef. Obviously, we would be accepting values for those parameters right? Now, this validateConfig is the place where we validate the values that are passed. For our HTTP service, all that we are accepting in the config is the services_list_url. Now, the implementation of this function would be to use a simple HTTP client to ping and check whether the service is up and running. 1 2 3 4 5 6 7 8 9 10 11 class RangerServiceHTTP extends RangerBaseService { override def validateConfig(): util.Map[String, AnyRef] = { if (configs.containsKey(\u0026#34;services_list_url\u0026#34;)) { val serviceUp = HttpServiceClient.isServiceUp(configs.get(\u0026#34;services_list_url\u0026#34;)) if (serviceUp) retSuccessMap() else returnFailureMap() } else { returnFailureMap() } } lookupResource: This is an interesting function. Consider the following screenshot. Later, when we configure an access policy, we would be configuring the resources in it. Now, this function is used to lookup and autofill those resources. Say, if we are entering a HDFS resource or Hive table, the number of options are quite a lot and it\u0026rsquo;s easy to do a typo. In case of Hive, this function would connect to the metastore and populate the tables and databases for us.\nIn the case of HTTP service, remember the service_list_url? That URL would just return a comma separated list of REST resources. For implementing this function, I am just calling the service again and tokenizing the response.\n1 2 3 4 override def lookupResource(resourceLookupContext: ResourceLookupContext): util.List[String] = { val serviceUrl = configs.get(\u0026#34;services_list_url\u0026#34;) HttpServiceClient.getServicePaths(serviceUrl).asJava } Now, as a final step to the code, we\u0026rsquo;ll need to tie this class RangerServiceHTTP and the servicedef configuration together. The way we do this is by configuring the class in the implClass property. Also notice that we are configuring the name of this ranger plugin as httpservice:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 { \u0026#34;name\u0026#34;: \u0026#34;httpservice\u0026#34;, \u0026#34;label\u0026#34;: \u0026#34;HTTP Service\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Rudimentary Ranger plugin to enforce security on top of a HTTP Service\u0026#34;, \u0026#34;guid\u0026#34;: \u0026#34;b8290b7f-6f69-44a9-89cc-06b6975ea676\u0026#34;, \u0026#34;implClass\u0026#34;: \u0026#34;com.arunma.ranger.http.RangerServiceHTTP\u0026#34;, * * \u0026#34;version\u0026#34;: 1, \u0026#34;isEnabled\u0026#34;: 1, \u0026#34;resources\u0026#34;: [ { \u0026#34;itemId\u0026#34;: 1, \u0026#34;name\u0026#34;: \u0026#34;path\u0026#34;, ... ... The full configuration looks like this.\nThere are two more minor administrative steps:\nIn order to ensure that our class is made available on the Ranger classpath, we\u0026rsquo;ll bundle it into a jar and drop it at \u0026lt;RANGER_HOME\u0026gt;/ews/webapp/WEB-INF/classes/ranger-plugins/httpservice. The name of the folder httpservice corresponds to the name that is declared in the servicedef configuration. Upload our configuration into Ranger so that our service becomes visible in the Ranger UI. 1 curl -u admin:admin -X POST -H \u0026#34;Accept: application/json\u0026#34; -H \u0026#34;Content-Type: application/json\u0026#34; --data @http-ranger.json http://localhost:6080/service/plugins/definitions Bounce the Ranger server.\nYaay! We now see HTTPSERVICE on our Ranger UI\nApplication-side components : On the application side, things couldn\u0026rsquo;t get any simpler. In order to use the policies used in Ranger, all that an application would need to do is to call Ranger and check if the user has access to a resource. The function is literally called isAccessAllowed.\nThe following code is pretty much all that needs to be written on the application side:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 package com.arunma.ranger import org.apache.ranger.plugin.audit.RangerDefaultAuditHandler import org.apache.ranger.plugin.policyengine.{RangerAccessRequestImpl, RangerAccessResourceImpl} import org.apache.ranger.plugin.service.RangerBasePlugin import scala.collection.JavaConverters._ object RangerAuthorizer { lazy val plugin = { val plg = new RangerBasePlugin(\u0026#34;httpservice\u0026#34;, \u0026#34;httpservice\u0026#34;) plg.setResultProcessor(new RangerDefaultAuditHandler) plg.init() plg } def authorize(path: String, accessType: String, userName: String, userGroups: Set[String] = Set(\u0026#34;public\u0026#34;)): Boolean = { val resource = new RangerAccessResourceImpl() resource.setValue(\u0026#34;path\u0026#34;, path) val request = new RangerAccessRequestImpl(resource, accessType, userName, userGroups.asJava) val result = plugin.isAccessAllowed(request) result != null \u0026amp;\u0026amp; result.getIsAllowed } } The RangerBasePlugin(\u0026quot;httpservice\u0026quot;, \u0026quot;httpservice\u0026quot;) and the init() function serves as our entry point into the Ranger service. Note the httpservice parameter inside the RangerBasePlugin. This must match the name that was given in the servicedef configuration.\nThe authorize function is the one that gets called by the interceptor just before the client is given access to a REST resource. The function simply constructs a AccessRequest - the RangerAccessRequestImpl and calls the plugin\u0026rsquo;s isAccessAllowed function, which returns a Boolean.\nThe interceptor directive authorize invokes the function isRangerAuthorized which then calls the authorize function in RangerAuthorizer.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 def isRangerAuthorized(path: String, httpMethod: String, userName: String): Boolean = RangerAuthorizer.authorize(path, httpMethod.toLowerCase, userName) lazy val userRoutes: Route = headerValueByName(\u0026#34;username\u0026#34;) { userName =\u0026gt; extractMethod { method =\u0026gt; pathPrefix(\u0026#34;users\u0026#34;) { extractMatchedPath { matchedPath =\u0026gt; authorize(isRangerAuthorized(matchedPath.toString(), method.name(), userName)) { concat( pathEnd { concat( get { val users: Future[Users] = (userRegistryActor ? GetUsers).mapTo[Users] complete(users) One last thing that we are required to do is to copy an audit and security xml into our classpath. These are like the site xmls for Ranger. For this exercise, we\u0026rsquo;ll just place the xmls in our resources directory.\nThe audit xml and the security xml could be copied from the ranger codebase. If you are running a local ranger, the audit XML can remain as-is but security xml needs to be changed for our service. The easiest way to achieve this is to copy a sample xml from the ranger code base and start replacing the service as httpservice like so:\nThere\u0026rsquo;s also one property that needs special attention. That\u0026rsquo;s the property called ranger.plugin.httpservice.service.name. This property\u0026rsquo;s value must be the same as the Service Name that you use in your Ranger UI.\n1 2 3 4 5 6 7 \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;ranger.plugin.httpservice.service.name\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;MyService\u0026lt;/value\u0026gt; \u0026lt;description\u0026gt; Name of the Ranger service containing policies for this httpservice instance \u0026lt;/description\u0026gt; \u0026lt;/property\u0026gt; Test Ride This would involve two steps\nConfigure a Ranger Policy Verifying your HTTP Service Configure a Ranger Policy Verifying your HTTP Service Let\u0026rsquo;s verify the policy by bringing up our HTTP Service - start the com.arunma.RangerManagedHttpServer\nPolicy-configured user\n1 curl -X GET -H \u0026#39;username:arunma\u0026#39; http://localhost:8080/users Invalid user\n1 curl -X GET -H \u0026#39;username:nobody\u0026#39; http://localhost:8080/users Summary The Ranger plugin has two parts to it - a server-side component and a client-side component. For the server-side component, we created a servicedeef json and a class that inherited the RangerBaseService. For the client side-component, we just called an isAccessAllowed function of the plugin.\nYou now have a working Ranger authorized HTTP Service.\nThanks for reading. Happy Hacking !\n","date":"2019-05-13T16:23:30Z","permalink":"/the-beautiful-simplicity-of-apache-ranger-plugin/","title":"The beautiful simplicity of Apache Ranger plugin"},{"content":"I have been recently playing with Apache Amaterasu, which is an amazing project that helps to deploy data pipelines. It\u0026rsquo;s still incubating and has a super-friendly team of engineers working on it. Some exciting features are lined up. Don\u0026rsquo;t take my word for it. Please check it out yourself.\nAmaterasu launches containers (on YARN/Mesos) all by itself for each of the stages in your data pipeline. All you need to provide is your repository and a YAML based configuration.\nI was just curious about launching containers on YARN and how the API works and thought I should give it a try myself. It\u0026rsquo;s really intuitive if we understand a small set of constructs.\nThis post is an attempt on a minimal example of a YARN application on Scala.\nNote : The complete code is available at github\nThere are three core classes in this App :\nSampleYarnClient ApplicationMaster DummyApplication (the business logic) SampleYarnClient This is the entry-point of the program. Does the following :\nInstantiates YarnClient. Negotiates resources for the ApplicationMaster container with the help of the YarnClient. The way it does it is to initiate an ApplicationSubmissionContext which is just a wrapper around the Resource, Priority and ContainerLaunchContext (among others). Let\u0026rsquo;s quickly look at the code and we\u0026rsquo;ll go over in detail on these three components of the SubmissionContext. 1 2 3 4 5 6 7 8 9 10 11 val yarnClient = YarnClient.createYarnClient() ... val application = yarnClient.createApplication() ... val context = application.getApplicationSubmissionContext context.setAMContainerSpec(amContainer) context.setApplicationName(\u0026#34;Scala Yarn App\u0026#34;) context.setResource(resource) context.setPriority(priority) yarnClient.submitApplication(context) Resource Resources is a simple wrapper around CPU and Memory\n1 val resource = Resource.newInstance(1024, 2) //1 GB memory and 2 cores Priority Priority is just an Integer - the higher the number, the higher the priority\n1 2 val priority = Records.newRecord(classOf[Priority]) priority.setPriority(1) ContainerLaunchContext The ContainerLaunchRequest has three primary parameters in this simple example :\nCommands (List[String]): The bootstrap command (ideally java \u0026lt;MainClass\u0026gt;) LocalResources (Map[String,LocalResource]) : The jars and the other artifacts (properties, libraries etc) that\u0026rsquo;s essential for running your command Environment (Map[String,String]): The environment variables essential for the program (the other important one is the Security Tokens which is not used here because my local cluster isn\u0026rsquo;t kerberized)\n1 2 3 4 5 6 7 def createContainerContext(commands: List[String], resources: Map[String, LocalResource], environment: Map[String, String]): ContainerLaunchContext = { val launchContext = Records.newRecord(classOf[ContainerLaunchContext]) launchContext.setCommands(commands.asJava) launchContext.setLocalResources(resources.asJava) launchContext.setEnvironment(environment.asJava) launchContext } Commands Like I said, the commands are just a sequence of instructions you would like to execute to run the ApplicationMaster from the shell.\n1 2 3 4 5 6 7 val commands = List( \u0026#34;$JAVA_HOME/bin/java \u0026#34; + \u0026#34; -Xmx256m \u0026#34; + s\u0026#34; com.arunma.ApplicationMaster \u0026#34; + \u0026#34; 1\u0026gt; \u0026#34; + ApplicationConstants.LOG_DIR_EXPANSION_VAR + \u0026#34;/stdout\u0026#34; + \u0026#34; 2\u0026gt; \u0026#34; + ApplicationConstants.LOG_DIR_EXPANSION_VAR + \u0026#34;/stderr\u0026#34; ) LocalResources As for this program, you don\u0026rsquo;t need any properties or configuration files. All you need is just the jar binary alone.\nNote :\nThe way that you make any binary or resource available for YARN is to place the binaries in a HDFS location. This particular step involving the setting up of local resources just means that we are telling YARN to download the binaries from the HDFS location and place in in the local path of the container when launched.\n1 2 3 val localResources = Map( \u0026#34;SampleYarnApp-assembly-0.1.jar\u0026#34; -\u0026gt; setUpLocalResourceFromPath(yarnPath) ) Environment These are custom environment variables or just the classpath that needs to be set for your bundle to run.\nApplicationMaster Now that we have discussed about the SampleYarnClient, let\u0026rsquo;s discuss the second. This class, as the name indicates, is your ApplicationMaster (Duh!). It\u0026rsquo;s responsible for launching the containers that is expected to run your \u0026ldquo;business logic\u0026rdquo;. The steps are :\nThe AppMaster uses the ResourceManager client (the AMRMClient) to raise a request for a container - ContainerRequest. This example uses the async version of the client - AMRMClientAsync that implements a series of callbacks - onContainersAllocated, onContainersCompleted, onError etc . When the RM allocates a container for the application, the onContainersAllocated callback gets invoked. Within the onContainersAllocated (now that we have the handle to the container), the AppMaster then uses the NMClientAsync to launch the \u0026ldquo;business\u0026rdquo; container (DummyApplication). This is achieved by constructing another ContainerLaunchContext (the one that wraps commands, local resources and environment variables). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 override def onContainersAllocated(containers: util.List[Container]): Unit = { val commands = List( \u0026#34;$JAVA_HOME/bin/java \u0026#34; + \u0026#34; -Xmx256m \u0026#34; + s\u0026#34; com.arunma.DummyApplication \u0026#34; + \u0026#34; 1\u0026gt; \u0026#34; + ApplicationConstants.LOG_DIR_EXPANSION_VAR + \u0026#34;/stdout\u0026#34; + \u0026#34; 2\u0026gt; \u0026#34; + ApplicationConstants.LOG_DIR_EXPANSION_VAR + \u0026#34;/stderr\u0026#34; ) val localResources = Map( \u0026#34;SampleYarnApp-assembly-0.1.jar\u0026#34; -\u0026gt; setUpLocalResourceFromPath(FileSystem.get(conf).makeQualified(new Path(sys.env(\u0026#34;ARUN_JAR_PATH\u0026#34;)))) ) val containerLaunchContext = createContainerContext(commands, localResources, buildEnvironment(Map())) containers.asScala.foreach { container =\u0026gt; nmClient.startContainerAsync(container, containerLaunchContext) } } DummyApplication This is the \u0026ldquo;business logic\u0026rdquo;. Not Scala in the purest sense but it helps us see the logs. Note that because this is an infinite loop, we\u0026rsquo;ll have to forcefully kill the application.\n1 2 3 4 5 6 object DummyApplication extends App { while(true) { println(\u0026#34;Niceeeeeeeeee !!! This is the core application that is running within the container that got negotiated by from Application Master !!!\u0026#34;) Thread.sleep(1000) } } Usage: 1 $ hadoop jar /Users/arun/IdeaProjects/SampleYarnApp/target/scala-2.11/SampleYarnApp-assembly-0.1.jar com.arunma.SampleYarnClient /Users/arun/IdeaProjects/SampleYarnApp/target/scala-2.11/SampleYarnApp-assembly-0.1.jar Alternatively, you could just do an\n1 sbt assembly and run the SampleYarnClient from your IDE with the absolute path of the assembly jar as the first argument.\n","date":"2018-05-24T15:27:43Z","permalink":"/yarn-app-scala/","title":"Creating a YARN Application using Scala"},{"content":"In the previous parts of this post, we discussed about Futures and Promises. In this last part, we\u0026rsquo;ll compose Futures using its powerful combinators.\nComposing Futures : In the first post, we saw how to extract a value from Future using onComplete, foreach and in testcases using Await.result. Extracting a value from a single Future is good but many a time we spawn more than one asynchronous operation and wait on multiple Futures to arrive at the final result. Even better, sometimes the result of one Future would be fed into another or a chain of Futures.\nFuture is a Monad. (I am sorry to drop the M-bomb here and I will take a stab at explaining my understanding on what a Monoid, Functor, Monad and an Applicative later). But for now, let\u0026rsquo;s live with this crude explanation :\nFuture is a container of a value(s) of some type (i.e it accepts a type as an argument and it can\u0026rsquo;t exist without it). You can have a Future[Int] or Future[String] or Future[AwesomeClass] - you can\u0026rsquo;t just have a plain Future. A fancy term for this is type-constructor. To compare, a List is a type constructor (and a Monad as well). A list is a container of values that are of type Int, String or any of other types. A List/Future without a contained type does not exist. Future has flatMap and unit functions (and consequentially a map function too). The reason I brought this up is that instead of using the onComplete callback or the foreach, we could simply map or flatMap the result of the Future just like we would do it against an Option or a List.\nNow, let\u0026rsquo;s look at the map and the flatMap combinators.\nMapping Futures that execute sequentially Let\u0026rsquo;s consider this simple task, which adds three numbers which are asynchronously calculated after some interval.\nWarning : The following code is messy and executes the Futures sequentially\nCode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 class FutureCombinators { def sumOfThreeNumbersSequentialMap(): Future[Int] = { Future { Thread.sleep(1000) 1 }.flatMap { oneValue =\u0026gt; Future { Thread.sleep(2000) 2 }.flatMap { twoValue =\u0026gt; Future { Thread.sleep(3000) 3 }.map { thirdValue =\u0026gt; oneValue + twoValue + thirdValue } } } } ... ... The first Future returns a 1 after 1 second, the second Future returns a 2 after 2 seconds and the third Future returns a 3 after 3 seconds. The nested block finally calculates the sum of three values and returns one single Future[Int].\nTestcase For the sake of calculating the time taken to compute the values, we have a small utility function (inside the ConcurrentUtils trait) called timed which calculates and prints the time taken by a block.\nWe Await.result to do a blocking wait on the result of futureCombinators.sumOfThreeNumbersSequentialMap. We also time the total execution and print it.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 class FutureCombinatorsTest extends FunSpec with Matchers with ConcurrentUtils { describe(\u0026#34;Futures\u0026#34;) { it(\u0026#34;could be composed using map\u0026#34;) { val futureCombinators = new FutureCombinators val result = timed(Await.result(futureCombinators.sumOfThreeNumbersSequentialMap(), 7 seconds)) result shouldBe 6 } } ... ... } trait ConcurrentUtils { def timed[T](block: =\u0026gt; T): T = { val start = System.currentTimeMillis() val result = block val duration = System.currentTimeMillis() - start println(s\u0026#34;Time taken : $duration\u0026#34;) result } } Output\n1 Time taken : 6049 The function took a little over 6 seconds to execute which indicates that the Futures are executed in sequence.\nUsing for-comprehension syntactic sugar instead of Map Scala gives a great way to work with classes that has map and flatMap (Monads) - for comprehensions. For comprehensions are just syntactic sugar, which gets de-sugared into flatMap and map.\nThe following code means exactly the same as the above, except that the uglification is done by the Scala compiler.\nCode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 def sumOfThreeNumbersSequentialForComprehension(): Future[Int] = { for { localOne \u0026lt;- Future { Thread.sleep(1000) 1 } localTwo \u0026lt;- Future { Thread.sleep(2000) 2 } localThree \u0026lt;- Future { Thread.sleep(3000) 3 } } yield localOne + localTwo + localThree } Testcase It\u0026rsquo;s the same as above.\n1 2 3 4 5 it(\u0026#34;could be composed using for comprehensions\u0026#34;) { val futureCombinators = new FutureCombinators val result = timed(Await.result(futureCombinators.sumOfThreeNumbersSequentialForComprehension(), 7 seconds)) result shouldBe 6 } Output\n1 Time taken : 6012 Executing Futures in parallel As we saw, the previous block of code runs the three Futures sequentially and therefore takes a total of 6 seconds to finish the computation. That isn\u0026rsquo;t good. Our Futures need to run in parallel. In order to achieve this, all we need to do is to extract the Future block out and declare them separately.\nCode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 val oneFuture: Future[Int] = Future { Thread.sleep(1000) 1 } val twoFuture: Future[Int] = Future { Thread.sleep(2000) 2 } val threeFuture: Future[Int] = Future { Thread.sleep(3000) 3 } Now, let\u0026rsquo;s use for-comprehension to calculate the value.\n1 2 3 4 5 def sumOfThreeNumbersParallelMapForComprehension(): Future[Int] = for { oneValue \u0026lt;- oneFuture twoValue \u0026lt;- twoFuture threeValue \u0026lt;- threeFuture } yield oneValue + twoValue + threeValue Testcase Let\u0026rsquo;s time the computation and assert the correct value using the following testcase.\n1 2 3 4 5 6 7 describe(\u0026#34;Futures that are executed in parallel\u0026#34;) { it(\u0026#34;could be composed using for comprehensions\u0026#34;) { val futureCombinators = new FutureCombinators val result = timed(Await.result(futureCombinators.sumOfThreeNumbersParallel(), 4 seconds)) result shouldBe 6 } } Output\n1 Time taken : 3005 As we see, that sumOfThreeNumbersParallel takes almost the same time as the longest Future (threeFuture), which is 3 seconds.\nJust for the sake of comparison, the above code could be written without using for-comprehension as :\n1 2 3 4 5 6 7 8 def sumOfThreeNumbersParallelMap(): Future[Int] = oneFuture.flatMap { oneValue =\u0026gt; twoFuture.flatMap { twoValue =\u0026gt; threeFuture.map { threeValue =\u0026gt; oneValue + twoValue + threeValue } } } Guards in for-comprehensions Just like we add a guarded if clause in for-comprehensions over List and other collections (aka other Monadic types), we could add guards against the generators of Future as well. The following if guard checks if the value returned by the twoFuture is more than 1, which it is.\n1 2 3 4 5 def sumOfThreeNumbersParallelWithGuard(): Future[Int] = for { oneValue \u0026lt;- oneFuture twoValue \u0026lt;- twoFuture if twoValue \u0026gt; 1 threeValue \u0026lt;- threeFuture } yield oneValue + twoValue + threeValue These guards get de-sugared as withFilter like so (I am 90% sure nobody wants to write this way):\n1 2 3 4 5 6 7 def sumOfThreeNumbersMapAndFlatMapWithFilter(): Future[Int] = oneFuture.flatMap { oneValue =\u0026gt; twoFuture.withFilter(_ \u0026gt; 1).flatMap { twoValue =\u0026gt; threeFuture.map { threeValue =\u0026gt; oneValue + twoValue + threeValue } } } Guards in for-comprehensions - Failure case If the guard evaluates to false thereby the generator yielding a failure, a NoSuchElementException would be thrown. Let\u0026rsquo;s change the guard condition to evaluate to false.\n1 2 3 4 5 def sumOfThreeNumbersParallelWithGuardAndFailure(): Future[Int] = for { oneValue \u0026lt;- oneFuture twoValue \u0026lt;- twoFuture if twoValue \u0026gt; 2 threeValue \u0026lt;- threeFuture } yield oneValue + twoValue + threeValue Output\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Future.filter predicate is not satisfied java.util.NoSuchElementException: Future.filter predicate is not satisfied at scala.concurrent.Future$$anonfun$filter$1.apply(Future.scala:280) at scala.util.Success$$anonfun$map$1.apply(Try.scala:237) at scala.util.Try$.apply(Try.scala:192) at scala.util.Success.map(Try.scala:237) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Exception handling Just like the NoSuchElementException thrown by the guard, code executing asynchronously inside a Future could throw a variety of exceptions. While one may argue that Exceptions are not very FP-like, chances are that with a distributed application or through usage of Java libraries inside your Future, exceptions do happen.\nCode Both the two functions below throw Exceptions - the first one throws a NoSuchElementException and the second one throws a LegacyException.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 //NoSuchElementException def throwsNoSuchElementIfGuardFails(): Future[Int] = for { oneValue \u0026lt;- oneFuture twoValue \u0026lt;- twoFuture if twoValue \u0026gt; 2 threeValue \u0026lt;- threeFuture } yield oneValue + twoValue + threeValue //LegacyException val futureCallingLegacyCode: Future[Int] = Future { Thread.sleep(1000) throw new LegacyException(\u0026#34;Danger! Danger!\u0026#34;) } def throwsExceptionFromComputation(): Future[Int] = for { oneValue \u0026lt;- oneFuture futureThrowingException \u0026lt;- futureCallingLegacyCode } yield oneValue + futureThrowingException case class LegacyException(msg: String) extends Exception(msg) Testcases 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 describe(\u0026#34;Futures that throw exception\u0026#34;) { it(\u0026#34;could blow up on the caller code when guard fails\u0026#34;) { val futureCombinators = new FutureCombinators intercept[NoSuchElementException] { val result = timed(Await.result(futureCombinators.throwsNoSuchElementIfGuardFails(), 4 seconds)) } } it(\u0026#34;could blow up on the caller code when exception comes from a computation executed inside the Future\u0026#34;) { val futureCombinators = new FutureCombinators intercept[LegacyException] { val result = timed(Await.result(futureCombinators.throwsExceptionFromComputation(), 4 seconds)) } } ... ... Note that even if one of the Futures result in an exception, the entire result of the composed computation will result in propagating the exception.\nRecovering from Exception : Using recover If a Future throws a scala.util.control.NonFatal and we would want to have a default fallback value instead of propagating the error to the caller, we could use the recover function. The recover is much like the catch block.\nLet\u0026rsquo;s modify the above function throwsExceptionFromComputation which throws a LegacyException. The recover function accepts a PartialFunction that maps from Throwable to the type that the Future wraps.\nCode In the below code, if the futureCallingLegacyCode throws an Exception (which it does), the value that is the result of this computation is set to be 200. If it hadn\u0026rsquo;t thrown an Exception, the resulting value would be the result of that computation itself.\n1 2 3 4 5 6 7 8 val futureCallingLegacyCodeWithRecover: Future[Int] = futureCallingLegacyCode.recover { case LegacyException(msg) =\u0026gt; 200 } def recoversFromExceptionUsingRecover(): Future[Int] = for { oneValue \u0026lt;- oneFuture futureThrowingException \u0026lt;- futureCallingLegacyCodeWithRecover } yield oneValue + futureThrowingException To reiterate, if the original Future yields a successful value, the recover block is never executed. Also, if the PartialFunction inside the recover function does not handle the exception that is originally thrown, the exception gets propagated to the caller.\nTestcase The testcase asserts that the result of the computation is the sum of values that are returned by oneFuture (which is 1) and the futureCallingLegacyCodeWithRecover (which is 200).\n1 2 3 4 5 it(\u0026#34;could be recovered with a recovery value\u0026#34;) { val futureCombinators = new FutureCombinators val result = timed(Await.result(futureCombinators.recoversFromExceptionUsingRecover(), 2 seconds)) result shouldBe 201 } Output\n1 Time taken : 1004 Using recoverWith Instead of recovering with a value when a Future results in an Exception, we might want to recover with a result of some other Future in some circumstances. Say, unavailability of a HTTP call to Server1 due to network failure could be recovered with a HTTP call to another service running on Server2.\nSimilar to recover, the recoverWith accepts a PartialFunction. However, the PartialFunction maps a Throwable to a Future of the same type as the original Future.\nJust like recover, if the main Future on which the recoverWith is called fails, then the Future that is mapped to in the PartialFunction gets called. If the second future results in a successful value, then the new result is returned.\nCode 1 2 3 4 5 6 7 8 9 10 11 val futureCallingLegacyCodeWithRecoverWith: Future[Int] = futureCallingLegacyCode.recoverWith { case LegacyException(msg) =\u0026gt; println(\u0026#34;Exception occurred. Recovering with a Future that wraps 1000\u0026#34;) Thread.sleep(2000) Future(1000) } def recoversFromExceptionUsingRecoverWith(): Future[Int] = for { oneValue \u0026lt;- oneFuture futureThrowingException \u0026lt;- futureCallingLegacyCodeWithRecoverWith } yield oneValue + futureThrowingException Testcase The oneFuture takes 1 second and the recovering Future takes 2 seconds. So, we set the Await.result timeout to 4 seconds. The final result 1001 is the sum of the result of oneFuture and futureCallingLegacyCodeWithRecoverWith.\n1 2 3 4 5 it(\u0026#34;could be recovered with a recovery Future\u0026#34;) { val futureCombinators = new FutureCombinators val result = timed(Await.result(futureCombinators.recoversFromExceptionUsingRecoverWith(), 4 seconds)) result shouldBe 1001 } Output\n1 Time taken : 3006 Note that just like the recover, if the second future also fails, then the error thrown by the second future gets propagated to the caller.\nCode In the following code, we create another Future that throws an Exception with message Dieded!! and we recover the first Future with this error-throwing-Future. The testcase would reveal that the exception from the second future (recovery one) gets thrown back to the caller.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 val anotherErrorThrowingFuture: Future[Int] = Future { Thread.sleep(1000) throw new LegacyException(\u0026#34;Dieded!!\u0026#34;) } val futureRecoveringWithAnotherErrorThrowingFuture: Future[Int] = futureCallingLegacyCode.recoverWith { case LegacyException(msg) =\u0026gt; anotherErrorThrowingFuture } def recoversFromExceptionUsingRecoverWithThatFails(): Future[Int] = for { oneValue \u0026lt;- oneFuture futureThrowingException \u0026lt;- futureRecoveringWithAnotherErrorThrowingFuture } yield oneValue + futureThrowingException Testcase 1 2 3 4 5 6 7 it(\u0026#34;when recovered with another Future that throws Exception would throw the error from the second Future\u0026#34;) { val futureCombinators = new FutureCombinators val exception = intercept[LegacyException] { timed(Await.result(futureCombinators.recoversFromExceptionUsingRecoverWithThatFails(), 4 seconds)) } exception.msg shouldBe \u0026#34;Dieded!!\u0026#34; } Using fallbackTo : fallbackTo works just like recoverWith when it comes to successful value. It uses the first Future\u0026rsquo;s value if it is successful or falls back to the second Future\u0026rsquo;s value. However, if both the first and the second Future fails, then the error that is propagated to the caller is that of the first Future and not the second Future.\nCode Let\u0026rsquo;s use the same Futures that we used in the recoverWith.\n1 2 3 4 5 6 7 val futureFallingBackToAnotherErrorThrowingFuture: Future[Int] = futureCallingLegacyCode.fallbackTo (anotherErrorThrowingFuture) def recoversFromExceptionUsingFallbackTo(): Future[Int] = for { oneValue \u0026lt;- oneFuture futureThrowingException \u0026lt;- futureFallingBackToAnotherErrorThrowingFuture } yield oneValue + futureThrowingException Notice that the fallbackTo function just accepts another Future and not a PartialFunction like recoverWith.\nTestcase 1 2 3 4 5 6 7 8 it(\u0026#34;when fallen back to another Future that throws Exception would throw the error from the first Future\u0026#34;) { val futureCombinators = new FutureCombinators val exception = intercept[LegacyException] { timed(Await.result(futureCombinators.recoversFromExceptionUsingFallbackTo(), 4 seconds)) } exception.msg shouldBe \u0026#34;Danger! Danger!\u0026#34; } Other interesting and useful combinators The following is a super-brief list of other Future combinators which I find very useful.\nzip zip works just like List.zip. It just merges two Futures and yields a Future of a Tuple.\n1 def zipTwoFutures:Future[(Int,Int)]=oneFuture zip twoFuture firstCompletedOf Ah! The firstCompletedOf comes in really handy when you have two equivalent services and you want to proceed once the fastest service returns a value.\n1 2 val listOfFutures=List(oneFuture,twoFuture,threeFuture) def getFirstResult():Future[Int]=Future.firstCompletedOf(listOfFutures) In the above case, the oneFuture returns the fastest.\nsequence The sequence is pure magic. Say, you have a List[Future[Int]] just like List(oneFuture,twoFuture,threeFuture) and you require that all the values are given back to you as a List[Int] instead of each Int wrapped inside a Future. The sequence takes your List[Future[Int]] and transforms into a Future[List[Int]]\n1 def getResultsAsList():Future[List[Int]]=Future.sequence(listOfFutures) The last time I used was for batching where I executed logic against chunks of data in parallel and combined them together with sequence.\nScala-async library The Scala Async library is an external project and could be added to the project by adding the dependency into our build.sbt\n1 \u0026#34;org.scala-lang.modules\u0026#34; %% \u0026#34;scala-async\u0026#34; % \u0026#34;0.9.6-RC2\u0026#34; The Async library has just two powerful functions in its scala.async.Async class - async and await.\nasync The async function is very similar to the Future.apply function. In fact their signatures are very much the same and we could comfortably replace the Future.apply with async wherever it is available.\nFuture.apply\n1 2 def apply[T](body: =\u0026gt;T)(implicit executor: ExecutionContext): Future[T] Async\n1 2 def async[T](body: =\u0026gt; T)(implicit execContext: ExecutionContext): Future[T] The foremost advantage of using async over Future.apply, other than the general ease of readability, is that for each Future generator (when used with a for comprehension), the compiler yields a separate anonymous class while with async it is just one single anonymous class.\nTherefore, we could re-write our oneFuture as,\n1 2 3 4 val oneFuture: Future[Int] = async { Thread.sleep(1000) 1 } await The await function accepts a Future and returns the result. But isn\u0026rsquo;t the same as Await.result which accepts a Future and returns the result as well? Nope. The key difference is that the Await.result is blocking and is strongly discouraged to be used in production code except for testcases. The await function, on the other hand is implemented using Scala macros and the implementation is that it returns the result of the Future using the onComplete callback.\nSince the async function returns a Future, all other error handling and recovery mechanisms stays the same as before.\nCode Let\u0026rsquo;s rewrite the previous sum-of-three-numbers with async/await :\n1 2 3 def sumOfThreeNumbersParallelWithAsyncAwait(): Future[Int] = async { await(oneFuture) + await(twoFuture) + await(threeFuture) } Testcase 1 2 3 4 5 it(\u0026#34;could be composed using async/await\u0026#34;) { val futureCombinators = new FutureCombinators val result = timed(Await.result(futureCombinators.sumOfThreeNumbersParallelWithAsyncAwait(), 4 seconds)) result shouldBe 6 } As we see, the code written this way is not only asynchronous but also looks natural (in fact, it looks synchronous). We could argue that for-comprehensions are a huge leap from using map and flatMap but async/await goes one big step further.\nCode The code and its corresponding testcase are on github.\n","date":"2016-06-12T14:26:28Z","permalink":"/scala-futures-3-combinators-and-async/","title":"Scala notes - Futures - 3 (Combinators and Async)"},{"content":"In the last post, we saw how to extract values from the Future upon onComplete and their counterparts - onSuccess and onFailure. We also saw how to use Await.result in Testcases to block and get the value from Future. In this post, we\u0026rsquo;ll discuss briefly about the relationship between a Promise and a Future.\nPromise The concepts Promise and a Future go hand in hand. A scala.concurrent.Promise is the one which sets a value for the Future. In other words, the Promise is the brain behind executing the computation asynchronously and Future is just a handle for reading the result when it becomes available. Crudely put, the Promise is the setter and the Future is the getter.\nMost often, we won\u0026rsquo;t need to explicitly create a Promise. However, we are required to understand what a Promise is, in order to truly understand how a Future works.\nLet\u0026rsquo;s use the following examples to understand how to create a Promise.\nCompleting a Promise In the following piece of code, we will see how a value is set in a promise and how it is read on the other side.\nWe\ncreate a Promise complete the Promise by setting a successful value then return the read side of the Promise - the Future back to the caller by using the promise.future There is no time consuming process happening behind the scenes. The value is set to the Promise immediately and therefore the value is immediately available via the Future.\nCode 1 2 3 4 5 6 7 8 9 class PromiseInternals { ... def aCompletedPromiseUsingSuccess(num:Int): Future[Int] = { val promise=Promise[Int]() promise.success(num) promise.future } ... ... Testcase When we run the Testcase code, the onComplete callback gets called immediately after the promise.success(100) is called.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 class PromiseInternalsTest extends FunSpec with Matchers { describe(\u0026#34;A Future\u0026#34;) { it(\u0026#34;gives out the correct value when a Promise is completed\u0026#34;) { val promiseInternals = new PromiseInternals val aCompletedPromise=promiseInternals.aCompletedPromiseUsingSuccess(100) assertValue(aCompletedPromise, 100) } ... ... def assertValueUsingOnComplete(future: Future[Int], expectedValue: Int): Unit = { future.onComplete { case Success(result) =\u0026gt; { println (s\u0026#34;Result is $result and expectedValue is $expectedValue\u0026#34;) result shouldBe expectedValue } case Failure (msg) =\u0026gt; fail(msg) } } The promise.success is just a shortcut for using promise.complete which accepts a Try[T] as an argument. So, we could have actually written the above function as :\n1 2 3 4 5 def aCompletedPromiseUsingComplete(num:Int): Future[Int] = { val promise=Promise[Int]() promise.complete(Success(num)) promise.future } Alternatively, if we would like to indicate a failure in computation, we could either use a promise.complete(Failure(throwable)) or\n1 2 3 4 5 def aCompletedPromiseUsingFailure(num:Int): Future[Int] = { val promise=Promise[Int]() promise.failure(new RuntimeException(\u0026#34;Evil Exception\u0026#34;)) promise.future } Let\u0026rsquo;s summarize the above in a picture :\nRunning a block asynchronously Now that we saw how to complete a Promise by setting a successful value or an exception, we\u0026rsquo;ll see how to execute a block of code asynchronously.\nIn the following Testcase, we pass a block of code to the someExternalDelayedCalculation to be executed asynchronously.\nTestcase Let\u0026rsquo;s look at the testcase first.\nWe pass a block as argument. The block of code simply sleeps for 2 seconds and then returns a 100. Assert the value after 3 seconds. Simple enough.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 it(\u0026#34;gives out the correct value when an asynchronous block is submitted and is completed through a Promise\u0026#34;) { val promiseInternals = new PromiseInternals val longCalculationFuture = promiseInternals.someExternalDelayedCalculation{()=\u0026gt; Thread.sleep(2000) 100 } println (s\u0026#34;We have submitted a block to be executed asynchronously ${longCalculationFuture.isCompleted}\u0026#34;) //false at this point assertValue(longCalculationFuture, 100) } def assertValue(future: Future[Int], expectedValue: Int): Unit = { val resultVal=Await.result(future, 3000 seconds) resultVal shouldBe expectedValue } Code The implementation of the someExternalDelayedCalculation is interesting :\nWe\ncreate a FixedThreadPool to execute our asynchronous code. create a Promise create a Runnable and wrap the block to be run asynchronously in the run method close the promise and complete the promise using the result of the run execute the Runnable in the somePool threadpool. return the promise.future from which the caller can read the value. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 val somePool=Executors.newFixedThreadPool(2) def someExternalDelayedCalculation(f:()=\u0026gt;Int): Future[Int] = { val promise=Promise[Int]() val thisIsWhereWeCallSomeExternalComputation = new Runnable { override def run(): Unit ={ promise.complete{ try(Success(f())) catch { case NonFatal (msg)=\u0026gt; Failure(msg) } } } } somePool.execute(thisIsWhereWeCallSomeExternalComputation) promise.future } That\u0026rsquo;s it !!\nHow is the Future.apply() actually implemented? Well, I cheated. The code in bullet 2 is actually stolen from the actual implementation of the Future.apply itself.\nRemember in the previous post, we saw that when a block of code is passed into the Future\u0026rsquo;s apply function, it gets executed asynchronously.\nNow, compare the code above in someExternalDelayedCalculation with the actual implementation of Future.apply and the Runnable that it wraps.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 def apply[T](body: =\u0026gt;T)(implicit executor: ExecutionContext): scala.concurrent.Future[T] = { val runnable = new PromiseCompletingRunnable(body) executor.prepare.execute(runnable) runnable.promise.future } class PromiseCompletingRunnable[T](body: =\u0026gt; T) extends Runnable { val promise = new Promise.DefaultPromise[T]() override def run() = { promise complete { try Success(body) catch { case NonFatal(e) =\u0026gt; Failure(e) } } } } To repeat the same steps as above, the apply function\nholds the ThreadPool that we provide as the implicit ExecutionContext creates a Promise by creating a PromiseCompletingRunnable that is a Runnable wraps the block to be run asynchronously in the run method closes the promise and completes the promise using the result of the run executes the Runnable using the ExecutionContext returns the promise.future from which the caller can read the value. Once written, twice error Once the promise gets completed either with a Success or a Failure, all we could do after that is to extract the value from its Future. Also, the onComplete callback of the Future gets called. The value wrapped inside the Future of the Promise is set in stone and cannot be changed.\nIf we attempt to set a new value by completing an already completed Promise, an IllegalStateException is thrown.\nCode Let\u0026rsquo;s look at this using a snippet. In the following code, we create a Promise and complete it with a value of 100. We then attempt to complete it with a failure.\n1 2 3 4 5 6 def alreadyCompletedPromise(): Future[Int] = { val promise = Promise[Int]() promise.success(100) //completed promise.failure(new RuntimeException(\u0026#34;Will never be set because an IllegalStateException will be thrown beforehand\u0026#34;)) promise.future } Testcase The testcase just asserts that the IllegalStateException gets thrown when an attempt to complete the Promise with a Failure.\n1 2 3 4 5 6 it(\u0026#34;should throw an error if a Promise is attempted to be completed more than once\u0026#34;) { val promiseInternals = new PromiseInternals intercept[IllegalStateException] { promiseInternals.alreadyCompletedPromise() } } Code The complete code backing this blog is available in github\n","date":"2016-06-05T14:29:24Z","permalink":"/scala-notes-futures-2-promises/","title":"Scala notes - Futures - 2 (Promises)"},{"content":"Almost all modern programming languages have a Future-Promise idiom for concurrent programming. I don\u0026rsquo;t intend to bore you with why we need higher level of concurrency abstractions. Instead, in this post, we\u0026rsquo;ll cut to the chase and discuss only about Scala\u0026rsquo;s approach to Futures.\nA scala.concurrent.Future is a representation of a value that is yet to be realized. This value is generally a result of a longer and/or parallel computation.\nIn crude terms, a block of code which runs synchronously, when wrapped in a Future runs asynchronously. All implementations of Future gives us a handle through which we could can get retrieve the resulting value of the computation of the block (or what\u0026rsquo;s the point in it!).\nIn this post, we\u0026rsquo;ll look at the basics of how to construct a Future and extract the value out of it through blocking wait and callbacks. In the next part, we\u0026rsquo;ll talk about composing Futures and other advanced constructs such as recover and fallback for exception handling.\nCreating an asynchronous computation Creating a computation that runs asynchronously using Future is super easy. We just need to throw in our logic into the apply function of the Future\n1 2 3 val aFuture: Future[Int] = Future { //Some massively huge super important computation } As an example, let\u0026rsquo;s create a oneFuture that returns a 1 after a delay of one second.\n1 2 3 4 val oneFuture: Future[Int] = Future { Thread.sleep(1000) 1 } Let\u0026rsquo;s pause and explore the apply function\n1 def apply[T](body: =\u0026gt;T)(implicit executor: ExecutionContext): scala.concurrent.Future[T] Yup. You got me there. The method also accepts something called an ExecutionContext as an implicit argument. When a block of code is used to construct a Future, the computation runs on an ExecutionContext. The ExecutionContext is synonymous with a ThreadPool and when a Future is started, it runs on a separate Thread.\nScala provides a ready-made static global ExecutionContext in scala.concurrent.ExecutionContext.global which we plan to use now. (More on this in a moment)\nSo, the entire code actually looks like this\n1 2 3 4 5 6 7 8 9 10 class PromisingFutures{ import scala.concurrent.ExecutionContext.Implicits.global val oneFuture: Future[Int] = Future { Thread.sleep(1000) 1 } ... ... On using scala.concurrent.ExecutionContext.global The global ExecutionContext is very convenient to use. However, the underlying ThreadPool is a ForkJoinPool. Nothing wrong with this and ForkJoinPool is amazing for short-lived computations but is strongly not recommended for Blocking IO such as database or webservice calls (or for that sake even long running computations or simply talking outside the JVM).\nThe good news is that we still can use Future but the workaround is to simply use a separate threadpool that is not ForkJoinPool - a fixed ThreadPool, for example, is a good option.\n1 2 3 4 implicit lazy val fixedThreadPoolExecutionContext: ExecutionContext = { val fixedThreadPool: ExecutorService = Executors.newFixedThreadPool(Runtime.getRuntime.availableProcessors * 2) //or some fixed number ExecutionContext.fromExecutor(fixedThreadPool) } Future States and values Before we go into the extracting the values out of Future, let\u0026rsquo;s see what are the various states of Future and when and what value is actually available during those states.\nThe Future just has two states - Not Completed and Completed.\nConsider the following image\nWhen given a computation, the resulting value that we could get from the Future is an Option[Try[T]].\nIn Not Complete state, the result of the computation is not yet realized. The value therefore would be a None. Once Completed, the result is a Some(Try[T]) which means it could be one of the following : A positive outcome of the computation or An Exception Let\u0026rsquo;s look at the Success case with an example\n1 2 3 4 5 6 7 8 9 10 11 def checkState(): Unit = { println(\u0026#34;Before the job finishes\u0026#34;) Thread.sleep(500) println(s\u0026#34;Completed : ${oneFuture.isCompleted}, Value : ${oneFuture.value}\u0026#34;) println(\u0026#34;After the job finishes\u0026#34;) Thread.sleep(1100) println(s\u0026#34;Completed : ${oneFuture.isCompleted}, Value : ${oneFuture.value}\u0026#34;) } The time taken for the delayedStringFuture Future to complete is 1 seconds. So, we first check after 500 milliseconds, whether the Future is complete and print its current value. The value function of the Future returns an Option[Try[T]]. Not surprisingly, we get a false for the isCompleted check and a None as the value itself.\nOutput for 500 ms:\n1 2 Before the job finishes Completed : false, Value : None Let\u0026rsquo;s check back again after 1100 milliseconds giving some leeway after that 1000 millisecond sleep. The output now is the result of the computation itself and the completion status is true now..\nOutput for 1100 ms:\n1 2 After the job finishes Completed : true, Value : Some(Success(1)) Now that we\u0026rsquo;ve got that out of our way, let\u0026rsquo;s see how to extract a value out of a Future\nExtracting value out of Future Other than composing futures, which we\u0026rsquo;ll be seeing next week, there are two ways to extract just the value out of a Future - Blocking wait and callback\nBlocking wait using Await.result : The scala.concurrent.Await\u0026rsquo;s result function has the following syntax :\n1 2 3 @throws(classOf[Exception]) def result[T](awaitable: Awaitable[T], atMost: Duration): T = blocking(awaitable.result(atMost)(AwaitPermission)) It accepts an implementation of the Awaitable trait, which a Future is. It also accepts a second parameter atMost which indicates the maximum time duration the caller thread has to block for the result. Upon expiry of the atMost duration, if the Future still didn\u0026rsquo;t complete, then a java.util.concurrent.TimeoutException would be thrown.\nIf the Future is complete, Await.result will extract us the actual value. If the Future is complete and if the Future\u0026rsquo;s result is a Throwable, the Exception gets propagated to the caller.\nUsing Await.result in production code is highly discouraged but this construct comes in handy for running testcases against Future.\nLet\u0026rsquo;s consider two Futures, the original oneFuture and a oneDangerousFuture which throws an Exception.\nCode Here\u0026rsquo;s how they look :\n1 2 3 4 5 6 7 8 9 10 11 val oneFuture: Future[Int] = Future { Thread.sleep(1000) 1 } val oneDangerousFuture=Future{ Thread.sleep(2000) throw new SomeComputationException(\u0026#34;Welcome to the Dark side !\u0026#34;) } case class SomeComputationException(msg: String) extends Exception(msg) Testcases We have three testcases here :\nThe first one is our happy day scenario - The computation takes 1 second, we wait for result for a maximum duration of 2 seconds. We obviously will have our result in hand. Our assertion that a value should be returned becomes true. In the second testcase, we spawn a Future that throws a SomeComputationException and we assert that the exception gets propagated to the caller when we await for the result. In the last testcase, we wait only for 500 milliseconds while the computation itself takes 1 second. As we saw from the implementation of the Await.result, this call throws a TimeoutException. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 class PromisingFutureTest extends FunSpec with Matchers { describe(\u0026#34;A PromisingFuture\u0026#34;) { it(\u0026#34;should hold a Int value if the Await.result is called after the Future completes\u0026#34;) { val promisingFuture = new PromisingFutures() val oneFuture = promisingFuture.oneFuture //Takes 1 second to compute val intValue = Await.result(oneFuture, 2 seconds) intValue should be(1) } it(\u0026#34;should propagate the Exception to the callee if the computation threw an exception\u0026#34;) { val promisingFuture = new PromisingFutures() val oneDangerousFuture = promisingFuture.oneDangerousFuture //throws exception intercept[SomeComputationException] { val intValue = Await.result(oneDangerousFuture, 2 seconds) } } it(\u0026#34;should throw a TimeOutException exception when an Await.result\u0026#39;s atMost parameter is lesser than the time taken for the Future to complete\u0026#34;) { val promisingFuture = new PromisingFutures() val oneDelayedFuture = promisingFuture.oneFuture //Takes 1 second to compute intercept[TimeoutException] { Await.result(oneDelayedFuture, 500 millis) } } } } Callback : The alternative and the clean way to extract a value from the Future (other than composing) is by way of callbacks. There are three different callbacks available on the Future - the onSuccess, the onFailure and the combined onComplete.\nThe onSuccess callback gets called only when the Future completes successfully with a result. The onFailure callback gets called only when there is an Exception. The onComplete is a combination of onSuccess and onFailure. It accepts a function that works on a Try[T] after getting the Option around the Future\u0026rsquo;s result unwrapped. 1 def onComplete[U](f: Try[T] =\u0026gt; U)(implicit executor: ExecutionContext): Unit Note that all callbacks return a Unit which means that they can\u0026rsquo;t be composed and are side-effecting.\nNow let\u0026rsquo;s see how we could use the onComplete callback. I have this little method called printFuture which just writes to the console the contents of the Future once it is complete. Let\u0026rsquo;s try to pass in both the oneFuture and the oneDangerousFuture into it.\n1 2 3 4 5 6 7 8 class PromisingFutures { ... ... def printFuture[T](future: Future[T]): Unit = future.onComplete { case Success(result) =\u0026gt; println(s\u0026#34;Success $result\u0026#34;) case Failure(throwable) =\u0026gt; println(s\u0026#34;Failure $throwable\u0026#34;) } ... 1 2 3 4 5 6 7 8 9 10 11 object PromisingFutures{ def main(args: Array[String]) { val promisingFutures=new PromisingFutures promisingFutures.printFuture(promisingFutures.oneFuture) promisingFutures.printFuture(promisingFutures.oneDangerousFuture) synchronized(wait(3000)) } } Output :\n1 2 Success 1 Failure SomeComputationException: Welcome to the Dark side ! As expected, the oneFuture goes into the Success case and yields 1 while the oneDangerousFuture goes into the Failure case and prints the Exception :\nTimeUnit Scala provides a very convenient DSL-like syntax to represent TimeUnit eg. 2 seconds, 5 minutes etc. Three implicit classes scala.concurrent.duration package - DurationInt, DurationLong and DurationDouble and the trait DurationConversions do this magic. All we need to do is to import scala.concurrent.duration._.\nAlso, the \u0026ldquo;500 millis\u0026rdquo; in our example could be represented as \u0026ldquo;500 milli\u0026rdquo;, \u0026ldquo;500 milliseconds\u0026rdquo; or \u0026ldquo;500 millisecond\u0026rdquo;. The entire list of aliases for various time units could be found in the sourcecode or Scaladoc of the trait DurationConversions.\nCode The complete code backing this blog is available in github\nAddendum @lightspeed7 pointed out that with Scala 2.12, the partial versions of onComplete, namely onSuccess and onFailure is deprecated. The deprecation is primarily to favor total versions such as onComplete and foreach in place of partials that handles either Success or Failure cases such as onSuccess and onFailure.\nLet\u0026rsquo;s explore a little more on Future.foreach with the following printing function.\n1 def printWithForEach[T](future: Future[T]): Unit = future.foreach(println) With foreach, the Future.foreach delegates itself to the Try.foreach. So, if the Future returns a Success, a Success(value) would be printed but for a Failure, nothing would be printed simply because the implementation of Failure goes like this :\n1 def foreach[U](f: T =\u0026gt; U): Unit = () ","date":"2016-05-29T14:21:18Z","permalink":"/scala-notes-futures-1/","title":"Scala Notes - Futures - 1"},{"content":"In the first part of notes on Akka FSM, we saw the basics of Akka FSM and the outline of the Coffee vending machine that we planned to build - the structure of the Actor and a list of messages we pass to the Actor. In this second and final part, we will go ahead and implement each of these States.\nRecap As a quick recap, let\u0026rsquo;s look at the structure of the FSM and the messages that can be sent to it.\nStates and Data The three States of the FSM and the Data being sent across the States are :\n1 2 3 4 5 6 7 8 9 10 11 object CoffeeMachine { sealed trait MachineState case object Open extends MachineState case object ReadyToBuy extends MachineState case object PoweredOff extends MachineState case class MachineData(currentTxTotal: Int, costOfCoffee: Int, coffeesLeft: Int) } Messages The Vendor and User Interaction messages that we send to the FSM are :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 object CoffeeProtocol { trait UserInteraction trait VendorInteraction case class Deposit(value: Int) extends UserInteraction case class Balance(value: Int) extends UserInteraction case object Cancel extends UserInteraction case object BrewCoffee extends UserInteraction case object GetCostOfCoffee extends UserInteraction case object ShutDownMachine extends VendorInteraction case object StartUpMachine extends VendorInteraction case class SetNumberOfCoffee(quantity: Int) extends VendorInteraction case class SetCostOfCoffee(price: Int) extends VendorInteraction case object GetNumberOfCoffee extends VendorInteraction case class MachineError(errorMsg:String) } Structure of FSM Actor Here\u0026rsquo;s the overall structure that we saw in Part 1\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 class CoffeeMachine extends FSM[MachineState, MachineData] { //What State and Data must this FSM start with (duh!) startWith(Open, MachineData(..)) //Handlers of State when(Open) { ... ... when(ReadyToBuy) { ... ... when(PoweredOff) { ... ... //fallback handler when an Event is unhandled by none of the States. whenUnhandled { ... ... //Do we need to do something when there is a State change? onTransition { case Open -\u0026gt; ReadyToBuy =\u0026gt; ... ... ... } Initial State As with any State Machine, an FSM needs an initial state to start with. This can be declared inside Akka FSM in a very intuitive method named startWith. The startWith accepts two arguments - the initial state and the initial data.\n1 2 3 4 5 6 7 class CoffeeMachine extends FSM[MachineState, MachineData] { startWith(Open, MachineData(currentTxTotal = 0, costOfCoffee = 5, coffeesLeft = 10)) ... ... The above code just says that the FSM\u0026rsquo;s initial state is Open and the initial data while the Coffee machine is opened is MachineData(currentTxTotal = 0, costOfCoffee = 5, coffeesLeft = 10).\nSince the machine has just started, the vending machine starts with a clean slate. It had no interaction with any user yet and therefore the current showing balance for this transaction is 0. The price of the coffee is set at $5 and the total number of coffees that the machine could vend in total is 10. Once the coffees has vended 10 coffees and is left with 0, the machine shuts down.\nImplementing the States Ah, Finally !!\nI felt that the easiest way to look at the interactions with the vending machine at different states is by grouping the interactions, write testcases around it and accompany that with the implementation in the FSM.\nIf you are referring to the github code, all tests are in the CoffeeSpec and the FSM is the CoffeeMachine\nAll the following tests are wrapped inside the CoffeeSpec test class whose declaration goes like :\n1 class CoffeeSpec extends TestKit(ActorSystem(\u0026#34;coffee-system\u0026#34;)) with MustMatchers with FunSpecLike with ImplicitSender Setting and Getting Price of coffee As we saw above, the MachineData is initiated at $5 dollars per coffee and with a capacity of 10 coffees. This is just an initial state. The Vendor must have the capacity to set the price of the coffee and the capacity of the machine at any point of time.\nSetting of price is achieved by sending the SetCostOfCoffee message to the Actor. We should also have the ability to get the price of coffee. This is done by using the GetCostOfCoffee message to which the Machine responds with the currently set price.\nTestcase 1 2 3 4 5 6 7 8 9 10 11 describe(\u0026#34;The Coffee Machine\u0026#34;) { it(\u0026#34;should allow setting and getting of price of coffee\u0026#34;) { val coffeeMachine = TestActorRef(Props(new CoffeeMachine())) coffeeMachine ! SetCostOfCoffee(7) coffeeMachine ! GetCostOfCoffee expectMsg(7) } ... ... ... Implementation Like we discussed in Part 1, every message that is being sent to the FSM is received and wrapped in an Event class which also wraps around the MachineData:\n1 2 3 4 5 6 7 when(Open) { case Event(SetCostOfCoffee(price), _) =\u0026gt; stay using stateData.copy(costOfCoffee = price) case Event(GetCostOfCoffee, _) =\u0026gt; sender ! (stateData.costOfCoffee); stay() ... ... } } There are a few words which are new in the above code - stay, using and stateData. Let\u0026rsquo;s look at them in detail.\nstay and goto The idea is that each of the case blocks in a state must return a State. This could either be done using stay which just means that at the end of processing this message (SetCostOfCoffee or GetCostOfCoffee), the CoffeeMachine remains in the same State, which is Open in our case.\nThe goto, on the other hand transitions to a different State. We\u0026rsquo;ll see how it is done while discussing the Deposit message.\nNot surprisingly, check out the implementation of the stay function\n1 final def stay(): State = goto(currentState.stateName) using As you might already guessed, the using function allows us to pass modified data to the next state. In case of the SetCostOfCoffee message, we set the costOfCoffee field of the MachineData to the incoming price wrapped inside the SetCostOfCoffee. Since State is a case class (immutability is strongly advised unless you have fun debugging at odd hours), we do a copy.\nstateData The stateData is just a function that gives us a handle to the data of the FSM, which is the MachineData itself. So, the following blocks of code are equivalent\n1 case Event(GetCostOfCoffee, _) =\u0026gt; sender ! (stateData.costOfCoffee); stay() 1 case Event(GetCostOfCoffee, machineData) =\u0026gt; sender ! (machineData.costOfCoffee); stay() The implementation of the maximum number of coffees to be dispensed using GetNumberOfCoffee and SetNumberOfCoffee is almost the same as the setting and getting the price itself. Let\u0026rsquo;s skip that and go for the interesting part - Buying coffee.\nBuying coffee So, the coffee enthusiast deposits money for the coffee but we can\u0026rsquo;t allow the machine to dispense coffee until he has entered the cost of a coffee. Also, if he has given extra cash, we\u0026rsquo;ll have to give him the balance. So, the various cases goes like this :\nUntil the User deposits money to cover a coffee, we\u0026rsquo;ll keep track of the cumulative amount he has deposited and stay in the Open state. Once the cumulative cash exceeds the price of a coffee, we\u0026rsquo;ll transition to the ReadyToBuy state and allow him to buy coffee. At this ReadyToBuy state, he could change his mind and Cancel the transaction during which all his cumulative Balance is returned. If he wishes to drink the coffee, he sends the machine a BrewCoffee message instead during which we dispense the coffee and give him back the Balance money as well. (Actually, in our code, we don\u0026rsquo;t dispense the coffee. We just subtract the price of the coffee from his deposit and give him the balance. Such a rip off !!) Let\u0026rsquo;s take each of the above cases\nCase 1 - User deposits cash but falls short of the price of a coffee Testcase The testcase starts by setting the cost of the coffee to the $5 and the total number of coffees in the machine to be 10. We then Deposit $2 which is less than the price of the coffee and check if the Machine is in the Open state and the total number of coffees in the machine remains at 10.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 it(\u0026#34;should stay at Transacting when the Deposit is less then the price of the coffee\u0026#34;) { val coffeeMachine = TestActorRef(Props(new CoffeeMachine())) coffeeMachine ! SetCostOfCoffee(5) coffeeMachine ! SetNumberOfCoffee(10) coffeeMachine ! SubscribeTransitionCallBack(testActor) expectMsg(CurrentState(coffeeMachine, Open)) coffeeMachine ! Deposit(2) coffeeMachine ! GetNumberOfCoffee expectMsg(10) } So, how exactly did we ensure that the machine is in the Open state?\nEach FSM can handle a special message called FSM.SubscribeTransitionCallBack(callerActorRef) which enables the caller to be notified of any State transitions. The first notification that gets sent on subscription is the CurrentState, which tells us what State the FSM is currently in. This is followed by several Transition messages when that happens.\nImplementation So, we add up deposit to the cumulative transaction total and stay in Open state waiting for more Deposit\n1 2 3 4 5 6 7 when(Open) { ... ... case Event(Deposit(value), MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) if (value + currentTxTotal) \u0026lt; stateData.costOfCoffee =\u0026gt; { val cumulativeValue = currentTxTotal + value stay using stateData.copy(currentTxTotal = cumulativeValue) } Case 2 and 4 - User deposits amount that covers the price of coffee Testcase 1 - Deposit equal to the price of coffee Our testcase bootstraps the machine, confirms if the current state is Open and then deposits $5 which is exactly the price of the coffee. We then assert that the machine has transitioned from Open to ReadyToBuy by way of expecting a Transition message which gives us information about the from and the to states of the coffee machine. In the first case, it is the transition from Open to ReadyToBuy.\nWe then go further and ask the machine to BrewCoffee during which we expect a transition again from ReadyToBuy to Open upon dispensing the coffee. Finally an assertion is made against the remaining number of coffees in the machine (which is 9 now).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 it(\u0026#34;should transition to ReadyToBuy and then Open when the Deposit is equal to the price of the coffee\u0026#34;) { val coffeeMachine = TestActorRef(Props(new CoffeeMachine())) coffeeMachine ! SetCostOfCoffee(5) coffeeMachine ! SetNumberOfCoffee(10) coffeeMachine ! SubscribeTransitionCallBack(testActor) expectMsg(CurrentState(coffeeMachine, Open)) coffeeMachine ! Deposit(5) expectMsg(Transition(coffeeMachine, Open, ReadyToBuy)) coffeeMachine ! BrewCoffee expectMsg(Transition(coffeeMachine, ReadyToBuy, Open)) coffeeMachine ! GetNumberOfCoffee expectMsg(9) } Testcase 2 - Deposit greater than the price of the coffee The second test case is 90% similar to the first testcase except that we deposit cash in increments that is greater than the price of the coffee ($6). Since we set the price of coffee to be $5, we now expect a Balance message with value $1\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 it(\u0026#34;should transition to ReadyToBuy and then Open when the Deposit is greater than the price of the coffee\u0026#34;) { val coffeeMachine = TestActorRef(Props(new CoffeeMachine())) coffeeMachine ! SetCostOfCoffee(5) coffeeMachine ! SetNumberOfCoffee(10) coffeeMachine ! SubscribeTransitionCallBack(testActor) expectMsg(CurrentState(coffeeMachine, Open)) coffeeMachine ! Deposit(2) coffeeMachine ! Deposit(2) coffeeMachine ! Deposit(2) expectMsg(Transition(coffeeMachine, Open, ReadyToBuy)) coffeeMachine ! BrewCoffee expectMsgPF(){ case Balance(value)=\u0026gt;value==1 } expectMsg(Transition(coffeeMachine, ReadyToBuy, Open)) coffeeMachine ! GetNumberOfCoffee expectMsg(9) } Implementation The implementation is much simpler than the test cases itself. If the deposit amount is greater than or equal to the cost of coffee, then we goto ReadyToBuy state using the accumulated amount.\n1 2 3 4 5 6 when(Open){ ... ... case Event(Deposit(value), MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) if (value + currentTxTotal) \u0026gt;= stateData.costOfCoffee =\u0026gt; { goto(ReadyToBuy) using stateData.copy(currentTxTotal = currentTxTotal + value) } Once transitioned to the ReadyToBuy state, when the user sends a BrewCoffee, we check if there is a balance to be dispensed. If not, we just transition to Open state after subtracting one coffee from the total number of coffees. Else, we disburse the balance and transition to Open state after subtracting the number of coffees. (Like I said earlier, we don\u0026rsquo;t actually dispense coffee in this example)\n1 2 3 4 5 6 7 8 9 10 11 when(ReadyToBuy) { case Event(BrewCoffee, MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) =\u0026gt; { val balanceToBeDispensed = currentTxTotal - costOfCoffee logger.debug(s\u0026#34;Balance is $balanceToBeDispensed\u0026#34;) if (balanceToBeDispensed \u0026gt; 0) { sender ! Balance(value = balanceToBeDispensed) goto(Open) using stateData.copy(currentTxTotal = 0, coffeesLeft = coffeesLeft - 1) } else goto(Open) using stateData.copy(currentTxTotal = 0, coffeesLeft = coffeesLeft - 1) } } That\u0026rsquo;s it !! We\u0026rsquo;ve covered the juice of the program.\nCase 3 - User wishes to Cancel the transaction Actually, the User should be in a position to Cancel the transaction at any point, whatever state he is in. Like we discussed in Part 1, the perfect place to hold these kind of generic messages is in the whenUnhandled block. We should also make sure that if the user has deposited some cash before canceling, we should give it back to them.\nImplementation 1 2 3 4 5 6 7 8 whenUnhandled { ... ... case Event(Cancel, MachineData(currentTxTotal, _, _)) =\u0026gt; { sender ! Balance(value = currentTxTotal) goto(Open) using stateData.copy(currentTxTotal = 0) } } Testcase The testcase is simply the same as the one we saw above except that the balance issued upon cancellation is the cumulative deposit.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 it(\u0026#34;should transition to Open after flushing out all the deposit when the coffee is canceled\u0026#34;) { val coffeeMachine = TestActorRef(Props(new CoffeeMachine())) coffeeMachine ! SetCostOfCoffee(5) coffeeMachine ! SetNumberOfCoffee(10) coffeeMachine ! SubscribeTransitionCallBack(testActor) expectMsg(CurrentState(coffeeMachine, Open)) coffeeMachine ! Deposit(2) coffeeMachine ! Deposit(2) coffeeMachine ! Deposit(2) expectMsg(Transition(coffeeMachine, Open, ReadyToBuy)) coffeeMachine ! Cancel expectMsgPF(){ case Balance(value)=\u0026gt;value==6 } expectMsg(Transition(coffeeMachine, ReadyToBuy, Open)) coffeeMachine ! GetNumberOfCoffee expectMsg(10) } Code I didn\u0026rsquo;t want to bore you to death and took the liberty of skipping the explanation for ShutDownMachine message and the PoweredOff state but if you are looking forward to an explanation for them, please leave a comment.\nAs always, the code is available on github.\n","date":"2016-05-22T16:09:56Z","permalink":"/akka-notes-finite-state-machines-2/","title":"Akka Notes - Finite State Machines - 2"},{"content":"I recently had the opportunity to play with Akka FSM at work for some really interesting use-case. The API (in fact, the DSL) is pretty awesome and the entire experience was amazing. Here\u0026rsquo;s my attempt to log my notes on building a Finite State Machine using Akka FSM. As an example, we\u0026rsquo;ll walk through the steps of building an (limited) Coffee vending machine.\nWhy not become and unbecome? We know that the plain vanilla Akka Actors can switch its behavior by using become/unbecome. Then, why do we need Akka FSM? Can\u0026rsquo;t a plain Actor just switch between the States and behave differently? Yes, it could. But while Akka\u0026rsquo;s become and unbecome is most often enough to switch the behavior of Actors with a couple of states involved, building a State Machine with more than a few states quickly makes the code hard to reason with (and even harder to debug).\nNot surprisingly, the popular recommendation is to switch to Akka FSM if there are more than 2 states in our Actor.\nWhat\u0026rsquo;s Akka FSM To expand on it further, Akka FSM is Akka\u0026rsquo;s approach to building Finite State Machines that simplifies management of behavior of an Actor in various states and transitions between those states.\nUnder the hood, Akka FSM is just a trait that extends Actor.\n1 2 trait FSM[S, D] extends Actor with Listeners with ActorLogging What this FSM trait provides is pure magic - it provides a DSL that wraps a regular Actor enabling us to focus on building the state machine that we have in hand, faster.\nIn other words, our regular Actor has just one receive function and the FSM trait wraps a sophisticated implementation of the receive method which delegates calls to the block of code that handles the data in a particular state.\nOne other good thing I personally noticed is that after writing, the complete FSM Actor still looks clean and easily readable.\nAlright, let\u0026rsquo;s get to the code. Like I said, we will be building a Coffee Vending Machine using Akka FSM. The State Machine looks like this :\nState and Data With any FSM, there are two things involved at any moment in a FSM - the State of the machine at any point in time and the Data that is shared among the states. In Akka FSM, in order to check which is our Data and which are the States, all we need to do is to check its declaration.\n1 2 class CoffeeMachine extends FSM[MachineState, MachineData] This simply means that all the States of the FSM extend from the MachineState and the data that is shared between these various States is just MachineData.\nAs a matter of style, just like with normal Actor where we declare all our messages in a companion object, we declare our States and Data in a companion object:\n1 2 3 4 5 6 7 8 9 10 11 object CoffeeMachine { sealed trait MachineState case object Open extends MachineState case object ReadyToBuy extends MachineState case object PoweredOff extends MachineState case class MachineData(currentTxTotal: Int, costOfCoffee: Int, coffeesLeft: Int) } So, as we captured in the State Machine diagram, we have three States - Open, ReadyToBuy and PoweredOff. Our data, the MachineData holds (in reverse order) the numbers of coffees that the vending machine could dispense before it shuts itself down (coffeesLeft), the price of each cup of coffee (costOfCoffee) and finally, the amount deposited by the vending machine user (currentTxTotal) - if it is less than the cost of the coffee, the machine doesn\u0026rsquo;t dispense coffee, if it is more, then we ought to give back the balance cash.\nThat\u0026rsquo;s it. We are done with the States and the Data.\nBefore we go through the implementation of each of the States that the vending machine can be in and the various interactions that the user can have with the machine at a particular State, we\u0026rsquo;ll have a 50,000 feet view of the FSM Actor itself.\nStructure of FSM Actor The structure of the FSM Actor looks very similar to the State Machine diagram itself and it looks like this :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 class CoffeeMachine extends FSM[MachineState, MachineData] { //What State and Data must this FSM start with (duh!) startWith(Open, MachineData(..)) //Handlers of State when(Open) { ... ... when(ReadyToBuy) { ... ... when(PoweredOff) { ... ... //fallback handler when an Event is unhandled by none of the States. whenUnhandled { ... ... //Do we need to do something when there is a State change? onTransition { case Open -\u0026gt; ReadyToBuy =\u0026gt; ... ... ... } What we understand from the structure: We have an initial State (which is Open) and any messages that is being sent to the Machine during Open State is handled in the when(Open) block, ReadyToBuy state is handled in when(ReadyToBuy) block and so on. The messages that I am referring to here are just like regular messages that we tell a plain Actor except that in case of FSMs, the message is wrapped along with the Data as well. The wrapper is called an Event (akka.actor.FSM.Event) and an example would look like Event(deposit: Deposit, MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) From the Akka documentation :\n1 2 3 4 5 6 /** * All messages sent to the [[akka.actor.FSM]] will be wrapped inside an * `Event`, which allows pattern matching to extract both state and data. */ case class Event[D](event: Any, stateData: D) extends NoSerializationVerificationNeeded We also notice that the when function accepts two mandatory parameters - the first being being the name of the State itself, eg. Open, ReadyToBuy etc and the second argument is a PartialFunction, just like an Actor\u0026rsquo;s receive where we do pattern matching. The most important thing to note here is that each of these pattern matching case blocks must return a State (more on this in next post). So, the code block would look something like 1 2 3 4 5 when(Open) { case Event(deposit: Deposit, MachineData(currentTxTotal, costOfCoffee, coffeesLeft)) =\u0026gt; { ... ... Generally, only those messages that match the patterns declared inside the when\u0026rsquo;s second argument gets handled at a particular State. If there is no matching pattern, then the FSM Actor tries to match our message to a pattern declared in the whenUnhandled block. Ideally, all the messages that are common across all the States is coded away in the whenUnhandled. (I am not the one to suggest style but alternatively, you could declare smaller PartialFunctions and compose them using andThen if you would like to reuse pattern matching across selected States) Finally, there is an onTransition function which allows you to react or get notified of changes in States. Interactions/Messages There are two kinds of people who interact with this Vending machine - Coffee drinkers, who need coffee and Vendors, who do the machine\u0026rsquo;s administrative tasks.\nFor sake of organization, I have introduced two traits for all the interactions with the Machine. (Just to refresh, an Interaction/Message is the first element wrapped inside the Event along with the MachineData). In plain old Actor terms, this is equivalent to the message that we send to the Actors.\n1 2 3 4 5 6 7 object CoffeeProtocol { trait UserInteraction trait VendorInteraction ... ... VendorInteraction Let\u0026rsquo;s also declare the various interactions that a Vendor can make with the machine.\n1 2 3 4 5 6 7 case object ShutDownMachine extends VendorInteraction case object StartUpMachine extends VendorInteraction case class SetCostOfCoffee(price: Int) extends VendorInteraction //Sets Maximum number of coffees that the vending machine could dispense case class SetNumberOfCoffee(quantity: Int) extends VendorInteraction case object GetNumberOfCoffee extends VendorInteraction So, the Vendor can\nstart and shutdown the machine set the price of the coffee and set and get the number of coffee remaining in the machine. UserInteraction 1 2 3 4 5 6 case class Deposit(value: Int) extends UserInteraction case class Balance(value: Int) extends UserInteraction case object Cancel extends UserInteraction case object BrewCoffee extends UserInteraction case object GetCostOfCoffee extends UserInteraction Now, for the UserInteraction, the User can\ndeposit money to buy a coffee get dispensed the extra cash if the deposited money is more than the cost of the coffee ask the machine to brew coffee if the deposit money is equal or more than the cost of the coffee cancel the transaction before brewing the coffee and get back all the money deposited query the machine for the cost of the coffee. In the next post, we\u0026rsquo;ll go through each of the States and explore on their interactions (along with testcases) in detail.\nCode For the benefit of the impatient, the entire code is available on github.\n","date":"2016-05-21T17:40:33Z","permalink":"/akka-notes-finite-state-machines-1/","title":"Akka Notes - Finite State Machines - 1"},{"content":"Failures are more like a feature among distributed systems. And with Akka\u0026rsquo;s let it crash fault tolerance model, you could achieve a clear separation between your business logic and your failure handling logic (supervision logic). All with very little effort. It\u0026rsquo;s pretty amazing. This is the topic of our discussion now.\nActor Supervision Imagine a method call stack and the top most method in your stack throws an Exception. What could be done by the methods down the stack?\nThe exception could be caught and handled in order to recover The exception could be caught, may be logged and kept quiet. The methods down the stack could also choose to duck the exception completely (or may be caught and rethrown) Imagine if all the methods until the main method doesn\u0026rsquo;t handle the exception. In that case, the program exits after writing an essay for an exception to the console.\nYou could also compare the same scenario with spawning Threads. If a child thread throws an exception and if the run or the call method doesn\u0026rsquo;t handle it, then the exception is expected to be handled by the parent thread or the main thread, whatever be the case. If the main thread doesn\u0026rsquo;t handle it, then the system exits.\nLet\u0026rsquo;s do it one more time - if the Child Actor which was created using the context.actorOf fails with an Exception, the parent actor (aka supervisor) could prefer to handle any failures of the child actor. If it does, it could prefer to handle it and recover (Restart/Resume). Else, duck the exception (Escalate) to its parent. Alternatively, it could just Stop the child actor - that\u0026rsquo;s the end of story for that child. Why did I say parent (aka supervisor)? Simply because Akka\u0026rsquo;s approach towards supervision is Parental supervision - which means that only the creators of the Actors could supervise over them.\nThat\u0026rsquo;s it !! We have pretty much covered all the supervision Directives (what could be done about the failures).\nStrategies Ah, I forgot to mention this one : You already know that an Akka Actor could create children and that they could create as many children as they want.\nNow, consider two scenarios :\n1. OneForOneStrategy\nYour Actor spawns multiple child actors and each one of these child actors connect to different datasources. Say you are running an app which translates an english word into multiple languages.\nSuppose, one child actor fails and you are fine to skip that result in the final list, what would you want to do? Shut down the service? Nope, you might want to just restart/stop only that child actor. Isn\u0026rsquo;t it? Now that\u0026rsquo;s called OneForOneStrategy in Akka supervision strategy terms - If one actor goes down, just handle one alone.\nDepending on your business exceptions, you would want to react differently (Stop, Restart, Escalate, Resume) to different exceptions. To configure your own strategy, you just override the supervisorStrategy in your Actor class.\nAn example declaration of OneForOneStrategy would be\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import akka.actor.Actor import akka.actor.ActorLogging import akka.actor.OneForOneStrategy import akka.actor.SupervisorStrategy.Stop class TeacherActorOneForOne extends Actor with ActorLogging { ... ... override val supervisorStrategy=OneForOneStrategy() { case _: MinorRecoverableException =\u0026gt; Restart case _: Exception =\u0026gt; Stop } ... ... 2. AllForOneStrategy\nAssume that you are doing an External Sort (One more example to prove that my creativity sucks!!), and each of your chunk is handled by a different Actor. Suddenly, one Actor fails throwing an exception. It doesn\u0026rsquo;t make any sense to continue processing the rest of the chunks because the final result wouldn\u0026rsquo;t be correct. So, it is logical to Stop ALL the actors.\nWhy did I say Stop instead of Restart in the previous line? Because Restarting would also not make any sense for this use-case considering the mailbox for each of these Actors would not be cleared on Restart. So, if we restart, the rest of the chunks would still be processed. That\u0026rsquo;s not what we want. Recreating the Actors with shiny new mailboxes would be the right approach here.\nAgain, just like the OneForOneStrategy, you just override the supervisorStrategy with an implementation of AllForOneStrategy\nAnd example would be\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import akka.actor.{Actor, ActorLogging} import akka.actor.AllForOneStrategy import akka.actor.SupervisorStrategy.Escalate import akka.actor.SupervisorStrategy.Stop class TeacherActorAllForOne extends Actor with ActorLogging { ... override val supervisorStrategy = AllForOneStrategy() { case _: MajorUnRecoverableException =\u0026gt; Stop case _: Exception =\u0026gt; Escalate } ... ... Directives The constructor of both AllForOneStrategy and the OneForOneStrategy accepts a PartialFunction[Throwable,Directive] called Decider which maps a Throwable to a Directive as you may see here :\n1 case _: MajorUnRecoverableException =\u0026gt; Stop There are simply just four kinds of directives - Stop, Resume, Escalate and Restart\nStop\nThe child actor is stopped in case of exception and any messages to the stopped actor would obviously go to the deadLetters queue.\nResume\nThe child actor just ignores the message that threw the exception and proceeds with processing the rest of the messages in the queue.\nRestart\nThe child actor is stopped and a brand new actor is initialized. Processing of the rest of the messages in the mailbox continue. The rest of the world is unaware that this happened since the same ActorRef is attached to the new Actor.\nEscalate\nThe supervisor ducks the failure and lets its supervisor handle the exception.\nDefault Strategy What if our Actor doesn\u0026rsquo;t specify any Strategy but has created child Actors. How are they handled? There is a default strategy declared in the Actor trait which (if condensed) looks like below :\n1 2 3 4 5 6 7 8 override val supervisorStrategy=OneForOneStrategy() { case _: ActorInitializationException=\u0026gt; Stop case _: ActorKilledException\t=\u0026gt; Stop case _: DeathPactException =\u0026gt; Stop case _: Exception =\u0026gt; Restart } So, in essence, the default strategy handles four cases :\nActorInitializationException =\u0026gt; Stop When the Actor could not be initialized, it would throw an ActorInitializationException. The Actor would be stopped then. Let\u0026rsquo;s simulate it by throwing an exception in the preStart callback :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 package me.rerun.akkanotes.supervision import akka.actor.{ActorSystem, Props} import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest import akka.actor.Actor import akka.actor.ActorLogging object ActorInitializationExceptionApp extends App{ val actorSystem=ActorSystem(\u0026#34;ActorInitializationException\u0026#34;) val actor=actorSystem.actorOf(Props[ActorInitializationExceptionActor], \u0026#34;initializationExceptionActor\u0026#34;) actor!\u0026#34;someMessageThatWillGoToDeadLetter\u0026#34; } class ActorInitializationExceptionActor extends Actor with ActorLogging{ override def preStart={ throw new Exception(\u0026#34;Some random exception\u0026#34;) } def receive={ case _=\u0026gt; } } Running the ActorInitializationExceptionApp would generate a ActorInitializationException (duh!!) and then move all the messages into the message queue of the deadLetters Actor:\nLog\n1 2 3 4 5 6 7 8 9 10 11 [ERROR] [11/10/2014 16:08:46.569] [ActorInitializationException-akka.actor.default-dispatcher-2] [akka://ActorInitializationException/user/initializationExceptionActor] Some random exception akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) ... ... Caused by: java.lang.Exception: Some random exception at me.rerun.akkanotes.supervision.ActorInitializationExceptionActor.preStart(ActorInitializationExceptionApp.scala:17) ... ... [INFO] [11/10/2014 16:08:46.581] [ActorInitializationException-akka.actor.default-dispatcher-4] [akka://ActorInitializationException/user/initializationExceptionActor] Message [java.lang.String] from Actor[akka://ActorInitializationException/deadLetters] to Actor[akka://ActorInitializationException/user/initializationExceptionActor#-1290470495] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings \u0026#39;akka.log-dead-letters\u0026#39; and \u0026#39;akka.log-dead-letters-during-shutdown\u0026#39;. ActorKilledException =\u0026gt; Stop When the Actor was killed using the Kill message, then it would throw an ActorKilledException. The default strategy would stop the child Actor if it throws the exception. At first, it seems that there\u0026rsquo;s no point in stopping an already killed Actor. However, consider this :\nActorKilledException would just be propagated to the supervisor. What about the lifecycle watchers or deathwatchers of this Actor that we saw during DeathWatch. The watchers won\u0026rsquo;t know anything until the Actor is Stopped. Sending a Kill on an Actor would just affect that particular actor which the supervisor knows. However, handling that with Stop would suspend the mailbox of that Actor, suspends the mailboxes of child actors, stops the child actors, sends a Terminated to all the child actor watchers, send a Terminated to all the immediate failed Actor\u0026rsquo;s watchers and finally stop the Actor itself. (Wow, that\u0026rsquo;s pretty awesome !!) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 package me.rerun.akkanotes.supervision import akka.actor.{ActorSystem, Props} import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest import akka.actor.Actor import akka.actor.ActorLogging import akka.actor.Kill object ActorKilledExceptionApp extends App{ val actorSystem=ActorSystem(\u0026#34;ActorKilledExceptionSystem\u0026#34;) val actor=actorSystem.actorOf(Props[ActorKilledExceptionActor]) actor!\u0026#34;something\u0026#34; actor!Kill actor!\u0026#34;something else that falls into dead letter queue\u0026#34; } class ActorKilledExceptionActor extends Actor with ActorLogging{ def receive={ case message:String=\u0026gt; log.info (message) } } Log\nThe logs just say that once the ActorKilledException comes in, the supervisor stops that actor and then the messages go into the queue of deadLetters\n1 2 3 4 5 6 INFO m.r.a.s.ActorKilledExceptionActor - something ERROR akka.actor.OneForOneStrategy - Kill akka.actor.ActorKilledException: Kill INFO akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://ActorKilledExceptionSystem/deadLetters] to Actor[akka://ActorKilledExceptionSystem/user/$a#-1569063462] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings \u0026#39;akka.log-dead-letters\u0026#39; and \u0026#39;akka.log-dead-letters-during-shutdown\u0026#39;. DeathPactException =\u0026gt; Stop From DeathWatch, you know that when an Actor watches over a child Actor, it is expected to handle the Terminated message in its receive. What if it doesn\u0026rsquo;t? You get the DeathPactException\nThe code shows that the Supervisor watches the child actor after creation but doesn\u0026rsquo;t handle the Terminated message from the child.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 package me.rerun.akkanotes.supervision import akka.actor.{ActorSystem, Props} import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest import akka.actor.Actor import akka.actor.ActorLogging import akka.actor.Kill import akka.actor.PoisonPill import akka.actor.Terminated object DeathPactExceptionApp extends App{ val actorSystem=ActorSystem(\u0026#34;DeathPactExceptionSystem\u0026#34;) val actor=actorSystem.actorOf(Props[DeathPactExceptionParentActor]) actor!\u0026#34;create_child\u0026#34; //Throws DeathPactException Thread.sleep(2000) //Wait until Stopped actor!\u0026#34;someMessage\u0026#34; //Message goes to DeadLetters } class DeathPactExceptionParentActor extends Actor with ActorLogging{ def receive={ case \u0026#34;create_child\u0026#34;=\u0026gt; { log.info (\u0026#34;creating child\u0026#34;) val child=context.actorOf(Props[DeathPactExceptionChildActor]) context.watch(child) //Watches but doesnt handle terminated message. Throwing DeathPactException here. child!\u0026#34;stop\u0026#34; } case \u0026#34;someMessage\u0026#34; =\u0026gt; log.info (\u0026#34;some message\u0026#34;) //Doesnt handle terminated message //case Terminated(_) =\u0026gt; } } class DeathPactExceptionChildActor extends Actor with ActorLogging{ def receive={ case \u0026#34;stop\u0026#34;=\u0026gt; { log.info (\u0026#34;Actor going to stop and announce that it\u0026#39;s terminated\u0026#34;) self!PoisonPill } } } Log\nThe logs tell us that the DeathPactException comes in, the supervisor stops that actor and then the messages go into the queue of deadLetters\n1 2 3 4 5 6 7 8 INFO m.r.a.s.DeathPactExceptionParentActor - creating child INFO m.r.a.s.DeathPactExceptionChildActor - Actor going to stop and announce that it\u0026#39;s terminated ERROR akka.actor.OneForOneStrategy - Monitored actor [Actor[akka://DeathPactExceptionSystem/user/$a/$a#-695506341]] terminated akka.actor.DeathPactException: Monitored actor [Actor[akka://DeathPactExceptionSystem/user/$a/$a#-695506341]] terminated INFO akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://DeathPactExceptionSystem/deadLetters] to Actor[akka://DeathPactExceptionSystem/user/$a#-1452955980] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings \u0026#39;akka.log-dead-letters\u0026#39; and \u0026#39;akka.log-dead-letters-during-shutdown\u0026#39;. Exception =\u0026gt; Restart For all other Exceptions, the default Directive is to Restart the Actor. Check the following app. Just to prove that the Actor is restarted, OtherExceptionParentActor makes the child throw an exception and immediately sends a message. The message reaches the mailbox and when the the child actor restarts, the message gets processed. Nice !!\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 package me.rerun.akkanotes.supervision import akka.actor.Actor import akka.actor.ActorLogging import akka.actor.ActorSystem import akka.actor.OneForOneStrategy import akka.actor.Props import akka.actor.SupervisorStrategy.Stop object OtherExceptionApp extends App{ val actorSystem=ActorSystem(\u0026#34;OtherExceptionSystem\u0026#34;) val actor=actorSystem.actorOf(Props[OtherExceptionParentActor]) actor!\u0026#34;create_child\u0026#34; } class OtherExceptionParentActor extends Actor with ActorLogging{ def receive={ case \u0026#34;create_child\u0026#34;=\u0026gt; { log.info (\u0026#34;creating child\u0026#34;) val child=context.actorOf(Props[OtherExceptionChildActor]) child!\u0026#34;throwSomeException\u0026#34; child!\u0026#34;someMessage\u0026#34; } } } class OtherExceptionChildActor extends akka.actor.Actor with ActorLogging{ override def preStart={ log.info (\u0026#34;Starting Child Actor\u0026#34;) } def receive={ case \u0026#34;throwSomeException\u0026#34;=\u0026gt; { throw new Exception (\u0026#34;I\u0026#39;m getting thrown for no reason\u0026#34;) } case \u0026#34;someMessage\u0026#34; =\u0026gt; log.info (\u0026#34;Restarted and printing some Message\u0026#34;) } override def postStop={ log.info (\u0026#34;Stopping Child Actor\u0026#34;) } } Log\nThe logs of this program is pretty neat.\nThe exception gets thrown. We see the trace The child restarts - Stop and Start gets called (we\u0026rsquo;ll see about the preRestart and postRestart callbacks soon) The message that was send to the Child actor before restart is processed. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 INFO m.r.a.s.OtherExceptionParentActor - creating child INFO m.r.a.s.OtherExceptionChildActor - Starting Child Actor ERROR akka.actor.OneForOneStrategy - I\u0026#39;m getting thrown for no reason java.lang.Exception: I\u0026#39;m getting thrown for no reason at me.rerun.akkanotes.supervision.OtherExceptionChildActor$$anonfun$receive$2.applyOrElse(OtherExceptionApp.scala:39) ~[classes/:na] at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.11-2.3.4.jar:na] ... ...\tINFO m.r.a.s.OtherExceptionChildActor - Stopping Child Actor INFO m.r.a.s.OtherExceptionChildActor - Starting Child Actor INFO m.r.a.s.OtherExceptionChildActor - Restarted and printing some Message Escalate and Resume We saw examples of Stop and Restart via the defaultStrategy. Now, let\u0026rsquo;s have a quick look at the Escalate.\nResume just ignores the exception and proceeds processing the next message in the mailbox. It\u0026rsquo;s more like catching the exception and doing nothing about it. Awesome stuff but not a lot to talk about there.\nEscalating generally means that the exception is something critical and the immediate supervisor would not be able to handle it. So, it asks help from its supervisor. Let\u0026rsquo;s take an example.\nConsider three Actors - EscalateExceptionTopLevelActor, EscalateExceptionParentActor and EscalateExceptionChildActor. If the child actor throws an exception and if the parent level actor could not handle it, it could Escalate it to the Top level Actor. The Top level actor could choose to react with any of the Directives. In our example, we just Stop. Stop would stop the immediate child (which is the EscalateExceptionParentActor). As you know, when a Stop is executed on an Actor, all its children would also be stopped before the Actor itself is stopped.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 package me.rerun.akkanotes.supervision import akka.actor.Actor import akka.actor.ActorLogging import akka.actor.ActorSystem import akka.actor.OneForOneStrategy import akka.actor.Props import akka.actor.SupervisorStrategy.Escalate import akka.actor.SupervisorStrategy.Stop import akka.actor.actorRef2Scala object EscalateExceptionApp extends App { val actorSystem = ActorSystem(\u0026#34;EscalateExceptionSystem\u0026#34;) val actor = actorSystem.actorOf(Props[EscalateExceptionTopLevelActor], \u0026#34;topLevelActor\u0026#34;) actor ! \u0026#34;create_parent\u0026#34; } class EscalateExceptionTopLevelActor extends Actor with ActorLogging { override val supervisorStrategy = OneForOneStrategy() { case _: Exception =\u0026gt; { log.info(\u0026#34;The exception from the Child is now handled by the Top level Actor. Stopping Parent Actor and its children.\u0026#34;) Stop //Stop will stop the Actor that threw this Exception and all its children } } def receive = { case \u0026#34;create_parent\u0026#34; =\u0026gt; { log.info(\u0026#34;creating parent\u0026#34;) val parent = context.actorOf(Props[EscalateExceptionParentActor], \u0026#34;parentActor\u0026#34;) parent ! \u0026#34;create_child\u0026#34; //Sending message to next level } } } class EscalateExceptionParentActor extends Actor with ActorLogging { override def preStart={ log.info (\u0026#34;Parent Actor started\u0026#34;) } override val supervisorStrategy = OneForOneStrategy() { case _: Exception =\u0026gt; { log.info(\u0026#34;The exception is ducked by the Parent Actor. Escalating to TopLevel Actor\u0026#34;) Escalate } } def receive = { case \u0026#34;create_child\u0026#34; =\u0026gt; { log.info(\u0026#34;creating child\u0026#34;) val child = context.actorOf(Props[EscalateExceptionChildActor], \u0026#34;childActor\u0026#34;) child ! \u0026#34;throwSomeException\u0026#34; } } override def postStop = { log.info(\u0026#34;Stopping parent Actor\u0026#34;) } } class EscalateExceptionChildActor extends akka.actor.Actor with ActorLogging { override def preStart={ log.info (\u0026#34;Child Actor started\u0026#34;) } def receive = { case \u0026#34;throwSomeException\u0026#34; =\u0026gt; { throw new Exception(\u0026#34;I\u0026#39;m getting thrown for no reason.\u0026#34;) } } override def postStop = { log.info(\u0026#34;Stopping child Actor\u0026#34;) } } Log\nAs you could see from the logs,\nThe child actor throws exception. The immediate supervisor (EscalateExceptionParentActor) escalates that exception to its supervisor (EscalateExceptionTopLevelActor) The resultant directive from EscalateExceptionTopLevelActor is to Stop the Actor. As a sequence, the child actors gets stopped first. The parent actor gets stopped next (only after the watchers have been notified) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 INFO m.r.a.s.EscalateExceptionTopLevelActor - creating parent INFO m.r.a.s.EscalateExceptionParentActor - Parent Actor started INFO m.r.a.s.EscalateExceptionParentActor - creating child INFO m.r.a.s.EscalateExceptionChildActor - Child Actor started INFO m.r.a.s.EscalateExceptionParentActor - The exception is ducked by the Parent Actor. Escalating to TopLevel Actor INFO m.r.a.s.EscalateExceptionTopLevelActor - The exception from the Child is now handled by the Top level Actor. Stopping Parent Actor and its children. ERROR akka.actor.OneForOneStrategy - I\u0026#39;m getting thrown for no reason. java.lang.Exception: I\u0026#39;m getting thrown for no reason. at me.rerun.akkanotes.supervision.EscalateExceptionChildActor$$anonfun$receive$3.applyOrElse(EscalateExceptionApp.scala:71) ~[classes/:na] ... ... INFO m.r.a.s.EscalateExceptionChildActor - Stopping child Actor INFO m.r.a.s.EscalateExceptionParentActor - Stopping parent Actor Please note that whatever directive that was issued would only apply to the immediate child that escalated. Say, if a Restart is issued at the TopLevel, only the Parent would be restarted and anything in its constructor/preStart would be executed. If the children of the Parent actor was created in the constructor, they would also be created. However, children that were created through messages to the Parent Actor would still be in the Terminated state.\nTRIVIA\nActually, you could control whether the preStart gets called at all. We\u0026rsquo;ll see about this in the next minor write-up. If you are curious, just have a look at the postRestart method of the Actor\n1 2 3 def postRestart(reason: Throwable): Unit = { preStart() } Code As always, code is on github\n(my .gitignore wasn\u0026rsquo;t setup right for this project. will fix it today. sorry)\n","date":"2014-11-10T09:26:35Z","permalink":"/akka-notes-actor-supervision-8/","title":"Akka Notes -  Actor Supervision - 8"},{"content":"When we talked about Actor lifecycle, we saw that Actors could be stopped by various means (using ActorSystem.stop or ActorContext.stop or sending a PoisonPill - there\u0026rsquo;s also the Kill and the gracefulStop).\nWhatever reason an Actor dies, there are cases when a few other actors in the system would like to know about it. Let\u0026rsquo;s take a trivial example of an Actor who talks to a database - let\u0026rsquo;s call it a RepositoryActor. For obvious reasons, there would be few other actors in the system who would be sending message to this RepositoryActor. These \u0026ldquo;interested\u0026rdquo; Actors would like to keep an eye on or watch this Actor if it goes down. Now, that in Actor terms is called DeathWatch. And the methods to watch and unwatch over this is intuitively ActorContext.watch and ActorContext.unwatch. If watched, the watchers would receive a Terminated message from the stopped Actor which they could comfortably add in to their receive function.\nUnlike Supervision, where there is a strict enforcement of parent-child hierarchy, any Actor could watch any other Actor in the ActorSystem.\nLet\u0026rsquo;s have a look at the code.\nCode QuoteRepositoryActor Our QueryRepositoryActor holds a bunch of quotes as a List and serves a random one upon receiving a QuoteRepositoryRequest. It keeps track of the number of messages received and if it receives more than 3 messages, it kills itself with a PoisonPill Nothing fancy here.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 package me.rerun.akkanotes.deathwatch import akka.actor.{PoisonPill, Actor, ActorLogging, actorRef2Scala} import me.rerun.akkanotes.protocols.QuoteRepositoryProtocol._ import scala.util.Random class QuoteRepositoryActor() extends Actor with ActorLogging { val quotes = List( \u0026#34;Moderation is for cowards\u0026#34;, \u0026#34;Anything worth doing is worth overdoing\u0026#34;, \u0026#34;The trouble is you think you have time\u0026#34;, \u0026#34;You never gonna know if you never even try\u0026#34;) var repoRequestCount:Int=1 def receive = { case QuoteRepositoryRequest =\u0026gt; { if (repoRequestCount\u0026gt;3){ self!PoisonPill } else { //Get a random Quote from the list and construct a response val quoteResponse = QuoteRepositoryResponse(quotes(Random.nextInt(quotes.size))) log.info(s\u0026#34;QuoteRequest received in QuoteRepositoryActor. Sending response to Teacher Actor $quoteResponse\u0026#34;) repoRequestCount=repoRequestCount+1 sender ! quoteResponse } } } } TeacherActorWatcher Again, nothing fancy with TeacherActorWatcher except that it creates the QuoteRepositoryActor and watches over it using a context.watch.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 package me.rerun.akkanotes.deathwatch import akka.actor.{Terminated, Props, Actor, ActorLogging} import me.rerun.akkanotes.protocols.TeacherProtocol.QuoteRequest import me.rerun.akkanotes.protocols.QuoteRepositoryProtocol.QuoteRepositoryRequest class TeacherActorWatcher extends Actor with ActorLogging { val quoteRepositoryActor=context.actorOf(Props[QuoteRepositoryActor], \u0026#34;quoteRepositoryActor\u0026#34;) context.watch(quoteRepositoryActor) def receive = { case QuoteRequest =\u0026gt; { quoteRepositoryActor ! QuoteRepositoryRequest } case Terminated(terminatedActorRef)=\u0026gt;{ log.error(s\u0026#34;Child Actor {$terminatedActorRef} Terminated\u0026#34;) } } } TestCases This is the interesting bit. Frankly, I never thought that these could be tested. akka-testkit FTW. We will analyze three testcases here :\n1. Assert receipt of Terminated message if watched\nThe QuoteRepositoryActor should send the testcase a Terminated message on receipt of the 4th message. The first three messages should go in fine.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 \u0026#34;A QuoteRepositoryActor\u0026#34; must { ... ... ... \u0026#34;send back a termination message to the watcher on 4th message\u0026#34; in { val quoteRepository=TestActorRef[QuoteRepositoryActor] val testProbe=TestProbe() testProbe.watch(quoteRepository) //Let\u0026#39;s watch the Actor within (1000 millis) { var receivedQuotes = List[String]() (1 to 3).foreach(_ =\u0026gt; quoteRepository ! QuoteRepositoryRequest) receiveWhile() { case QuoteRepositoryResponse(quoteString) =\u0026gt; { receivedQuotes = receivedQuotes :+ quoteString } } receivedQuotes.size must be (3) println(s\u0026#34;receiveCount ${receivedQuotes.size}\u0026#34;) //4th message quoteRepository!QuoteRepositoryRequest testProbe.expectTerminated(quoteRepository) //Expect a Terminated Message } } 2. Assert non-receipt of Terminated message if not watched/unwatched\nActually, we are over-doing things just to showcase the context.unwatch. The testcase would work just fine if we remove the testProbe.watch and testProbe.unwatch lines.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 \u0026#34;not send back a termination message on 4th message if not watched\u0026#34; in { val quoteRepository=TestActorRef[QuoteRepositoryActor] val testProbe=TestProbe() testProbe.watch(quoteRepository) //watching within (1000 millis) { var receivedQuotes = List[String]() (1 to 3).foreach(_ =\u0026gt; quoteRepository ! QuoteRepositoryRequest) receiveWhile() { case QuoteRepositoryResponse(quoteString) =\u0026gt; { receivedQuotes = receivedQuotes :+ quoteString } } testProbe.unwatch(quoteRepository) //not watching anymore receivedQuotes.size must be (3) println(s\u0026#34;receiveCount ${receivedQuotes.size}\u0026#34;) //4th message quoteRepository!QuoteRepositoryRequest testProbe.expectNoMsg() //Not Watching. No Terminated Message } } 3. Assert receipt of Terminated message in the TeacherActorWatcher\nWe subscribe to the EventStream and check for a specific log message to assert termination.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 \u0026#34;end back a termination message to the watcher on 4th message to the TeacherActor\u0026#34; in { //This just subscribes to the EventFilter for messages. We have asserted all that we need against the QuoteRepositoryActor in the previous testcase val teacherActor=TestActorRef[TeacherActorWatcher] within (1000 millis) { (1 to 3).foreach (_=\u0026gt;teacherActor!QuoteRequest) //this sends a message to the QuoteRepositoryActor EventFilter.error (pattern=\u0026#34;\u0026#34;\u0026#34;Child Actor .* Terminated\u0026#34;\u0026#34;\u0026#34;, occurrences = 1).intercept{ teacherActor!QuoteRequest //Send the dangerous 4th message } } } The pattern property of the EventFilter, not surprisingly, expects a regex pattern. The pattern=\u0026quot;\u0026quot;\u0026quot;Child Actor .* Terminated\u0026quot;\u0026quot;\u0026quot; is expected to match a log message which is of the format Child Actor {Actor[akka://TestUniversityMessageSystem/user/$$d/quoteRepositoryActor#-1905987636]} Terminated\nGithub\nAs always, the code is available at github. Watch for the deathwatch package.\n","date":"2014-10-31T08:09:54Z","permalink":"/akka-notes-deathwatch-7/","title":"Akka Notes -  DeathWatch - 7"},{"content":"Actors are completely hierarchical. Whatever Actors that you create HAS to be a child of some other Actor.\nLet\u0026rsquo;s analyze that a bit :\nPath Say, we create an ActorRef using ActorSystem.actorOf and try to print its path.\n1 2 3 val actorSystem=ActorSystem(\u0026#34;SupervisionActorSystem\u0026#34;) val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor]) println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/$a As you see, a path looks very similar to a file path in a file system.\nakka here is fixed because all these are addresses of Akka Actors - more like file:// or http:// prefix (nothing to do with protocol though). The SupervisionActorSystem is just the name of your ActorSystem that you created. We\u0026rsquo;ll talk about the user in the next section. The $a is the name of your Actor that the system generated for you. How would you like if your operating system generated random file names for your files? You\u0026rsquo;d obviously hate it because you would want to refer to that name in future. So, let\u0026rsquo;s give it a proper meaningful name to it : 1 2 3 val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor], \u0026#34;teacherActor\u0026#34;) println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/teacherActor That\u0026rsquo;s it. Now, the path actually makes sense.\nChild Actors Similar to top level actors which we create out of ActorSystem, we could also create child actors out of the ActorContext. In fact, the power of Actors\u0026rsquo;s fault tolerance primarily lie within leveraging the Actor hierarchy and the ability of a parent to manage the life of child actors.\nAssume you have a TeacherSupervisor and you would like to create a TeacherActor to be a child of the Supervisor, you do a ActorContext.actorOf instead of a ActorSystem.actorOf:\n1 2 3 4 class TeacherSupervisor extends Actor with ActorLogging { val teacherActor=context.actorOf(Props[TeacherActor], \u0026#34;teacherActor\u0026#34;) ... ... Frankly, in any application, you will be creating a whole lot of child actors than top level actors - which means that you\u0026rsquo;ll be calling a lot more actorContext.actorOf than actorSystem.actorOf.\nYou\u0026rsquo;ll notice that the path of the child actor is akka://SupervisionActorSystem/user/teacherSupervisor/teacherActor, which is very similar to the path you get when you create a child folder within a parent folder.\nWhen do you create child Actors?\nYou generally create child actors when a particular task is composed of a subtask or multiple subtasks. You also create a child actor when a particular task to be executed by the parent is error-prone and you would want to isolate it (so that if it fails, you could recover). When there is no parent child relationship between tasks, then you DON\u0026rsquo;T create child actors.\nAlso, there\u0026rsquo;s nothing which is stopping a child actor from creating children to delegate its subtasks. Actors and their creation are really cheap but the power that comes with it is amazing (we\u0026rsquo;ll see about this while we talk about supervision).\nNow what is that user in the path? For lack of a creativity, let me compare the ActorSystem to a Unix file system - with a / root folder and all those /etc, /usr, /bin and various other folders.\nActorSystem are much like that. It creates a few top level Actors - the most important being the root Actor with the path /, the user Actor with the path /user and a system Actor with the path /system. (there\u0026rsquo;s also a /deadLetters that represent the DeadLetterActorRef. We saw this in our previous post)\nCodewise, the ActorSystem composes three Actors inside it (via ActorRefProvider). Those are the root for ALL the Actors that gets created under the ActorSystem.\nsystemGuardian actor - root of all actors under /system guardian actor - root of all actors under /user and rootGuardian Actor - root of both the systemGuardian and the userGuardian actors. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 /** * Reference to the supervisor of guardian and systemGuardian; .... */ def rootGuardian: InternalActorRef /** * Reference to the supervisor used for all top-level user actors. */ def guardian: LocalActorRef /** * Reference to the supervisor used for all top-level system actors. */ def systemGuardian: LocalActorRef /user (aka) user guardian\nAny Actor that you create in your program like the StudentActor or the TeacherActor using the ActorSystem\u0026rsquo;s actorOf method would directly fall under /user. That\u0026rsquo;s the reason your teacherActor in the first part of this write-up had the path slug /user/teacherActor.\n/system (aka) system guardian\nThe system guardian shuts itself down when it notices that the userGuardian is dead. Makes sense considering if the userGuardian is down, then all the business actors under it is also down and therefore all administrative actors needs to be gone too.\nWe could see two distinct places where System Actors are being created - I mean actors under the /system hierarchy.\nLike we saw earlier, any message that you send to an Actor that is terminated gets forwarded to the mailbox of an internal Actor called DeadLetterActor. The DeadLetter Actor wraps each message as a DeadLetter and publishes it to the EventStream. One other Actor called DeadLetterListener consumes all DeadLetters and publishes that as a log message. Now, the DeadLetterListener is a system Actor with path /system/deadLetterListener. Remember the TestEventListener that we created in our previous write-up to subscribe to log messages in the EventStream? They are System actors too. In fact, all akka.loggers are created as System actors. 1 2 3 class TeacherTest extends TestKit(ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;, ConfigFactory.parseString(\u0026#34;\u0026#34;\u0026#34;akka.loggers = [\u0026#34;akka.testkit.TestEventListener\u0026#34;]\u0026#34;\u0026#34;\u0026#34;))) ... ... The documentation here says that any Actor that is configured in the configuration files and created and deployed into the ActorSystem while it starts, falls under the /system umbrella. Let me update this post when I find something interesting around this\n/ (aka) root guardian\nAs we saw earlier, the / Actor is the parent of the user and the system guardians.\nTRIVIA\nTechnically, there\u0026rsquo;s a superfluous parent for the root actor too. This Actor\u0026rsquo;s only job is to shut down the entire ActorSystem if the root actor fails. Since it\u0026rsquo;s strictly not considered within the Actor hierarchy, the Akka team calls it :\n1 2 private[akka] val theOneWhoWalksTheBubblesOfSpaceTime: InternalActorRef = new MinimalActorRef { ... ","date":"2014-10-21T17:51:09Z","permalink":"/akka-notes-child-actors-and-path/","title":"Akka Notes - Child Actors and ActorPath - 6"},{"content":"(Please note that this lifecycle write-up does not cover the preRestart or the postRestart methods. We\u0026rsquo;ll talk about them when we discuss supervision)\nThe basic Actor lifecycle is very much intuitive. You could actually compare the basic Actor lifecycle with a Java servlet lifecycle with one special difference.\nJust like any other regular class, we have a Constructor The preStart method gets called back next. Here, you could initialize resources that you would like to clean-up in postStop The \u0026ldquo;servicing\u0026rdquo; or the message handling by the receive method occupies the major chunk of time and that happens in between. Let\u0026rsquo;s look at a simple actor which prints the lifecycle.\nDumb Lifecycle Actor 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 package me.rerun.akkanotes.lifecycle import akka.actor.{ActorLogging, Actor} import akka.event.LoggingReceive class BasicLifecycleLoggingActor extends Actor with ActorLogging{ log.info (\u0026#34;Inside BasicLifecycleLoggingActor Constructor\u0026#34;) log.info (context.self.toString()) override def preStart() ={ log.info(\u0026#34;Inside the preStart method of BasicLifecycleLoggingActor\u0026#34;) } def receive = LoggingReceive{ case \u0026#34;hello\u0026#34; =\u0026gt; log.info (\u0026#34;hello\u0026#34;) } override def postStop()={ log.info (\u0026#34;Inside postStop method of BasicLifecycleLoggingActor\u0026#34;) } } App The LifecycleApp just initiates, sends a message to the Actor and shuts down the ActorSystem.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 import akka.actor.{ActorSystem, Props} object LifecycleApp extends App{ val actorSystem=ActorSystem(\u0026#34;LifecycleActorSystem\u0026#34;) val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],\u0026#34;lifecycleActor\u0026#34;) lifecycleActor!\u0026#34;hello\u0026#34; //wait for a couple of seconds before shutdown Thread.sleep(2000) actorSystem.shutdown() } Output 1 2 3 4 5 6 7 8 9 10 Inside BasicLifecycleLoggingActor Constructor Actor[akka://LifecycleActorSystem/user/lifecycleActor#-2018741361] Inside the preStart method of BasicLifecycleLoggingActor hello Inside postStop method of BasicLifecycleLoggingActor What\u0026rsquo;s that special difference between Servlets and the basic Actor lifecycle?\nThat there is no difference between constructor and preStart in Actor lifecycle - more or less.\nThe reason why I printed the context.self in the constructor is this - unlike Servlets, Actors have access to the ActorContext even inside the constructor. The difference between the preStart and the constructor then becomes very subtle. We\u0026rsquo;ll revisit the difference while we talk about supervision but if you are curious - calling the preStart when the Actor restarts (in case of failure) could be controlled. With constructor, that isn\u0026rsquo;t possible.\nWhen is postStop called? As we saw from the program, the postStop gets called when the ActorSystem shuts down. There are a couple of other times when the callback gets invoked too.\nActorSystem.stop() We could stop an Actor using the stop method of the ActorSystem and the ActorContext\n1 2 3 4 5 6 7 8 9 10 11 object LifecycleApp extends App{ val actorSystem=ActorSystem(\u0026#34;LifecycleActorSystem\u0026#34;) val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],\u0026#34;lifecycleActor\u0026#34;) actorSystem.stop(lifecycleActor); ... ... } ActorContext.stop 1) Either by way of a message (externally or passing a message to itself)\n1 2 3 4 5 6 7 class BasicLifecycleLoggingActor extends Actor with ActorLogging{ ... ... def receive = LoggingReceive{ case \u0026#34;hello\u0026#34; =\u0026gt; log.info (\u0026#34;hello\u0026#34;) case \u0026#34;stop\u0026#34; =\u0026gt; context.stop(self) } and\n1 2 3 4 5 6 7 8 9 object LifecycleApp extends App{ val actorSystem=ActorSystem(\u0026#34;LifecycleActorSystem\u0026#34;) val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],\u0026#34;lifecycleActor\u0026#34;) lifecycleActor!\u0026#34;stop\u0026#34; ... ... 2) OR kill itself for no reason (this is just for fun. No Actor with an ambition would do this)\n1 2 3 4 5 6 7 class BasicLifecycleLoggingActor extends Actor with ActorLogging{ log.info (\u0026#34;Inside BasicLifecycleLoggingActor Constructor\u0026#34;) log.info (context.self.toString()) context.stop(self) ... ... PoisonPill In the previous example, we passed a message called stop from the LifecycleApp to the Actor. The Actor received that message and killed itself using a context.stop. We could achieve the same thing by passing a PoisonPill message to the target actor. Please note that the PoisonPill message, just like the previous stop message gets enqueued in the regular mailbox and will be processed when it turn comes up.\n1 2 3 4 5 6 7 8 object LifecycleApp extends App{ val actorSystem=ActorSystem(\u0026#34;LifecycleActorSystem\u0026#34;) val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],\u0026#34;lifecycleActor\u0026#34;) lifecycleActor!PoisonPill ... ... Kill Instead of sending a PoisonPill, you could also send a Kill message to the target Actor.\n1 lifecycleActor ! Kill Between sending PoisonPill or Kill messages, the difference is subtle but important.\nWith PoisonPill, a Terminated message is sent to all watchers (we\u0026rsquo;ll see that shortly in our DeathWatch part). With a Kill message, an ActorKilledException is thrown by the host Actor which gets propagated to its Supervisor (we\u0026rsquo;ll see this shortly in our Supervision part) TRIVIA\nWhat do I mean regular mailbox? Is there a \u0026ldquo;special\u0026rdquo; mailbox too? Yup. There is. And we\u0026rsquo;ll talk about it when we talk about it when we discuss supervision and system messages.\nTermination Once the Actor is stopped, it is said to enter into a Terminated state. The immediate question that would come up to your mind is what would happen to the messages that is sent to an Actor which is already terminated?\nLet\u0026rsquo;s see that :\nApp 1 2 3 4 5 6 7 8 9 10 object LifecycleApp extends App{ val actorSystem=ActorSystem(\u0026#34;LifecycleActorSystem\u0026#34;) val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],\u0026#34;lifecycleActor\u0026#34;) lifecycleActor!\u0026#34;hello\u0026#34; lifecycleActor!\u0026#34;stop\u0026#34; lifecycleActor!\u0026#34;hello\u0026#34; //Sending message to an Actor which is already stopped } Actor - Just as before 1 2 3 4 5 6 7 8 class BasicLifecycleLoggingActor extends Actor with ActorLogging{ def receive = LoggingReceive{ case \u0026#34;hello\u0026#34; =\u0026gt; log.info (\u0026#34;hello\u0026#34;) case \u0026#34;stop\u0026#34; =\u0026gt; context.stop(self) } } Output 1 2 3 BasicLifecycleLoggingActor - hello akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://LifecycleActorSystem/deadLetters] to Actor[akka://LifecycleActorSystem/user/lifecycleActor#-569230546] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings \u0026#39;akka.log-dead-letters\u0026#39; and \u0026#39;akka.log-dead-letters-during-shutdown\u0026#39;. From the logs, you see that there are some references of deadletters. Any message that you send to an Actor that is terminated gets forwarded to the mailbox of an internal Actor called DeadLetterActor.\nCurious as to what happens next? The DeadLetter Actor processes the messages in its mailbox, wraps each message as a DeadLetter and publishes it to the EventStream.\nOne other Actor called DeadLetterListener consumes all DeadLetters and publishes that as a log message. Check this out.\nRemember, when we talked about logging, we saw that all log messages gets published to the EventStream and that we are free to subscribe to that EventStream - just that the subscriber also needs to be an Actor. Let\u0026rsquo;s try that now.\nFor our example, we\u0026rsquo;ll subscribe to the EventStream and watch out for all DeadLetter messages and will print to the console (so much for creativity??!!). Frankly, we are free to do anything from generating an alert, storing it into a database or even feeding into analytics.\nSubscribing to DeadLetters in EventStream 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 import akka.actor.ActorSystem import akka.actor.Props import akka.actor.PoisonPill import akka.actor.DeadLetter import akka.actor.Actor object LifecycleApp extends App { val actorSystem = ActorSystem(\u0026#34;LifecycleActorSystem\u0026#34;) val lifecycleActor = actorSystem.actorOf(Props[BasicLifecycleLoggingActor], \u0026#34;lifecycleActor\u0026#34;) val deadLetterListener = actorSystem.actorOf(Props[MyCustomDeadLetterListener]) actorSystem.eventStream.subscribe(deadLetterListener, classOf[DeadLetter]) lifecycleActor ! \u0026#34;hello\u0026#34; lifecycleActor ! \u0026#34;stop\u0026#34; lifecycleActor ! \u0026#34;hello\u0026#34; } class MyCustomDeadLetterListener extends Actor { def receive = { case deadLetter: DeadLetter =\u0026gt; println(s\u0026#34;FROM CUSTOM LISTENER $deadLetter\u0026#34;) } } Output\n1 2 3 4 5 164 [LifecycleActorSystem-akka.actor.default-dispatcher-4] INFO BasicLifecycleLoggingActor - hello 167 [LifecycleActorSystem-akka.actor.default-dispatcher-4] INFO akka.actor.RepointableActorRef - Message [java.lang.String] from Actor[akka://LifecycleActorSystem/deadLetters] to Actor[akka://LifecycleActorSystem/user/lifecycleActor#-782937925] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings \u0026#39;akka.log-dead-letters\u0026#39; and \u0026#39;akka.log-dead-letters-during-shutdown\u0026#39;. FROM CUSTOM LISTENER DeadLetter(hello,Actor[akka://LifecycleActorSystem/deadLetters],Actor[akka://LifecycleActorSystem/user/lifecycleActor#-782937925]) ","date":"2014-10-21T07:16:18Z","permalink":"/akka-notes-actor-lifecycle-basic/","title":"Akka Notes - Actor Lifecycle - Basic - 5"},{"content":"As we saw from our previous posts, we could create an Actor using the actorOf method of the ActorSystem. There\u0026rsquo;s actually much more you could do with ActorSystem. We\u0026rsquo;ll touch upon just the Configuration and the Scheduling bit in this write-up\nLet\u0026rsquo;s look at the subsets of methods available in the ActorSystem.\nConfiguration Management Remember the application.conf file we used for configuring our log level in the previous write-up? This configuration file is just like those .properties files in Java applications and much more. We\u0026rsquo;ll be soon seeing how we could use this configuration file to customize our dispatchers, mailboxes etc. (I am not even closely doing justice to the power of the typesafe config. Please go through some examples to really appreciate its awesomeness)\nSo, when we create the ActorSystem using the ActorSystem object\u0026rsquo;s apply method without specifying any configuration, it looks out for application.conf, application.json and application.properties in the root of the classpath and loads them automatically.\nSo,\n1 val system=ActorSystem(\u0026#34;UniversityMessagingSystem\u0026#34;) is the same as\n1 val system=ActorSystem(\u0026#34;UniversityMessagingSystem\u0026#34;, ConfigFactory.load()) To provide evidence to that argument, check out the apply method in ActorSystem.scala\n1 2 3 4 5 def apply(name: String, config: Option[Config] = None, classLoader: Option[ClassLoader] = None, defaultExecutionContext: Option[ExecutionContext] = None): ActorSystem = { val cl = classLoader.getOrElse(findClassLoader()) val appConfig = config.getOrElse(ConfigFactory.load(cl)) new ActorSystemImpl(name, appConfig, cl, defaultExecutionContext).start() } Overriding default configuration If you are not keen on using the application.conf (as in testcases) or would like to have your own custom configuration file (as in testing againt different configuration or deploying to different environments), you are free to override this by passing in your own configuration instead of wanting the one from the classpath.\nConfigFactory.parseString is one option\n1 val actorSystem=ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;, ConfigFactory.parseString(\u0026#34;\u0026#34;\u0026#34;akka.loggers = [\u0026#34;akka.testkit.TestEventListener\u0026#34;]\u0026#34;\u0026#34;\u0026#34;)) or\nsimply in your Testcase as\n1 2 3 4 class TeacherTestLogListener extends TestKit(ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;, ConfigFactory.parseString(\u0026#34;\u0026#34;\u0026#34;akka.loggers = [\u0026#34;akka.testkit.TestEventListener\u0026#34;]\u0026#34;\u0026#34;\u0026#34;))) with WordSpecLike with MustMatchers with BeforeAndAfterAll { There\u0026rsquo;s also a ConfigFactory.load\n1 val system = ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;, ConfigFactory.load(\u0026#34;uat-application.conf\u0026#34;)) If you need access to your own config parameters in runtime, you could do it via its API like so :\n1 2 3 val system=ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;, ConfigFactory.parseString(\u0026#34;\u0026#34;\u0026#34;akka.loggers = [\u0026#34;akka.testkit.TestEventListener\u0026#34;]\u0026#34;\u0026#34;\u0026#34;)) println (system.settings.config.getValue(\u0026#34;akka.loggers\u0026#34;)) // Results in \u0026gt; SimpleConfigList([\u0026#34;akka.testkit.TestEventListener\u0026#34;]) Extending default configuration Other than overriding, you could also extend the default configuration with your custom configuration using the withFallback method of the Config.\nLet\u0026rsquo;s say your application.conf looks like :\n1 2 3 4 5 akka{ loggers = [\u0026#34;akka.event.slf4j.Slf4jLogger\u0026#34;] loglevel = DEBUG arun=\u0026#34;hello\u0026#34; } and you decide to override the akka.loggers property like :\n1 2 val config=ConfigFactory.parseString(\u0026#34;\u0026#34;\u0026#34;akka.loggers = [\u0026#34;akka.testkit.TestEventListener\u0026#34;]\u0026#34;\u0026#34;\u0026#34;) val system=ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;, config.withFallback(ConfigFactory.load())) You end up with a merged configuration of both :\n1 2 3 println (system.settings.config.getValue(\u0026#34;akka.arun\u0026#34;)) //\u0026gt; ConfigString(\u0026#34;hello\u0026#34;) println (system.settings.config.getValue(\u0026#34;akka.loggers\u0026#34;)) //\u0026gt; SimpleConfigList([\u0026#34;akka.testkit.TestEventListener\u0026#34;]) So, why did I tell this whole story on configuration? Because our ActorSystem is the one which loads and provides access to all the configuration information.\nIMPORTANT NOTE :\nWatch out the order of falling back here - which is the default and which is the extension configuration. Remember, you have to fall back to the default configuration. So,\n1 config.withFallback(ConfigFactory.load()) would work\nbut\n1 ConfigFactory.load().withFallback(config) would not get the results that you may need.\nScheduler As you can see from the API of ActorSystem, there is a powerful little method in ActorSystem called scheduler which returns a Scheduler. The Scheduler has a variety of schedule methods with which we could do some fun stuff inside the Actor environment.\nSchedule something to execute once Taking our Student-Teacher example, assume our StudentActor would want to send message to the teacher only after 5 seconds of it receiving the InitSignal from our Testcase and not immediately, our code looks like :\n1 2 3 4 5 6 7 8 9 10 11 12 13 class StudentDelayedActor (teacherActorRef:ActorRef) extends Actor with ActorLogging { def receive = { case InitSignal=\u0026gt; { import context.dispatcher context.system.scheduler.scheduleOnce(5 seconds, teacherActorRef, QuoteRequest) //teacherActorRef!QuoteRequest } ... ... } } Testcase\nLet\u0026rsquo;s cook up a testcase to verify this :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 \u0026#34;A delayed student\u0026#34; must { \u0026#34;fire the QuoteRequest after 5 seconds when an InitSignal is sent to it\u0026#34; in { import me.rerun.akkanotes.messaging.protocols.StudentProtocol._ val teacherRef = system.actorOf(Props[TeacherActor], \u0026#34;teacherActorDelayed\u0026#34;) val studentRef = system.actorOf(Props(new StudentDelayedActor(teacherRef)), \u0026#34;studentDelayedActor\u0026#34;) EventFilter.info (start=\u0026#34;Printing from Student Actor\u0026#34;, occurrences=1).intercept{ studentRef!InitSignal } } } Increasing the timeout for Eventfilter interception\nOuch. The default timeout for the EventFilter to wait for the message to appear in the EventStream is 3 seconds. Let\u0026rsquo;s increase that to 7 seconds now to verify our testcase. The filter-leeway configuration property helps us achieve that.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 class RequestResponseTest extends TestKit(ActorSystem(\u0026#34;TestUniversityMessageSystem\u0026#34;, ConfigFactory.parseString(\u0026#34;\u0026#34;\u0026#34; akka{ loggers = [\u0026#34;akka.testkit.TestEventListener\u0026#34;] test{ filter-leeway = 7s } } \u0026#34;\u0026#34;\u0026#34;))) with WordSpecLike with MustMatchers with BeforeAndAfterAll with ImplicitSender { ... ... Schedule something to execute repeatedly In order to execute something repeatedly, you use the schedule method of the Scheduler.\nOne of the frequently used overload of the schedule method is the one which sends a message to the Actor on a regular basis. It acccepts 4 parameters :\nHow long should be initial delay be before the first execution begins Frequency of subsequent executions The target ActorRef that we are going to send a message to The Message 1 2 3 4 5 case InitSignal=\u0026gt; { import context.dispatcher context.system.scheduler.schedule(0 seconds, 5 seconds, teacherActorRef, QuoteRequest) //teacherActorRef!QuoteRequest } TRIVIA\nThe import import context.dispatcher is very important here.\nThe schedule methods requires a very important implicit parameter - ExecutionContext, the reason for which would be pretty obvious once we see the implementation of the schedule method :\n1 2 3 4 5 6 7 8 9 10 11 12 13 final def schedule( initialDelay: FiniteDuration, interval: FiniteDuration, receiver: ActorRef, message: Any)(implicit executor: ExecutionContext, sender: ActorRef = Actor.noSender): Cancellable = schedule(initialDelay, interval, new Runnable { def run = { receiver ! message if (receiver.isTerminated) throw new SchedulerException(\u0026#34;timer active for terminated actor\u0026#34;) } }) The schedule method just wraps the tell in a Runnable which eventually is executed by the ExecutionContext that we pass in.\nIn order to make an ExecutionContext available in scope as an implicit, we leverage upon the implicit dispatcher available on the context.\nFrom ActorCell.scala (Context)\n1 2 3 4 5 /** * Returns the dispatcher (MessageDispatcher) that is used for this Actor. * Importing this member will place an implicit ExecutionContext in scope. */ implicit def dispatcher: ExecutionContextExecutor Code As always, the entire project could be downloaded from github here.\n","date":"2014-10-06T14:31:09Z","permalink":"/akka-notes-actorsystem-in-progress/","title":"Akka Notes - ActorSystem (Configuration and Scheduling) - 4"},{"content":"Last time when we saw Actor messaging, we saw how fire-n-forget messages are sent (Meaning, we just send a message to the Actor but don\u0026rsquo;t expect a response from the Actor).\nTechnically, we fire messages to Actors for its side-effects ALL THE TIME. It is by design. Other than not responding, the target Actor could ALSO do the following with that message -\nSend a response back to the sender (in our case, the TeacherActor would respond with a quote back to the StudentActor OR Forward a response back to some other Actor who might be the intended audience which in turn might respond/forward/have a side-effect. Routers and Supervisors are examples of those cases. (we\u0026rsquo;ll look at them very soon) Request \u0026amp; Response In this write-up, we\u0026rsquo;ll be focussing only on Point 1 - the request-response cycle.\nThe picture conveys what we are trying to achieve this time. For sake of brevity, I didn\u0026rsquo;t represent the ActorSystem, Dispatcher or Mailboxes in the picture.\nThe DriverApp sends an InitSignal message to the StudentActor. The StudentActor reacts to the InitSignal message and sends a QuoteRequest message to the TeacherActor. The TeacherActor, like we saw in the first discussion, responds with a QuoteResponse. The StudentActor just logs the QuoteResponse to the console/logger. We\u0026rsquo;ll also cook up a testcase to verify it.\nLet\u0026rsquo;s look at these 4 points in detail now :\nThe DriverApp sends an InitSignal message to the StudentActor By now, you would have guessed what would the DriverApp do. Just 4 things :\nInitialize the ActorSystem 1 2 //Initialize the ActorSystem val system = ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;) Create the TeacherActor 1 2 //create the teacher actor val teacherRef = system.actorOf(Props[TeacherActor], \u0026#34;teacherActor\u0026#34;) Create the StudentActor 1 2 //create the Student Actor - pass the teacher actorref as a constructor parameter to StudentActor val studentRef = system.actorOf(Props(new StudentActor(teacherRef)), \u0026#34;studentActor\u0026#34;) You\u0026rsquo;ll notice that I am passing in the ActorRef of the TeacherActor to the constructor of the StudentActor so that the StudentActor could use the ActorRef for sending messages to the TeacherActor. There are other ways to achieve this (like passing in the Props) but this method would come in handy when we look at Supervisors and Routers in the following write-ups. We\u0026rsquo;ll also be looking at child actors pretty soon but that wouldn\u0026rsquo;t semantically be the right approach here - Student creating Teacher doesn\u0026rsquo;t sound nice. Does it?\nLastly,\nThe DriverApp would then send an InitSignal to the StudentActor, so that the StudentActor could start sending the QuoteRequest message to the TeacherActor. 1 2 //send a message to the Student Actor studentRef ! InitSignal That\u0026rsquo;s pretty much the DriverClass. The Thread.sleep and the ActorSystem.shutdown are just to wait for a couple of seconds for the message sending to finish before we finally shut down the ActorSystem.\nDriverApp.scala 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 package me.rerun.akkanotes.messaging.requestresponse import akka.actor.ActorSystem import akka.actor.Props import me.rerun.akkanotes.messaging.protocols.StudentProtocol._ import akka.actor.ActorRef object DriverApp extends App { //Initialize the ActorSystem val system = ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;) //construct the teacher actor val teacherRef = system.actorOf(Props[TeacherActor], \u0026#34;teacherActor\u0026#34;) //construct the Student Actor - pass the teacher actorref as a constructor parameter to StudentActor val studentRef = system.actorOf(Props(new StudentActor(teacherRef)), \u0026#34;studentActor\u0026#34;) //send a message to the Student Actor studentRef ! InitSignal //Let\u0026#39;s wait for a couple of seconds before we shut down the system Thread.sleep(2000) //Shut down the ActorSystem. system.shutdown() } The StudentActor reacts to the InitSignal message and sends a QuoteRequest message to the TeacherActor AND\nThe StudentActor receives the QuoteResponse from TeacherActor and just logs to the console/logger Why did I combine Points 2 and 4? Because it is so simple you\u0026rsquo;ll hate me if I separate them.\nSo, Point 2 - the StudentActor receives the InitSignal message from the DriverApp and sends QuoteRequest to the TeacherActor.\n1 2 3 4 5 6 def receive = { case InitSignal=\u0026gt; { teacherActorRef!QuoteRequest } ... ... That\u0026rsquo;s it !!!\nPoint 4 - The StudentActor logs the message that it receives from the TeacherActor.\nJust, as promised :\n1 2 3 4 case QuoteResponse(quoteString) =\u0026gt; { log.info (\u0026#34;Received QuoteResponse from Teacher\u0026#34;) log.info(s\u0026#34;Printing from Student Actor $quoteString\u0026#34;) } I am sure you\u0026rsquo;d agree that it almost looks like pseudocode now.\nSo, the entire StudentActor class looks like :\nStudentActor.scala 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 package me.rerun.akkanotes.messaging.requestresponse import akka.actor.Actor import akka.actor.ActorLogging import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._ import me.rerun.akkanotes.messaging.protocols.StudentProtocol._ import akka.actor.Props import akka.actor.ActorRef class StudentActor (teacherActorRef:ActorRef) extends Actor with ActorLogging { def receive = { case InitSignal=\u0026gt; { teacherActorRef!QuoteRequest } case QuoteResponse(quoteString) =\u0026gt; { log.info (\u0026#34;Received QuoteResponse from Teacher\u0026#34;) log.info(s\u0026#34;Printing from Student Actor $quoteString\u0026#34;) } } } The TeacherActor responds with a QuoteResponse. This is the exact same code as we saw in the fire-n-forget write-up.\nThe TeacherActor receives a QuoteRequest message and sends QuoteResponse back.\nTeacherActor.scala 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 package me.rerun.akkanotes.messaging.requestresponse import scala.util.Random import akka.actor.Actor import akka.actor.ActorLogging import akka.actor.actorRef2Scala import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._ class TeacherActor extends Actor with ActorLogging { val quotes = List( \u0026#34;Moderation is for cowards\u0026#34;, \u0026#34;Anything worth doing is worth overdoing\u0026#34;, \u0026#34;The trouble is you think you have time\u0026#34;, \u0026#34;You never gonna know if you never even try\u0026#34;) def receive = { case QuoteRequest =\u0026gt; { import util.Random //Get a random Quote from the list and construct a response val quoteResponse = QuoteResponse(quotes(Random.nextInt(quotes.size))) //respond back to the Student who is the original sender of QuoteRequest sender ! quoteResponse } } } Testcases Now, our testcase would simulate the DriverApp. Since, the StudentActor just logs the message and we won\u0026rsquo;t be able to assert on the QuoteResponse itself, we\u0026rsquo;ll just assert the presence of the log message in the EventStream (just like we talked last time)\nSo, our testcase looks like :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 \u0026#34;A student\u0026#34; must { \u0026#34;log a QuoteResponse eventually when an InitSignal is sent to it\u0026#34; in { import me.rerun.akkanotes.messaging.protocols.StudentProtocol._ val teacherRef = system.actorOf(Props[TeacherActor], \u0026#34;teacherActor\u0026#34;) val studentRef = system.actorOf(Props(new StudentActor(teacherRef)), \u0026#34;studentActor\u0026#34;) EventFilter.info (start=\u0026#34;Printing from Student Actor\u0026#34;, occurrences=1).intercept{ studentRef!InitSignal } } } Code The entire project could be downloaded from github here.\nNext up, we\u0026rsquo;ll see how to use schedulers in Akka and monitoring your Akka app using Kamon\n","date":"2014-10-06T02:01:43Z","permalink":"/akka-notes-actor-messaging-request-and-response-3/","title":"Akka Notes - Actor Messaging - Request and Response - 3"},{"content":"In the first two parts (one, two), we briefly talked about Actors and how messaging works. In this part, let\u0026rsquo;s look at fixing up Logging and Testing our TeacherActor.\nRecap This is how our Actor from the previous part looked like :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 class TeacherActor extends Actor { val quotes = List( \u0026#34;Moderation is for cowards\u0026#34;, \u0026#34;Anything worth doing is worth overdoing\u0026#34;, \u0026#34;The trouble is you think you have time\u0026#34;, \u0026#34;You never gonna know if you never even try\u0026#34;) def receive = { case QuoteRequest =\u0026gt; { import util.Random //Get a random Quote from the list and construct a response val quoteResponse=QuoteResponse(quotes(Random.nextInt(quotes.size))) println (quoteResponse) } } } Logging Akka with SLF4J You notice that in the code we are printing the quoteResponse to the standard output which you would obviously agree is a bad idea. Let\u0026rsquo;s fix that up by enabling logging via the SLF4J Facade.\nFix the Class to use Logging Akka provides a nice little trait called ActorLogging to achieve it. Let\u0026rsquo;s mix that in :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 class TeacherLogActor extends Actor with ActorLogging { val quotes = List( \u0026#34;Moderation is for cowards\u0026#34;, \u0026#34;Anything worth doing is worth overdoing\u0026#34;, \u0026#34;The trouble is you think you have time\u0026#34;, \u0026#34;You never gonna know if you never even try\u0026#34;) def receive = { case QuoteRequest =\u0026gt; { import util.Random //get a random element (for now) val quoteResponse=QuoteResponse(quotes(Random.nextInt(quotes.size))) log.info(quoteResponse.toString()) } } //We\u0026#39;ll cover the purpose of this method in the Testing section def quoteList=quotes } A small detour here :\nInternally, when we log a message, the the logging methods in the ActorLogging (eventually) publishes the log message to an EventStream. Yes, I did say publish. So, what actually is an EventStream?\nEventStream and Logging\nEventStream behaves just like a message broker to which we could publish and receive messages. One subtle distinction from a regular MOM is that the subscribers of the EventStream could only be an Actor.\nIn case of logging messages, all log messages would be published to the EventStream. By default, the Actor that subscribes to these messages is the DefaultLogger which simply prints the message to the standard output.\n1 2 3 4 5 6 class DefaultLogger extends Actor with StdOutLogger { override def receive: Receive = { ... case event: LogEvent ⇒ print(event) } } So, that\u0026rsquo;s the reason when we try to kick off the StudentSimulatorApp, we see the log message written to the console.\nThat said, EventStream isn\u0026rsquo;t suited only for logging. It is a general purpose publish-subscribe mechanism available inside the ActorWorld inside a VM (more on that later).\nBack to SLF4J setup :\nConfigure Akka to use SLF4J 1 2 3 4 5 6 akka{ loggers = [\u0026#34;akka.event.slf4j.Slf4jLogger\u0026#34;] loglevel = \u0026#34;DEBUG\u0026#34; logging-filter = \u0026#34;akka.event.slf4j.Slf4jLoggingFilter\u0026#34; } We store this information in a file called application.conf which should be in your classpath. In our sbt folder structure, we would throw this in your main/resources directory.\nFrom the configuration, we could derived that\nthe loggers property indicates the Actor that is going to subscribe to the log events. What Slf4jLogger does is to simply consume the log messages and delegate that to the SLF4J Logger facade. the loglevel property simply indicates the minimum level that should be considered for logging. the logging-filter compares the currently configured loglevel and incoming log message level and chucks out any log message below the configured loglevel before publishing to the EventStream. But Why didn\u0026rsquo;t we have an application.conf for the previous example?\nSimply because Akka provides some sane defaults so that we needn\u0026rsquo;t build a configuration file before we start playing with it. We\u0026rsquo;ll revisit this file too often here on for customizing various things. There are a whole bunch of awesome parameters that you could use inside the application.conf for logging alone. They are explained in detail here.\nThrow in a logback.xml We\u0026rsquo;ll be configuring an SLF4J logger backed by logback now.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;appender name=\u0026#34;FILE\u0026#34; class=\u0026#34;ch.qos.logback.core.rolling.RollingFileAppender\u0026#34;\u0026gt; \u0026lt;encoder\u0026gt; \u0026lt;pattern\u0026gt;%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n\u0026lt;/pattern\u0026gt; \u0026lt;/encoder\u0026gt; \u0026lt;rollingPolicy class=\u0026#34;ch.qos.logback.core.rolling.TimeBasedRollingPolicy\u0026#34;\u0026gt; \u0026lt;fileNamePattern\u0026gt;logs\\akka.%d{yyyy-MM-dd}.%i.log\u0026lt;/fileNamePattern\u0026gt; \u0026lt;timeBasedFileNamingAndTriggeringPolicy class=\u0026#34;ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP\u0026#34;\u0026gt; \u0026lt;maxFileSize\u0026gt;50MB\u0026lt;/maxFileSize\u0026gt; \u0026lt;/timeBasedFileNamingAndTriggeringPolicy\u0026gt; \u0026lt;/rollingPolicy\u0026gt; \u0026lt;/appender\u0026gt; \u0026lt;root level=\u0026#34;DEBUG\u0026#34;\u0026gt; \u0026lt;appender-ref ref=\u0026#34;FILE\u0026#34; /\u0026gt; \u0026lt;/root\u0026gt; \u0026lt;/configuration\u0026gt; I threw this inside the main/resources folder too along with application.conf. Please ensure that the main/resources is now in your eclipse or other IDE\u0026rsquo;s classpath. Also include logback and slf4j-api to your build.sbt.\nAnd when we kick off our StudentSimulatorApp and send a message to our new TeacherLogActor, the akkaxxxxx.log file that we configured looks like this.\nTesting Akka Please note that this is by no means an exhaustive coverage of Testing Akka. We would be building our tests on more features of Testing in the following parts under their respective topic headers. These testcases are aimed to cover the Actors we wrote earlier.\nWhile the StudentSimulatorApp does what we need, you would agree that it should be driven out of testcases.\nTo ease the testing pain, Akka came up with an amazing testing toolkit with which we could do some magical stuff like probing directly into the Actor implementation\u0026rsquo;s internals.\nEnough talk, let\u0026rsquo;s see the testcases.\nLet\u0026rsquo;s first try to map the StudentSimulatorApp to a Testcase.\nLet\u0026rsquo;s look at the declaration alone now.\n1 2 3 4 class TeacherPreTest extends TestKit(ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;)) with WordSpecLike with MustMatchers with BeforeAndAfterAll { So, from the definition of the TestCase class we see that :\nThe TestKit trait accepts an ActorSystem through which we would be creating Actors. Internally, the TestKit decorates the ActorSystem and replaces the default dispatcher too. We use WordSpec which is one of the many fun ways to write testcases with ScalaTest. The MustMatchers provide convenient methods to make the testcase look like natural language We mixin the BeforeAndAfterAll to shutdown the ActorSystem after the testcases are complete. The afterAll method that the trait provides is more like our tearDown in JUnit 1, 2 - Sending message to Actors The first testcase just sends a message to the PrintActor. It doesn\u0026rsquo;t assert anything :-( The second case sends message to the Log actor which uses the log field of the ActorLogging to publish the message to the EventStream. This doesn\u0026rsquo;t assert anything too :-( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 //1. Sends message to the Print Actor. Not even a testcase actually \u0026#34;A teacher\u0026#34; must { \u0026#34;print a quote when a QuoteRequest message is sent\u0026#34; in { val teacherRef = TestActorRef[TeacherActor] teacherRef ! QuoteRequest } } //2. Sends message to the Log Actor. Again, not a testcase per se \u0026#34;A teacher with ActorLogging\u0026#34; must { \u0026#34;log a quote when a QuoteRequest message is sent\u0026#34; in { val teacherRef = TestActorRef[TeacherLogActor] teacherRef ! QuoteRequest } - Asserting internal state of Actors The third case uses the underlyingActor method of the TestActorRef and calls upon the quoteList method of the TeacherActor. The quoteList method returns the list of quotes back. We use this list to assert its size.\nIf reference to quoteList throws you back, refer to the TeacherLogActor code listed above and look for\n1 2 3 //From TeacherLogActor //We\u0026#39;ll cover the purpose of this method in the Testing section def quoteList=quotes 1 2 3 4 5 6 7 //3. Asserts the internal State of the Log Actor. \u0026#34;have a quote list of size 4\u0026#34; in { val teacherRef = TestActorRef[TeacherLogActor] teacherRef.underlyingActor.quoteList must have size (4) teacherRef.underlyingActor.quoteList must have size (4) } - Asserting log messages As we discussed earlier in the EventStream and Logging section (above), all log messages go to the EventStream and the SLF4JLogger subscribes to it and uses its appenders to write to the log file/console etc. Wouldn\u0026rsquo;t it be nice to subscribe to the EventStream directly in our testcase and assert the presence of the log message itself? Looks like we can do that too.\nThis involves two steps :\nYou need to add an extra configuration to your TestKit like so : 1 2 3 4 5 class TeacherTest extends TestKit(ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;, ConfigFactory.parseString(\u0026#34;\u0026#34;\u0026#34;akka.loggers = [\u0026#34;akka.testkit.TestEventListener\u0026#34;]\u0026#34;\u0026#34;\u0026#34;))) with WordSpecLike with MustMatchers with BeforeAndAfterAll { Now that we have a subscription to the EventStream, we could assert it from our testcase as : 1 2 3 4 5 6 7 8 9 //4. Verifying log messages from eventStream \u0026#34;be verifiable via EventFilter in response to a QuoteRequest that is sent\u0026#34; in { val teacherRef = TestActorRef[TeacherLogActor] EventFilter.info(pattern = \u0026#34;QuoteResponse*\u0026#34;, occurrences = 1) intercept { teacherRef ! QuoteRequest } } The EventFilter.info block just intercepts for 1 log message which starts with QuoteResponse (pattern='QuoteResponse*). (You could also achieve it by using a start='QuoteResponse'. If there is no log message as a result of sending a message to the TeacherLogActor, the testcase would fail.\n- Testing Actors with constructor parameters Please note that the way we create Actors in the testcase is via the TestActorRef[TeacherLogActor] and not via system.actorOf. This is just so that we could get access to the Actor\u0026rsquo;s internals through the underlyingActor method in the TeacherActorRef. We wouldn\u0026rsquo;t be able to achieve this via the ActorRef that we have access during the regular runtime. (That doesn\u0026rsquo;t give us any excuse to use TestActorRef in production. You\u0026rsquo;ll be hunted down).\nIf the Actor accepts parameters, then the way we create TestActorRef would be like :\n1 2 val teacherRef = TestActorRef(new TeacherLogParameterActor(quotes)) The entire testcase would then look something like :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 //5. have a quote list of the same size as the input parameter \u0026#34; have a quote list of the same size as the input parameter\u0026#34; in { val quotes = List( \u0026#34;Moderation is for cowards\u0026#34;, \u0026#34;Anything worth doing is worth overdoing\u0026#34;, \u0026#34;The trouble is you think you have time\u0026#34;, \u0026#34;You never gonna know if you never even try\u0026#34;) val teacherRef = TestActorRef(new TeacherLogParameterActor(quotes)) //val teacherRef = TestActorRef(Props(new TeacherLogParameterActor(quotes))) teacherRef.underlyingActor.quoteList must have size (4) EventFilter.info(pattern = \u0026#34;QuoteResponse*\u0026#34;, occurrences = 1) intercept { teacherRef ! QuoteRequest } } Shutting down ActorSystem And finally, the afterAll lifecycle method\n1 2 3 4 override def afterAll() { super.afterAll() system.shutdown() } CODE As always, the entire project could be downloaded from github here.\n","date":"2014-09-29T07:36:01Z","permalink":"/akka-notes-logging-and-testing/","title":"Akka Notes - Logging and Testing Actors - 2"},{"content":"From the introductory first part of the Akka Notes, we saw a bird\u0026rsquo;s eye view of Actors in the Akka Toolkit. In this second part of the Akka Notes, we\u0026rsquo;ll look at the messaging part of Actors. As for the example, we would use the same Student-Teacher example that we discussed earlier.\nIn this first part of Actor Messaging, we\u0026rsquo;ll create the Teacher Actor and instead of the Student Actor, we\u0026rsquo;ll use a main program called StudentSimulatorApp.\nRevisiting Student-Teacher in Detail Let\u0026rsquo;s for now consider the message sent by the StudentSimulatorApp to the TeacherActor alone. When I say StudentSimulatorApp, I just mean a normal main program.\nThe picture conveys this :\n(if the terms are overwhelming, don\u0026rsquo;t worry, we\u0026rsquo;ll go through them in detail)\nStudent creates something called an ActorSystem It uses the ActorSystem to create something called as ActorRef. The QuoteRequest message is sent to the ActorRef (a proxy to TeacherActor) Actor ref passes the message along to a Dispatcher The Dispatcher enqueues the message in the target Actor\u0026rsquo;s MailBox. The Dispatcher then puts the Mailbox on a Thread (more on that in the next section). The MailBox dequeues a message and eventually delegates that to the actual Teacher Actor\u0026rsquo;s receive method. Like I said, don\u0026rsquo;t worry about it. Let\u0026rsquo;s look at each step in detail now. You can come back and revisit these five steps once we are done.\nThe StudentSimulatorApp program We would use this StudentSimulatorApp to bring up the JVM and initialize the ActorSystem.\nAs we understand from the picture, the StudentSimulatorApp\nCreates an ActorSystem Uses the ActorSystem to create a proxy to the Teacher Actor (ActorRef) Sends the QuoteRequest message to the proxy. Let\u0026rsquo;s focus on these three points alone now.\n1. Creating an ActorSystem\nActorSystem is the entry point into the ActorWorld. ActorSystems are through which you could create and stop Actors. Or even shutdown the entire Actor environment.\nOn the other end of the spectrum, Actors are hierarchical and the ActorSystem is also similar to the java.lang.Object or scala.Any for all Actors - meaning, it is the root for all Actors. When you create an Actor using the ActorSystem\u0026rsquo;s actorOf method, you create an Actor just below the ActorSystem.\nThe code for initializing the ActorSystem looks like\n1 2 val system=ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;) The UniversityMessageSystem is simply a cute name you give to your ActorSystem.\n2. Creating a Proxy for TeacherActor?\nLet\u0026rsquo;s consider the following snippet :\n1 2 val teacherActorRef:ActorRef=actorSystem.actorOf(Props[TeacherActor]) The actorOf is the Actor creation method in ActorSystem. But, as you can see, it doesn\u0026rsquo;t return a TeacherActor which we need. It returns something of type ActorRef.\nThe ActorRef acts as a Proxy for the actual Actors. The clients do not talk directly with the Actor. This is Actor Model\u0026rsquo;s way of avoiding direct access to any custom/private methods or variables in the TeacherActor or any Actor for that sake.\nTo repeat, you send messages only to the ActorRef and it eventually reaches your actual Actor. You can NEVER talk to your Actor directly. People will hate you to death if you find some mean ways to do that.\n3. Send a QuoteRequest to the Proxy\nIt\u0026rsquo;s an one-liner again. You just tell the QuoteRequest message to the ActorRef. The tell method in Actor is actually !. (there\u0026rsquo;s also a tell method in ActorRef which just delegates the call back to !)\n1 2 3 //send a message to the Teacher Actor teacherActorRef!QuoteRequest That\u0026rsquo;s it !!!\nIf you think I am lying, check the entire code of the StudentSimulatorApp below :\nStudentSimulatorApp.scala 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 package me.rerun.akkanotes.messaging.actormsg1 import akka.actor.ActorSystem import akka.actor.Props import akka.actor.actorRef2Scala import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._ object StudentSimulatorApp extends App{ //Initialize the ActorSystem val actorSystem=ActorSystem(\u0026#34;UniversityMessageSystem\u0026#34;) //construct the Teacher Actor Ref val teacherActorRef=actorSystem.actorOf(Props[TeacherActor]) //send a message to the Teacher Actor teacherActorRef!QuoteRequest //Let\u0026#39;s wait for a couple of seconds before we shut down the system Thread.sleep (2000) //Shut down the ActorSystem. actorSystem.shutdown() } Well, I cheated a little. You\u0026rsquo;ll have to shutdown the ActorSystem or otherwise, the JVM keeps running forever. And I am making the main thread sleep for a little while just to give the TeacherActor to finish off its task. I know this sounds stupid. Don\u0026rsquo;t worry about it. We\u0026rsquo;ll write some neat testcases in the next part in order to avoid this hack.\nThe Message We just told a QuoteRequest to the ActorRef but we didn\u0026rsquo;t see the message class at all !!\nHere it comes :\n(It is a recommended practice to wrap your messages in a nice object for easier organization)\nTeacherProtocol\n1 2 3 4 5 6 7 8 9 package me.rerun.akkanotes.messaging.protocols object TeacherProtocol{ case class QuoteRequest() case class QuoteResponse(quoteString:String) } As you know, the QuoteRequest is for the requests that come to the TeacherActor. The Actor would respond back with a QuoteResponse.\nDispatcher and a MailBox The ActorRef delegates the message handling functionality to the Dispatcher. Under the hood, while we created the ActorSystem and the ActorRef, a Dispatcher and a MailBox was created. Let\u0026rsquo;s see what they are about.\nMailBox\nEvery Actor has one MailBox (we\u0026rsquo;ll see one special case later). Per our analogy, every Teacher has one mailbox too. The Teacher has to check the mailbox and process the message. In Actor world, it\u0026rsquo;s the other way round - the mailbox, when it gets a chance uses the Actor to accomplish its work.\nAlso the mailbox has a queue to store and process the messages in a FIFO fashion - a little different from our regular inbox where the most latest is the one at the top.\nNow, the dispatcher\nDispatcher does some really cool stuff. From the looks of it, the Dispatcher just gets the message from the ActorRef and passes it on to the MailBox. But there\u0026rsquo;s one amazing thing happening behind the scenes :\nThe Dispatcher wraps an ExecutorService (ForkJoinPool or ThreadPoolExecutor). It executes the MailBox against this ExecutorService.\nCheck out this snippet from the Dispatcher\n1 2 3 4 5 6 7 protected[akka] override def registerForExecution(mbox: Mailbox, ...): Boolean = { ... try { executorService execute mbox ... } What? Did you just say you execute the MailBox?\nYup. We already saw that the MailBox holds all the messages in a Queue. Also since the Executor runs the MailBox, the MailBox must be a Thread. You\u0026rsquo;re right. That\u0026rsquo;s pretty much MailBox\u0026rsquo;s declaration and constructor.\nHere\u0026rsquo;s the signature of the Mailbox\n1 2 private[akka] abstract class Mailbox(val messageQueue: MessageQueue) extends SystemMessageQueue with Runnable Teacher Actor The MailBox, when it gets its run method fired, dequeues a message from the message queue and passes it to the Actor for processing.\nThe method that eventually gets called when you tell a message to an ActorRef is the receive method of the target Actor.\nThe TeacherActor is a rudimentary class which has a List of quotes and obviously the receive method which handles the messages.\nCheck this out :\nTeacherActor.scala\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 package me.rerun.akkanotes.messaging.actormsg1 import scala.util.Random import akka.actor.Actor import me.rerun.akkanotes.messaging.protocols.TeacherProtocol._ /* * Your Teacher Actor class. * * The class could use refinement by way of * using ActorLogging which uses the EventBus of the Actor framework * instead of the plain old System out * */ class TeacherActor extends Actor { val quotes = List( \u0026#34;Moderation is for cowards\u0026#34;, \u0026#34;Anything worth doing is worth overdoing\u0026#34;, \u0026#34;The trouble is you think you have time\u0026#34;, \u0026#34;You never gonna know if you never even try\u0026#34;) def receive = { case QuoteRequest =\u0026gt; { import util.Random //Get a random Quote from the list and construct a response val quoteResponse=QuoteResponse(quotes(Random.nextInt(quotes.size))) println (quoteResponse) } } } The TeacherActor\u0026rsquo;s receive method pattern matches for just one Message - the QuoteRequest (actually, it is a good practice to pattern match the default case but there\u0026rsquo;s an interesting story to tell there)\nAll that the receive method does is to\npattern match for QuoteRequest pick a random quote from the static list of quotes construct a QuoteResponse print the QuoteResponse to the console Code The entire project could be downloaded from github here.\nWe expand this example to Logging and Testing in here.\n","date":"2014-09-19T10:30:31Z","permalink":"/akka-notes-actor-messaging-1/","title":"Akka Notes - Actor Messaging - 1"},{"content":"Anyone who has done multithreading in the past won\u0026rsquo;t deny how hard and painful it is to manage multithreaded applications. I said manage because it starts out simple and it became a whole lot of fun once you start seeing performance improvements. However, it aches when you see that you don\u0026rsquo;t have a easier way to recover from errors in your sub-tasks OR those zombie bugs that you find hard to reproduce OR when your profiler shows that your threads are spending a lot of time blocking wastefully before writing to a shared state.\nI prefer not to talk about how Java concurrency API and their collections made it better and easier because I am sure if you are here, you probably needed more control over the sub-tasks or simply because you don\u0026rsquo;t like to write locks and synchronized blocks and would prefer a higher level of abstraction.\nIn this series of Akka Notes, we would go through simple Akka examples to explore the various features that we have in the toolkit.\nWhat are Actors? Akka\u0026rsquo;s Actors follow the Actor Model (duh!).\nTreat Actors like People. People who don\u0026rsquo;t talk to each other in person. They just talk through mails.\nLet\u0026rsquo;s expand on that a bit.\nMessaging Consider two persons - A wise Teacher and Student. The Student sends a mail every morning to the Teacher and the wise Teacher sends a wise quote back.\nPoints to note :\nThe student sends a mail. Once sent, the mail couldn\u0026rsquo;t be edited. Talk about natural immutability. The Teacher checks his mailbox when he wishes to do so. The Teacher also sends a mail back (immutable again). The student checks the mailbox at his own time. The student doesn\u0026rsquo;t wait for the reply. (no blocking) That pretty much sums up the basic block of the Actor Model - passing messages.\nConcurrency Now, imagine there are 3 wise teachers and 3 students - every student sends notes to every other teacher. What happens then? Nothing changes actually. Everybody has their own mailbox. One subtle point to note here is this :\nBy default, Mails in the mailbox are read/processed in the order they arrived.\nInternally, by default it is a ConcurrentLinkedQueue. And since nobody waits for the mail to be picked up, it is simply a non-blocking message. (There are a variety of built-in mailboxes including bounded and priority based. In fact, we could build one ourself too)\nFailover Imagine these 3 teachers are from three different departments - History, Geography and Philosophy.\nHistory teachers replies with a note on an Event in the past, Geography teachers sends an Interesting Place and Philosophy teachers, a quote. Each student sends message to each teacher and gets responses. The student doesnt care which teacher in the department sends the reply back. What if one day, a teacher falls sick? There has to be at least one teacher handling the mails from the department. In this case, another teacher in the department steps up and does the job.\nPoints to note :\nThere could be a pool of Actors who does different things. An Actor could do something that causes an exception. It wouldn\u0026rsquo;t be able to recover by itself. In which case a new Actor could be created in place of the old one. Alternatively, the Actor could just ignore that one particular message and proceed with the rest of the messages. These are called Directives and we\u0026rsquo;ll discuss them later. Multitasking For a twist, let\u0026rsquo;s assume that each of these teachers also send the exam score through mail too, if the student asks for it. Similarly, an the Actor could handle more than one type of message comfortably.\nChaining What if the student would like to get only one final consolidated trivia mail instead of three?\nWe could do that too with Actors too. We could chain the teachers as a hierarchy. We\u0026rsquo;ll come back to that later when we talk about Supervisors and revisit the same thought when we talk about Futures.\nAs requested by Mohan, let\u0026rsquo;s just try to map the analogy components with the the components in the Actor Model.\nStudents and the Teachers becomes our Actors. The Email Inbox becomes the Mailbox component. The request and the response can\u0026rsquo;t be modified. They are immutable objects. Finally, the MessageDispatcher component in Actor manages the mailbox and routes the messages to the respective Mailbox.\nEnough talk, let\u0026rsquo;s cook up some code\u0026hellip;.\n","date":"2014-09-11T09:04:06Z","permalink":"/introducing-actors-akka-notes-part-1/","title":"Akka Notes - Introducing Actors"},{"content":"I found the Knapsack problem tricky and interesting at the same time. I am sure if you are visiting this page, you already know the problem statement but just for the sake of completion :\nProblem : Given a Knapsack of a maximum capacity of W and N items each with its own value and weight, throw in items inside the Knapsack such that the final contents has the maximum value. Yikes !!!\nLink to the problem page in wiki\nHere\u0026rsquo;s the general way the problem is explained - Consider a thief gets into a home to rob and he carries a knapsack. There are fixed number of items in the home - each with its own weight and value - Jewellery, with less weight and highest value vs tables, with less value but a lot heavy. To add fuel to the fire, the thief has an old knapsack which has limited capacity. Obviously, he can\u0026rsquo;t split the table into half or jewellery into 3/4ths. He either takes it or leaves it.\nExample :\n1 2 3 4 5 6 7 Knapsack Max weight : W = 10 (units) Total items : N = 4 Values of items : v[] = {10, 40, 30, 50} Weight of items : w[] = {5, 4, 6, 3} A cursory look at the example data tells us that the max value that we could accommodate with the limit of max weight of 10 is 50 + 40 = 90 with a weight of 7.\nApproach : The way this is optimally solved is using dynamic programming - solving for smaller sets of knapsack problems and then expanding them for the bigger problem.\nLet\u0026rsquo;s build an Item x Weight array called V (Value array)\n1 V[N][W] = 4 rows * 10 columns Each of the values in this matrix represent a smaller Knapsack problem.\nBase case 1 : Let\u0026rsquo;s take the case of 0th column. It just means that the knapsack has 0 capacity. What can you hold in them? Nothing. So, let\u0026rsquo;s fill them up all with 0s\nBase case 2 : Let\u0026rsquo;s take the case of 0 row. It just means that there are no items in the house. What do you do hold in your knapsack if there are no items. Nothing again !!! All zeroes.\nSolution : Now, let\u0026rsquo;s start filling in the array row-wise. What does row 1 and column 1 mean? That given the first item (row), can you accommodate it in the knapsack with capacity 1 (column). Nope. The weight of the first item is 5. So, let\u0026rsquo;s fill in 0. In fact, we wouldn\u0026rsquo;t be able to fill in anything until we reach the column 5 (weight 5). Once we reach column 5 (which represents weight 5) on the first row, it means that we could accommodate item 1. Let\u0026rsquo;s fill in 10 there (remember, this is a Value array) Moving on, for weight 6 (column 6), can we accommodate anything else with the remaining weight of 1 (weight - weight of this item =\u0026gt; 6 - 5). Hey, remember, we are on the first item. So, it is kind of intuitive that the rest of the row will just be the same value too since we are unable to add in any other item for that extra weight that we have got. So, the next interesting thing happens when we reach the column 4 in third row. The current running weight is 4. We should check for the following cases.\nCan we accommodate Item 2 - Yes, we can. Item 2\u0026rsquo;s weight is 4. Is the value for the current weight is higher without Item 2? - Check the previous row for the same weight. Nope. the previous row* has 0 in it, since we were not able able accommodate Item 1 in weight 4. Can we accommodate two items in the same weight so that we could maximize the value? - Nope. The remaining weight after deducting the Item2\u0026rsquo;s weight is 0. Why previous row ? Simply because the previous row at weight 4 itself is a smaller knapsack solution which gives the max value that could be accumulated for that weight until that point (traversing through the items).\nExemplifying,\nThe value of the current item = 40 The weight of the current item = 4 The weight that is left over = 4 - 4 = 0 Check the row above (the Item above in case of Item 1 or the cumulative Max value in case of the rest of the rows). For the remaining weight 0, are we able to accommodate Item 1? Simply put, is there any value at all in the row above for the given weight) The calculation goes like so :\nTake the max value for the same weight without this item\nprevious row, same weight = 0\n=\u0026gt; V[item-1][weight]\nTake the value of the current item + value that we could accommodate with the remaining weight\nValue of current item\nvalue in previous row with weight 4 (total weight until now (4) - weight of the current item (4)) =\u0026gt; val[item-1] + V[item-1][weight-wt[item-1]]\nMax among the two is 40 (0 and 40)\nThe next and the most important event happens at column 9 and row 2. Meaning we have a weight of 9 and we have two items. Looking at the example data we could accommodate the first two items. Here, we consider few things\nThe value of the current item = 40 The weight of the current item = 4 The weight that is left over = 9 - 4 = 5 Check the row above. At the remaining weight 5, are we able to accommodate Item 1. So, the calculation is :\nTake the max value for the same weight without this item\nprevious row, same weight = 10\nTake the value of the current item + value that we could accumulate with the remaining weight\nValue of current item (40)\nvalue in previous row with weight 5 (total weight until now (9) - weight of the current item (4)) = 10\n10 vs 50 = 50\nAt the end of solving all these smaller problems, we just need to return the value at V[N][W] - Item 4 at Weight 10\nComplexity Analyzing the complexity of the solution is pretty straight-forward. We just have a loop for W within a loop of N =\u0026gt; O (NW)\nImplementation : Here comes the obligatory implementation code in Java\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 class Knapsack { public static void main(String[] args) throws Exception { int val[] = {10, 40, 30, 50}; int wt[] = {5, 4, 6, 3}; int W = 10; System.out.println(knapsack(val, wt, W)); } public static int knapsack(int val[], int wt[], int W) { //Get the total number of items. //Could be wt.length or val.length. Doesn\u0026#39;t matter int N = wt.length; //Create a matrix. //Items are in rows and weight at in columns +1 on each side int[][] V = new int[N + 1][W + 1]; //What if the knapsack\u0026#39;s capacity is 0 - Set //all columns at row 0 to be 0 for (int col = 0; col \u0026lt;= W; col++) { V[0][col] = 0; } //What if there are no items at home. //Fill the first row with 0 for (int row = 0; row \u0026lt;= N; row++) { V[row][0] = 0; } for (int item=1;item\u0026lt;=N;item++){ //Let\u0026#39;s fill the values row by row for (int weight=1;weight\u0026lt;=W;weight++){ //Is the current items weight less //than or equal to running weight if (wt[item-1]\u0026lt;=weight){ //Given a weight, check if the value of the current //item + value of the item that we could afford //with the remaining weight is greater than the value //without the current item itself V[item][weight]=Math.max (val[item-1]+V[item-1][weight-wt[item-1]], V[item-1][weight]); } else { //If the current item\u0026#39;s weight is more than the //running weight, just carry forward the value //without the current item V[item][weight]=V[item-1][weight]; } } } //Printing the matrix for (int[] rows : V) { for (int col : rows) { System.out.format(\u0026#34;%5d\u0026#34;, col); } System.out.println(); } return V[N][W]; } } ","date":"2014-05-27T13:25:00Z","permalink":"/the-knapsack-problem/","title":"The Knapsack problem"},{"content":"While the awesome Apache Camel team is busy fixing the handling of the multiple parameters in the query, here\u0026rsquo;s a workaround. Hopefully, this post will become obsolete with the next versions of Camel. (Currently, I use 2.7.5)\nProblem Query parameters more than 1 is passed as a null value into a Camel-CXF service. Say, if the URL has four query parameters as in\n1 name=arun\u0026amp;email=arun@arunma.com\u0026amp;age=10\u0026amp;phone=123456 only the first one gets populated when you do a\n1 2 3 4 5 6 7 8 9 @GET @Path(\u0026#34;search\u0026#34;) @Produces(MediaType.APPLICATION_JSON) public String sourceResultsFromTwoSources(@QueryParam(\u0026#34;name\u0026#34;) String name, @QueryParam(\u0026#34;age\u0026#34;) String age, @QueryParam(\u0026#34;phone\u0026#34;) String phone, @QueryParam(\u0026#34;email\u0026#34;) String email ); All other parameters are null.\nFinal Output For url\nhttp://localhost:8181/cxf/karafcxfcamel/search?name=arun\u0026amp;email=arun@arunma.com\u0026amp;age=31\u0026amp;phone=232323\nthe result expected is :\nWorkaround Interestingly, we could get the entire query string in the header.\n1 String queryString = exchange.getIn().getHeader(Exchange.HTTP_QUERY, String.class); We could then do a\n1 MultivaluedMap\u0026lt;String, String\u0026gt; queryMap = JAXRSUtils.getStructuredParams(queryString, \u0026#34;\u0026amp;\u0026#34;, false, false); to get the query parameters as a multi valued Map.\nThe query parameters could then be set as a property to the Exchange and used across the exchange.\nCode The entire code could be downloaded from github\nPlease note that I am running Camel as part of OSGi inside the Karaf container. While the workaround does not differ because of the environment in which you are using Camel-CXF, please be wary of this fact when you download the code from github. Watch out for the blueprint xmls for Camel configuration.\nThe most important piece here is the Router\nRouter 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 package me.rerun.karafcxfcamel.camel.beans; import org.apache.camel.Exchange; import org.apache.camel.Processor; import org.apache.camel.builder.RouteBuilder; import org.apache.camel.model.dataformat.JsonLibrary; import org.apache.cxf.jaxrs.utils.JAXRSUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import javax.ws.rs.core.MultivaluedMap; import java.util.List; import java.util.Map; public class RestToBeanRouter extends RouteBuilder { private static Logger logger= LoggerFactory.getLogger(RouteBuilder.class); @Override public void configure() throws Exception { from (\u0026#34;cxfrs://bean://rsServer\u0026#34;) .process(new ParamProcessor()) .multicast() .parallelProcessing() .aggregationStrategy(new ResultAggregator()) .beanRef(\u0026#34;restServiceImpl\u0026#34;, \u0026#34;getNameEmailResult\u0026#34;) .beanRef(\u0026#34;restServiceImpl\u0026#34;, \u0026#34;getAgePhoneResult\u0026#34;) .end() .marshal().json(JsonLibrary.Jackson) .to(\u0026#34;log://camelLogger?level=DEBUG\u0026#34;); } private class ParamProcessor implements Processor { @Override public void process(Exchange exchange) throws Exception { String queryString = exchange.getIn().getHeader(Exchange.HTTP_QUERY, String.class); MultivaluedMap\u0026lt;String, String\u0026gt; queryMap = JAXRSUtils.getStructuredParams(queryString, \u0026#34;\u0026amp;\u0026#34;, false, false); for (Map.Entry\u0026lt;String, List\u0026lt;String\u0026gt;\u0026gt; eachQueryParam : queryMap.entrySet()) { exchange.setProperty(eachQueryParam.getKey(), eachQueryParam.getValue()); } } } } Interface 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 package me.rerun.karafcxfcamel.rest; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.core.MediaType; public interface RestService { @GET @Path(\u0026#34;search\u0026#34;) @Produces(MediaType.APPLICATION_JSON) public String sourceResultsFromTwoSources(); } Implementation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 package me.rerun.karafcxfcamel.rest; import me.rerun.karafcxfcamel.model.AgePhoneResult; import me.rerun.karafcxfcamel.model.NameEmailResult; import me.rerun.karafcxfcamel.service.base.AgePhoneService; import me.rerun.karafcxfcamel.service.base.NameEmailService; import me.rerun.karafcxfcamel.service.impl.AgePhoneServiceImpl; import org.apache.camel.Exchange; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.List; public class RestServiceImpl implements RestService { private static Logger logger= LoggerFactory.getLogger(AgePhoneServiceImpl.class); private NameEmailService nameEmailService; private AgePhoneService agePhoneService; public RestServiceImpl(){ } //Do nothing. Camel intercepts and routes the requests public String sourceResultsFromTwoSources() { return null; } public NameEmailResult getNameEmailResult(Exchange exchange){ logger.info(\u0026#34;Invoking getNameEmailResult from RestServiceImpl\u0026#34;); String name=getFirstEntrySafelyFromList(exchange.getProperty(\u0026#34;name\u0026#34;, List.class)); String email=getFirstEntrySafelyFromList(exchange.getProperty(\u0026#34;email\u0026#34;, List.class)); return nameEmailService.getNameAndEmail(name, email); } public AgePhoneResult getAgePhoneResult(Exchange exchange){ logger.info(\u0026#34;Invoking getAgePhoneResult from RestServiceImpl\u0026#34;); String age=getFirstEntrySafelyFromList(exchange.getProperty(\u0026#34;age\u0026#34;, List.class)); String phone=getFirstEntrySafelyFromList(exchange.getProperty(\u0026#34;phone\u0026#34;, List.class)); return agePhoneService.getAgePhoneResult(age, phone); } public NameEmailService getNameEmailService() { return nameEmailService; } public AgePhoneService getAgePhoneService() { return agePhoneService; } public void setNameEmailService(NameEmailService nameEmailService) { this.nameEmailService = nameEmailService; } public void setAgePhoneService(AgePhoneService agePhoneService) { this.agePhoneService = agePhoneService; } private String getFirstEntrySafelyFromList(List\u0026lt;String\u0026gt; list){ if (list!=null \u0026amp;\u0026amp; !list.isEmpty()){ return list.get(0); } return null; } } Reference Camel Mailing List Question\n","date":"2013-11-14T19:35:00Z","permalink":"/camel-cxf-service-with-multiple-query-parameters/","title":"Camel CXF Service with Multiple Query Parameters"},{"content":"For reasons best left untold, in my day job, I was expected to provide an SLF4J Adapter for ADF Logger Oracle ADF. Not surprisingly, slf4j does not have an adapter for ADFLogger but since ADFLogger was just a gentle wrapper over Java Util Logging, it took a little over an hour to fill that gap.\nThe testcases (more like main programs) in the repository will confirm that the adapter framework plays well with the Oracle Diagnostic Logging \u0026ndash;without breaking the format of the log messages. (which is more or less the only advantage which the ADFLogger provides).\nYou could download the entire codebase from the repository at github. Optionally, if you are interested only in the binding jar, please download it here.\nLog Levels : Considering there are many log levels on the ADF Logger (7) and fewer levels in SLF4J (5), a compromise was made on certain levels on the ADF Logger. The mapping is made as follows :\nNOTE : Support for Log levels FINER and CONFIG were dropped during this adaptation.\nUsage : Just go ahead and replace your ADFLogger with SLF4J instantiations.\n1 2 3 private static final Logger slfLogger = LoggerFactory.getLogger(LoggingChecker.class); instead of\n1 2 private static final ADFLogger adfLogger= ADFLogger.createADFLogger(LoggingChecker.class); Logging methods Logging to various levels 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 @Test public void testLoggerLevels(){ adfLogger.finest(\u0026#34;finest Message from ADF Logger\u0026#34;); slfLogger.trace(\u0026#34;finest Message from SLF Logger\u0026#34;); adfLogger.fine(\u0026#34;fine Message from ADF Logger\u0026#34;); slfLogger.debug(\u0026#34;fine Message from SLF Logger\u0026#34;); adfLogger.info(\u0026#34;info Message from ADF Logger\u0026#34;); slfLogger.info(\u0026#34;info Message from SLF Logger\u0026#34;); adfLogger.warning(\u0026#34;warning Message from ADF Logger\u0026#34;); slfLogger.warn(\u0026#34;warning Message from SLF Logger\u0026#34;); adfLogger.severe(\u0026#34;severe Message from ADF Logger\u0026#34;); slfLogger.error(\u0026#34;severe Message from SLF Logger\u0026#34;); } Logging exceptions where dummyException inherits a Throwable\nLogging parameters Development Notes : The ADF Logger Adapter framework composes of two core classes (the ADFLoggerFactory.java and the ADFLoggerAdapter.java) and three other helper classes. The Adapter can support multiple loggers including the ROOT logger of the ADFLogger (which is the default and has an empty string as its name). It is no surprise that SLF4J uses the ADFLoggerFactory.java to instantiate the ADFLoggerAdapter. For each Logger instance, the ADFLoggerAdapter.java composes an instance of ADFLogger (bound to the logger name) and delegates the call to the ADFLogger itself. The fillCallerData method of the ADFLoggerAdapter.java filters the stackframes of the ADFLoggerAdapter from the entire logging stack so that the caller class and method name is retained as the one from the host application and not of the Adapter framework classes. Testcase (well, sort of) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 public class TestClass { private static final ADFLogger adfLogger= ADFLogger.createADFLogger(TestClass.class); private static final Logger slfLogger = LoggerFactory.getLogger(TestClass.class); private List\u0026lt;String\u0026gt; dummyList1=null; private List\u0026lt;Integer\u0026gt; dummyList2=null; private Exception dummyException=null; @Before public void setUp(){ dummyList1=new ArrayList\u0026lt;String\u0026gt;(); dummyList1.add(\u0026#34;Rock\u0026#34;); dummyList1.add(\u0026#34;Paper\u0026#34;); dummyList1.add(\u0026#34;Scissors\u0026#34;); dummyList2=new ArrayList\u0026lt;Integer\u0026gt;(); dummyList2.add(21); dummyList2.add(22); dummyList2.add(23); dummyException=new Exception(\u0026#34;Mind blowing Exception\u0026#34;); } @Test public void testLoggerLevels(){ adfLogger.finest(\u0026#34;finest Message from ADF Logger\u0026#34;); slfLogger.trace(\u0026#34;finest Message from SLF Logger\u0026#34;); adfLogger.fine(\u0026#34;fine Message from ADF Logger\u0026#34;); slfLogger.debug(\u0026#34;fine Message from SLF Logger\u0026#34;); adfLogger.info(\u0026#34;info Message from ADF Logger\u0026#34;); slfLogger.info(\u0026#34;info Message from SLF Logger\u0026#34;); adfLogger.warning(\u0026#34;warning Message from ADF Logger\u0026#34;); slfLogger.warn(\u0026#34;warning Message from SLF Logger\u0026#34;); adfLogger.severe(\u0026#34;severe Message from ADF Logger\u0026#34;); slfLogger.error(\u0026#34;severe Message from SLF Logger\u0026#34;); } @Test public void testException(){ adfLogger.severe(\u0026#34;severe Message from ADF Logger\u0026#34;, dummyException); slfLogger.error(\u0026#34;severe Message from SLF Logger\u0026#34;, dummyException); } @Test public void testParameters(){ adfLogger.severe(\u0026#34;severe Message from ADF Logger Param :{0}\u0026#34;, dummyList1 ); slfLogger.error(\u0026#34;severe Message from SLF Logger Param :{}\u0026#34;, dummyList1); adfLogger.severe(\u0026#34;severe Message from ADF Logger Param 1 :[{0}] \\n [{1}]\u0026#34;, new Object[]{dummyList1, dummyList2} ); slfLogger.error(\u0026#34;severe Message from SLF Logger Param :{} \\n {} \u0026#34;, dummyList1, dummyList2); slfLogger.error(\u0026#34;severe Message from SLF Logger Param :[{}] \\n [{}] \u0026#34;, new Object[]{dummyList1, dummyList2}); } Output : **Logger levels : **\n1 2 3 4 5 6 7 8 9 10 11 12 Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels INFO: info Message from ADF Logger Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels INFO: info Message from SLF Logger Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels WARNING: warning Message from ADF Logger Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels WARNING: warning Message from SLF Logger Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels SEVERE: severe Message from ADF Logger Jul 01, 2013 4:21:18 PM org.slf4j.test.TestClass testLoggerLevels SEVERE: severe Message from SLF Logger Testing Parameters :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters SEVERE: severe Message from ADF Logger Param :[Rock, Paper, Scissors] Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters SEVERE: severe Message from SLF Logger Param :[Rock, Paper, Scissors] Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters SEVERE: severe Message from ADF Logger Param 1 :[[Rock, Paper, Scissors]] [[21, 22, 23]] Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters SEVERE: severe Message from SLF Logger Param :[Rock, Paper, Scissors] [21, 22, 23] Jul 01, 2013 4:32:04 PM org.slf4j.test.TestClass testParameters SEVERE: severe Message from SLF Logger Param :[[Rock, Paper, Scissors]] [[21, 22, 23]] Testing Exception :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Jul 01, 2013 4:32:28 PM org.slf4j.test.TestClass testException SEVERE: severe Message from ADF Logger java.lang.Exception: Mind blowing Exception at org.slf4j.test.TestClass.setUp(TestClass.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) Jul 01, 2013 4:32:28 PM org.slf4j.test.TestClass testException SEVERE: severe Message from SLF Logger java.lang.Exception: Mind blowing Exception at org.slf4j.test.TestClass.setUp(TestClass.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) ","date":"2013-11-03T22:48:00Z","permalink":"/slf4j-binding-for-adflogger-the-missing-piece/","title":"SLF4J binding for ADFLogger - the missing piece"},{"content":"Please check out my other post on building plain CXF services (without Camel) in OSGi on Karaf.\nThis is a basic tutorial on how to\ncreate a CXF REST service multicast (and parallelize) the incoming request using Camel source data from two different services aggregate the response and finally return the consolidated result as JSON to the the end user. You could download the entire codebase from github.\nWhat this application does, in simple terms The result expected from this service is a hardcoded response which looks like\nAs you could see from the image, the top portion of the response is sourced from a service called NameEmailService and the second portion of the response is sourced from a service called AgePhoneService. The calls to enrich both the data are done concurrently and the consolidated result entity - ConsolidatedSearchResult is populated.\nThe Project structure looks like this :\nThere are two baby steps for Step 1.\nStep 1.a - Create a CXF REST Service As you might have guessed, there\u0026rsquo;s nothing complicated in this step. Just an interface and an implementation.\nInterface\n1 2 3 4 5 6 7 8 9 10 11 @Path(\u0026#34;rest\u0026#34;) public interface RestService { @GET @Path(\u0026#34;query/{queryString}\u0026#34;) @Produces(MediaType.APPLICATION_JSON) public String sourceResultsFromTwoSources(@PathParam(\u0026#34;queryString\u0026#34;) String queryString); } Implementation\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 public class RestServiceImpl implements RestService { private static Logger logger= LoggerFactory.getLogger(AgePhoneServiceImpl.class); private NameEmailService nameEmailService; private AgePhoneService agePhoneService; public RestServiceImpl(){ } //Do nothing. Camel intercepts and routes the requests public String sourceResultsFromTwoSources(String queryString) { return null; } public NameEmailResult getNameEmailResult(String queryString){ logger.info(\u0026#34;Invoking getNameEmailResult from RestServiceImpl\u0026#34;); return nameEmailService.getNameAndEmail(queryString); } public AgePhoneResult getAgePhoneResult(String queryString){ logger.info(\u0026#34;Invoking getAgePhoneResult from RestServiceImpl\u0026#34;); return agePhoneService.getAgePhoneResult(queryString); } public NameEmailService getNameEmailService() { return nameEmailService; } public AgePhoneService getAgePhoneService() { return agePhoneService; } public void setNameEmailService(NameEmailService nameEmailService) { this.nameEmailService = nameEmailService; } public void setAgePhoneService(AgePhoneService agePhoneService) { this.agePhoneService = agePhoneService; } } Note that the method implementation sourceResultsFromTwoSources returns a null. The truth is that this method doesn\u0026rsquo;t even get called when making a REST call. Camel intercepts all requests to the URL and routes it to various endpoints (calls two methods - getNameEmailResult() and getAgePhoneResult(), in our case).\nStep 1.b - Create the Service Implementation Kiddish implementations of the NameEmailService and the AgePhoneService are below :\n1 2 3 4 5 6 7 8 9 10 public class NameEmailServiceImpl implements NameEmailService { public NameEmailResult getNameAndEmail(String queryString){ return new NameEmailResult(\u0026#34;Arun\u0026#34;, \u0026#34;arun@arunma.com\u0026#34;); } } 1 2 3 4 5 6 7 public class AgePhoneServiceImpl implements AgePhoneService { public AgePhoneResult getAgePhoneResult(String queryString){ return new AgePhoneResult(32, \u0026#34;111-222-333\u0026#34;); } } Step 2, 3, 4 \u0026amp; 5 Well, I lied when I said 2,3,4 and 5 were 4 steps. They are all done as a single step using Camel routing and its Enterprise Integration Pattern implementations.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 public class RestToBeanRouter extends RouteBuilder { @Override public void configure() throws Exception { from (\u0026#34;cxfrs://bean://rsServer\u0026#34;) .multicast() .parallelProcessing() .aggregationStrategy(new ResultAggregator()) .beanRef(\u0026#34;restServiceImpl\u0026#34;, \u0026#34;getNameEmailResult\u0026#34;) .beanRef(\u0026#34;restServiceImpl\u0026#34;, \u0026#34;getAgePhoneResult\u0026#34;) .end() .marshal().json(JsonLibrary.Jackson) .to(\u0026#34;log://camelLogger?level=DEBUG\u0026#34;); } } Our Routing explained Simply put, what our routerbuilder does is that it\nfrom (\u0026quot;cxfrs://bean://rsServer\u0026quot;) intercepts all requests to a JAX-RS server endpoint defined in the rest-blueprint.xml as 1 2 3 4 5 \u0026lt;cxf:rsServer id=\u0026#34;rsServer\u0026#34; address=\u0026#34;/karafcxfcamel\u0026#34; serviceClass=\u0026#34;me.rerun.karafcxfcamel.rest.RestServiceImpl\u0026#34; loggingFeatureEnabled=\u0026#34;true\u0026#34; /\u0026gt; The .multicast() forwards the original request untouched to\ngetNameEmailResult \u0026amp; getAgePhoneResult methods in RestServiceImpl The .parallelProcessing() places concurrent calls to the methods.\nThe .aggregationStrategy(new ResultAggregator()) specifies how the results from various multicasted sources should be, well, aggregated.\nOur aggregator looks like :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 public class ResultAggregator implements AggregationStrategy { @Override public Exchange aggregate(Exchange oldExchange, Exchange newExchange) { ConsolidatedSearchResult consolidatedSearchResult=null; if (oldExchange==null){ consolidatedSearchResult=new ConsolidatedSearchResult(); } else{ consolidatedSearchResult=oldExchange.getIn().getBody(ConsolidatedSearchResult.class); } NameEmailResult nameEmailResult=newExchange.getIn().getBody(NameEmailResult.class); AgePhoneResult agePhoneResult=newExchange.getIn().getBody(AgePhoneResult.class); if (nameEmailResult!=null){ consolidatedSearchResult.setNameEmailResult(nameEmailResult); } if (agePhoneResult!=null){ consolidatedSearchResult.setAgePhoneResult(agePhoneResult); } newExchange.getIn().setBody(consolidatedSearchResult); return newExchange; } } Our Aggregator explained The aggregate method in our ResultAggregator is a little crude but does the job.\nThe aggregate method gets called for all multicasted endpoints whenever they finish.\nSo, the first time, the oldExchange will be null. We take that as an opportunity to construct the final consolidated result entity that we wanted to respond to the user.\nWe check whether the newExchange that comes in is the result of a call to the NameEmailService or AgePhoneService and populate the consolidated entity accordingly.\nFinally, we return the consolidated entity - the returning does two jobs.\nThe consolidated entity comes in as oldExchange for the next call to the aggregate method. (more like chaining - the last returned object from the entity is the one which comes in as the incoming exchange for the next call) Gets returned back to the user if it is the last call of aggregate (all multicast endpoints calls are complete). ","date":"2013-10-20T16:02:00Z","permalink":"/building-camel-cxf-rest-service-in-osgi-for-karaf-multicasting-and-aggregation/","title":"Building Camel-CXF REST Service in OSGi for Karaf - Multicasting and Aggregation"},{"content":"I\u0026rsquo;ll leave it to the experts to tell how awesome OSGi is. Among the many benefits, I could tell you why we picked up OSGi for a pet project - Modularity, avoiding JAR hell and dynamic updates (hey, why not?)\nWe chose Apache Felix (an OSGi framework specification implementation) and Apache Karaf (ummm, how do I put this - something like an app server for OSGi applications). Besides serving as an OSGi container, Karaf has a lot of awesome features (pun intended). And we like the idea of managing multiple Karaf instances managed through Zookeeper.\nEnough talk, let\u0026rsquo;s see some code. This is a rudimentary tutorial on how to run a basic OSGi CXF-JAX Rest Service on Karaf. Given a REST parameter (a name), the service would just return Hello, (name). That\u0026rsquo;s it !! The entire project could be downloaded here\nSo, if the request is http://localhost:8181/cxf/karafsimple/say/hello/arun, the response would be Hello, arun\nPlease note that this project does not have any domain models and therefore has only two projects - one for the REST (Controller) and the other for the actual service implementation.\nThe project structure looks like this :\nStep 1 - Service Implementation This project just has two \u0026lsquo;useful\u0026rsquo; things - the HelloService interface and the HelloServiceImpl class.\n1 2 3 4 5 6 7 package me.rerun.karafcxf.service.impl; public interface HelloService { public String sayHello(String name); } 1 2 3 4 5 6 7 8 9 10 11 package me.rerun.karafcxf.service.impl; public class HelloServiceImpl implements HelloService { @Override public String sayHello(String name) { return \u0026#34;Hello, \u0026#34;+name; } } Stupid right?\nStep 2 - REST project Similar to the Service project, this project also has just two notable things - the HelloRestService interface and the HelloRestServiceImpl class.\nHelloRestService\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 package me.rerun.karafcxf.rest; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.PathParam; //Maps for the `say` in the URL @Path(\u0026#34;say\u0026#34;) public interface HelloRestService { @GET @Path(\u0026#34;hello/{name}\u0026#34;) //Maps for the `hello/John` in the URL public String handleGet(@PathParam(\u0026#34;name\u0026#34;) String name); } HelloRestServiceImpl\nThe HelloRestServiceImpl does nothing but calls the HelloService which gets injected through Blueprint Dependency Injection. Hey, does the inject look very familiar to Spring DI? Exactly !! The Blueprint DI is heavily influenced by Spring DI. In fact, the original work for blueprint is done by Spring.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 package me.rerun.karafcxf.rest; import me.rerun.karafcxf.service.impl.HelloService; public class HelloRestServiceImpl implements HelloRestService{ //Just like Spring. Please add Getters/Setters. Blueprint annotations are still work in progress private HelloService helloService; public String handleGet(String name){ return helloService.sayHello(name); } /* Constructor */ public HelloRestServiceImpl(){ } /* Getters and Setters */ public HelloService getHelloService() { return helloService; } public void setHelloService(HelloService helloService) { this.helloService = helloService; } } Step 3 - Injections XMLs (of any name) inside OSGI-INF/blueprint folder will get picked up for DI scanning.\nserviceimpl.xml Does two things in one tag :\nRegisters the HelloService into the service registry for lookup Says that its implementation is HelloServiceImpl 1 2 3 4 5 6 7 8 9 10 11 \u0026lt;blueprint xmlns=\u0026#34;http://www.osgi.org/xmlns/blueprint/v1.0.0\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34; xsi:schemaLocation=\u0026#34;http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd\u0026#34;\u0026gt; \u0026lt;service ref=\u0026#34;helloServiceBean\u0026#34; interface=\u0026#34;me.rerun.karafcxf.service.impl.HelloService\u0026#34;\u0026gt; \u0026lt;bean class=\u0026#34;me.rerun.karafcxf.service.impl.HelloServiceImpl\u0026#34;/\u0026gt; \u0026lt;/service\u0026gt; \u0026lt;/blueprint\u0026gt; In fact, you could do this in two separate steps. More on that here.\n1 2 3 4 5 \u0026lt;service id=\u0026#34;helloServiceBean\u0026#34; ref=\u0026#34;helloServiceImpl\u0026#34; interface=\u0026#34;me.rerun.karafcxf.service.impl.HelloService\u0026#34; /\u0026gt; \u0026lt;bean id=\u0026#34;helloServiceImpl\u0026#34; class=\u0026#34;me.rerun.karafcxf.service.impl.HelloServiceImpl\u0026#34; /\u0026gt; rest.xml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 \u0026lt;blueprint xmlns=\u0026#34;http://www.osgi.org/xmlns/blueprint/v1.0.0\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34; xmlns:cm=\u0026#34;http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0\u0026#34; xmlns:jaxws=\u0026#34;http://cxf.apache.org/blueprint/jaxws\u0026#34; xmlns:jaxrs=\u0026#34;http://cxf.apache.org/blueprint/jaxrs\u0026#34; xmlns:cxf=\u0026#34;http://cxf.apache.org/blueprint/core\u0026#34; xsi:schemaLocation=\u0026#34; http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd http://cxf.apache.org/blueprint/jaxws http://cxf.apache.org/schemas/blueprint/jaxws.xsd http://cxf.apache.org/blueprint/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd\u0026#34;\u0026gt; \u0026lt;!-- 1 --\u0026gt; \u0026lt;cxf:bus id=\u0026#34;cxfBus1\u0026#34;\u0026gt; \u0026lt;cxf:features\u0026gt; \u0026lt;cxf:logging/\u0026gt; \u0026lt;/cxf:features\u0026gt; \u0026lt;/cxf:bus\u0026gt; \u0026lt;!-- 2 --\u0026gt; \u0026lt;jaxrs:server address=\u0026#34;/karafsimple\u0026#34; id=\u0026#34;someRestService\u0026#34;\u0026gt; \u0026lt;jaxrs:serviceBeans\u0026gt; \u0026lt;ref component-id=\u0026#34;restServiceImpl\u0026#34;/\u0026gt; \u0026lt;/jaxrs:serviceBeans\u0026gt; \u0026lt;/jaxrs:server\u0026gt; \u0026lt;!-- 3 --\u0026gt; \u0026lt;!-- Implementation of the rest service --\u0026gt; \u0026lt;bean id=\u0026#34;restServiceImpl\u0026#34; class=\u0026#34;me.rerun.karafcxf.rest.HelloRestServiceImpl\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;helloService\u0026#34; ref=\u0026#34;helloServiceBean\u0026#34;/\u0026gt; \u0026lt;!--Points to the reference below --\u0026gt; \u0026lt;/bean\u0026gt; \u0026lt;!-- 4 --\u0026gt; \u0026lt;!-- This has to point to the service registered through serviceimpl.xml in the service.impl project --\u0026gt; \u0026lt;reference id=\u0026#34;helloServiceBean\u0026#34; interface=\u0026#34;me.rerun.karafcxf.service.impl.HelloService\u0026#34;\u0026gt; \u0026lt;/reference\u0026gt; \u0026lt;/blueprint\u0026gt; cxf-bus is the bus configuration for CXF. It is like the manager for all CXF services. The most common use as far as I know is to configure custom interceptors (for auditing, request/response manipulation, headers manipulation etc) the jaxrs:server initiates a server which would start listening to the URLs that we mapped for. Of course, we would want to map the handlers for the URLs and those go under the serviceBeans. The third note in the XML is the just the restServiceImpl configuration and the injection of the helloService as a property inside the HelloRestServiceImpl The reference tag will lookup the service registry for a bean with the same id. Step 4 - KAR - Karaf Archive (Optional but easier this way) Technically, you could just start throwing the bundles generated through the Maven into to the deploy directory of karaf. However, as the project goes big and your dependencies are becoming many, it is advisable to use the .kar archive. Picture .kar as your .war or .ear bundle. A .kar archive, internally, looks something like this :\nBuilding .kar Building the .kar is composed of two steps :\nStep 1 feature.xml\nSimilar to your web descriptor or application descriptor, we have the features descriptor in Karaf to represent the entire repository that we are bundling together to compose our application.\nIn our case, we don\u0026rsquo;t have any external .jar dependencies except for the cxf and the http service for which we are using the in-built features available inside karaf.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 \u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;features xmlns=\u0026#34;http://karaf.apache.org/xmlns/features/v1.0.0\u0026#34; name=\u0026#34;${project.artifactId}-${project.version}\u0026#34;\u0026gt; \u0026lt;feature name=\u0026#34;karafcxf\u0026#34; description=\u0026#34;karaf cxf hello\u0026#34; version=\u0026#34;1.0-SNAPSHOT\u0026#34; resolver=\u0026#34;(obr)\u0026#34;\u0026gt; \u0026lt;details\u0026gt;${project.description}\u0026lt;/details\u0026gt; \u0026lt;feature\u0026gt;http\u0026lt;/feature\u0026gt; \u0026lt;feature\u0026gt;cxf\u0026lt;/feature\u0026gt; \u0026lt;feature\u0026gt;this-project-dependants\u0026lt;/feature\u0026gt; \u0026lt;bundle\u0026gt;mvn:karafcxf/karafcxf.service.impl/1.0-SNAPSHOT\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;mvn:karafcxf/karafcxf.rest/1.0-SNAPSHOT\u0026lt;/bundle\u0026gt; \u0026lt;/feature\u0026gt; \u0026lt;feature name=\u0026#34;this-project-dependants\u0026#34;\u0026gt; \u0026lt;!-- Dependent bundles if you have any. This project doesn\u0026#39;t have one \u0026lt;bundle\u0026gt;mvn:org.apache.commons/commons-lang3/3.1\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;mvn:org.codehaus.jackson/jackson-core-asl/1.9.5\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;mvn:org.codehaus.jackson/jackson-mapper-asl/1.9.5\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;mvn:org.codehaus.jackson/jackson-jaxrs/1.9.5\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;mvn:com.google.guava/guava/14.0.1\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;wrap:mvn:org.apache.httpcomponents/httpcore/4.2.4\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;wrap:mvn:org.apache.httpcomponents/httpmime/4.2.5\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;wrap:mvn:org.noggit/noggit/0.5\u0026lt;/bundle\u0026gt; \u0026lt;bundle\u0026gt;wrap:mvn:org.apache.solr/solr-solrj/4.4.0\u0026lt;/bundle\u0026gt; --\u0026gt; \u0026lt;/feature\u0026gt; \u0026lt;/features\u0026gt; Step 2 Maven plugin configuration to create .kar, which indicates where your feature.xml is located (Note that this file is located inside your karaf project)\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 \u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.apache.karaf.tooling\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;features-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.3.2\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;id\u0026gt;create-kar\u0026lt;/id\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;create-kar\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;featuresFile\u0026gt;${project.basedir}/src/main/resources/feature.xml\u0026lt;/featuresFile\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; Wraps Notice the wrap protocol in front of the mvn protocol in few bundles as in\n\u0026lt;bundle\u0026gt;wrap:mvn:org.apache.httpcomponents/httpmime/4.2.5\u0026lt;/bundle\u0026gt;\nNot all Jars are OSGi ready but they would obviously be used as a dependency in our project. In those cases, Karaf notices the wrap protocol and bundles the JAR into an OSGi bundle. In case of doubt, just drop off the wrap and Karaf would complain that the jar is not OSGi compatible. (Alternatively, you could open up each of the dependant jars and check their manifest files)\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 013-08-28 01:38:48,669 | WARN | raf-2.3.2/deploy | KarArtifactInstaller | eployer.kar.KarArtifactInstaller 192 | 24 - org.apache.karaf.deployer.kar - 2.3.2 | Unable to install Kar feature xx-xxx-xxxxxx/0.0.0 org.osgi.framework.BundleException: Jar is not a bundle, no Bundle-SymbolicName mvn:org.apache.httpcomponents/httpcore/4.2.4 at org.apache.karaf.features.internal.FeaturesServiceImpl.installBundleIfNeeded(FeaturesServiceImpl.java:836)[26:org.apache.karaf.features.core:2.3.2] at org.apache.karaf.features.internal.FeaturesServiceImpl.doInstallFeature(FeaturesServiceImpl.java:618)[26:org.apache.karaf.features.core:2.3.2] at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:414)[26:org.apache.karaf.features.core:2.3.2] at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:402)[26:org.apache.karaf.features.core:2.3.2] at Proxy508d2419_d21e_4a93_b7fb_26e28d2f03a6.installFeature(Unknown Source)[:] at org.apache.karaf.deployer.kar.KarArtifactInstaller.installFeatures(KarArtifactInstaller.java:189)[24:org.apache.karaf.deployer.kar:2.3.2] at org.apache.karaf.deployer.kar.KarArtifactInstaller.install(KarArtifactInstaller.java:134)[24:org.apache.karaf.deployer.kar:2.3.2] at org.apache.karaf.deployer.kar.KarArtifactInstaller.update(KarArtifactInstaller.java:348)[24:org.apache.karaf.deployer.kar:2.3.2] at org.apache.felix.fileinstall.internal.DirectoryWatcher.update(DirectoryWatcher.java:1103)[6:org.apache.felix.fileinstall:3.2.6] at org.apache.felix.fileinstall.internal.DirectoryWatcher.update(DirectoryWatcher.java:898)[6:org.apache.felix.fileinstall:3.2.6] at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:482)[6:org.apache.felix.fileinstall:3.2.6] at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:291)[6:org.apache.felix.fileinstall:3.2.6] Changing Log Levels For all our development environment, we would want to increase our log level to get more feedback from Karaf. This could be achieved by modifying the org.ops4j.pax.logging.cfg file located in your \u0026lt;karaf-installation-directory\u0026gt;/etc\nStep 5 - Other Maven configuration Not technically a step because we would have covered this from the beginning anyway. And nothing fancy here.\nParent pom.xml Has the rest of the sub-modules configured. The various library dependencies configured 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 \u0026lt;project xmlns=\u0026#34;http://maven.apache.org/POM/4.0.0\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34; xsi:schemaLocation=\u0026#34;http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\u0026#34;\u0026gt; \u0026lt;modelVersion\u0026gt;4.0.0\u0026lt;/modelVersion\u0026gt; \u0026lt;groupId\u0026gt;karafcxf\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;karafcxf\u0026lt;/artifactId\u0026gt; \u0026lt;packaging\u0026gt;pom\u0026lt;/packaging\u0026gt; \u0026lt;version\u0026gt;1.0-SNAPSHOT\u0026lt;/version\u0026gt; \u0026lt;properties\u0026gt; \u0026lt;cxf.version\u0026gt;2.7.5\u0026lt;/cxf.version\u0026gt; \u0026lt;/properties\u0026gt; \u0026lt;modules\u0026gt; \u0026lt;module\u0026gt;karafcxf.rest\u0026lt;/module\u0026gt; \u0026lt;module\u0026gt;karafcxf.service.impl\u0026lt;/module\u0026gt; \u0026lt;module\u0026gt;karafcxf.kar\u0026lt;/module\u0026gt; \u0026lt;/modules\u0026gt; \u0026lt;dependencies\u0026gt; … … Rest and ServiceImpl pom.xml The other poms.xmls aren\u0026rsquo;t interesting\nThey just affiliate themselves to the parent pom with the parent tag as in\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.apache.felix\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;maven-bundle-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.3.7\u0026lt;/version\u0026gt; \u0026lt;extensions\u0026gt;true\u0026lt;/extensions\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;instructions\u0026gt; \u0026lt;Bundle-SymbolicName\u0026gt;${project.artifactId}\u0026lt;/Bundle-SymbolicName\u0026gt; \u0026lt;Bundle-Version\u0026gt;${project.version}\u0026lt;/Bundle-Version\u0026gt; \u0026lt;Bundle-Activator\u0026gt;com.nutraspace.coreservices.search.rest.Activator\u0026lt;/Bundle-Activator\u0026gt; \u0026lt;Export-Package\u0026gt;com.nutraspace.coreservices.search.rest*;version=${project.version}\u0026lt;/Export-Package\u0026gt; \u0026lt;Import-Package\u0026gt;*\u0026lt;/Import-Package\u0026gt; \u0026lt;/instructions\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; Step 6 - Bring up Karaf My Karaf installation is located at : /Users/Gabriel/apps/apache-karaf-2.3.2\n**Start Karaf : **\n\u0026lt;installation-dir\u0026gt;/bin/./karaf\nInstalling CXF and HTTP services\nfeatures:chooseurl cxf 2.7.5\nfeatures:install http cxf\nChecking whether your bundle is installed and your service running\nosgi:list\ndxf:list-endpoints\nStop Karaf\nCtrl +D (Please note that Ctrl +C closes the connection abruptly instead of stopping Karaf)\nIn case of accidental Ctrl+C and if Karaf isn\u0026rsquo;t starting properly, do a\nrm -rf \u0026lt;installation-dir\u0026gt;/data/cache/*\nURL Mapping Like I mentioned earlier, the target request URL will be something like http://localhost:8181/cxf/karafsimple/say/hello/arun and the response would be Hello, arun\nThe HelloRestService interface has all the JAX RS annotations for the URL mapping. Well, technically, the interfaces just maps for say/hello/(name) The karafsimple in the URL is derived from the the JAX RS address in blueprint.xml (about that later in Step 3 - rest.xml). The cxf is a default if you deploy a CXF service on Karaf which obviously you could change. ","date":"2013-09-01T15:50:00Z","permalink":"/building-cxf-rest-service-in-osgi-for-karaf/","title":"Building CXF REST Service in OSGi for Karaf"},{"content":"It\u0026rsquo;s no news that Quicksort is considered one of the most important algorithms of the century and that it is the defacto system sort for many languages, including the Arrays.sort in Java.\nSo, what\u0026rsquo;s new about quicksort?\nWell, nothing except that I figured just now (after 2 damn years of release of Java 7) that the Quicksort implementation of the Arrays.sort has been replaced with a variant called Dual-Pivot QuickSort. This thread is not only awesome for this reason but also how humble Jon Bentley and Joshua Bloch really are.\nWhat did I do next?\nJust like everybody else, I wanted to implement it and do some benchmarking - against some 10 million integers (random and duplicate).\nOddly enough, I found the following results :\nRandom Data :\nQuick Sort Basic : 1222 millis Quick Sort 3 Way : 1295 millis (seriously !!) Quick Sort Dual Pivot : 1066 millis Duplicate Data :\nQuick Sort Basic : 378 millis Quick Sort 3 Way : 15 millis Quick Sort Dual Pivot : 6 millis Stupid Question 1 :\nI am afraid that I am missing something in the implementation of 3-way partition. Across several runs against random inputs (of 10 million) numbers, I could see that the single pivot always performs better (although the difference is less than 100 milliseconds for 10 million numbers).\nI understand that the whole purpose of making the 3-way Quicksort as the default Quicksort until now is that it does not give 0(n^2) performance on duplicate keys - which is very evident when I run it against duplicate input. But is it true that for the sake of handling duplicate data, a small penalty is taken by 3-way? Or is my implementation bad?\nStupid Question 2\nMy Dual Pivot implementation (link below) does not handle duplicates well. It takes a sweet forever (0(n^2)) to execute. Is there a good way to avoid this? Referring to the Arrays.sort implementation, I figured out that ascending sequence and duplicates are eliminated well before the actual sorting is done. So, as a dirty fix, if the pivots are equal I fast-forward the lowerIndex until it is different than pivot2. Is this a fair implementation?\n1 2 3 4 5 6 7 else if (pivot1==pivot2){ while (pivot1==pivot2 \u0026amp;\u0026amp; lowIndex\u0026lt;highIndex){ lowIndex++; pivot1=input[lowIndex]; } } That\u0026rsquo;s it. That is all I did?\nI always find the tracing of the algorithm interesting but with the number of variables in Dual Pivot quicksort, my eyes found it overwhelming while debugging. So, I also went ahead and created trace-enabled implementations (for all 3) so that I could see where the variable pointers currently are.\nThese trace-enabled classes just overlay where the pointers are below the array values. I hope you find these classes useful.\neg. for a Dual Pivot iteration\nWhere can you download the code?\nThe entire project (along with a few lame implementations of DSA) is available on github here. The quicksort classes alone could be found here.\nHere\u0026rsquo;s my implementation of the SinglePivot (Hoare), 3-way (Sedgewick) and the new Dual-Pivot (Yaroslavskiy)\nSingle Pivot\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 package basics.sorting.quick; import static basics.sorting.utils.SortUtils.exchange; import static basics.sorting.utils.SortUtils.less; import basics.shuffle.KnuthShuffle; public class QuickSortBasic { public void sort (int[] input){ //KnuthShuffle.shuffle(input); sort (input, 0, input.length-1); } private void sort(int[] input, int lowIndex, int highIndex) { if (highIndex\u0026lt;=lowIndex){ return; } int partIndex=partition (input, lowIndex, highIndex); sort (input, lowIndex, partIndex-1); sort (input, partIndex+1, highIndex); } private int partition(int[] input, int lowIndex, int highIndex) { int i=lowIndex; int pivotIndex=lowIndex; int j=highIndex+1; while (true){ while (less(input[++i], input[pivotIndex])){ if (i==highIndex) break; } while (less (input[pivotIndex], input[--j])){ if (j==lowIndex) break; } if (i\u0026gt;=j) break; exchange(input, i, j); } exchange(input, pivotIndex, j); return j; } } 3-way\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 package basics.sorting.quick; import static basics.shuffle.KnuthShuffle.shuffle; import static basics.sorting.utils.SortUtils.exchange; import static basics.sorting.utils.SortUtils.less; public class QuickSort3Way { public void sort (int[] input){ //input=shuffle(input); sort (input, 0, input.length-1); } public void sort(int[] input, int lowIndex, int highIndex) { if (highIndex\u0026lt;=lowIndex) return; int lt=lowIndex; int gt=highIndex; int i=lowIndex+1; int pivotIndex=lowIndex; int pivotValue=input[pivotIndex]; while (i\u0026lt;=gt){ if (less(input[i],pivotValue)){ exchange(input, i++, lt++); } else if (less (pivotValue, input[i])){ exchange(input, i, gt--); } else{ i++; } } sort (input, lowIndex, lt-1); sort (input, gt+1, highIndex); } } Dual Pivot\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 package basics.sorting.quick; import static basics.shuffle.KnuthShuffle.shuffle; import static basics.sorting.utils.SortUtils.exchange; import static basics.sorting.utils.SortUtils.less; public class QuickSortDualPivot { public void sort (int[] input){ //input=shuffle(input); sort (input, 0, input.length-1); } private void sort(int[] input, int lowIndex, int highIndex) { if (highIndex\u0026lt;=lowIndex) return; int pivot1=input[lowIndex]; int pivot2=input[highIndex]; if (pivot1\u0026gt;pivot2){ exchange(input, lowIndex, highIndex); pivot1=input[lowIndex]; pivot2=input[highIndex]; //sort(input, lowIndex, highIndex); } else if (pivot1==pivot2){ while (pivot1==pivot2 \u0026amp;\u0026amp; lowIndex\u0026lt;highIndex){ lowIndex++; pivot1=input[lowIndex]; } } int i=lowIndex+1; int lt=lowIndex+1; int gt=highIndex-1; while (i\u0026lt;=gt){ if (less(input[i], pivot1)){ exchange(input, i++, lt++); } else if (less(pivot2, input[i])){ exchange(input, i, gt--); } else{ i++; } } exchange(input, lowIndex, --lt); exchange(input, highIndex, ++gt); sort(input, lowIndex, lt-1); sort (input, lt+1, gt-1); sort(input, gt+1, highIndex); } } ","date":"2013-06-13T12:35:00Z","permalink":"/quicksorting-3-way-and-dual-pivot/","title":"Quicksorting - 3-way and Dual Pivot"},{"content":"While everybody would naturally accept the following lines of code on grounds of reference equality and value equality and that String and wrappers override the equals method, it takes some effort at first to accept the behavior of Arrays.equals and Arrays.deepEquals\n1 2 3 4 5 6 7 8 9 10 11 12 Object obj1=new Object(); Object obj2=new Object(); String hello1=new String(\u0026#34;hello\u0026#34;); String hello2=new String(\u0026#34;hello\u0026#34;); System.out.println(hello1.equals(hello2)); //returns true System.out.println(hello1==hello2); //returns false System.out.println(obj1.equals(obj2)); //returns false System.out.println(obj1==obj2); //returns false Boring Stuff :-(\nFirst, the similarities\nBoth Arrays.equals and Arrays.deepEquals are similar in certain behaviors as\nIf they are the same object (reference equality), they return true If either of the compared objects are null, then return false If the array lengths are not equal, then return false They care about order (position) Next, the differences\nArrays.equals is really just skin deep\nOpening up the souce, we could see that the lousy Array.equals just does this\n1 2 3 4 5 6 7 8 for (int i=0; i\u0026lt;length; i++) { Object o1 = a[i]; Object o2 = a2[i]; if (!(o1==null ? o2==null : o1.equals(o2))) return false; } So, it just loops through the given arrays, does a equals on each of the pairs. This means that if you are passing in a String/Wrapper array or any other arrays whose equals method is overridden, then they are equal. Non-equals-overridden classes (derived from Object) will return false.\nArrays.deepEquals looks really deep\nFrom the source, we could understand that Arrays.deepEquals\nLoops through the input arrays, gets each pair Analyses the type of each pair Delegates the equal deciding logic to one of the overloaded Arrays.equals if they are one of the primitive arrays Delegates recursively to Arrays.deepEquals if it is an Object array Calls the respective object\u0026rsquo;s equals, for any other object 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 for (int i = 0; i \u0026lt; length; i++) { Object e1 = a1[i]; Object e2 = a2[i]; if (e1 == e2) continue; if (e1 == null) return false; // Figure out whether the two elements are equal boolean eq; if (e1 instanceof Object[] \u0026amp;\u0026amp; e2 instanceof Object[]) eq = deepEquals ((Object[]) e1, (Object[]) e2); else if (e1 instanceof byte[] \u0026amp;\u0026amp; e2 instanceof byte[]) eq = equals((byte[]) e1, (byte[]) e2); … … else eq = e1.equals(e2); if (!eq) return false; } return true; Now, the Awesome stuff !!\nEquals not overridden (nested and non-nested)\nWhile doing an Arrays.equals for nested or non-nested \u0026rsquo;non-overridden equals\u0026rsquo; objects, it is safe to assume that if Arrays.equals return false, then Arrays.deepEquals also return false.\nNon-Nested\nSo, given two arrays\n1 2 3 private YourClass[] equalsNotOverriddenArrayNonNested1={new YourClass(), new YourClass()}; private YourClass[] equalsNotOverriddenArrayNonNested2={new YourClass(), new YourClass()}; where YourClass is simply\n1 2 3 public class YourClass { } then the following assertions are true\n1 2 3 assertFalse(Arrays.equals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2)); assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2)); Nested\nAlso given two arrays,\n1 2 3 private Object[] equalsNotOverriddenArrayNested1={new YourClass(), new YourClass[]{new YourClass()}}; private Object[] equalsNotOverriddenArrayNested2={new YourClass(), new YourClass()}; the following assertions are true\n1 2 3 assertFalse(Arrays.equals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2)); assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2)); Equals overridden\nNon-Nested\nWhile doing an Arrays.equals for non-nested \u0026lsquo;overridden equals\u0026rsquo; objects, it can be said that if Arrays.equals is true, then Arrays.deepEquals also return true.\nGiven two String arrays\n1 2 3 private Object[] equalsOverriddenArrayNonNested1={\u0026#34;101\u0026#34;,\u0026#34;201\u0026#34;}; private Object[] equalsOverriddenArrayNonNested2={\u0026#34;101\u0026#34;,\u0026#34;201\u0026#34;}; the following assertions are true\n1 2 3 assertTrue(Arrays.equals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2)); assertTrue(Arrays.deepEquals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2)); since they are just one-to-one equals call on each pair.\nNested\nInteresting scenario : While doing an Arrays.equals for nested \u0026lsquo;overridden equals\u0026rsquo; objects, if Arrays.equals is false, then Arrays.deepEquals need not be false.\nConsider two Object arrays which has two values - a String and a String array\n1 2 3 private Object[] equalsOverriddenArrayNested1={new String(\u0026#34;hello\u0026#34;), new String[]{new String(\u0026#34;hello\u0026#34;)}}; private Object[] equalsOverriddenArrayNested2={new String(\u0026#34;hello\u0026#34;), new String[]{new String(\u0026#34;hello\u0026#34;)}}; you can notice that the Arrays.equals return false while Arrays.deepEquals return true\n1 2 3 assertFalse(Arrays.equals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2)); assertTrue(Arrays.deepEquals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2)); The result for Arrays.deepEquals is logical since each (from the source), the method loops through each pair of elements, checks whether it is an array type and calls deepEquals on each of the pair. If it is an non-array type, then it just calls the equals on the object.\nHowever, the result of Arrays.equals is tricky but at the same time obvious. The Arrays.equals method blindly calls equals on each pair and since the second arguments String[] are of Object type (whose equals is not overridden), it checks for reference equality and fails !!\nThe entire testcase can be found below :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 import static org.testng.Assert.assertFalse; import static org.testng.Assert.assertTrue; import java.util.Arrays; import org.testng.annotations.Test; public class DeepEquals { private Object[] equalsOverriddenArrayNonNested1={\u0026#34;101\u0026#34;,\u0026#34;201\u0026#34;}; private Object[] equalsOverriddenArrayNonNested2={\u0026#34;101\u0026#34;,\u0026#34;201\u0026#34;}; private YourClass[] equalsNotOverriddenArrayNonNested1={new YourClass(), new YourClass()}; private YourClass[] equalsNotOverriddenArrayNonNested2={new YourClass(), new YourClass()}; private Object[] equalsNotOverriddenArrayNested1={new YourClass(), new YourClass[]{new YourClass()}}; private Object[] equalsNotOverriddenArrayNested2={new YourClass(), new YourClass()}; private Object[] equalsOverriddenArrayNested1={new String(\u0026#34;hello\u0026#34;), new String[]{new String(\u0026#34;hello\u0026#34;)}}; private Object[] equalsOverriddenArrayNested2={new String(\u0026#34;hello\u0026#34;), new String[]{new String(\u0026#34;hello\u0026#34;)}}; @Test public void stringArrayTest(){ assertFalse (equalsOverriddenArrayNonNested1==equalsOverriddenArrayNonNested2); assertFalse(equalsOverriddenArrayNonNested1.equals(equalsOverriddenArrayNonNested2)); assertTrue(Arrays.equals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2)); assertTrue(Arrays.deepEquals(equalsOverriddenArrayNonNested1, equalsOverriddenArrayNonNested2)); } @Test public void objectArrayTestNonNested(){ assertFalse (equalsNotOverriddenArrayNonNested1==equalsNotOverriddenArrayNonNested2); assertFalse(equalsNotOverriddenArrayNonNested1.equals(equalsNotOverriddenArrayNonNested2)); assertFalse(Arrays.equals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2)); assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNonNested1, equalsNotOverriddenArrayNonNested2)); } @Test public void objectArrayTestNested(){ assertFalse (equalsNotOverriddenArrayNested1==equalsNotOverriddenArrayNested2); assertFalse(equalsNotOverriddenArrayNested1.equals(equalsNotOverriddenArrayNested2)); assertFalse(Arrays.equals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2)); assertFalse(Arrays.deepEquals(equalsNotOverriddenArrayNested1, equalsNotOverriddenArrayNested2)); } @Test public void objectArrayTest2(){ assertFalse (equalsOverriddenArrayNested1==equalsOverriddenArrayNested2); assertFalse(equalsOverriddenArrayNested1.equals(equalsOverriddenArrayNested2)); assertFalse(Arrays.equals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2)); assertTrue(Arrays.deepEquals(equalsOverriddenArrayNested1, equalsOverriddenArrayNested2)); } } ","date":"2012-11-04T18:04:00Z","permalink":"/your-s-deeply-why-arrays-deepequals-when-we-have-arrays-equals/","title":"Your's deeply - Why Arrays.deepEquals when we have Arrays.equals"},{"content":"If you are looking for evaluating an infix expression with parantheses, don\u0026rsquo;t waste your time here. Visit my other fresh write up here\nThese images have been running around Facebook for a while now. Though it is eye-damaging primary level arithmetic, it is kind of sweet to write it as a program. Just for the fun of it, I created an html page driven by a javascript implementation. Here is the link.\nSo, essentially the idea is to evaluate the infix notation in a natural way - respecting the operator precedence. (If you consider this as a puzzle, then there are other variations which involves evaluating the expression in such a way that the result is a maximum value/minimum value)\nPlease note that the algorithm (and therefore, the program), as it stands does not handle parentheses. This is just a two-stack algorithm with a small variation to accommodate numbers more than a digit.\nThe algorithm goes like this :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1. Initialize two stacks - one for holding values and one for holding operators 2. Read the input expression from left to right - one character at a time. 3. If the character is a space, ignore 4. If the character is a number, then append it to an accumulating variable (don\u0026#39;t push it to the stack yet) 5. If the character is an operator, push the accumulated number to the value stack and reinitialize the accumulator. 5a. If the current operator is of higher precedence than the first item in the operator stack, then safely ignore and read further. 5b. If the current operator is of lower precedence than the first item in the operator stack (peek view), then evaluate the previous set (two from value stack and one from operator stack) and insert the result into the value stack 5c. Finally, insert the current operator to the operator stack A Groovy implementation is here :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 /** * Only basic arithmetic operators are supported. However, could easily be * extended to support any binary operators * * This program is not an infix to postfix converter. However, this program does the following * * 1) Evaluates a non-parenthesized infix expression and drives it to a final value * 2) Does not support parentheses * 3) Supports multiple digits as operands * 4) Empty space allowed * 5) Floating point numbers not supported * 6) Supports only the 4 basic arithmetic operators but could easily be extended to support any other * binary operation * * @author Arun Manivannan */ public class InfixToValue { public static final String OPERATORS=\u0026#34;+-/*)\u0026#34; //ignore the \u0026#34;(\u0026#34; as an operator public static final Map OPERATOR_LEVELS= [\u0026#34;*\u0026#34;:1, \u0026#34;/\u0026#34;:1, \u0026#34;+\u0026#34;:2,\u0026#34;-\u0026#34;:2] public static void main(String[] args) { String inputExpression=\u0026#34;9 - 5+5*0+3\u0026#34; //610 InfixToValue ifixToPfix=new InfixToValue() ifixToPfix.infixToValue(inputExpression); } def infixToValue(String inputExpression) { Stack\u0026lt;String\u0026gt; expressionStack = new Stack\u0026lt;String\u0026gt;() Stack\u0026lt;Long\u0026gt; valueStack = new ArrayDeque\u0026lt;Long\u0026gt;() //to hold only non-decimal values String eachNumber=\u0026#34;\u0026#34; //to hold multiple digit values. lame idea, i know, but can\u0026#39;t think of anything else at 2 AM int totalCharInInput=inputExpression.length() inputExpression.each { eachChar-\u0026gt; totalCharInInput-- println (\u0026#34;each char : \u0026#34;+eachChar) if (eachChar.trim()==\u0026#34;\u0026#34;){ //ignore } else if (isValidOperator(eachChar)){ valueStack.push(Long.parseLong(eachNumber)) //push the till now accumulated number to the value stack eachNumber=\u0026#34;\u0026#34; println (\u0026#34;Iteration Value Stack 1: \u0026#34;+valueStack) println (\u0026#34;Iteration Expression Stack 1: \u0026#34;+expressionStack) if (expressionStack.isEmpty()){ //first item expressionStack.push(eachChar) } //if the current operator has higher precedence than the first item in the stack, then it is fine. //Just insert the incoming else if (getHigherPrecedenceOperator(eachChar, expressionStack.peek())==eachChar){ expressionStack.push(eachChar) } else if (isRightParen(eachChar)){ evaluateAndPushValueToValueStack() } //if the current operator is lesser, then the previous higher precedence expression have to be //evaluated. Else, we would be making wrong calculations while popping off for final evaluation else{ println (\u0026#34;Value Stack 1: \u0026#34;+valueStack) println (\u0026#34;Expression Stack 1: \u0026#34;+expressionStack) //the current operator has higher precedence. Evaluate expression evaluateAndPushValueToValueStack(valueStack, expressionStack) //after evaluation of the higher pair, don\u0026#39;t forget to insert the current character into the expression stack expressionStack.push(eachChar) } } else if (eachChar.isNumber()){ eachNumber+=eachChar if (totalCharInInput==0){ valueStack.push(Long.parseLong(eachNumber)) //the last element } } /*else { //other input (alphabets and special chars) are treated as garbage }*/ } println (\u0026#34;Expression stack \u0026#34; +expressionStack); println (\u0026#34;Value stack : \u0026#34;+valueStack) while (!expressionStack.empty){ evaluateAndPushValueToValueStack(valueStack,expressionStack) } println (\u0026#34;Final value : \u0026#34;+valueStack) } boolean isRightParen(String operator) { operator==\u0026#34;)\u0026#34;?true:false } boolean isLeftParen(String operator) { operator==\u0026#34;(\u0026#34;?true:false } private void evaluateAndPushValueToValueStack(Stack\u0026lt;Long\u0026gt; valueStack, Stack\u0026lt;String\u0026gt; expressionStack) { Long firstOperand=valueStack.pop() Long secondOperand=valueStack.pop() println (\u0026#34;Value stack : \u0026#34;+firstOperand+ \u0026#34;:\u0026#34;+ secondOperand ) Long evaluatedValue = this.evaluate(secondOperand, firstOperand, expressionStack.pop()) //intermediate result valueStack.push(evaluatedValue) } Long evaluate(Long firstOperand, Long secondOperand, String operator) { Long returnValue switch (operator) { case \u0026#34;+\u0026#34; : returnValue=firstOperand+secondOperand break case \u0026#34;-\u0026#34; : returnValue=firstOperand-secondOperand break; case \u0026#34;*\u0026#34;: returnValue=firstOperand*secondOperand break case \u0026#34;/\u0026#34;: if (secondOperand==0){ //cheating death by zero returnValue=0 } else{ returnValue=firstOperand/secondOperand } break; } } boolean isValidOperator(String input) { OPERATORS.contains(input)?true:false } String getHigherPrecedenceOperator(String firstOperator, String secondOperator){ OPERATOR_LEVELS[firstOperator]\u0026lt;OPERATOR_LEVELS[secondOperator]?firstOperator:secondOperator } } ","date":"2012-10-14T16:06:00Z","permalink":"/evaluating-infix-expression-multiple-digits/","title":"Evaluating Infix expression - multiple digits"},{"content":"Extremely sorry about the delay on Part 2 of this series. Graduate exams are just round the corner and I am unable to find time for quality research. Exams get over by end of November.\nI thoroughly enjoy Geb and I think it is the most stylish way to write functional tests.\n**So, the Prerequisites : **\nSome Groovy Magic. Most of all that you need to learn Groovy is covered in this manual but for obvious reasons if you get obsessed with the language you might want to consider Groovy in Action. If you are from a Java (except for closures) or a Python background, you could probably skim through the tutorial for 15 minutes and you are there already). A little Selenium. The more, the better but fear not this single page tells you all that you need to know about the Selenium Webdriver that you would generally use. jQuery selectors (everybody says that it is easy but frankly, I refer to the manual at least twice per hour. I am dumb, so…). If you are new to jQuery, you would want to start from basic selectors and click on the left navigation menu for more. PageObjects is actually not a prerequisite for Geb but PageObjects are so awesome that you never wanted to think about Geb outside of it. Interestingly, PageObjects is not a technology that you need to pick. It is cute little pattern which says that you wrap the structure of your HTML page into an Object so that the actual test does not have to deal with it. Hah. Got you. Let me put that in plain English. Say, you have a registration form with input textbox which has an id of \u0026ldquo;nametext\u0026rdquo;. How would you get the handle of the textbox? In DOM terms, in javascript, you would just do a\n1 document.getElementById(\u0026#34;nametext\u0026#34;) In Selenium, you would do a very similar thing\n1 driver.findElement(By.id(\u0026#34;nametext\u0026#34;)) So, if you would want to populate Jason in your text box in Selenium, you would do a\n1 driver.findElement(By.id(\u0026#34;nametext\u0026#34;)).sendKeys(\u0026#34;Jason\u0026#34;); If you do that for all your input fields, very soon your testcases become ugly and hateful. Instead of that, in OO terms, you encapsulate. You create a new class, say RegistrationPage and wrap your findElement and sendKeys as in :\n1 2 3 4 5 6 7 8 9 10 11 12 public class RegistrationPage{ … public RegistrationPage fillRegistrationForm(String name, String email){ driver.findElement(By.id(\u0026#34;nametext\u0026#34;)).sendKeys(name); driver.findElement(By.id(\u0026#34;emailtext\u0026#34;)).sendKeys(email); } } and from your testcase, you would say\n1 2 3 RegistrationPage regPage=new RegistrationPage(); regPage.fillRegistrationForm(\u0026#34;Jason\u0026#34;,\u0026#34;jason@bourne.com\u0026#34;); (Even better idea is to wrap your input values into a class and pass it to the fillRegistrationForm)\nIn fact, Geb leverages PageObjects in a much better way - jQuery selectors to the rescue\n1 2 3 4 5 6 7 8 9 10 class InputFormPage extends Page{ … static content={ name {$(\u0026#34;input\u0026#34;, id:\u0026#34;entry_0\u0026#34;)} emailAddress {$(\u0026#34;input\u0026#34;, id:\u0026#34;entry_1\u0026#34;)} } } and in your testcase, you would just say\n1 2 name.value (\u0026#34;Jason\u0026#34;) emailAddress.value (\u0026#34;jason@bourne.com\u0026#34;) (you could do even better. Stay tuned)\n","date":"2012-10-01T19:10:00Z","permalink":"/grokking-geb-part-1-prerequisites/","title":"Grokking Geb  - Prerequisites"},{"content":"I love reading code.\nFor two reasons :\nIf the code is bad, it is an awesome ego boost PLUS you get to foul mouth someone who has a good reputation for designing amazing things. If the code is good, then you get to learn some new tricks and some cool patterns yet to be published anywhere I am sure you read a lot of code and I am sure you\u0026rsquo;ll love this awesome code reading tool which I came across a few days ago - Architexa. Architexa also has collaborative documentation support, as I could clearly see with their codemaps project, but I\u0026rsquo;ve been using primarily for code reading.\nHey, forgot to tell you - Architexa is absolutely free for individual users.\nBefore going any further, a WOW video is here\nA UML diagram is worth a thousand lines of code :\nArchitexa does not bring in a lot of diagrams but only the most relevant.\nClass diagram - for Static view Sequence diagram - for Dynamic view Layered diagram (package diagram + awesomeness) - for overall project static view Class first - Class diagrams :\nImagine you are assigned to a new project or you are simply opening up an open source library to check its implementation.\nIf you are like me, you would probably\ngo directly to the class that you are interested and start from there check all the methods of the class in the Outline window go to the method of choice and start drawing a Class dependency graph on your mind using the caller and the callee hierarchy. With Architexa, all you need to do is to throw all your classes into a diagram. Inheritence hierarchy gets automatically mapped by default.\nIf you would like to map the associations (which I always do), then all you need to do is a right click.\nYou get to choose what associates/members you would want to display in your diagram\nThe best part I like about Architexa is that you could add/remove items that are not of interest to you. No more noisy A3 print outs.\nThe big picture - Layered diagrams\nIf you dont have any Class of interest, then you would probably do a package scan - you simply open up all packages and check how the files in the project are arranged. Again, you are trying to construct a Package Dependency graph and identify commonly used design patterns or frameworks. From my personal experience, I always found that idenfifying package dependency is more difficult than individual classes. However, Architexa does it naturally.\nThe following is a layered representation of a Google web toolkit project where the client and the server folder are dependent on the shared package (for models, of course). The white components indicate that they are siblings.\nSay 1, 2, 3 - Sequence diagrams\nI love the way that the messages are marked with not only its name but also the complete parameter list. And I absolutely love the way that the classes are marked with either a \u0026ldquo;C\u0026rdquo; (for class with source), \u0026ldquo;I\u0026rdquo; and \u0026ldquo;E\u0026rdquo; for interfaces and enums and a \u0026ldquo;jar icon\u0026rdquo; if the class is sourced from a bundle. Generics support in messages. FTW !!\nNo more lonely notes\nThe best part is that the diagrams does not live in isolation. You could just double click on your class component in your diagram and go to the respective class, or double click on your sequence message and go to that method in the code directly. This helps in switching back and forth between code and diagram (Ctrl+F6). Cool huh?\nOptionally, you could export your diagram and have a hard copy printed out for reference - in case you hate switching windows.\nNotes :\nA little disappointment when you are writing in language other than Java. Architexa currently supports only Java. And it is a plugin only for Eclipse. You might want to check with them for support for your other favourite IDEs. Mine is eclipse. I am good. Referenced constants being a part of the sequence diagrams when you select the \u0026ldquo;Add All\u0026rdquo; references is strange but probably would come in handy. You can always remove the Constants files off the sequences, if you dont like them. I see that the time taken to draw the diagram is stunning and the UI too looks light-weight. It takes no lag to drag and drop a class or remove a component off the diagram. I am running a 16GB + quad core though (show off time !!). YMMV You could export the diagram as an image in your local file or in their community server (publicly available). However, I understand that if you need to have your own private secured server setup in your intranet you might want to go for an upgraded edition. No support of Generics on the class diagram could be painful for some but I am not deeply concerned there. Links :\nArchitexa Project site\nCodeMaps Beta site (Interesting)\n","date":"2012-09-24T19:51:00Z","permalink":"/architexa-a-fine-code-reading-tool/","title":"Architexa - A fine code reading tool"},{"content":"I would make a fool out of myself if I tell you that util.concurrent APIs kicks cheetah\u0026rsquo;s ass when the classes are available since 2004. However, there are some cool features which I would like to revisit. Concurrency experts, now is the time for you to close this window. All others, stay tight for the fun ride.\nThou shall not forget your roots\nExecutor is the root interface with a single execute method. Anything that implements a Runnable interface can passed as a parameter. Silly Executor, however, has no support for Callable though.\nGood news : ExecutorService interface, which extends Executor, adds support for Callable. Its implementation class is ThreadPoolExecutor.\nI am going to pretend that the ScheduledExecutorService interface and its implementation class ScheduledThreadPoolExecutor does not exist as they just add scheduling capabilities on top of the ExecutorService and ThreadPoolExecutor. But remember this class when the powerful but boring java.util.Timer is not enough and a full blown external scheduler is just too much.\nIf you are new to concurrency or forgot the difference between Callable and Runnable, you might want to read up a little before reading further. A dummy guide is here\nThe ExecutorService.submit Facts :\nThe three submit variants :\n1 2 3 4 Future submit(Callable task) \u0026lt;br\u0026gt; Future submit(Runnable task) \u0026lt;br\u0026gt; Future submit(Runnable task, T result) The submit method of the ExecutorService is overloaded and accepts either a Callable or Runnable. Since the run method of the Runnable returns void, it is no surprise that Future.get always returns a null when the task is complete. 1 Future\u0026lt;?\u0026gt; submit(Runnable task) The other overloaded submit method that accepts a Runnable and a generic returns whatever you passed in as the second parameter as the result. 1 \u0026lt;T\u0026gt; Future\u0026lt;T\u0026gt;\tsubmit(Runnable task, T result) In fact, opening up the code (FutureTask), you\u0026rsquo;ll notice that the RunnableAdapter top level nested class of Executors simply holds the result and returns the same result after the run method is complete.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 static final class RunnableAdapter\u0026lt;T\u0026gt; implements Callable\u0026lt;T\u0026gt; { final Runnable task; final T result; RunnableAdapter(Runnable task, T result) { this.task = task; this.result = result; } public T [More ...] call() { task.run(); return result; } } RunnableAdapter source\nIn both the cases, if you would like to (you should !) terminate the program instead of your executor thread blocking the program and entering a busy loop, you should call the shutdown method as in\n1 executorService.shutdown() shutDown facts\nYou could imagine shutdown as a half-closed door of a mall. No new customers will be let in but the existing customers can leave the mall once they are done.\nTo reiterate,\nshutdown is a polite method. It does not actually shut down the tasks in the pool immediately. It just says that no new tasks will be accepted. Unless you are executing your tasks using invokeAll, you would need to wait for all tasks in progress to complete. This is achieved by calling the awaitTermination method.\n(invokeAll and submit examples at the bottom of the post) Once all the current tasks are done, the executor service shuts down. If you are in need of an impolite, intruding method which doesn\u0026rsquo;t care whether the current threads are done with their tasks, then shutdownNow is your guy. However, there\u0026rsquo;s no guarantee that the method will shutdown the service on the dot but it is the closest you have to immediate shutdown.\nOn awaitTermination, you could specify the timeout period until which the main thread should wait for the pool threads to complete its tasks.\n1 2 3 4 5 6 7 8 9 10 ExecutorService executorService=Executors.newFixedThreadPool(10); … future = executorService.submit(getInstanceOfCallable(count,sum)); … executorService.shutdown(); if (executorService.awaitTermination(10, TimeUnit.SECONDS)){ System.out.println(\u0026#34;All threads done with their jobs\u0026#34;); } Executors - the factory guy\nThe classes above are all awesome. But, say, you wanted to create a single thread executor, you would write something like\n1 new ThreadPoolExecutor(1, 1,0L, TimeUnit.MILLISECONDS,new LinkedBlockingQueue\u0026lt;Runnable\u0026gt;()); Compare that to\n1 Executors.newSingleThreadExecutor() So, here you go. Executors is a class with just factory methods for creating various forms of executor service with some commonly used defaults. Note that, other than the awesome factory methods, it doesn\u0026rsquo;t bring any new features to the table.\nIt is recommended that you have a quick look at the implementation of the factory methods and check if it suits your needs.\nThe invokeAll and the submit\nThe All part of invokeAll method of the ExecutorService gives no surprise. It just says that you need to pass in a Collection of Callables. Again, as expected, the method does not return until all the Threads are done with their tasks. So, for cases when you are interested in the result only after all the jobs are complete, invokeAll is your guy.\nOn the other hand, the submit method returns immediately after the callable is submitted to the executor service. Unless you are doing nothing at all in your call method of your Callable, your worker threads should ideally be running when the submit method returns.\nThe following samples might be useful for your reference. The programs just tries to find the sum of all the natural numbers till 100 (Brute force, of course)\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 package me.rerun.incubator; import java.util.ArrayList; import java.util.List; import java.util.concurrent.Callable; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.atomic.AtomicInteger; public class ExecutorInvokeAll { public void runApp() throws InterruptedException, ExecutionException{ //variable to store the sum AtomicInteger sum=new AtomicInteger(); //Use our friendly neighbourhood factory method of the Executors. ExecutorService executorService=Executors.newFixedThreadPool(10); List\u0026lt;Callable\u0026lt;AtomicInteger\u0026gt;\u0026gt; callableList=new ArrayList\u0026lt;Callable\u0026lt;AtomicInteger\u0026gt;\u0026gt;(); for (int count = 0; count \u0026lt;= 100;count++) { callableList.add(getInstanceOfCallable(count,sum)); } //returns only after all tasks are complete List\u0026lt;Future\u0026lt;AtomicInteger\u0026gt;\u0026gt; resultFuture = executorService.invokeAll(callableList); //Prints 5050 all through for (Future\u0026lt;AtomicInteger\u0026gt; future : resultFuture) { //Didn\u0026#39;t deliberately put a timeout here for the get method. Remember, the invoke All does not return until the task is done. System.out.println(\u0026#34;Status of future : \u0026#34; + future.isDone() +\u0026#34;. Result of future : \u0026#34;+future.get().get()); } executorService.shutdown(); // You might as well call a resultFuture.get(0).get().get() and that would give you the same //result since all your worker threads hold reference to the same atomicinteger sum. System.out.println(\u0026#34;Final Sum : \u0026#34;+sum); } //Adds count to the sum and returns the reference of the sum as the result private Callable\u0026lt;AtomicInteger\u0026gt; getInstanceOfCallable(final int count, final AtomicInteger sum) { Callable\u0026lt;AtomicInteger\u0026gt; clientPlanCall=new Callable\u0026lt;AtomicInteger\u0026gt;(){ public AtomicInteger call() { sum.addAndGet(count); System.out.println(\u0026#34;Intermediate sum :\u0026#34;+sum); return sum; } }; return clientPlanCall; } public static void main(String[] args) throws ExecutionException { try { new ExecutorInvokeAll().runApp(); } catch (InterruptedException e) { e.printStackTrace(); } } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 package me.rerun.incubator; import java.util.concurrent.Callable; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicInteger; public class ExecutorSubmit { public void runApp() throws InterruptedException, ExecutionException{ //holder for the total sum AtomicInteger sum=new AtomicInteger(); //Use the factory method of Executors ExecutorService executorService=Executors.newFixedThreadPool(10); Future\u0026lt;AtomicInteger\u0026gt; future = null; for (int count = 0; count \u0026lt;= 100; count++) { future = executorService.submit(getInstanceOfCallable(count,sum)); //prints intermediate sum try { System.out.println(\u0026#34;Status of future : \u0026#34; + future.isDone() +\u0026#34;. Result of future : \u0026#34;+future.get(1000, TimeUnit.MILLISECONDS).get()); } catch (TimeoutException e) { System.out.println(\u0026#34;\u0026lt;IGNORE\u0026gt; Timeout exception for count : \u0026#34;+count); //e.printStackTrace(); } //System.out.println(\u0026#34;Result of future : \u0026#34;+future.get().get() +\u0026#34;.Status of future : \u0026#34; + future.isDone()); } executorService.shutdown(); if (executorService.awaitTermination(10, TimeUnit.SECONDS)){ System.out.println(\u0026#34;All threads done with their jobs\u0026#34;); } //exec System.out.println(\u0026#34;Final Sum : \u0026#34;+sum); } //Adds count to the sum and returns the reference of the sum as the result private Callable\u0026lt;AtomicInteger\u0026gt; getInstanceOfCallable(final int count, final AtomicInteger sum) { Callable\u0026lt;AtomicInteger\u0026gt; clientPlanCall=new Callable\u0026lt;AtomicInteger\u0026gt;(){ public AtomicInteger call() { sum.addAndGet(count); //System.out.println(\u0026#34;Intermediate sum :\u0026#34;+sum); return sum; } }; return clientPlanCall; } public static void main(String[] args) throws ExecutionException { try { new ExecutorSubmit().runApp(); } catch (InterruptedException e) { e.printStackTrace(); } } } Further reading :\nAlex Miller\u0026rsquo;s amazing blog Alex Miller\u0026rsquo;s concurrency gotchas Vogella\u0026rsquo;s article on comparison with original API Good introduction to concurrency in general Highly recommended book on Java concurrency ","date":"2012-09-15T20:31:00Z","permalink":"/a-lazy-developers-introduction-to-java-concurrency-executors/","title":"A lazy developers introduction to Java Concurrency Executors"},{"content":"Let\u0026rsquo;s be sure of what the Future holds\nFuture, which is a part of the Java concurrency Task execution framework, is the result of your computation as the javadoc claims. And more.\nWhen you are executing a task in a different thread, there is a lot of information you would need from it soon after you submit it to the pool.\nWhat can the Future give?\nResult : If you are spawning multiple threads and you are interested in the consolidated/incremental result, obviously you wanted a way to get it. So, here you go. Future.get() Cancel : Just in case if you spawned 10 threads and you are happy with just one result and wanted to kill the others (or) you wanted to kill that rogue thread (or) you simply loved to power to cancel any thread at will, you have a nice Future.cancel(). Status : Future.isDone() is your way of asking Are you done with your work? Interesting fact :\nThe Future.get() method waits for the result forever until the result gets returned. For some cases, you might need this. However, if you don\u0026rsquo;t want to wait forever, you might choose the overloaded get method\n1 V get(long timeout, TimeUnit unit) Fun Fact :\nIf you are just spawning a single thread in your Executor and calling the Future.get() (the one which waits forever), you are actually mimicking a Thread.join() call on a spawned single thread by the main thread.\nFurther reading :\nAlex Miller\u0026rsquo;s amazing blog Alex Miller\u0026rsquo;s concurrency gotchas Vogella\u0026rsquo;s article on comparison with original API Good introduction to concurrency in general Highly recommended book on Java concurrency ","date":"2012-09-15T20:18:00Z","permalink":"/what-does-java-util-concurrent-future-hold/","title":"What does java.util.concurrent.Future hold?"},{"content":"Runnable : I am the Yoda of multithreading. For generations, I was the only way (other than lang.Thread) that people could do parallel activities in Java. Remember the cool method that I have - run. Too bad, it returns a void though.\nCallable : I am a Runnable, just better and cooler. I have this amazing method called call which returns the result of my work. The return type of call is a cool generic and the result could be got using Future. Now, top that Runnable !!\nRunnable : Yeah, Yeah, Whatever. Probably, you just came into existence to fill the gaps left in my design. Like, you know, my run method does not throw any Exception which would just mean that if the task wants to throw an exception, it can throw only a Runtime exception.\nCallable : I agree. I do stand on your shoulders. And it\u0026rsquo;s no wonder the call method throws an Exception which enables the task to throw a checked exception back to the caller.\nRunnable API\n1 void run() Callable API\n1 V call() throws Exception More details on the Future is here\nUnity in Diversity\nFutureTask is an interesting class which can accept both Runnable and Callable in its constructor. Since FutureTask implements Runnable, it can safely pass by the Executor.execute door. Remember, the Executor.execute accepts only a Runnable. I still need to figure out why somebody would want that considering the execute method has a void return type and cannot return results\nShamelessly copied and modified from javadoc\n1 2 3 4 5 6 7 8 9 10 11 Executor executor= … FutureTask\u0026lt;String\u0026gt; future = new FutureTask\u0026lt;String\u0026gt;(new Callable\u0026lt;String\u0026gt;() { public String call() { … do your thing… }}); executor.execute(future); Further reading :\nAlex Miller\u0026rsquo;s amazing blog Alex Miller\u0026rsquo;s concurrency gotchas Vogella\u0026rsquo;s article on comparison with original API Good introduction to concurrency in general Highly recommended book on Java concurrency ","date":"2012-09-15T20:15:00Z","permalink":"/callable-vs-runnable-the-brawl-of-the-runners/","title":"Callable vs Runnable - The brawl of the runners"},{"content":"A couple of days ago, a friend of mine was interested to know how to filter a log file (in my case a log4j log file) for ERRORs alone.\nAn hour of tricks sharing followed and here is the gist of the conversation that you will be interested in.\nFind me wherever I am\nListing all the lines in the log file which has occurrences of ERROR is as simple as executing the following command\n1 grep \u0026#34;ERROR\u0026#34; \u0026lt;YOUR LOG FILE NAME\u0026gt; When there is a beginning\nAnd if you are interested in the lines which start with ERROR, then it just involves the usage of the regular expression for \u0026ldquo;begin\u0026rdquo;\n1 grep \u0026#34;^ERROR\u0026#34; \u0026lt;YOUR LOG FILE NAME\u0026gt; \u0026hellip;there\u0026rsquo;s an end\nA similar trick applies if you are interested in searching lines which has occurrences of \u0026ldquo;Exception\u0026rdquo; at the end of the line (specially in case of Java stack traces), then you would use\n1 grep \u0026#34;Exception$\u0026#34; \u0026lt;YOUR LOG FILE NAME\u0026gt; Don\u0026rsquo;t be so sensitive\nGood news is that you need not be case sensitive to grep with regard to search string. You could just use\n1 2 3 grep -i \u0026#34;exception$\u0026#34; \u0026lt;YOUR LOG FILE NAME\u0026gt; (i as in insensitive) Second thoughts\nThat said, imagine you are printing the username or a unique trace id of your request in your logs and you would want to filter your log based on that. And suppose this username/trace id is located as the 3rd word in every line. To print the lines which has the third word as say \u0026ldquo;12345\u0026rdquo; and your log file delimiter is a space (which is generally the case unless you are logging as xml or json), then you could use\n1 2 3 4 5 cut -d \u0026#34; \u0026#34; -f3 \u0026lt;YOUR LOG FILE NAME\u0026gt; | grep \u0026#34;12345\u0026#34; where -d \u0026#34; \u0026#34; says that your delimiter is a space -f3\tsays that you interested in the 3rd delimiter tokenized string Moving target\nWhat if you wanted to filter lines on a live log file. Just add some \u0026ldquo;tail\u0026rdquo; to it and you are good to go\n1 tail -f \u0026lt;YOUR LOG FILE NAME\u0026gt; | grep \u0026#34;ERROR\u0026#34; Too bad, this does not take care of rolling files.\nLess is more\nYour production box threw some exception and you are expected to check it out. So, you know that the exception logs exist somewhere towards the end of the file. How do you search for the exception?\nOpen the file\n1 less \u0026lt;YOUR LOG FILE NAME\u0026gt; (Don\u0026rsquo;t get me wrong, I love the vi editor but vi tries to load the entire file in its buffer and for large files like our typical log files, performance suffers)\nSo, yeah less \u0026lt;YOUR FILENAME\u0026gt; will open the file and keep you at the first page. If you need to go to the last page, press\n1 Shift + G Now, you are on the last line.\nSift though the last pages by pressing\n1 b (as in back) and\n1 f (as in forward) Elementary, Dr.Watson\nSo, if you would like to search for \u0026ldquo;Exception\u0026rdquo;, just type\n1 /Exception You would get a\n1 \u0026#34;Pattern not found\u0026#34; error because the default search direction is forward.\nTo search backward, press\n1 Shift + N Keep pressing Shift + N to repeat search upwards\nIf you went past the search string upwards and would like to continue search downwards, then just press\n1 n and keep pressing n to repeat search downwards\nCase-insensitive less search\nIf you like to make your search keywords insensitive, just type\n1 -i (remember -i toggles the case insensitivity)\nBetter to be a head of a dog\ntail -f \u0026lt;YOUR LOG FILE\u0026gt; is awesome but if you are analysing the log files inside the less buffer and would like to do a tail of your file, all you need to do is to press\n1 Shift + F Well, that\u0026rsquo;s all it is for now. I am sure there are many more awesome tricks that you can do in a shell which I don\u0026rsquo;t know or didn\u0026rsquo;t mention here. Please share your favourite trick so that I can do some serious show off to my friends.\nPS : There is a way to open the file at the last line with less instead of opening the file at the first page and pressing Shift+G. Can\u0026rsquo;t remember it and can\u0026rsquo;t find it too.\n","date":"2012-08-30T18:50:00Z","permalink":"/filter-lines-in-log-file-with-error/","title":"Filter lines in log file with ERROR"},{"content":"Update : Previous version of the code failed for some inputs, as pointed out by @Thinker in the first comment. Changes to the program and the presentation were made. Tested to satisfy most conditions that I could google for.\nIn a desperate attempt to increase my sad looking stackoverflow reputation, I replied to an old but interesting problem.\nThe problem goes like this :\nGiven Random integers in array. Find the greatest sum of a continuous subset\nThe problem is otherwise known as the Maximum subarray problem and is beaten to death numerous times.\nSo, instead of just translating a Python program into Java, I decided to pull together some slides which explains how this algorithm works.\nAnd to bore you to death, here is the Java version of the modified algorithm :\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 package me.rerun; public class Kadane { public static void main(String[] args) { int[] intArr={3, -1, -1, -1, -1, -1, 2, 0, 0, 0 }; //int[] intArr = {-1, 3, -5, 4, 6, -1, 2, -7, 13, -3}; //int[] intArr={-6,-2,-3,-4,-1,-5,-5}; findMaxSubArray(intArr); } public static void findMaxSubArray(int[] inputArray){ int maxStartIndex=0; int maxEndIndex=0; int maxSum = Integer.MIN_VALUE; int cumulativeSum= 0; int maxStartIndexUntilNow=0; for (int currentIndex = 0; currentIndex \u0026lt; inputArray.length; currentIndex++) { int eachArrayItem = inputArray[currentIndex]; cumulativeSum+=eachArrayItem; if(cumulativeSum\u0026gt;maxSum){ maxSum = cumulativeSum; maxStartIndex=maxStartIndexUntilNow; maxEndIndex = currentIndex; } else if (cumulativeSum\u0026lt;0){ maxStartIndexUntilNow=currentIndex+1; cumulativeSum=0; } } System.out.println(\u0026#34;Max sum : \u0026#34;+maxSum); System.out.println(\u0026#34;Max start index : \u0026#34;+maxStartIndex); System.out.println(\u0026#34;Max end index : \u0026#34;+maxEndIndex); } } Rajesh was kind enough to write the implementation in Scala. Sweet huh?\n","date":"2012-08-29T17:47:00Z","permalink":"/find-continuous-subarray-with-maximum-sum-problem-kadane-s-algorithm/","title":"Find Continuous subarray with maximum sum problem - Kadane's algorithm"},{"content":"A binary heap (generally referred as heap) is a rooted left-complete binary tree which has two properties\n1 2 3 1)\theap property 2)\tshape property Wow. Now, that is a lot of jargon. Let\u0026rsquo;s see what each word in the definition means.\nWhat is a tree?\nA tree is just a set of nodes connected by edges. Yikes !\nLet\u0026rsquo;s put this in a picture\nWhat is a binary tree?\nA binary tree is simply a tree with utmost 2 nodes.\nWhat is a rooted binary tree?\nA root is just a node which is specially qualified as root. For clarity, a root is generally drawn as the topmost node when graphically representing the datastructure.\nWhat is a left-complete binary tree?\nWhen all the levels of the binary tree has nodes except the last level, then it is a complete binary tree.\nEven if the last one does not have all the nodes, then it should be filled in such a way that the left most nodes are filled before the right ones. ie. There should not be any gaps in the middle. Now, that makes a left-complete binary tree\nA legal heap\nNow that clears up \u0026ldquo;A binary heap (generally referred as heap) is a rooted complete binary tree\u0026rdquo;.\nLet\u0026rsquo;s see what heap and the shape properties mean.\nShape-property :\nWell, I cheated. The shape property of a heap simply says that the heap must be a left-complete binary tree. So, dangling nodes can appear only on the left-side. No gaps in between.\nHeap property :\nThe children/child of each node should be lesser than the node itself (in case of max heap)\nThe children/child of each node should be greater than the node itself (in case of min heap)\nRepresenting Heap in an array :\nThe easiest and the general way of representing a heap is using an array. You\u0026rsquo;ll see how the complete/shape property of the heap makes it easier to represent in an array.\nJust read the tree left to right (level-order traversal, if you like it that way)\nCalculating location of children and parent in array :\nThe root is the first item of the array\nIndices of Children can be calculated, known a parent index, using the simple calculation :\n1 2 3 4 Child 1 = 2i +1 Child 2 = 2i +2 where i is the index of the parent Again, the index of the parent can be calculated by reversing the formula\n1 2 3 Index of parent given a child index is Parent = i/2 (round the result to the lower bound) So, considering the image above, the children of 5 at index 2 are items located in indices 5 and 6 which are 2 and 3\nFun facts :\n1) A sorted array is a min heap\n2) Unlike a Stack, a JVM heap has nothing to do with the heap data structure. Though rumor has it that the usage of name \u0026ldquo;heap\u0026rdquo; for its memory area comes from one of the early languages using Heap datastructure to store objects in memory at runtime.\n1 million nodes = 20 levels :\nSince Heap is a complete binary tree (remember : no spaces in between), the maximum number of levels is easy to calculate.\n1 2 3 4 5 6 7 First level =\t1 Second level =\t2 =\t2^1 Third level =\t4 = 2^2 Fourth level = 8\t=\t2^3 \u0026hellip;\neg. until 4th level, there are 15 nodes.\nor\n1 total number of nodes until 4th level = 2^4 -1. In other words, the height of the heap = log base 2 (n) (round to the upper bound)\nApplications of Heaps:\nA heap could be used to implement a priority queue, which is a queue that orders entities not a on first-come first-serve basis, but on a priority basis. So, since the heap has the minimum or the maximum key at the top, the next \u0026ldquo;priority\u0026rdquo; item is at the top with a removal time of 0(1).\nAlso, Heaps are used in the Heap Sorting algorithm which has the worst case performance of 0 (n log n). No wonder Heap sort is the preferred sorting algorithm for all the mission critical applications.\n","date":"2012-08-25T02:58:00Z","permalink":"/heap-datastructure-in-pictures/","title":"Heap datastructure in pictures"},{"content":" Note : For 3-way partition and Dual Pivot Quicksort (with programs to trace the sort process), please refer to this recent post.\nQuick sort is the fastest known non-hybrid comparision sort for arrays which has no knowledge of the data beforehand. To top it, it could be done in-place for arrays. For Linked Lists, Merge Sort might be a better option. Also since Quicksort improves its performance by using random numbers, it also becomes an example of a breed of algorithms named \u0026ldquo;Randomized Algorithms\u0026rdquo;.\nThe naive version of the Algorithm goes like this :\n1 2 3 4 5 6 7 8 9 1)\tStart with list I of n items 2)\tChoose pivot v from I 3)\tPartition I into 2 unsorted lists I1 and I2 such that I1 : all keys smaller than pivot I2 : all keys larger than pivot Items same as pivot goes to either list 4)\tSort I1 recursively, yielding sorted list S1 5)\tSort I2 recursively, yielding sorted list S2 6)\tConcatenate S1,v,S2 yielding sorted list S Pseudocode in a Video\nHow Long does Quick sort take ?\nInterestingly, Quicksort has a worst case running time of 0(n^2). However, with careful implementations, n^2 never happens and it almost always runs in 0(n log n). As compared to Heapsort, whose best and worst case is guaranteed to be n log n, Quicksort beats Heapsort in clock time for two reasons :\nThe constant factor surrounding the 0(nlogn) is lesser than Heapsort due to the very thin core logic. Heap sort\u0026rsquo;s detailed comparison and exchange mechanism inside loops increases the constant factor. Quicksort could leverage on the locality of reference (and therefore, memory access) advantages provided by the hardware and the operating system How n log n?\nIf you choose the pivot right, then, on an average, you get a 1/4, 3/4 split which gives an average running time of 0(n log n)\nSource : Goodrich and Tamassia\nPivot Choosing strategy :\nThere are many methods to choose a pivot\n1 2 3 4 5 1)\tFirst Item 2)\tLast item 3)\tMedian 4)\tMedian of k (generally of 3) 5)\tRandom item Choosing a random key as the pivot is the preferred method because of the guaranteed worst case possibility with the other selection methods with following kinds of sequences :\n1 2 3 4 5 1)\tPresorted list when chosing first item as your pivot in case of natural order last item as your pivot in case of reverse order 2)\tRepeated elements 3)\tUnimodal sequences* (when median is chosen as pivot) Bottom line : When you choose a pivot in such a way that one portion of your split (either I1 or I2) becomes empty AND if you are doing it all through the list, then you are essentially doing the split N times.\nQuicksort in Java\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 package me.rerun; import java.util.Arrays; public class QuickSort { public static void quickSort (Comparable comparableArray[], int lowIndex, int highIndex){ //at least one item must exist in the array if (lowIndex\u0026gt;=highIndex){ return; } int pivotIndex=getMedianIndexAsPivotIndex(lowIndex, highIndex); //1) Choose pivot from the sublist Comparable pivot=comparableArray[pivotIndex]; System.out.println(\u0026#34;Pivot : \u0026#34;+pivot); //2) Swap the pivot to the last item in the array swapItemsWithIndices(comparableArray, pivotIndex, highIndex); /*\tGet the border indices sandwiching the unsorted items alone (ignore pivot (now, in the highIndex)) set \u0026#39;i\u0026#39; to point to the item before the first Index set \u0026#39;j\u0026#39; to point to the item before pivot Notice that this way, the following invariant gets maintained all through the sorting procedure a. All items left of Index \u0026#39;i\u0026#39; have a value \u0026lt;=pivot b. All items right of Index \u0026#39;j\u0026#39; have a value \u0026gt;=pivot */ int i=lowIndex-1; int j=highIndex; do{ //Notice the \u0026lt;j (pivot item is ignored). We stop when both the counters cross //compareTo will return 0 when it reaches the pivot - will exit loop do {i++;} while (comparableArray[i].compareTo(pivot)\u0026lt;0); //we dont have the protection as the previous loop. //So, add extra condition to prevent \u0026#39;j\u0026#39; from overflowing outside the current sub array do {j--;} while (comparableArray[j].compareTo(pivot)\u0026gt;0 \u0026amp;\u0026amp;(j\u0026gt;lowIndex)); if (i\u0026lt;j){ swapItemsWithIndices(comparableArray, i, j); } System.out.println(\u0026#34;I :\u0026#34;+i + \u0026#34; J :\u0026#34;+j); } while (i\u0026lt;j); swapItemsWithIndices(comparableArray, highIndex, i);//bring pivot to i\u0026#39;s position\tSystem.out.println(\u0026#34;Comparable array : \u0026#34;+Arrays.asList(comparableArray)); //the big subarray is partially sorted (agrees to invariant). Let\u0026#39;s recurse and bring in more hands quickSort(comparableArray, lowIndex, i-1); //sort subarray between low index and one before the pivot quickSort(comparableArray, i+1, highIndex); //sort subarray between low index and one before the pivot } //... since swapping with array is the easiest way to swap two objects private static void swapItemsWithIndices(Comparable[] comparableArray, int firstItem, int secondItem) { System.out.println(\u0026#34;Swapping \u0026#34;+comparableArray[firstItem] +\u0026#34; and \u0026#34;+comparableArray[secondItem]); final Comparable tempItem=comparableArray[firstItem]; comparableArray[firstItem]=comparableArray[secondItem]; comparableArray[secondItem]=tempItem; System.out.println(\u0026#34;After swap array : \u0026#34;+Arrays.asList(comparableArray)); } //Variation 1 - chose median as pivot private static int getMedianIndexAsPivotIndex(int lowIndex, int highIndex) { return lowIndex+((highIndex-lowIndex)/2); } public static void main(String[] args) { //Integer[] unsortedArray=new Integer[]{1,32,121,1424,2,1214,121214,3535,754,343}; //Integer[] unsortedArray=new Integer[]{4,4,8,0,8,9,7,3,7,6}; Integer[] unsortedArray=new Integer[]{5,5,5,5,5,5,5,5,5,5}; long startTime = System.nanoTime(); System.out.println(\u0026#34;Original array : \u0026#34;+Arrays.asList(unsortedArray)); quickSort(unsortedArray, 0, unsortedArray.length-1); System.out.println(\u0026#34;Sorted array : \u0026#34;+Arrays.asList(unsortedArray)); System.out.println(System.nanoTime()-startTime); } } References\nGoodrich and Tamassia Three Beautiful Quicksorts Quicksort is Optimal Unimodal sequences ","date":"2012-08-20T19:06:00Z","permalink":"/quicksort-the-easy-way/","title":"Quicksort - the easy way"},{"content":"Never send a human to do a machine’s job Testing a CRM application which I am part of became very difficult over the recent months. There were simply too many usecases. Despite unit testing on the backend services (SOA), bugs crept in on a regular basis considering a lot of logic is on the javascript and other layers of the application.\nThis is Part 1 of this series explaining how the following problems that I faced during testing were addressed :\nThere were too many combinations of test scenarios Application logic was spread over javascript, front-end, service and database tiers Existing testcases for services/portals are not exhaustive Our Unit testcases verified against static data Enabling/disabling/visibility/invisibility of fields on the front end could not be verified by plain Unit tests Incremental failure built inside JUnit testcases is not always useful. A report of all the issues would be nice. The itch and the scratch The Tool was built upon Selenium, Spring and Velocity.\nThe following features were built into the tool\nSelenium As part of the web automation, a series of utility methods was written/stolen from various sources. I am sure you\u0026rsquo;ll find the SeleniumUtils class interesting and useful.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 import java.io.File; import java.io.IOException; import java.util.ArrayList; import java.util.List; import org.apache.commons.collections.ListUtils; import org.apache.commons.io.FileUtils; import org.apache.commons.lang.StringUtils; import org.apache.log4j.Logger; import org.openqa.selenium.By; import org.openqa.selenium.JavascriptExecutor; import org.openqa.selenium.OutputType; import org.openqa.selenium.TakesScreenshot; import org.openqa.selenium.TimeoutException; import org.openqa.selenium.WebDriver; import org.openqa.selenium.WebElement; import org.openqa.selenium.support.ui.ExpectedCondition; import org.openqa.selenium.support.ui.ExpectedConditions; import org.openqa.selenium.support.ui.Select; import org.openqa.selenium.support.ui.Wait; import org.openqa.selenium.support.ui.WebDriverWait; /** * @author Arun Manivannan * */ public class SeleniumUtils { private static Logger logger=Logger.getLogger(SeleniumUtils.class); public static void populateDropDown(WebDriver driver, By by, String value) { Select select = new Select(driver.findElement(by)); select.selectByValue(value); } public static ExpectedCondition\u0026lt;WebElement\u0026gt; visibilityOfElementLocated(final By locator) { return new ExpectedCondition\u0026lt;WebElement\u0026gt;() { public WebElement apply(WebDriver driver) { WebElement toReturn = driver.findElement(locator); if (toReturn.isDisplayed()) { return toReturn; } return null; } }; } public static void populateTextBox(WebDriver driver, By by, String value) { WebElement inputElement = driver.findElement(by); if (\u0026#34;\u0026#34;.equals(value)) { inputElement.clear(); } else { inputElement.sendKeys(value); } } public static void checkRadio(WebDriver driver, By by) { WebElement inputElement = driver.findElement(by); inputElement.click(); } public static void goToTab(WebDriver driver, By by) { waitUntilClickable(driver, by); driver.findElement(by).click(); } public static WebElement waitForVisibility(WebDriver driver, By by) { return waitForVisibility(driver, by, 45); } public static WebElement waitUntilClickable(WebDriver driver, By by){ WebDriverWait wait = new WebDriverWait(driver, 45); WebElement element = wait.until(ExpectedConditions.elementToBeClickable(by)); return element; } public static WebElement waitForVisibility(WebDriver driver, By by, int waitTime) { Wait\u0026lt;WebDriver\u0026gt; wait = new WebDriverWait(driver, waitTime); WebElement divElement = wait.until(visibilityOfElementLocated(by)); return divElement; } public static WebElement switchToNewWindow(WebDriver driver, String iframeId) { driver.switchTo().frame(iframeId); WebElement window = driver.switchTo().activeElement(); return window; } public static String getTextFromId(WebDriver driver, WebElement navigator, String id) { //String text = navigator.findElement(By.id(id)).getText(); System.out.print(id); final WebElement element = findElement(driver, By.id(id),3); ElementType currentElementType=getCurrentElement(element); String text=StringUtils.EMPTY; switch (currentElementType) { case INPUT: text=element.getAttribute(\u0026#34;value\u0026#34;); break; case DIV: text=element.getText().trim(); break; case TEXTAREA: //text=element.getText(); text = (String)((JavascriptExecutor) driver).executeScript(\u0026#34;return document.getElementById(\u0026#39;\u0026#34;+id+\u0026#34;\u0026#39;).value\u0026#34;,\u0026#34;\u0026#34;); break; case SELECT: //Select select=new Select(element); //text=select.getFirstSelectedOption().getAttribute(\u0026#34;value\u0026#34;); text = (String)((JavascriptExecutor) driver).executeScript(\u0026#34;return document.getElementById(\u0026#39;\u0026#34;+id+\u0026#34;\u0026#39;).value\u0026#34;,\u0026#34;\u0026#34;); break;\tdefault: break; } System.out.println(\u0026#34; : Text value : \u0026#34;+text); return text; } private static ElementType getCurrentElement(WebElement element) { String tagName=element.getTagName(); ElementType elementType=null; if (StringUtils.equalsIgnoreCase(tagName, Constants.INPUT)){ elementType=ElementType.INPUT; } else if (StringUtils.equalsIgnoreCase(tagName, Constants.SELECT)){ elementType=ElementType.SELECT; } else if (StringUtils.equalsIgnoreCase(tagName, Constants.DIV)){ elementType=ElementType.DIV; } else if (StringUtils.equalsIgnoreCase(tagName, Constants.TEXTAREA)){ elementType=ElementType.TEXTAREA; } else{ logger.error(\u0026#34;Unhandled element type : \u0026#34;+element.getTagName()); } return elementType; } public static WebElement findElement(WebDriver driver, By by, int timeoutInSeconds){ try { WebDriverWait wait = new WebDriverWait(driver, timeoutInSeconds); wait.until(ExpectedConditions.presenceOfElementLocated(by)); } catch (TimeoutException e) { logger.error(e.getMessage(),e); e.printStackTrace(); return null; } return driver.findElement(by); } public static boolean isEnabled(WebDriver driver, String eachField) { return (driver.findElement(By.id(eachField)).isEnabled()); } public static boolean isDisabled(WebDriver driver, String eachField) { return (!driver.findElement(By.id(eachField)).isEnabled()); } public static boolean isVisible(WebDriver driver, String eachField) { return (driver.findElement(By.id(eachField)).isDisplayed()); } public static boolean isInvisible(WebDriver driver, String eachField) { return (!driver.findElement(By.id(eachField)).isDisplayed()); } public static List\u0026lt;Message\u0026gt; checkEnabledFields(WebDriver driver, List\u0026lt;String\u0026gt; enabledFields) { System.out.println(\u0026#34;Check enabled fields\u0026#34;); if (enabledFields==null) return ListUtils.EMPTY_LIST; List\u0026lt;Message\u0026gt; messages=new ArrayList\u0026lt;Message\u0026gt;(); boolean result=false; for (String eachField: enabledFields) { result=isEnabled(driver, eachField); messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.ENABLED, Constants.DISABLED, eachField)); } return messages; } public static List\u0026lt;Message\u0026gt; checkDisabledFields(WebDriver driver, List\u0026lt;String\u0026gt; fields) { if (fields==null) return ListUtils.EMPTY_LIST; List\u0026lt;Message\u0026gt; messages=new ArrayList\u0026lt;Message\u0026gt;(); boolean result=false; for (String eachField: fields) { result=isDisabled(driver, eachField); messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.DISABLED, Constants.ENABLED, eachField)); } return messages; } public static List\u0026lt;Message\u0026gt; checkVisibleFields(WebDriver driver, List\u0026lt;String\u0026gt; fields) { if (fields==null) return ListUtils.EMPTY_LIST; List\u0026lt;Message\u0026gt; messages=new ArrayList\u0026lt;Message\u0026gt;(); boolean result=false; for (String eachField: fields) { result=isVisible(driver, eachField); messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.VISIBLE, Constants.INVISIBLE, eachField)); } return messages; } public static List\u0026lt;Message\u0026gt; checkInvisibleFields(WebDriver driver, List\u0026lt;String\u0026gt; fields) { if (fields==null) return ListUtils.EMPTY_LIST; List\u0026lt;Message\u0026gt; messages=new ArrayList\u0026lt;Message\u0026gt;(); boolean result=false; for (String eachField: fields) { result=isInvisible(driver, eachField); messages.add(ValidationReportUtils.constructFieldValidationMessageFromResult(result, Constants.INVISIBLE, Constants.VISIBLE, eachField)); } return messages; } public static String captureScreenshot(WebDriver driver, String folder, String fileName) { File screenshotFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE); File targetFile=new File(folder, fileName+\u0026#34;.png\u0026#34;); try { FileUtils.copyFile(screenshotFile,targetFile ); } catch (IOException e) { logger.error (\u0026#34;Error while writing file \u0026#34;,e); } return targetFile.getAbsolutePath(); } } ","date":"2012-07-30T06:31:00Z","permalink":"/testing-automation-with-selenium-part-1/","title":"Testing automation with Selenium"}]