Log Files

In today's world, the amount of log data generated by any application is
so much that it is not possible to open the said log files in a text
editor of choice. The modern logging framework like log4net make it
possible for the log files to be split based on size, so that, one can atleast
open them in a text editor. But the problem with this is, there is most often,
a case of explosion of number of log files and pretty soon we will have GBs
worth of log files all over the place. The whole logging issue becomes a mess
once we start to have a distributed system. Looking through multiple log
files from different sources and making sense of them is a pain.

Recently, I faced exactly the same situation. I found myself sifting
through GBs of log data generated by a system which had been running for
many days.

In a multi threaded environment, logs are written to a log file by
multiple threads and sequence of operations from a single thread will not
be available in consecutive lines. If the logging is extensive, then the
sequence of operations that we are interested in, might also be a subset
of the entire log file. Sifting through the log file for this is very time
consuming if done manually. Especially, if the text editor used is a very
basic one, with no advanced features to do marking, highlighting and
searching.

I use VIM, so I get by very well, for sifting through huge log files.

Nevertheless, I found myself doing this activity so often, that I had to think
of making a small tool for this.

The basic set of needs that I had were :
- Should be able to look into multiple files
- should be able to look into large files
- should be able to filter out only those lines that I am interested in based
  on REGEX pattern matching.
- should produce a consolidated result file.

So I came up with this small utility class to do this for me :

Code 

1. public class LineFilter : ILineFilter
2. {
3.     private readonly string[] fileNames;
4.     private readonly List<string> searchFilter = new List<string>();
5.  
6.     private LineFilter(params string[] fileNames)
7.     {
8.         this.fileNames = fileNames;
9.     }
10.  
11.     public ILineFilter WithFilter(string filter)
12.     {
13.         searchFilter.Add(filter);
14.         return this;
15.     }
16.  
17.     public void WriteResults(string outFileName)
18.     {
19.         var regex = string.Join("|", searchFilter);
20.         var g = new Regex(regex, RegexOptions.IgnoreCase);
21.         var linesToWrite = new List<string>();
22.         foreach (var f in fileNames)
23.         {
24.             Console.WriteLine(f);
25.             var lines = File.ReadAllLines(f);
26.             foreach (var line in lines)
27.             {
28.                 var m = g.Match(line);
29.                 if (m.Success)
30.                 {
31.                     linesToWrite.Add(line);
32.                 }
33.             }
34.         }
35.         File.WriteAllLines(outFileName, linesToWrite);
36.     }
37.  
38.     public static ILineFilter CreateLineFilter(params string[] fileNames)
39.     {
40.         return new LineFilter(fileNames);
41.     }
42. }

The way to use this class is as follows :

Code

1. LineFilter.CreateLineFilter(
2.             @"d:\MyLogFile-Debug-2015-04-29.11.txt",
3.             @"d:\MyLogFile-Debug-2015-04-29.10.txt",
4.             @"d:\MyLogFile-Debug-2015-04-29.9.txt",
5.             @"d:\MyLogFile-Debug-2015-04-29.8.txt",
6.             @"d:\MyLogFile-Debug-2015-04-29.7.txt",
7.             @"d:\MyLogFile-Debug-2015-04-29.6.txt",
8.             @"d:\MyLogFile-Debug-2015-04-29.5.txt",
9.             @"d:\MyLogFile-Debug-2015-04-29.4.txt",
10.             @"d:\MyLogFile-Debug-2015-04-29.3.txt",
11.             @"d:\MyLogFile-Debug-2015-04-29.2.txt",
12.             @"d:\MyLogFile-Debug-2015-04-29.1.txt",
13.             @"d:\MyLogFile-Debug-2015-04-29.txt")
14.     .WithFilter(@"starting task A [A-Z0-9]{8}-([A-Z0-9]{4}-){3}[A-Z0-9]{12}")
15.     .WithFilter(@"total DB connections are")
16.     .WithFilter(@"task A [A-Z0-9]{8}-([A-Z0-9]{4}-){3}[A-Z0-9]{12} actions")
17.     .WithFilter(@"task A [A-Z0-9]{8}-([A-Z0-9]{4}-){3}[A-Z0-9]{12} ends")
18.     .WriteResults(FileName);

As you can see above, it is very convenient to use this. No matter how
many files I have, it will filter out the unwanted lines and generate
a single file from it. Of course, if the filter is letting through a lot
of lines then the number of lines in the generated file is going to be
large. This means the generated file will also be large. We can split the
writing to the output file and make multiple files and this is pretty easy
to do. In the above example I have used the REGEX for filtering the GUID.

I am a big fan of REGEX, particularly for the searching of log files.
A big use of this log filtering utility is that I can take output of the
tool and feed it to a graphing software to generate graphs. All logs have
a timestamp as their first column. So if we can filter logs indicating
some type of start and end of actions, then the generated log file will
look like:

Log

1. 2015-04-29 13:54:12,123 [Thread 1] Starting Task A 9e8fa525-5eeb-4653-b9f6-a489017eada8:
2. 2015-04-29 13:54:42,123 [Thread 1] Task A 9e8fa525-5eeb-4653-b9f6-a489017eada8 ends

This can used by the graphing software to generate graphs of when
"task A" started and ended and how long "task A" took.

This class is not perfect, but it does the job for me. I see many improvements
that can be done to this utility class :
- Splitting of the output file into many files if the size of the output
  file is large.
- Speeding up the whole process by using multiple
  threads.
- Reordering the lines of the generated log file based on
  further pattern matching.

I have ideas with respect to 3rd point above, I shall write about it soon..

I only wrote this small utility to scratch my own itch. Of course
I recommend a better logging strategy,like logstash.net for highly
distributed systems . The utility here is only for making offline processing of
log files and generate meaning from them. For example, go through 4 days
of worker logs and make a graph of the timings of actions.

Practice makes a man perfect - CodeKata

A kata is an exercise in karate where you repeat one particular form, multiple
times, each time making little improvements. The intent behind code kata is
similar.

How did Sachin Tendulkar or Roger Federer become the greatest players in their
sport?

Through practice. They practiced everyday. They did, even after they became
no.1 in their sport. It is an accepted norm to practice in the sports industry.
However, this doesn't happen in the software industry. A software
engineer doesn't practice his profession daily. Everyone thinks  "I write software
daily, so why practice writing more software ??"

Sadly, this is not wise to say. One has to practice .. to get better... to learn ..
to stay relevant .. to stay current

And so begins my journey on Code Kata. Inspired by codekata.com, which has many
interesting problems to practice on. Today, I am attempting to solve problem 1.

Kata01: Supermarket Pricing

Using some of the concepts of DDD, I tried to come up with a ubiquitous language
that might fit the model needed here based on the information presented.

"A super market order contains many items, which depict the products, their
quantity and their billing amount based on a pricing scheme"

This leads to the following model:

    - An order has many items
    - An item has a product, its quantity
    - A product has the pricing scheme used to calculate the amount.
    - A pricing scheme has many types
        - Single e.g. this can of beans costs $0.65
        - Multiple e.g three for a dollar
        - NonCountable e.g $1.99/Pound
        - MultipleWithOffer e.g buy two, get one free

This is shown in the UML below :

The clear advantage is that calculation of the total bill amount and individual
line item's amount easily fits in a strategy design pattern. Adding more
pricing schemes becomes easy, Software becomes more composable.

Let us now answer the questions posed in the code kata 1 :
   
    - In the case of Multiple pricing scheme, like "three for a dollar", what is
      the price if I buy 4, or 5 pieces?
            A: It is the responsibility of the algorithm in the "Multiple"
            pricing scheme to decide this based on the business rules. for
            e.g. 4th and 5th one will have price of what a single piece would
            have if only one piece was bought.
    - In the case of  NonCountable scheme, like $1.99/pound, what does 4
      ounces cost?
            A: Again calculated by the "NonCountable" algorithm, which would
            have conversion methods to different fractions of the measure and
            also to different units of the measurement. Adding a conversion
            algorithm for a newer unit later would just mean adding a method
            and/or decorator pattern.
    - For MultipleWithOffer, like "buy two, get one free", does the third item
      have a price?
            A: maybe, depends on the business rules.

This was my kata for today, trying to practice my DDD and modelling skills.
I don't know if this approach mentioned is the best way to model the
supermarket pricing. However, I know  practicing daily will surely make me
better at modelling software.

Happy Practicing !!!

To-do and task managers - plenty of fish in the sea

There is no dearth of to-do and task manager applications in this world. All boast a lot of features, and most of them get it wrong, Fancy UI is not the key point in such apps, but the workflow that it provides is.

If one has a look at the typical workflows of people with respect to tasks and to-dos, then it becomes abundantly clear, what are the features that these apps must be concentrating on.

Primary reason why people use to-do managers is that human brain is not designed to remember things and do it before the task becomes overdue. For e.g. if we need to buy new batteries for the torch, human mind will only remind us that we need new batteries, when the power goes off and not when we are near the shop. For this reason, people use to-do managers so that they can look at the shopping list when they are near shop to check if the need any purchases to be made. Thus, To-dos must be accessible everywhere, on all my devices.

Let us look into the anatomy of a "task" There are many types of task that one usually performs.

• Something that one has to do on a given day. e.g. implement feature "x"
• Something that one does many times periodically. e.g. pay bill on 1st of month.
• Something to respond to. e.g. email, phone call, article on a browser.. etc
• Hierarchical to-dos, like sub projects under projects etc,
• Merging of calendar and to-do view to get a better view of the day

In my opinion, Point 2 is extremely important, If a to-do manager does not have recurring tasks, then it is not a good to-do manager. By not having recurring tasks with good customizable scheduling rules, it is only subjecting the user to a lot of manual scheduling and wasting the users time. Imagine a task/time management software wasting time !!!

This means, a good to-do manager, is one, which runs/syncs on multiple platforms, with features such as recurring tasks. It should have ability of importing tasks from various sources like email, phone, browser etc. It should also support hierarchical tasks and single view of calendar and tasks.

I recently discovered this great app called "todoist", which satisfies all the above conditions. It is a very nice app and developed by understanding the problems of the users instead of just a fancy UI.

Kudos to the todoist guys !!