Audit and Security are not meant to be interceptors

I was working on an administrative web interface for a user management product, and was trying to find out how best to do audit and security.

I should throw up the conclusion first: Audit and Security are not meant to be interceptors.

And now for the journey to that conclusion.

We are in the year 2009. Java annotations are on the rampant. Every problem looks like a nail to that hammer. And it is no surprise that there are many projects which implement audit and security annotations, and run them through interceptors like AOP.

Which I tried as well. It worked great initially. Marking methods to be auditable and securing methods with roles were simple, and does not obstruct code logic.

Until the security got more 'realistic' and requring the audit to be 'useful'.

In the original concept, a user could be an administrator, and able to modify any users. In the updated requirement, a user could manage multiple group of users. No longer that simple to just check by a single role a user has. The role would be dynamic too. To support multiple dynamic groups after deployment, the role name would be built up from the name of the group of users. And annotations for role name support only strings that can be determined from compile time!

Granted, a more complicated security annotation might work. Something along the line of annotating a method with a security policy instead of a role name...

But let's move on to audit first. The most common audit approach is probably marking methods as auditable, and then record all input arguments and output result of that method.

That seemed to me to be a rather... technical audit rather than a business audit trail. Translation of the audit information to business information is not so easy. And there are definitely times when the required audit information might not be part of the input arguments nor output result. And this might violate the SRP to some people, but it's entirely possible that a single method is in fact, multiple events and require two audit entry instead of one.

My point here is probably this, that for a given business scenario, there are definitely clear requirements on what audit information to capture. And these information usually go beyond the simple 'method invocation with arguments, results, or exceptions'.

I'm definitely not against annotations and interceptors. They are excellent idea, and work great for many cases. My grip is that audit and security functional requirements are never that trival, in the sense that many out of the box interceptors will fulfill the requirements. And there is just too much hype in audit and security interceptors as the holy grail.

Danger of being overly focused (to a skillset)

I started playing with XCode, Objective-C and iPhone Application Development recently (which was about 3 days ago).

Some background on my skillset first.

I started out learning Pascal and Foxpro, and then proceed to C++ with Visual C++. Along the way I picked up VB, and then Java. Then I worked with Java professional for about 3 years, and then out came C# (within that 3 years), which I then decided to make a jump for it, but ended up with VB.Net and Visual Studio for about a year. And then I decided that Java had a better career opportunity back then. So it was back to Java and Eclipse for another 2+ years.

And this is where I am now. Which is to say that I am strongly comfortable with Java and Eclipse (but seriously, Eclipse is an excellent IDE).

During the time I read up on Ruby and Python, but resisted developing with them, blaming the lack of an IDE with the quality of Eclipse.

And now that I want to try out iPhone Application Development, I had to pick up Objective-C and XCode.

I started reading up on both topics. And with every read, Objective-C felt like an excellent language. Named parameter was a big plus for me. Categories were a good way to slice up code into separate files (much like .Net's extension methods). Other than that, much was the same as old time (to me). XCode itself was a good tool, with code completion support (though I have yet to read anything on refactoring, and not much books talk about unit testing though they are available online).

And then came the time of actual usage. I was confident that with my experience in C++ I could tame that Objective-C beast. And the tool did seem good.

I was wrong. Not on the account of Objective-C. That was pretty easy. I am a firm believer that language syntax are simply decorations. And so I had concentrated on honing the more important skills of writing clean code, design patterns, building stronger foundation. But development still take a while, because I was so used to Java. Coding in Objective-C requires some mindset changes. Recognizing new symbols.

XCode was a much bigger killer. Code completion works, but code 'suggestion' was a different key. I had to retrain my instincts. I have yet to figure out unit testing, but probably will complain when it get working, citing it as different and weird compared to Eclipse (but that is definitely due to my overly used to Eclipse, not because of XCode). Checking for compilation errors was different and had to get used to. And looking at the logs while debugging was... well just wonder why I had to switch my view in XCode. Perhaps there is a different project layout view (I was sure I read something on it), and I should try that.

So why so much pain? I became overly reliant on a single tool and language. And that is definitely a very dangerous thing. Regardless if you read up on different tools or languages, if one does not actually try using them, they will not be able to operate under different environment. And as developers, we MUST be as flexible as possible. One can never know what tools one might use in the future of their career. (I am currently cursing on using Visual Studio in Windows XP VM on my Mac... GRRRR).

And after years of using Java, a garbage collecting language, I had to revert back to a non-garbage collected language. More care is put into each line, and there is always a wondering thought on if I should release the memory of something returned from a system call, or not if I am returning them as return arguments. It takes some getting used to.

I probably should have tried out Ruby and Python more often, and C++ too, with a variety of tools. Life would be better.

Stop anticipating requirements

As developers, we want an easier life. We worry and fear of unstable client requirements. We think of any possible ideas the client might come up with, and implement hooks for them in our current code (as long as they are not too tough, and even that is relative to a developer's skill).

Half the time, we are right, and we felt good about saving ourselves a lot of hell by putting in the hooks early.

The other half, the requirements never come. But we still felt good because we WILL be saving ourselves a lot of hell by putting in the hooks early.

But in general? The extensions and hooks we introduced made the system more complex. Code does not read that naturally, hidden within implementations of Template Pattern, nested one within another. The runtime structure of an application is so remotely different from the static structure of an application (class diagrams), that documentation no longer seem to be useful. In fact, they seem to be talking about totally different applications! The growing unnecessary hooks only add to more documentation, which seemed to scare every other developer away except yourself. And we wonder why.

But what could be worse? Well, that the requirements for a new feature came. And it is totally in conflict with the hook we planned. Well, we could argue that the hook was not well designed, and some change would be necessary, and then we could wait for the 'real' requirements to come. So for now, we add a hook to the hook, happy that we have added even more flexibility. And while we are at it, we thought of another possible client requirement. So we add another hook too.

And the story goes on. Hooks on hooks on hooks. on hooks. Half of them probably would never be used.

So what exactly went wrong? Obviously, the problem is us.

We tried to act like we are the client. Except we are not. We are probably the worse candidate to act like the client, since in most cases we are developing for a problem domain of which we are neither expert, nor even remotely familiar with.

Which is why as developers, we should always keep our solution simple. A simple solution is a flexible solution. Add hooks only if the actual client requirement comes. The value added service we provide here is faster delivery of solution, instead of a monolith and extensible solution but would require months to add new features.

Remember, in most cases the clients are the domain experts, not us. They may not always know what they want, but they do know what they do day-to-day. Spend time with them, understand their need. Do not give in to their desires. And do not give them something that they do not need.

Towards Better Code: No side effects, shorter methods, exit early, and log every invocations

Recall earlier that I had mentioned that I learnt through logs of existing program/framework.

It's weird, but I cannot emphasize enough about the importance and need to log.

I have been using a rather new approach to logging. Well, not exactly to logging, but a new approach combining logging and coding style.

First off, I keep my methods short. For example, given the following algorithm:

public void login(final String username, final String password) {
  // authenticate
  // check if user container is active
  // check if user is active
}

It will most likely evolve into the following code:

public void login(final String username, final String password) {
  authenticate(username, password);
  verifyUserContainerIsActive(username);
  verifyUserIsActive(username);
}
void authenticate(final String username, final String password) {
  final boolean valid = checkUserCredentials(username, password);
  if (!valid) throw new AuthenticationException();
}
void verifyUserContainerIsActive(final String username) {
  final Container container = getUserContainer(username);
  if (container.isInactive()) throw new UserContainerInactiveException();
}
void verifyUserIsActive(final String username) {
  final User user = getUser(username);
  if (user.isInactive()) throw new UserInactiveException();
}

That's just a rough idea. Basically, each method is composed of many more methods, which serve to describe what each step do. Benefits? Eliminate the need to comment complicated algorithms. Comments run the highest risk of being out of sync with the actual code intent.

Also, notice the use of exceptions in the inner methods. I had started out with an alternative idea, that the inner methods would return true/false values, which would. It basically create the following nasty code block:

public void login(final String username, final String password) {
  if (isValidCredentials(username, password)) {
    if (isUserContainerIsActive(username)) {
      if (isUserIsActive(username)) return;
      throw new UserInactiveException();
    }
    throw new UserContainerInactiveException();
  }
  throw new AuthenticationException();
}

Of course, the better way is as follows:

public void login(final String username, final String password) {
  if (isNotValidCredentials(username, password)) throw new AuthenticationException();
  if (isUserContainerNotIsActive(username)) throw new UserContainerInactiveException();
  if (isUserIsNotActive(username)) throw new UserInactiveException();
}

Personally, I still felt that my first example up there produce a much more readable version of the algorithm, which communicates the intent much better.

But the basic idea/rule here is, exit early. What is most important is to communicate the basic intent of the code. The other possible cases are, well, exceptions (pun intended), and should not clutter the original intent of the code.

And finally, on to logging.

Now, I log all entry and exit of a method, as well as method arguments and method return values. Ideally this should be done with AOP, but I am not doing that yet.

What benefit does this logging bring me?

I can actually, from tracing the method entry logs, form a very clear view of how a request pass through the entire system. An example of the above log might be as follows:

enter: login {username:kentlai, password:xxx}
enter: authenticate {username:kentlai, password:xxx}
exit: authenticate
enter: verifyUserContainerIsActive {username: kentlai}
exit: verifyUserContainerIsActive
enter: verifyUserIsActive
exception caught: UserInactiveException

With shorter methods, logging every method invocations, I can trace the flow very accurately. Sure, there is an increase in the size of logs, but the return in such investment is easier debugging. Without even hooking up to the debugger.

With the logged method arguments, and knowing which method is causing a problem (from the logs), I can actually write a quick unit test to fix any potential problem found.

But there is one last magic elixer in this combination. No side effects.

Basically, each class should not have mutable state. The only mutable state should be from external storage (eg. database). And if a function need to use any values from such storage, they should, themselves be encapsulated into methods of their own, with the return values logged.

Why? This allow you to put together a complete picture, the context and environment of which a method was executed under. When every variable becomes known and can be supplied once again to a method during unit testing, a method can be guaranteed to work consistently and constantly as expected, which allows for more robust code.

Funny it took me so long to realize such principals.

Where can Twitter get money?

Twitter has recently annouced that they might offer a premium twitter service at the end of the year. But before then, there were endless thoughts and advices on how to run twitter. How various company should buy twitter, why twitter should sell, and wonders on how long twitter will go down the deadpool.

Except seriously, twitter has nothing to be afraid about. Various attempts have been made to create alternative microblogging services, tools, and software, but they did not seem to gain the same amount of traction and adoption rate.

Twitter is an infrastructure. Messaging delivery infrastructure.

The rate of growth is unbeliverable fast. The ecosystem of tools and services around it is huge as well.

People are using it for real time news and search. Recently, if I have to find information, I use twitter search first before I even try google.

Companies are monitoring twitter for keywords to do direct sales and product support. In fact, they are more responsive than me sending them an email for support!

Imagine if twiiter goes bankrupt and into the deadpool. Imagine all the companies that will be affected.

Which is why all the alternatives to twitter keep coming up. And the advices to business models and selling to a rich company like microsoft or google.

People are afraid of losing such an important tool. And people do recognize its importance!
And so the investments come. People who are betting on a future with twitter. They might want direct paybacks on the long term. Or they might have interest in the twitter ecosystem. Or they might be using twitter and am convinced how it will help them in the long run.
That is an interesting growth/survival strategy.
Start with something free. Make it easy to use, and easy to create tools and services around it (the API). As long as it gains popularity with the netizens, and the ecosystem starts to grow, investments (or donations) will come.
But that's easier said than done I guess.

The next developer is probably not as good

Of course, this is an unfair judgement call, since you probably had not even met the next developer.

But this is something my colleague kept trying to hammer into me, which has its valid point.

Why does he say so?

Well, simply because my code is really, rather hard to dive into.

We are actually working on an in-house project, where we want to set up a simple framework, and have the future development 'outsourced' if possible. We had a quick stint with an intern, but it took really long for him to get into.

There are various technology used.

Spring DI. Spring JDBC. Spring LDAP. Spring JavaMail.

Apache Quartz. Apache Struts 2. Apache Tiles. Apache Velocity. Apache Maven.

Logback. JUnit. Mockito. Selenium.

All rather simple to learn and use... to me that is.

He made an interesting point. That most developers do not learn new stuff on their own. They are component with Java, but perhaps as a language. They do not go experiment with the bleeding edge. They are comfortable with the basics. Some of them are probably still in the plain old Servlet and JSP era! Annotations? What are those? How do we use them?

I came initially from the 'productive rock star' perspective. Anything that keeps the stack lean and mean, and allow me to quickly deliver business values, the better it is.

But given his perspective, I honestly doubt a newcomer can easily pick it up. I will have to explain Maven, set it up for him. Explain unit testing and Selenium. Mocks. What Spring DI is. Where are the Struts localization files. How Tiles work. Logging with logback. Struts interceptors. Struts validation... blah blah blah.

Which brings me back to reiterate the same point in a different word from him again. Assuming that there are component developers in the company. But being in a company, there are definitely programmers of different level and standards. Being on the bleeding edge might also mean that the knowledge of using it being constraint to the 'elite'. The rest 'inferior' developers might find it harder to pick up. Or maybe even impossible and never.

And as we saw in the phrase, "It is all very much good to place all decision making in the hand of the perfect man, but what if the perfect man has a bellyache?"

We are using Java. Why Java? Sure, it takes more line to do the same work in Java than probably in Ruby or Scala. But it is 'clear and simple', and 'mass-market'. It is very much easier to get java developers here than the others.

So right now I'm taking a step back. Annotations might have to go. Various Struts specific concepts might have to go too. An abstraction layer of the above mentioned stack might have to be developed, such that we can have a consolidated framework and usage pattern, along with documentations on how to use that.

That was his idea, but it definitely has merit. Given a one to one mapping (even though it is a thin redirection layer), we can simply tell the incomer to use 'our libraries', complete with tutorials and example. It promotes a standardization of company code, and probably is much more helpful in the long run.

Learning new frameworks through logs and reading source code

I was having lunch with a colleague, and commenting on how slow another colleague was in learning new frameworks. I wondered if it had to do with experience, since he was relatively new. But still, the rate he was picking up was unimaginable (to me at least).

I cited how I had to help in resolving a compilation and deployment problem, which by just googling a similar pattern of the error message (since the exact message would not yield that close a match with our project names, user folders, etc in it) gave me a hit on the first page (maybe not the first result, but at least it was on the first page).

I wondered if it had to do with the educational system, where they are more spoon-fed and focused on passing exams. Back in the older days we had a genuine desire to learn new stuff. We would borrow and devour books (figuratively speaking)

And then I mentioned about how I went about learning Struts2 and Tiles for this current project.

Which led me to notice the approach I took to learning it (as well as the previous JAX-RS framework).

1. Reading official documents and tutorials. The framework usually have some form of documentations. Even if they are incomplete or few, they are still a source of learning.

2. Google. For something you want to be done, there is a high chance that someone else has done it as well.

Usually these two would be sufficient, but recently, I go a step further.

I just came back from an IDM project. Sun Identity Management product. It was heavily customized. And at times, we had to deal with insufficient documentations, and find out if something is a product bug or a customization done wrong.

And how we do that is to turn on log tracing, and trace the calls, and then check the product source code (decompiled).

After the project, I had just unknowingly do that whenever I hit a problem in using a framework.

I fire up the logging, and find out which method call I had to use, and go through to look at the source.

In fact, it helped me understand the framework much better.

Of course, some would argue that reading source is hard. But I did not find it that way. I could still find my way around it, even though it might take a while.

Let me share my latest experience.

I was using Struts2 with Tiles, and I ended up with a problem of wildcard matching of definition names, and cascading attributes. The attribute was not found in the nested definition.

First, I tried to check the documents. They cited examples of wildcards, and then of cascading attributes, but not together. So I was not able to determine if I was using it correctly.

Next I tried to google. It took a while, but I finally managed to find someone with the same problem. Apparently it was a bug the guy reported, and he was asked to produce a JIRA (not sure if he did), and it was dated April 2009.

But this did not solve my problem, since it was a deliverable project. So I had to fix it myself. I dug through the code, and realized that the problem was in ResolvingLocaleUrlDefinitionDAO.replaceDefinition, where the definition was not duplicating the cascading attributes, only the local.

Now to figure out how to fix it. I just need a custom DAO. From my experiences with the various frameworks, a good framework usually allow you to provide custom components, defined in a property file or web.xml. There was no property to provide a custom DAO (I traced through the DAO creation path by code), but there was a way to provide a custom DefinitionsFactory, which create the DAO.

So I did that, and solved my bug.

If I had gave this task to that new colleague, I doubt that the bug can be resolved. Somehow, decompiling might not have crossed his mind.

I am trying to find out, what was the reason of the difference.

Was it because I was placed in a situation where I had to read the source and debug and fix?

Was it because I had more years of experience working? (approaching 6 this year)

Was it because of inadequate googling skills? (That proved to be a problem for another of our ex-colleague, who was not that good in English)

Was it because I am used to reading logs?

Was it because I am comfortable reading source code?

But in any case, learning new framework and reading source code does give me various new ideas on how to design and develop. A case in point: if it is a framework, most components should be allowed to be defined and replaced by the consumer of the framework, just like the Tiles case above.