Friday, September 11, 2015

IntelliJ Feature Request: Find Escaped Strings


IntelliJ has some automatic String escaping built in.

IntelliJ automatic String escaping on copy/paste


Example: pasting TO IntelliJ
Write some code:
String s = "";
Then paste this sequence between the double quotes: <li><a href="#">
What you have is:
String s = "<li><a href=\"#\">";
IntelliJ intelligently escaped the double quotes for you.

Example: copying FROM IntelliJ
Select the string literal above, including the surrounding double quotes ("<li><a href=\"#\">"), and paste it into Notepad. What you get is exactly that.
Select the string content, excluding the double quotes (<li><a href=\"#\">"), and paste it into Notepad. What you get is the string unescaped: <li><a href="#">

That's pretty smart, and most always what I want.


IntelliJ automatic String escaping on search


Example: ctrl-alt-f searching for <li><a href="#"> does not find our code String s = "<li><a href=\"#\">";



Here's an idea for the IDEA team: how about a checkbox "Find escaped strings"?
Of course, in practice it'll get a bit more complicated than just handling the double quotes. Depending on the programming language. JavaScript has single quotes, there are regular expression and SQL escapes, etc. And I'd want to be careful with the search-and-replace functionality (or just not offer this feature there).


Conclusion

The Java editor has already proven that it can handle the escapes to the advantage of the developer. So why not extend to search?

Wednesday, June 17, 2015

French OVH vs German Hetzner: the Stereotypes Are Alive

This is an open letter describing my experiences and offering some suggestions to the French hosting company OVH.

I have had 2 cheap VPS at OVH for a year now. (And services from Hetzner for longer.)

Every once in a while, one of the virtual private servers at OVH is unreachable. Today was such a case; no information from OVH announcing the downtime, no alert while offline. My own monitoring shows one VPS unreachable for 1.5 hours in the afternoon. 1.5 hours. I wonder what it is that takes so long to be fixed.

So let's check the OVH status:

 
D'Accord.

I prefer how the competing German ISP Hetzner has a completely separate domain name for the status page at http://www.hetzner-status.de/ because you know maybe something is wrong with the primary domain or DNS record or whatever and then hell breaks loose.

Quite a while later I was able to get that page, here's how it looks:



The colors look scary, issues everywhere. But that's probably explainable by the large size of the ISP.

Although it's not visually visible, I have clicked and therefore filtered by the VPS category on the screen.

Unfortunately, with the information given in the table, I am not able to identify by which of the many cases my VPS could be affected. So let's log in to the admin panel of my hosting account.

There is no alert message here either. After a while I'm able to find charts that prove the downtime, and they indicate a restart:



The CPU was off and then a bump (boot). Ram was off too, and traffic was zero.

Recommendation to OVH:

  1. The backend tells me the VPS is in zone vz-rbx2-005. If you'd list that on the status page in the table, I'd have a chance identifying issues that affect me.
  2. Even better, show the issue in the customer control panel overview, and on the VPS overview page. With a nice history of all past issues.
  3. If it was a predictable/planned downtime, inform before. Hetzner does this. The last notice was about replacing a Switch. It came 1 week (!) before. And after all there was 0 downtime. German planning.
  4. During downtime, send an info pointing to the task. At least if it is not resolved quickly.
  5. After prolonged downtime (1.5 hours!) and an OS reboot (!) send an info maybe? That it happened, what the cause was, and what you intend to do in the future to prevent it. After finishing your croissant maybe.

Regarding email, OVH used to send alerts. There was a period with annoying alerts for high CPU load caused by Rsyslog on Ubuntu. OVH fixed it and they are gone. Other than that, I have not received alerts.

Out of curiosity, let's check the uptime of my second OVH virtual server.

> uptime
> up 2 days, 54 min,  1 user,  load average: 0.08, 0.03, 0.01

Quelle surprise! Considering that I have not touched this server in months...


Final words

I have no affiliation with any of the 2 providers, other than being a customer with both.

At OVH, everything feels a bit "French". The admin panels; confusing, one redirecting to the next and back. Parts are translated, parts appear in Français. The website. The payment process. I would not host anything uptime critical there.

Hetzner is not perfect. In a subjective classification I'd give them a "good to very good" - for my needs. Great value for the price. And I don't see anyone else on the horizon with a better service offering at competitive costs.

If you need a place for a game server, or other leet services, please choose OVH with their DDoS protection. I'd rather have you not too close to my important machines ;-)

If anyone else asks, then I personally recommend Hetzner.

Tuesday, May 5, 2015

Interpretations of end-of-file, and Linux ad hoc open heart surgery

Today's occasional Linux sysadmin task for a Java developer (me) is to loop all lines in a text file and run a task based on each. Can't be that hard, can it?

Off we go the usual route:

1) Google the task. Aha, looks like a common requirement.

2) Find the usual stackoverflow, askubuntu, superuser and some other forums and blog posts.

And the offered snippets looks simple and clear:

while read line; do
    echo "line is: $line"
done </my/file.txt

or alternatively

cat /my/file.txt | while read line; do
    echo "line is: $line"
done

And it works! Mostly. Occasionally the last line is missing. This happens when there is no trailing blank line in the file. A detailed explanation to that problem is here: http://stackoverflow.com/questions/12916352/shell-script-read-missing-last-line

So finally here's a snipped that works for all files:

cat /tmp/testfile | while read line || [ -n "$line" ]; do
    echo "line is: $line"
done

 

Fragile solutions


This most recent experience is very much like so many others I've had with Linux system administration tasks. Faced with a seemingly simple problem, I find many solutions, but there are hidden pitfalls. An implemented solution must be stable, and not depend on assumptions and best practices and current conditions. That's what we practice in programming. Why not in system administration?

In this case one can argue that a text file must end with a newline character. Some pro arguments:
  • Apparently there's some C specification (from a time before you were born).
  • When storing a text file in the Linux editor vi it's done, the newline is appended. (vi also being from the 70s)
  • When writing a file by code there's usually a println(string) method. This also results in the trailing blank line.
But there's also contra:
  • Files can come from other sources. For example from a Windows Notepad++ and the newline can be missing. 
  • Or if the file is generated by a program, it's easy for a developer to change some logic and remove the trailing newline, without being aware of the consequences.
  • In PHP, if you include a script file that ends with a newline after the closing "?>" end tag, it sends white space to the client, and prevents you from adding header()s. You must break that C standard. (PHP is written in C you see ...)
  • It just sounds like a silly specification, way too easy to break, and not necessary.
Ask yourself the question: if you'd write the specification where a file ends, would you choose "the last newline character" or "where the last bit ends"?


Stackoverflow helps but pay attention to the comments


Another case I recently run into was with a Linux cleanup process. The task definition was simple again: "delete all but the most recent 5 files from a folder matching a file name pattern". An accepted solution is here http://stackoverflow.com/questions/25785/delete-all-but-the-most-recent-x-files-in-bash with plenty of upvotes. The problems, which are not mentioned in the answer:
  • If there are only 5 or less files, it deletes them all
  • It doesn't work if there are folders present
  • It won't work with file names with spaces
In time I've learned to pay attention to the user's comments. And one saying "This one fails if there are no files to delete." made me suspicious. Again, it doesn't fail, the command goes through like a knife cuts through soft butter... no complaint, just removes them all.

Imagine this scenario (mine): It's the backup folder, and the last 5 backups should be kept. For some reason the backup process fails, and no new files are created. The separate cleanup script still runs nightly and removes all but the last 5 files. With this little glitch in the script you'll end up having no backup at all.


All ad hoc, open heart surgery


Linux sysadmin is not in my job description. I see the situation from a bit of distance. What I realize is that programming has come a long way in the last 20 years. Open source libraries are used instead of quickly hacked untested functions. Everything is version controlled. Nothing gets deployed without thorough testing. Pair programming. Code reviews.

System administration? Still the same. Copy pasting some commands found on the internet. On production machines. Works? No? Try another one. Works? Fine. Document changes? Nah.

Any sysadmin of +5 years who hasn't locked himself out of an ssh server by misconfiguring sshd or iptables in da house please stand up.


Conclusions


It's easier these days to find answers and solutions on the internet thanks to the Q&A format of Stack Exchange. But more than in  programming, in sysadmin the comments and secondary answers are important to read.

At the company we stick to some simple rules
  • Changes made to machines must to be documented. Remine for tasks and especially the Wiki works well.
  • Mission critical machines have a sibling.
  • System changes on production machines must be applied one at a time, with a few days in between.
  • Most apps run on VMs. It's not like git but it's as good as it gets for the time being.


End-of-file Marker


Back to the initial task for looping a file to the end: when reading mission critical changeable files by Java we use an end of file marker "eof". If the file ends with that line then we can be sure the file was read completely. If not, then it could be broken, and the program throws an exception.





Sunday, April 5, 2015

Is that Googlebot user agent really from Google?

How do you know if the client identifying as Googlebot is actually from Google? And Bingbot? And all the others? On Github there's a newly published library, written in Java with Maven that performs the checks for you. And why would you care?

There are different kinds of visitors to your website. Some webmasters don't mind any of the traffic, don't discriminate, and let them all pass. Others block or at least limit or throttle some of the requests. But why?
In the end it all boils down to the visitors you want, and those that you don't want. Those that bring you potential benefit, and those that only cost you traffic, and possibly worse duplicate your site content, collect email addresses, or align your site with bad reputation.

There are 2 kinds of users visiting your site. Humans, and robots. This article (and the software) is about the bots only.

The robots can be classified as follows:
- search engine crawlers: googlebot, bingbot, baiduspider, ...
- other purpose crawlers: archive.org_bot, ahrefsbot, ...
- custom crawlers: data scrapers, email harvesters, offline readers, ...

Search engines

Search engines operate crawlers to feed their indexes. You generally want to let these through because they drive visitors to your site. The more pages they index, the better for you.

However, there may be exceptions. Dan Birken has an interesting read on this: Baidu is a greedy spider that possibly gives you nothing in return, depending on your target audience.

And these days search engines don't only operate text search, there is also images, videos, and more. Some sites may have not interest in appearing in the Google image search, and therefore lock out the Googlebot-Image crawler.

Other purpose crawlers

The web archive is a nice thing. Some webmasters choose to not have a pubic history of their site, and block the bot.

Commons Crawl is another one that some don't mind, and others forbid, depending on their site content.

Blocking blood suckers including the ahrefsbot, a6, ADmantX and more has a larger fan base.

And finally blocking email harvesters should be on every webmaster's menu.

Custom crawlers

This is software from the shelf or even custom tailored to grab parts or your entire website. These bots are not run from a specific website or service provider. Any script kiddie can fire them up. If you operate a blog, on a shared hosting with unlimited traffic, you probably don't mind. In the best case you end up being listed on some blogging directory. In a worse case parts of your content appear remixed on a seo index spammer's fake portal, alined with links to sites of bad reputation.

The problem: impersonators

As long as all visitors identify with the correct and honest user-agent they are, the task is simple. You define in your /robots.txt which crawlers are allowed. Task accomplished. But as we all know (that's where you nod), the user-agent is just and http header field that anyone can modify. Even and especially from PHP. Pun intended.

Some masquerade as a common web browser, eg Chrome or Firefox. It's a bit more tricky to identify these, but there are ways.

The well known spider trap trick can help: a link in the html to /dont-go-here that humans can't see thanks to CSS, and that nice bots won't follow because you forbid the area in the robots.txt. Then, if someone hits that page, he's either a vampire bot, or a curious web developer who views the source, or a blind person agnostic to CSS.

Other ideas are limiting the amount of web traffic per user (ip/network, session). Some sites implement a Captcha after too many requests per time.

Other custom crawlers impersonate as one of the well known spiders; Googlebot. Chances are there are no request limits in place. After all who wants to risk being deleted from the Google index.

The solution: spider identification

That's where this software library comes in. The users that identify as one of the top search sites need to be verified.

For a long time the way of identification used to be by maintaining ip lists. Good news! For many years the major players have been supporting and advocating a zero-maintenance system. It works by validating the ip with a reverse and forward DNS lookup.

For those search engines and sites that don't support this method, identification still needs ip lists.
History has shown that this way is problematic. The ip ranges change, the lists always lag behind. Search for lists on Google and you'll find totally outdated lists in the tops. Lists that still mention msnbot (now bingbot) and Yahoo! Slurp (now defunct).

A library, used as a dependency by many developers, hosted on Github where it's easy to contribute, hopefully solves that problem.

Actions taken on spoofers

What action you take if verification returns negative is up to you. Block the user, throttle the connection, display a Captcha, alert the webmaster.

You can run the verification check live as the request comes in. There's an initial price you pay on the first request for the DNS lookups. Or you can run a watchdog on your webserver's logfiles and take action a bit delayed.

Call for action

Head over to https://github.com/optimaize/webcrawler-verifier and replace your custom Googlebot verification hack, using outdated ip lists, with this dependency.

Thursday, January 29, 2015

Manual PHP Installation on Windows: for Masochists only?


Yesterday I needed to quickly apply some changes to a PHP library hosted on GitHub. I haven't used PHP since I switched to a new PC, so there's some setup to do. But for me as a former PHP developer it won't take too long, will it?

This article won't teach you how to install PHP on Windows. It illustrates my experience, the troubles I went through, and suggests improvements to the process.


The requirements: PHP and PHPUnit


Because I should test my code before committing, I need a runtime environment. IntelliJ IDEA (the big brother of PhpStorm) is here already. What's left is PHP and PHPUnit. The library to lay my hands on doesn't require web. No web server, just scripting.


Step 1:  Install PHP

 

Choosing the right version

Visiting php.net, I learn that the latest releases are 5.4.37, 5.6.5 and 5.5.21. Hrm. What to choose? The downloads page http://php.net/downloads.php tells me that the current stable is 5.6.5, so probably that. Just to be sure, I'd like to know a bit more about the compatibility and feature sets of these versions. On the PHP page, including the documentation, I don't find that kind of data. Wikipedia is my friend: http://en.wikipedia.org/wiki/PHP#Release_history so the 5.6 branch it is.

There are more choices to be done on http://windows.php.net/download#php-5.6


32 or 64 bit? Thread safe or not? Zip or Debug Pack? Or download source code?
Ah, there's a helping hand on the left "Which version do I choose?"
Since there isn't an easy answer, I go with the rule-out logic to limit the choices.
  1. IIS? no.
  2. Apache? no.
  3. VC9 and VC11? There's only VC11 for PHP 5.6.5, so I'll need to install that.
  4. Regarding TS vs NTS since I want CLI I'll go with NTS.
  5. x86 vs x64, don't want experimental, so x86.
I guess we have a winner: php-5.6.5-nts-Win32-VC11-x86.zip

Installing it

After extracting the archive, the novice user might be looking for a setup or install exe. I also checked quickly because years passed by since I had last installed this thing. But no luck. So the good old install.txt file needs to be consulted. And by old I mean old. It seems that this file is an accumulated pile of information that grew in all directions. It's almost 2000 lines long. It explains how to handle PHP 4 throughout the file in 26 places. PHP 4 support ended in 2008, 7 years ago. It's a lot to read.

Now at this point I wish there was an installer. There are installers. I've used them in the past. But I don't want web server integration. Just a simple scripting environment. That's what PHP is, according to the first sentence of the front page:

PHP is a popular general-purpose scripting language that is especially suited to web development.

And the install instructions clearly advice me not to use an installer:

Warning

There are several all-in-one installers over the Internet, but none of
those are endorsed by PHP.net, as we believe that the manual
installation is the best choice to have your system secure and
optimised.

Secure and optimized... I need a development environment.  

After some ini file renaming and environment variable modification and installing VC11 redistributable (2x, took the wrong version on the first attempt) the PHP command would finally respond.


Step 2: Installing PHPUnit


There are instructions at https://www.jetbrains.com/idea/help/enabling-phpunit-support.html and at https://phpunit.de/manual/current/en/installation.html and as always there isn't just one way to Rome.

I was successful with the PEAR way in the past, and JetBrains still lists that as the first option. But as of December 2014, PHPUnit stopped supporting it. Dead end.

After some trial and error with the Composer I've downloaded the latest phpunit.phar and told IntelliJ to use it. But tests wouldn't run:
PHP Fatal error: Class IDE_PHPUnit_Framework_TestListener contains 1 abstract method and must therefore be declared abstract or implement the remaining methods
According to http://stackoverflow.com/questions/22799619/intellij-idea-wont-run-phpunit-4-0-tests/22799620 I can patch my IntelliJ (ugh!) then I rather try an older PHPUnit as written here http://stackoverflow.com/questions/22531584/how-to-setup-phpunit-for-intellij-idea-13-1-on-ubuntu-13-10

Finding that was a bit of a challenge too, since https://phpunit.de/ has no link to older downloads. Here they are: https://phar.phpunit.de/
 



And finally the library's unit tests would execute in my IDE.

 

Conclusions



Such installation and configuration procedures are the reason why I hesitate each time before finally replacing my PC, or installing a newer OS. 

All in all it took me, a former PHP developer, 3 hours. Too many brain cycles were burned, too much frustration hit the desk. That was more than a flow stopper - it marked the end of the day.

I'm sure that experienced PHP people can't relate. They know exactly how it works, and are done setting up a new machine in no time. And that's the reason for my writing... try to put yourself into the shoes of a noob.

Does it have to be that complicated? Other scripting languages think not.

PHP used to be the easy alternative. Looking at this mess of installation, and putting it in relation to what one expects these days and what the alternatives offer, I have some serious doubts that some beginners even manage to get started.

But then again maybe it's good the way it is. (If you know what I mean. That's a sarcastic comment in case you don't.)

 

Suggestions



For manual installation, I believe that the install instructions should be cleaned up completely. A new, shortish, simple file, without the burden of all historical facts of how things once were. But that's just applying patches and fixes.

It's time for the PHP Group to provide or promote an official Windows installer. That's what would really fix the situation.



Wednesday, December 10, 2014

Graphical Visualizations in JavaDoc

Having code documentation is crucial for a software. It helps the developers to understand the functions and APIs. Code that is well understood is easier to improve, and therefore more likely to live on, instead of being thrown away in favor of a complete rewrite. It serves as the specification in case the code contradicts the text, and helps identifying possible bugs.

Sometimes explaining a situation only by words is difficult and results in lengthy, hard to comprehend blathering. What options for visualizations are there?


Option 1: Images

Although most Javadoc is plain text, it permits the use of HTML formatting. Images can be embedded. Put them into a doc-files folder and link it:

/**
 * Visualization:
 * <p><img src="doc-files/my-uml-diagram.png"/></p>
 */


There is some cost involved in keeping this up to date; the next programmer touching the code will need to have access to the originals, and have the graph software (UML, image editor, ...) installed. This takes time.

Also, chances are your IDE will not show the image. 

Javadoc history and today's use

When Javadoc was created in the early 90s, source code comments were nothing new. The pioneering feature was the auto-generation of a technical documentation directly from the source code. Prior to such documentation generators, the technical documentation used to be (badly) maintained separate from the source code. Ouch. HTML was brand new, and the build putting those files on a web server was cutting edge technology.

Now, Javadoc is over 20 years old. At the time it was created, virtually all business software was closed source. Today, all software libraries we are using at the company are open source.
Software projects reorganized from lengthy release cycles to release-early-release-often. And software is written in many smaller, modular pieces instead of monolithic systems.

Fact is, I rarely look at Javadoc in the web browser. Most often I see it directly in the IDE. Whether it's one of our own ~40 software projects, or a library checked out from GitHub or from Maven central, or code from the JDK itself. It's just so much better to have a complete view; up to date source code and Javadoc.


Option 2: ASCII Art

An alternative to images is the good old ascii art. I personally use and recommend asciidraw.

Here's an example:

/**
 * <pre>
 *
 *                 +---------gn-hypo-+                +--------------gn-+
 *                 |                 | +------------> |                 |
 *                 | Osquitar        |                | Óscar           |
 *                 |                 |       +------> |                 |
 *                 +-----------------+       |        +-----------------+
 *                                           |
 *                                           |
 *                                           |
 *                 +---------gn-hypo-+       |
 *                 |                 | +-----+
 *                 | Osqui           |                +--------------gn-+
 *                 |                 | +-------+      |                 |
 *                 +-----------------+         +----> | Osgar           |
 *                                                    |                 |
 *                                                    +-----------------+
 *
 * </pre>
 */


The advantages:
  1. It's quick, create a drawing within minutes.
  2. It's super simple, no special knowledge required.
  3. No "file originals" nor special software required.
  4. Every developer can maintain it.
  5. Any IDE or editor is guaranteed to display it in line.
  6. Because of the technical limitations, one is forced to keep it simple and focus on the main components.

Conclusion

You may want to consider ascii visualizations for your Javadoc the next time you're struggling with explaining a circumstance. It works great for us.



Monday, September 22, 2014

Java SLF4J: Dynamic Log Level

In short

Problem: SLF4J doesn't support to set the log level at runtime.
Solution: Use Lidalia Extensions to SLF4J

The whole story

My Java projects that aren't older than 2008 all use the same logging setup: SLF4J for the interface, Logback for the implementation. It's a widely accepted standard with benefits.

Today I had the need to dynamically set the log level at run time. Just like in Log4j logger.log(priority, message). But there's no such or similar method in the API. With one eyebrow raised I started typing into Google:


Ah, good, apparently I'm not the only one. There must be a simple solution. Stackoverflow already has the matching question. But this time the answer is a surprise:
"There is no way to do this with slf4j."
That's not a statement a programmer comes across often. I shared my reaction with this popular user comment:
"WTF? SLF4J is heralded as the future of java's logging, and this simply use case is not supported?"
The SLF4J API team has arguments not to include these methods.

I wasn't too much interested in the architectural design decisions. I just needed a quick and clean solution. And here is a nice one:

Lidalia Extensions to SLF4J "An extension to SLF4J allowing logging at a level determined at run, rather than compile, time via a Level enum."

It's small, available from Maven central, adds exactly the missing functionality, and works.

        <dependency>
            <groupId>uk.org.lidalia</groupId>
            <artifactId>lidalia-slf4j-ext</artifactId>
            <version>1.0.0</version>
        </dependency>


Example:

import uk.org.lidalia.slf4jext.Level;
import uk.org.lidalia.slf4jext.Logger;
import uk.org.lidalia.slf4jext.LoggerFactory;

public class Example {

    private static final Logger logger = LoggerFactory.getLogger(Example.class);

    public static void main(String[] args) {
        Level level = Level.valueOf(args[0]);
        logger.log(level, "Logged at a level configured at runtime");
    }
}





Tuesday, May 20, 2014

If your project is still hosted on SourceForge, I automatically assume it's dead.


SourceForge was the first free project hosting platform for open source software. It was immensely popular. It was the hub during the early open source years on the web - it was the GitHub of today.

Because of its historical popularity, there is still a large amount of projects
hosted there. Some are popular, some are dead - just like in any source code
repository. Many developers moved their code to more modern platforms since then.
At first to Google Code, now to GitHub.


Here are 3 reasons why you should move away from SourceForge.

1) SourceForge feels dead. Many active projects moved away. 

When I look for a software, I don't mind too much where it's hosted. Maturity,
stability, community size and active development are the factors I look for.

I don't advocate moving your whole project with source code and bugtracker every
couple of years to the next hip kid on the block.

But the fact that so many moved on means that what's left is probably abandoned. When I search for a software, and a GitHub and a SourceForge project comes up, I tend to click the GitHub first. Also, because of my search history, I probably get to see the GitHub links first. I'm certainly not alone here.

2)  Developers have GitHub accounts

Open Source needs contributors and users. And the easiest way for developers to pitch in or file a bug report is when they already know the system and have an account. If it's complicated, I pass.

3) Scam ads: this stinks!

Here's the reason for my blog post.

Once again I was downloading a (totally outdated and abandoned, but familiar and working) software that is still on SourceForge, and then I got to see this ad:



It leads to a suspicious looking website, with Microsoft Partner logo and Windows logo, offering Windows driver updates. The kind my mother would click, thinking it's from Microsoft.


And here's the profile on web of trust: https://www.mywot.com/en/scorecard/driverupdate.net


The site even manages to get a top rating on WoT by spamming good ratings, even though all the important comments say it's a scam. WoT, please fix.
"they say I need over a dozen drivers updated. Confirming with the manufacturer, I am up to date. ..." --pmheart6
"why do all the positive comments use poor english with typical chinese mistakes?? A dishonest sales trap. It's difficult to uninstall and must be done manually." --KITRVL



SF changed ownership in 2012, and as the Wikipedia article writes "More recently additional revenue generation schemes, such as bundleware models have been trialled, with the goal of further improving sourceforge's revenue."

This looks to me like beating a dead horse. There's only downhill from here.

Please spare other people from such frustration. Even if the platform was good once, now is the time to move the code away. To BitBucket or Google Code or GitHub or wherever. Thanks.

Tuesday, April 29, 2014

Java: .equals() or == on enum values?

Both do pretty much the same thing: checking whether 2 enum values are the same. And if you search your code base, you probably find both kinds. There are some subtle differences. What are they - and which syntax should I use?

Same same - but different

Enum values are guaranteed singletons - and thus the == reference comparison is safe. The enum itself just delegates to Object.equals(), which does exactly that.



So if they execute the same code, why should I bother?

Learning the hard way - actual bug

I produced a bug which broke the build. Luckily we did have a unit test in place that caught it. But still I would have seen it earlier, had I used the "correct" syntax.

So which one is it?

And the winner is: ==

It provides compile-time safety. While MyColor.BLACK.equals(acceptsAnything)gives a warning at most, the check MyColor.BLACK == something won't even compile if something is not an instance of MyColor.



This makes it refactoring-safe. If you go - like me - and change the type of something later on, you'll notice instantly.

Another difference is the support for null. The check with == allows null on both sides. It doesn't throw, however, it may also "hide" a null value where there shouldn't be one.
The check with equals() allows null only on the right side, and throws a NullPointerException if the left side is null. That may be desired sometimes. Anyway, I vote for not letting null slip into this comparison at all. It should be checked/handled before, instead of being smart. (See my article about null handling.)
So the comparison with == is "safer" also at run-time in that it never throws a NPE, but is possibly hiding a bug when the left side should never have been null.

Then there's the argument of performance. It's irrelevant. I'm not going there.

And the visual aspect. Which one looks nicer?
When I can have compile-time safety, I don't care about the looks. I was used to .equals() simply because that's how Strings are compared. But in retrospect that's a pretty lame explanation.
On this StackOverflow question, which is exactly about the topic,  Kevin Bourrillion from Guava commented that "== may appear incorrect to the reader until he looks at the types" and concludes that "In that sense, it's less distracting to read ".equals()". Au contraire! When I see CONSTANTS around == I instantly know they're either enums, or primitives, of matching types. If not, the code is either red because it doesn't compile, or with a yellow warning from my intelligent IDE saying that .equals() should be used. For example Strings or primitive wrappers like Integer.

After that incident at the company we've decided to go with reference equality comparison ==.

Tuesday, April 22, 2014

Java: Evolution of Handling Null References

Using null in programming is playing with fire. It's powerful and sometimes the right choice, but the infamous NullPointerException can sneak in quickly. That's why I advocate for every software project to include a section about handling null in the coding guidelines.

The first time I've seen a Java exception was in the 90s in the web browser's status bar when surfing the net: a NPE caused by an applet. I had no idea what it meant and what I had done wrong. Today I know that the programmer let a null reference slip in where it wasn't expected to happen.

When looking at open source software written in Java I often come across code where null references are used but not documented. Sometimes preconditions are in place. When stepping deeper it gets really tricky to figure out which variable is now allowed to be null and which is not. For the author it was obvious at the time of writing the code... but software becomes better when written and looked over by many, and thus it is important to make it clear for everyone, everywhere.

Software is constantly shipped with NPEs detected later on. Redeployments and bugfix releases are expensive - and it's a pity because this kind of bug could be eliminated almost completely.


Here's my personal evolution of dealing with null in Java.

Level 1: No Plan

Using it wherever, no information about it in the Javadoc.
Fixing NPEs as they occur, seeing no problem in that.
public class Foo {
    
    public String getText() {
        return null;
    }
    
}

Level 2: Document Null - Sometimes

Realizing that NPEs are a problem that could and should be avoided. Detecting them late is expensive, especially after deployment.
Starting to document variables, arguments and return values that can be null... incomplete.
public class Foo {
    
    /**
     * @return the text, or null
     */
    public String getText() {
        return null;
    }
    
}

Level 3: Add "Null" to Method Names

Realizing it's still a problem. Having Javadoc is nice, but useless when not read.
Starting to name methods with null such as getFooOrNull() to force it to be seen.
public class Foo {
    
    /**
     * @return the text, or null
     */
    public String getTextOrNull() {
        return null;
    }
    
}

Level 4: Code Annotations

Using Jetbrains' null annotations: Annotating all variables with @Nullable and @NotNull. This is a huge step forward. The crippled method name pattern 'OrNull' is obsolete. The code is automatically documented.
public class Foo {
    
    @Nullable
    public String getText() {
        return null;
    }
    
}
And the best part: the IDE checks the code and warns on a mistake.


Using these annotations strictly I don't remember causing a single NPE in years. The drawback is more typing, more text on the screen. But then we use static typing, and not defining null is just incomplete.

Level 5: Guava's Optional

The concept: Instead of using the null reference directly, wrap it in another object that permits null. The getter methods on it myOptional.get(), myOptional.orNull() and myOptional.or(alternative) force the user to think about what to do when it's null. The API becomes extremely clean.

If you don't know about it yet, read the Guava page about using and avoiding null.

It took me a couple of days to get used to this. And I did produce a handful of bugs initially because I've used Optional.of() instead of Optional.fromNullable() by mistake.

Although step 4 with annotations already got rid of NPEs and improved the code confidence, this was another big step forward. The API of the Optional class is clean, and the API of code using it is consistent and clear. A developer only has to learn the concept once, and because it's from Guava, sooner or later everyone will know it.

The Guava team uses @Nullable annotations for the few places where null is permitted. Anywhere else there are no annotations.

Level 6: Java 8 has Optional built in

Oracle has a nice article on it. The API is slightly different, and I have not made the transition to Java 8 yet.



Finger Pointing

The bugtracker for the Glassfish software currently has 1309 matches for NullPointerException. Wow. (Go to the issue navigator, select the 11 projects on the left starting with "glassfish", and add the query NullPointerException, hit enter. The software is session-based, can't paste a link...)

The Grizzly project has 69.

This is a codebase that's still on level 1 regarding null. Exactly 2 years ago I had recommended to improve the Javadoc and start using null annotations. The task was accepted, got priority "Major" assigned, but is still open and things haven't changed.

I had also mentioned in that task the undocumented method int getQueueLimit() which would return the magical -1 for 'no limit'. Nowadays - being a level 5 guru ;-) - I'd instantly turn this into a self-documenting Optional. This forces users to think about the exceptional case. No horrible bugs from computations with -1 can occur. Users of it would then just do something like getQueueLimit().or(Integer.MAX_INT) or whatever suits their case - short, clear and safe.



My Recommendations

1) Avoid Null when possible.

Consider the null object pattern.

Return empty collections, not null.

Throw UnsupportedOperationException() instead of returning null.
Don't do this:
public Object foo() {
     return null; //todo
}
It is done sometimes as a quick way to satisfy the compiler. It's the no-impl pattern.

Do this instead:
public Object foo() {
     throw new UnsupportedOperationException(); //todo
}
If you return null then it (null) can go a long way until it throws a nasty NullPointerException somewhere. By throwing UOE directly the cause is clear in the stack trace.

2) Use Guava's Optional.

Wherever you would use null, use Optional. 
Except in very low level code.

3) Use Jetbrains's Null Annotations.


If allowing null then use the @Nullable annotation.This is a must. It automatically tells the user and the IDE that null is possible/permitted.

Use the @NotNull annotation everywhere else. This is optional. The Guava people do not, I do.

To get the annotations using Maven:

        <dependency>
            <groupId>com.intellij</groupId>
            <artifactId>annotations</artifactId>
            <version>12.0</version>
        </dependency>


Tuesday, March 11, 2014

Software Versioning and Bugfixes

There are many ways how to give version identifiers to software builds.

The goals of software versioning, as I understand them:
  1. Give each build (release, snapshot) a unique identifier.
  2. Apparent order: it must be obvious which is newer when comparing 2.
  3. Apparent changes: contains breaking changes, new functionality, or just bug fixes.
  4. Possibility to release updates later on that "fit in between".
I like the increasingly popular semantic versioning. It accommodates all these goals.

Apparent Order

1.9.0 -> 1.10.0 -> 1.11.0
This is important for me as a user, it lets me select the newest version quickly.

Apparent Changes

Going from 1.9.0 to 2.0.0 may mean that some of my use cases need to be adjusted, some things don't work the same anymore. I might prefer to stay on the "1" branch.

Going from 1.9.0, 1.10.0 contains more functionality, it seems pretty safe. Experience taught me that new features bring new bugs - even if I don't make use of the new stuff.

Going from 1.9.0 to 1.9.1 means there are just bug fixes. It is almost a safe bet to upgrade this. A quick look over the change-list helps me to decide where I might be affected. The risk is very low that this introduces new bugs.

Release Updates

When after 1.9 the version 1.10 has been released, and then bugs are found in 1.9.0 that should be fixed, not everyone will be happy with a 1.11 that now contains new functionality and bug fixes. Hence the 3 numbers.

H2 Database thinks otherwise

Unfortunately this software library that I use, like and value follows a different version system. Just like Google Chrome, they have been publishing new versions with what is effectively a single continuous version number.

For a recent history see http://www.h2database.com/html/changelog.html
And longer: http://mvnrepository.com/artifact/com.h2database/h2

There are no bug fix releases. Each new version brings new features, improvements, bug fixes, and new bugs, which becomes clear when you follow the issues. But to learn in which version they were introduced, and so on, you need to dig and read the bug tracker.

For a web browser, this release cycle OK. Give me the latest, I can cope with some level of annoyances and crashes now and then.

The database however is the back bone of many applications. It is the piece of software that one relies on, that is the worst to break. And as such, I really wish it would maintain two code branches. One for latest developments, and one that lags behind where just fixes are applied.

Whenever I consider updating this library in my code, I hesitate. Is it really necessary? Most times I answer no and move on. Updating means reading all recent changes, and running tests.

I only use this database for read-only scenarios, not as a live database with active changes.

My suggestion to H2 and in general

  1. Move code from Google Code to GitHub
  2. Change to semantic versioning
  3. Create a branch that lags behind, and let contributors do the work of maintaining it (applying just the fixes)

Monday, December 2, 2013

IntelliJ IDEA: add Project Files to Version Control?

Do the contents of the .idea project folder and the .iml module files belong into version control?

Yes, says the IntelliJ team:
Here is what you need to share:
  • All the files under .idea directory in the project root except the workspace.xml and tasks.xml files which store user specific settings
  • All the .iml module files that can be located in different module directories

No for .iml files say Maven users who are annoyed by ever-changing iml files, see these 2 older but still open bugs:

What are the drawbacks if I don't share these files? None, from my experience. When doing a fresh checkout of a project, IDEA asks whether I'd like to create a new project. Yes please, and all files are generated correctly. 

The only annoyance we have had at the company from time to time is when someone accidentally adds an .iml file to the VCS. There's a feature in IDEA to stop it from asking:

Settings -> Version Control -> Ignored Files

This setting has to be applied in every project separately. And thus it's not very helpful in this case. Because every new project I check out, IDEA asks me whether I want to add the .iml file to version control (one by one), and I can just tell it to stop asking:


Conclusion

For the time being we keep IDEA files out of the VCS. Our projects are usually multi-module maven, with lots of iml files. Your mileage may vary.



Friday, November 29, 2013

Tabs and Saved Session Folders for PuTTY

For many years I've been using the PuTTY network terminal to connect to remote servers when doing sysadmin tasks. And when I do, my monitors quickly fill up with those black windows. Remember web browsing before tabs? Like that. I totally lose track.

The other thing that's annoying is that it does not let me categorize my saved sessions. All are in one alphabetical list. That might work when you just have a handful, but my list grew to dozens. Meh.


So I checked again today on the official site for an update: 



With such an user interface of the site, it would be too much to expect gimmicks in the program. Yes there was an update. Security fixes.

So I googled for what else there is. And apparently there are plenty of wrappers around PuTTY that bring exactly what I want: Tabs and Session Folders. I don't know how they could hide from me for so long - maybe because neither the official site nor the Wikipedia page mention any.

I've decided to go with this open source and actively maintained one: superputty


Conclusion

Same putty, more gui. Awesome.

Friday, November 8, 2013

Java: Should I assert or throw AssertionError?

Assertions were introduced to the Java language in JDK 1.4, that was in 2002. It brought the assert statement
assert x != null : "x was null!";
and the throwable error class
if (x==null) throw new AssertionError("x was null!");
Which one should I use in my code?



Well then, what's the difference?

The assert statement

Unless assertions are actively turned on when running the program (-ea), they are not executed. They are still in the bytecode, but they don't run.
  • Bugs go unnoticed.
    These bugs should have been detected already in unit tests and the testing phase before deployment when running with assertions on. But experience taught us that some slip through.
  • No performance loss.
    There are a few rare scenarios where it matters.
  • Side effects are possible.
    Badly programmed assertions can cause nasty side effects. Because sometimes the assert code is executed, and other times it's not, the program may behave slightly different.
Using the assert statement gives control to the runtime config whether they run or not. The choice is left to the person who runs the program. The choice can be made from run to run, no recompile is needed.

The AssertionError

The asserting code is always executed. Changing your mind about execution means a recompile and re-deployment.
  • Bugs always show up.
    Even in production they abort the program.
  • Performance loss.
    In most cases it's irrelevant.
  • Guaranteed to have no side effects.
    The executed code is always the same.

 

Vital or nice to have?

Programs were written in Java long before assertions were available. Some came up with their own self-made assertions logic - which was not as clean and short as the real thing. And most probably didn't.
Assertions are not allowed to alter the program logic, were not always available, and are not mandatory. One can very well run the same program with them turned off. Conclusion: nice to have.

However, it is not guaranteed that a specific bug ever shows up without assertions running. Let's take a flight simulator for example. Assume a computation bug that only occurs in a rarely executed code path. The bug can cause the airplane to fly at slightly reduced speed, and no one ever notices. Or it can be a number overflow, causing the plane to fly backwards. That will surely be seen each time.

Would assertions help in production code?

It depends on the domain. Assertion means abort. Is that what you'd want, to prevent the worst? Or would you rather try to go on, hoping that it's a minor, neglegtible bug?
  1. End-of-day accounting program: rather abort, alert the technicians, they fix it, and go on. No damage done, and bug fixed.
  2. Real-time program where thousands of employees depend on it, and an abort means a big loss of work time. No abort. Hope that the bug eventually shows up as a side effect, and it can be traced and fixed.
  3. Program where an abort is the worst case scenario, like a flight simulator: go on.

Conclusions

assert, not AssertionError

When writing library code, assert is the right choice. It gives the power to the user whether assertions should be on or off, whether he favors detection + abort, or to go unnoticed.

When writing an application, use assert as well. If you want to enforce assertions, you can still hack in this piece of code, and if you change your mind later, you don't have to go through all assertions and replace them:
static {
    boolean assertsEnabled = false;
    assert assertsEnabled = true; // Intentional side effect
    if (!assertsEnabled)
        throw new RuntimeException("Asserts must be enabled (-ea)!");
} 
Also, with assert, they can be enabled on a per-package level
java -ea:com.example... MyApp

AssertionError to satisfy compilation

Sometimes, when reaching code that should never be reached, an AssertionError is thrown to make the code compile. Example:
switch (TrafficLight) {
    case GREEN:
        return doGreen();
    case ORANGE:
        return doOrange();
    case RED:
        return doRed();
    default: 
        //Ugh! I gotta do something, but have no clue what to do!
        //Let's abort the app and turn the semaphore offline.
        throw new AssertionError("Dead code reached");
} 
An assert statement would not compile when a return is required.

However, in some cases, it might be more suited to throw an exception instead of an error, for example an UnsupportedOperationException. In the above case an exception barrier could catch that, then turn all semaphores at the intersection to blinking orange for a minute, and then restart the green interval as it would from a clean program start and continue normal operation. That would have several advantages:
  • be cheaper than sending a technician
  • the semaphore is off for short time only
  • while it's off it's blinking orange, rather than being totally off, that seems more secure
The problem would be detected in both cases. With an AssertionError it's in the output for free. And with UnsupportedOperationException it's the task of the one catching the exception to log it.

So again, the choice is between a hard abort, or giving the program the chance to recover and continue if a higher level decides to do so.


What I'm missing: detection and logging!

In the production scenarios 2 and 3 from above I'd want a 3rd way of handling assertions, which is not offered by Java's assertion feature.

Java has 2 strategies:
  1. Don't even check, hence no abort
  2. Check, and conditionally abort or continue
And I'm missing the 3rd one with the behavior:
  1. Run assertion code to detect bugs and don't abort, but instead log.
    It's a clear bug, it's detected, so log it. There's no cheaper way to detect it than right here, in the assert code that was written already.
 

Configure your IDE to run with assertions

It strikes me that the default runconfig in the most common IDEs does not have assertions enabled. It actually happened to me that I had spent way too much time chasing a bug, when the assertion would have shown it instantly - but they weren't on.

In IntelliJ IDEA: You need to modify the Defaults for all kinds that you're using: Application, JUnit, TestNG, ... and changing the defaults does not change the runconfigs you've created earlier. And you need to do this in every project, separately. (Please add a comment if you know how to set this once and for all!)


For Eclipse there's an explanation here http://stackoverflow.com/questions/5509082/eclipse-enable-assertions also see the 2nd answer about JUnit tests.

Monday, November 4, 2013

Java toString(): the Program Logic vs. Debug Dilemma

I'll start with my real world problem of how to implement toString() for my new class, followed by an analysis of how Java uses toString(), and finishing with my conclusions.


Real world problem: my class

Here's the (stripped down for simplicity) version of my new class:

1:  /**  
2:   * Represents a possibly multi-byte character and provides information about it.  
3:   * <p>A UnicodeCharacter is the equivalent of a Java "code point".</p>  
4:   */  
5:  public final class UnicodeCharacter {  
6:    private final int codepoint;  
7:    public UnicodeCharacter(int codepoint) {  
8:      this.codepoint = codepoint;  
9:    }  
10:    public int getCodepoint() {  
11:      return codepoint;  
12:    }  
13:    /**  
14:     * @return Array with usually 1 character, 2 characters for multi-byte.  
15:     */  
16:    public char[] getChars() {  
17:      return UCharacter.toChars(codepoint);  
18:    }  
19:    /**  
20:     * @return Tells if the {@link #getCodepoint codepoint} is for a surrogate pair. 
21:     */  
22:    public boolean isMultiByte() {  
23:      return getChars().length>1;  
24:    }  
25:    /**  
26:     * @return For example the category UCharacterCategory.UPPERCASE_LETTER  
27:     */  
28:    public Byte getCategory() {  
29:      return (byte) UCharacter.getType(codepoint);  
30:    }  
31:    /**  
32:     * See {UCharacter#getName}  
33:     */  
34:    public String getName() {  
35:      return UCharacter.getName(codepoint);  
36:    }  
37:  }  


Now for toString(), what should it be?

Suggestion 1: for the human user, debug output

1:    public String toString() {  
2:      return "UnicodeCharacter{" +  
3:          "cp=" + codepoint +  
4:          " ,string='" + getString() +  
5:          " ,name=" + getName() +  
6:          '}';  
7:    }  
Example output:
  • UnicodeCharacter{cp=65,string='A',name=LATIN CAPITAL LETTER A}
  • UnicodeCharacter{cp=1040,string='А',name=CYRILLIC CAPITAL LETTER A}
  • UnicodeCharacter{cp=32,string=' ',name=SPACE} 

Suggestion 2: for the machine, program logic, concatenateable

1:    public String toString() {  
2:      return UTF16.valueOf(codepoint);  
3:    }  

Example output for the same 3 (yes 3) characters:
  • A
  • А
  •  

Both have their advantages and drawbacks. It certainly needs a concatenateable method, but that can be named "getString()". The debug method is nice to have - you see, just by looking at the characters you can't tell whether it's the Latin or Cyrillic A, or what kind of whitespace it is.

Unfortunately, in this case, expanding the object in the debugger doesn't help, because solely the code point is a property of the object. The other information (character and name) are computed:




Analysis of Java's toString()

Here's what Java has to say about Object.toString():
Returns a string representation of the object. In general, the toString method returns a string that "textually represents" this object. The result should be a concise but informative representation that is easy for a person to read.
Both my suggestions follow the specification. The one with debug info is easier to read for a person. But I'm not sold yet.

Who calls toString()?

  • Java itself:
    When doing string concatenation: String s = myString + myCar;
    Is the same as doing String s = myString.toString().concat(myCar.toString());
  • JDK methods:
    String.valueOf(Object obj), and thus every method that uses this such as PrintStream.print(Object o).
    Arrays.toString(Object[] a) for every object in the array a.
    StringBuilder and StringBuffer: the toString() method is used as the build method.
  • Logging:
    System.out.println().
    Your favorite logging framework.
  • Debugging:
    Your favorite IDE in the debugger.

Hrm. So there are mainly 2 uses:
  1. String representation: toString() returns the object's value "as string" as close as possible.
    It is absolutely required to override toString(), and to do it in this way.
  2. Debug information: the object's values for the human.
    For example IntelliJ IDEA's default toString() template generates this kind.
    It's just nice to have.
Sometimes, as an additional benefit, the object provides a constructor accepting that string as a parameter to re-create it. Example: Integer.toString() and new Integer(String).

How does Java in the JDK define toString() in their classes?

For some simple value classes there's not much choice. Integer for example: returning "-43" makes sense.

Character could return more than just the character as string, but it does not. String could tell the length and cut it if it's too much, but it does not. StringBuilder and StringBuffer could report the appended chunks separately, and tell how many, and cut, but they don't. If they would, the classes would need a separate method for string concatenation, and concatenation with + would not work anymore. Now here's an observation: They all implement CharSequence, which was added in JDK 1.4, and it overrides the toString() method signature just to say something about it:
Returns a string containing the characters in this sequence in the same order as this sequence. The length of the string will be the length of this sequence.
So that's why.

Conclusions

My class UnicodeCharacter is a wrapper around a unicode codepoint just like Character is a wrapper around the char primitive. It's a character supporting those that don't fit into a char. And as such it really should implement the CharSequence interface. Then the decision is made: toString() must be suggestion 2, only returning the character's string value.

In some rare cases it would be nice to have 2 different methods: one for the string value (toString()) and one for the debug info (toDebug() or toDebugString()). The method could be defined in Object, with a default implementation: calling toString().


Wednesday, October 30, 2013

Password-protect Play2 Framework Webapp

There are numerous extensions to bring authentication and access control to your Play based website. For example the Deadbolt 2 authorization system lets you define access rights per controller and method. Or you can roll your own, as this step by step guide shows.

What to do if you just want basic http authentication for the whole site? As of today there's no such thing built in. (If this changes, let me know...)

If you're serving your Play pages through Apache as reverse proxy, you're lucky.

Going through a reverse proxy is a good idea anyway:
  • You get the option for load balancing and failover: run multiple instances.
  •  Run multiple Play sites on the same machine, on whatever port number, and expose them all on port 80 to the outside.
 Your apache website definition looks simething like this:
<VirtualHost *:80>
  ServerName my-play-app.example.com
  ProxyPreserveHost on
  <Location />
    ProxyPass http://192.168.56.100:9000/ connectiontimeout=9999999 timeout=9999999
    ProxyPassReverse http://192.168.56.100:9000/

    AuthType Basic
    AuthName "whatever" 
    AuthUserFile /etc/apache2/sites-available/.htpasswd-mysite
    Require valid-user
  </Location>
</VirtualHost>
I'm running my Play app in a virtual machine (192.168.56.100) on standard port 9000. The connection timeout is there so that long running tasks are not aborted by the proxy.

Create the password file as usual:
cd /etc/apache2/sites-available/
htpasswd -c .htpasswd-mysite newuser

And then refresh Apache, and you're done:
service apache2 reload