The Java ByteBuffer – a crash course

Posted: December 14, 2012 | Author: | Filed under: Uncategorized | Tags: , , , , , | 11 Comments

In my experience, java.nio.ByteBuffer is a source of confusion and subtle bugs when developers first encounter it, because it is not immediately obvious how to use it correctly. It took some repeated reading of the API docs and some experience to realize some subtleties, before I felt comfortable with them. This post is a short crash in how to use them correctly, to hopefully save others some trouble.

Since all of this is inferred (rather than based on explicit documentation), and based on experience, I cannot claim the information is necessarily authoritative. I welcome feedback to point out mistakes or alternative points of view. I also welcome suggestions for additional pitfalls/best practices to cover.

I do assume that the reader will be looking at the API documentation to go with this post. I am not going to be exhaustive on all the possible things you can dowith a ByteBuffer.

The ByteBuffer abstraction

Look at a ByteBuffer as providing a view into some (undefined) underlying storage of bytes. The two most common concrete types of byte buffers are those backed by byte arrays, and those backed by direct (off-heap, native) byte buffers. In both cases, the same interface can be used to read and write contents of the buffer.

Some parts of the API of a ByteBuffer is specific to some types of byte buffers. For example, a byte buffer can be read-only, restricting usage to a subset of methods. The array() method will only work for a byte buffer backed by a byte array (which can be tested with hasArray()) and should generally only be used if you know exactly what you are doing. A common mistake is to use array() to “convert” a ByteBuffer into a byte array. Not only does this only work for byte array backed buffers, but it is easily a source of bugs because depending on how the buffer was created, the beginning of the returned array may or may not correspond to the beginning of the ByteBuffer. The result tends to be a subtle bug where the behavior of code differs depending on implementation details of the byte buffer and the code that created it.

A ByteBuffer offers the ability to duplicate itself by calling duplicate(). This does not actually copy the underlying bytes, it only creates a new ByteBuffer instance pointing to the same underlying storage. A ByteBuffer representing a subset of another ByteBuffer may be created using slice().

Key differences from byte arrays

  • A ByteBuffer has value semantics with respect to hashCode()/equals() and as a result can be more conveniently used in containers.
  • A ByteBuffer has offers the ability to pass around a subset of a byte buffer as a value without copying the bytes, by instantiating a new ByteBuffer.
  • The NIO API makes extensive use of ByteBuffer:s.
  • The bytes in a ByteBuffer may potentially reside outside of the Java heap.
  • A ByteBuffer has state beyond the bytes themselves, which facilitate relative I/O operations (but with caveats, talked about below).
  • A ByteBuffer offers methods for reading and writing various primitive types like integers and longs (and can do it in different byte orders).

Key properties of a ByteBuffer

The following three properties of a ByteBuffer are critical (I’m quoting the API documentation on each):

  • A buffer’s capacity is the number of elements it contains. The capacity of a buffer is never negative and never changes.
  • A buffer’s limit is the index of the first element that should not be read or written. A buffer’s limit is never negative and is never greater than its capacity.
  • A buffer’s position is the index of the next element to be read or written. A buffer’s position is never negative and is never greater than its limit.

Here is a visualization of an example ByteBuffer which is (in this case) backed by a byte array, and the value of the ByteBuffer is the word “test” (click it to zoom):

ByteByffer_example

That ByteBuffer would be equal to (in the sense of equals()) to any other ByteBuffer whose contents in between [position,limit) is the same.

Suppose the byte buffer visualized above was bb, and we did this:

final ByteBuffer other = bb.duplicate();
other.position(bb.position() + 4);

We would now have two ByteBuffer instances both referring to the same underlying byte array, but their contents would be different (other would be empty):

ByteBuffer_after_duplicate_example

The buffer/stream duality of byte buffers

There are two ways of accessing the contents of a byte buffer – absolute and relative access. For example, suppose I have a ByteBuffer that I know contains two integers. In order to extract the integers using absolute positioning, one might do this:

int first = bb.getInt(0)
int second = bb.getInt(4)

Alternatively one can extract them using relative positioning:

int first = bb.getInt();
int second = bb.getInt();

The second option is often convenient, but at the cost of having a side-effect on the buffer (i.e., changing it). Not the contents itself, but the ByteBuffers view into that content.

In this way, ByteBuffers can behave similarly to a stream, if used as such.

Best practices and pitfalls

flip() the buffer

If you are building up a ByteBuffer by repeatedly writing into it, and then want to give it away, you must remember to flip() it. For example, here is a method that copies a byte array into a ByteBuffer, assuming default encoding (note that ByteBuffer.wrap(), which is used here, creates a ByteBuffer that wraps the specified byte array, as opposed to copy the contents of it into a new ByteBuffer):

public static ByteBuffer fromByteArray(byte[] bytes) {
    final ByteBuffer ret = ByteBuffer.wrap(new byte[bytes.length]);

    ret.put(bytes);
    ret.flip();

    return ret;
}

If we did not flip() it, the returned ByteBuffer would be empty because the position would be equal to the limit.

Don’t consume the buffer

Be careful not to “consume” a byte buffer when reading it, unless you specifically intend on doing so. For example, consider this method to convert a ByteBuffer into a String, assuming default encoding:

public static String toString(ByteBuffer bb) {
    final byte[] bytes = new byte[bb.remaining()];

    bb.duplicate().get(bytes);

    return new String(bytes);
}

Unfortunately, there is no method provided to do absolute positioning reads of a byte array (but there does exist absolute positioning reads for the primitives).

Notice the use of duplicate() when reading the bytes. If we did not, the function would have a side-effect on the input ByteBuffer. The cost of doing this is the extra allocation of a new ByteBuffer just for the purpose of the one call to get(). You could record the position of the ByteBuffer prior to the get() and restore it afterwards, but that has thread-safety issues (see next section).

It is worth noting that this only applies when you are trying to treat ByteBuffer:s as values. If you are writing code whose purpose is do side-effect on ByteBuffers, treating them more like streams, you would of course be intending to do so and this section does not apply.

Don’t mutate the buffer

In the context of general-purpose code which is not intimately specific to a particular use-case, it is (in my opinion) good practice for a method that does an (abstractly) read-only operation (such as reading a byte buffer), to not mutate its input. This is a stronger requirement than “Don’t consume the ByteByffer”. Take the example from the previous section, but with an attempt to avoid the extra allocation of the ByteBuffer:

public static String toString(ByteBuffer bb) {
    final byte[] bytes = new byte[bb.remaining()];

    bb.mark();      // NOT RECOMMENDED, don't do this
    bb.get(bytes);
    bb.reset();     // NOT RECOMMENDED, don't do this

    return new String(bytes);
}

In this case, we record the state of the ByteBuffer prior to our call to get() and restore it afterwards (see the API documentation for mark() and reset()). There are two problems with this approach. The first problem is that the function above does not compose. A ByteBuffer only has one “mark”, and your (very general, not context aware) toString() method cannot safely assume that the caller is not trying to use mark() and reset() for its own purposes. For example, imagine this caller which is de-serializing a length-prefixed string:

bb.mark();
int length = bb.getInt();
... sanity check length
final String str = ByteBufferUtils.toString(bb);
... do something
bb.reset(); // OOPS - reset() will now point 4 bytes off, because toString() modified the mark

(As an aside, this is a very contrived and strange example, because I found it hard to come up with a realistic example of code that uses mark()/reset(), which is typically used when treating the buffer in a stream-like faction, which would also feel the need to call toString() on the remainder of said buffer. I’d be interested in hearing what solutions people have come up with here. For example, one could imagine clear policies in a code base that allow mark()/reset() in value oriented contexts like toString() – but even if you did (and it smells likely to be inadvertently violated) you would still suffer the mutation problem mentioned later.)

Let’s look at an alternative version of toString() that avoids this problem:

public static String toString(ByteBuffer bb) {
    final byte[] bytes = new byte[bb.remaining()];

    bb.get(bytes);
    bb.position(bb.position() - bytes.length);     // NOT RECOMMENDED, don't do this

    return new String(bytes);
}

In this case we are not modifying the mark, so we do compose. However, we are still committing the “crime” of mutating our input. This is a problem in multi-threaded situations; you don’t want reading something to imply mutating it, unless the abstraction implies it (for example, with a stream or when using ByteBuffers in a stream-like fashion). If you’re passing around a ByteBuffer treating it as a value, putting it in containers, sharing them, etc – mutating them will introduces subtle bugs unless you are guaranteed that two threads never ever use the same ByteBuffer at the same time. Typically, the result of this type of bug is strange corruption of values or unexpected BufferOverFlowException:s.

A version that suffers from neither of this appears in the “Don’t consume the buffer” section above, which uses duplicate() to construct a temporary ByteBuffer instance on which it is safe to call get().

compareTo() is subject to byte signedness

bytes in Java are signed, contrary to what one typically expects. What is easy to miss though, is the fact that this affects ByteBuffer.compareTo() as well. The Java API documentation for that method reads:

“Two byte buffers are compared by comparing their sequences of remaining elements lexicographically, without regard to the starting position of each sequence within its corresponding buffer.”

A quick reading might lead one to believe the result is what you would typically expect, but of course given the definition of a byte in Java, this is not the case. The result is that the order of byte buffers that contains values with the highest order bit set, will diverge from what you may be expecting.

Google’s excellent Guava library has an UnsignedBytes helper to mitigate your pain.

array() is usually the wrong method to use

Generally, don’t use array() casually. In order for it to be used correctly you either have to know for a fact that the byte buffer is array backed, or you have to test it with hasArray() and have two separate code paths for either case. Additionally, when you use it, you must use arrayOffset() in order to determine what the zeroth position of the ByteBuffer corresponds to in the byte array.

In typical application code, you would not use array() unless you really know what you are doing and you specifically need it. That said, there are cases where it’s useful. For example, supposing you were implementing a ByteBuffer version of UnsignedBytes.compare() (again, from Guava) – you may wish to optimize the case where either or both of the arguments are array backed, to avoid unnecessary copying and frequent calls to the buffers. For such a generic and potentially heavily used method, such an optimization would make sense.

Ending the dark age of telecom with the help of Google

Posted: November 17, 2012 | Author: | Filed under: Uncategorized | 2 Comments

(Warning: This is more of a rant than anything else. Now you know, up-front.)

I was extremely excited to read that it appears Google will be going into the wireless carrier business. Coming off the heels of Nexus 4 (which is causing me to ditch my current carrier even though it will mean I will drop from LTE to HSPA+, just to stop supporting innovation stifling business practices on the part of the carriers), it’s perfect timing. Suddenly, it looks Google might be in a position to finally realize my long-held fantasy of the wireless carrier as a bit pipe. Gone would be the days of artificial pricing structures that are orthogonal to the actual cost of services rendered (an unfortunate reality not limited to carriers).

I will avoid going into a long rant about appropriate pricing structures and the negative consequences of artificial restrictions, because the main topic I want to discuss is this: Google becoming a wireless carrier would fuel anti-Google sentiment held by those who are not comfortable with one single company dominating and controlling a large portion of the services and technology used in day-to-day life. It’s a valid concern; regardless of past track record, entrusting a single entity with too much power is risky (that includes, btw, entities that happen to be governments, even if elected by the people – but I’m diverging…).

In the end, while I agree that it is a valid concern, I am still extremely excited about developments like this. Android (and the iPhone) have already hit all old-style legacy phone manufacturers with a clue bat, forcing them to either change or stop dominating the market with outright crappy products (and they were, even before iPhone/Android came out as a contrast). Selling their own phones directly to consumers without carrier intervention is the first step in doing the same to the carriers, and launching their own wireless provider may be the final nail in the coffin that forces legacy carriers to change or see their market domination slowly erode to non-existence. Here’s hoping.

Why is Google seemingly taking over so much of our lives (Android, gmail, Chrome, Google Docs, Google Calendar, etc)? While Google is obviously trying to make a profit, I am not going for the “big Company” conspiracy theory cop-out. Rather, in my opinion, while Google does a lot of things, some of them failures, they have a track record of coming out with products or services that simply blow the competition completely away in game-changing ways. The first example of course was the search that started it all – remember back in the days of AltaVista, Hotbot, Lycos? Remember how suddenly within the span of a few months the way you searched the internet changed, with the introduction of Google?

In this particular case, working on the premise that Google will indeed be launching a carrier, they are doing something which has been the obvious end-game for several years. It’s not that they are doing something original in the sense of an idea, or that they’re doing something no one else thought of. They are doing the reasonable thing; the enabling factor here is that they can in combination with whatever internal Google workings allow the company to actually want to do it (instead of behaving like the typical legacy company). Myself and friends were discussing fantasy of the future data-only world back when 3G was new. We obviously weren’t alone; I’m sure hundreds of thousands of people had similar discussions. The key is that none of those people were able to make it happen.

Is it a good or a bad thing that Google will start taking over another area of our day-to-day lives (the wireless carrier)? My claim is that if it is to be considered bad, the root cause is not that Google is being evil, irresponsible, or anything else of that sort. The root cause is that all other options suck and Google is the first company to offer an alternative that does not. In an area where you have to be huge to compete, innovation will tend to suffer because you are relying on one out of a few very large companies getting their act together and doing something innovative – even if it means a seeming short-term drop in profits. It is rare for a large corporation to “get it” and do the reasonable thing, no matter how obvious it might seem to lots of people who cannot do anything about it.

Here’s my request: To everyone who is uncomfortable with Google taking over more and more, I urge you not to complain about Google. Instead, complain about everyone else. Refuse to accept the stupidity of the alternatives out there. Argue with your carrier’s customer support when they try to make you pay a “tethering” charge after you already payed (supposedly) for a certain amount of bandwidth. Argue with your carrier when they try to tell you that you can only get the latest and greatest phone if you also commit to a contract. Encourage your friends and family not to accept these things. Support the alternative options, and make it known why. Explain to people who are not knee deep in technical knowledge why it makes no sense whatsoever to pay $0.1 to send a text message (nor did it 10 years ago), and how the prices only serve to lure you into more expensive “unlimited text” plans – by making the alternative be crap, instead of providing something good. Complain when your carrier tells you that you are now allowed to use VoIP on their network. Explain to them that they should provide a superior voice calling service if they want you to use it, rather than block superior alternatives that exist. Carrot, not the stick. Complain when they give you a crippled version of an expensive phone (such as removing Google Wallet). Complain when they require you to sign up for a minimum $40/month plan for minutes, if all you want is data and are likely to spend a maximum of 5 minutes per months actually using your minutes. Complain when they delay updates to your Android phone by months. Support phones like the Nexus 4 that are sold directly to consumers without any lock-in.

Let’s all do our best to end the dark age of telecom.

End of rant.

Practical Garbage Collection, part 1 – Introduction

Posted: December 26, 2011 | Author: | Filed under: Uncategorized | Tags: , , | 46 Comments

Introduction

This is the first part of a series of blog posts I intend to write, whose aim will be to explain how garbage collection works in the real world (in particular, with the JVM). I will cover some theory that I believe is necessary to understand garbage collection enough for practical purposes, but will keep it to a minimum. The motivation is that garbage collection related questions keeps coming up in variety of circumstances, including (for example) on the Cassandra mailing list. The problem when trying to help is that explaining the nuances of garbage collection is too much of an effort to do ad-hoc in a mailing list reply tailored to that specific situation, and you rarely have enough information about the situation to tell someone what their particular problem is caused by.

I hope that this guide will be something I can point to in answering these questions. I hope that it will be detailed enough to be useful, yet easy to digest and non-academic enough for a broad audience.

I very much appreciate any feedback on what I need to clarify, improve, rip out completely, etc.

Much of the information here is not specific to Java. However, in order to avoid constantly invoking generic and abstract terminology, I am going to speak in concrete terms of the Hotspot JVM wherever possible.

Why should anyone have to care about the garbage collector?

That is a good question. The perfect garbage collector would do its job without a human ever noticing that it exists. Unfortunately, there exists no known perfect (whatever perfection means) garbage collection algorithm. Further, the selection of garbage collectors practically available to most people is additionally limited to a subset of garbage collection algorithms that are in fact implemented. (Similarly, malloc is not perfect either and has its issues, with multiple implementations available with different characteristics. However, this post is not trying to contrast automatic and explicit memory management, although that is an interesting topic.)

The reality is that, as with many technical problems, there are some trade-offs involved. As a rule of thumb, if you’re using the freely available Hotspot based JVM:s (Oracle/Sun, OpenJDK), you mostly notice the garbage collector if you care about latency. If you do not, chances are the garbage collector will not be a bother – other than possibly to select a maximum heap size different from the default.

By latency, in the context of garbage collection, I mean pause times. The garbage collector needs to pause the application sometimes in order to do some of its work; this is often referred to as a stop-the-world pause (the “world” being the observable universe from the perspective of the Java application, or mutator in GC speak (because it is mutating the heap while the garbage collector is trying to collect it). It is important to note that while all practically available garbage collectors impose stop-the-world pauses on the application, the frequency and duration of these pauses vary greatly with the choice of garbage collector, garbage collector settings, and application behavior.

As we shall see, there exists garbage collection algorithms that attempt to avoid the need to ever collect the entire heap in a stop-the-world pause. The reason this is an important property is that if at any point (even if infrequent), you stop the application for a complete collection of the heap, the pause times suffered by the application scale proportionally to the heap size. This is typically the main thing you want to avoid when you care about latency. There are other concerns as well, but this is usually tbe big one.

Tracing vs. reference counting

You may have heard of reference counting being used (for example, cPython uses a reference counting scheme for most of it’s garbage collection work). I am not going to talk much about it because it is not relevant to JVM:s, except to say two things:

  • One property that reference counting garbage collection has is that an object will be known to be unreachable  immediately at the point where the last reference is removed.
  • Reference counting will not detect as unreachable cyclic data structures, and has some other problems that cause it to not be the end-all be-all garbage collection choice.

The JVM instead uses what is known as a tracing garbage collector. It is called tracing because, at least at an abstract level, the process of identifying garbage involves taking the root set (things like your local variables on your stack or global variables) and tracing a path from those objects to all objects that are directly or indirectly reachable from said root set. Once all reachable (live) objects have been identified, the objects eligible for being freed by the garbage collector have been identified by a proces of elimination.

Basic stop-the-world, mark, sweep, resume

A very simple tracing garbage collector works using the following process:

  1. Pause the application completely.
  2. Mark all objects that are reachable (from the root set, see above) by tracing the object graph (i.e., following references recursively).
  3. Free all objects that were not reachable.
  4. Resume the application.

In a single-threaded world, this is pretty easy to imagine: The call that is responsible for allocating a new object will either return the new object immediately, or, if the heap is full, initiate the above process to free up space, followed by completing the allocation and returning the object.

None of the JVM garbage collectors work like this. However, it is good to understand this basic form of a garbage collector, as the available garbage collectors are essentially optimizations of the above process.

The two main reasons why the JVM does not implement garbage collection like this are:

  • Every single garbage collection pause will be long enough to collect the entire heap; in other words, it has very poor latency.
  • For almost all real-world applications, it is by far not the most efficient way to perform garbage collection (it has a high CPU overhead).

Compacting vs. non-compacting garbage collection

An important distinction between garbage collectors is whether or not they are compacting. Compacting refers to moving objects around (in memory) to as to collect them in one dense region of memory, instead of being spread out sparsely over a larger region.

Real-world analogy: Consider a room full of things on the floor in random places. Taking all these things and stuffing them tightly in a corner is essentially compacting them; freeing up floor space. Another way to remember what compaction is, is to envision one of those machines that take something like a car and compact it together into a block of metal, thus taking less space than the original car by eliminating all the space occupied by air (but as someone has pointed out, while the car id destroyed, objects on the heap are not!).

By contrast a non-compacting collector never moves objects around. Once an object has been allocated in a particular location in memory, it remains there forever or until it is freed.

There are some interesting properties of both:

  • The cost of performing a compacting collection is a function of the amount of live data on the heap. If only 1% of data is live, only 1% of data needs to be compacted (copied in memory).
  • By contrast, in a non-compacting collector objects that are no longer reachable still imply book keeping overhead as their memory locations must be kept track of as being freed, to be used in future allocations.
  • In a compacting collector, allocation is usually done via a bump-the-pointer approach. You have some region of space, and maintain your current allocation pointer. If you allocate an object of n bytes, you simply bump that pointer by n (I am eliding complications like multi-threading and optimizations that implies).
  • In a non-compacting collector, allocation involves finding where to allocate using some mechanism that is dependent on the exact mechanism used to track the availability of free memory. In order to satisfy an allocation of n bytes, a contiguous region of n bytes free space must be found. If one cannot be found (because the heap is fragmented, meaning it consists of a mixed bag of free and allocated space), the allocation will fail.

Real-world analogy: Consider your room again. Suppose you are a compacting collector. You can move things around on the floor freely at your leisure. When you need to make room for that big sofa in the middle of the floor, you move other things around to free up an appropriately sized chunk of space for the sofa. On the other hand, if you are a non-compacting collector, everything on the floor is nailed to it, and cannot be moved. A large sofa might not fit, despite the fact that you have plenty of floor space available – there is just no single space large enough to fit the sofa.

Generational garbage collection

Most real-world applications tend to perform a lot allocation of short-lived objects (in other words, objects that are allocated, used for a brief period, and then no longer referenced). A generational garbage collector attempts to exploit this observation in order to be more CPU efficient (in other words, have higher throughput). (More formally, the hypothesis that most applications have this behavior is known as the weak generational hypothesis.)

It is called “generational” because objects are divided up into generations. The details will vary between collectors, but a reasonable approximation at this point is to say that objects are divided into two generations:

  • The young generation is where objects are initially allocated. In other words, all objects start off being in the young generation.
  • The old generation is where objects “graduate” to when they have spent some time in the young generation.

The reason why generational collectors are typically more efficient, is that they collect the young generation separately from the old generation. Typical behavior of an application in steady state doing allocation, is frequent short pauses as the young generation is being collected – punctuated by infrequent but longer pauses as the old generation fills up and triggers a full collection of the entire heap (old and new). If you look at a heap usage graph of a typical application, it will look similar to this:

Typical sawtooth behavior of heap usage

Typical sawtooth behavior of heap usage with the throughput collector

The ongoing sawtooth look is a result of young generation garbage collections. The large dip towards the end is when the old generation became full and the JVM did a complete collection of the entire heap. The amount of heap usage at the end of that dip is a reasonable approximation of the actual live set at that point in time. (Note: This is a graph from running a stress test against a Cassandra instance configured to use the default JVM throughput collector; it does not reflect out-of-the-box behavior of Cassandra.)

Note that simply picking the “current heap usage” at an arbitrary point in time on that graph will not give you an idea of the memory usage of the application. I cannot stress that point enough. What is typically considered the memory “usage” is the live set, not the heap usage at any particular time. The heap usage is much more a function of the implementation details of the garbage collector; the only effect on heap usage from the memory usage of the application is that it provides a lower bound on the heap usage.

Now, back to why generational collectors are typically more efficient.

Suppose our hypothetical application is such that 90% of all objects die young; in other words, they never survive long enough to be promoted to the old generation. Further, suppose that our collection of the young generation is compacting (see previous sections) in nature. The cost of collecting the young generation is now roughly that of tracing and copying 10% of the objects it contains. The cost associated with the remaining 90% was quite small. Collection of the young generation happens when it becomes full, and is a stop-the-world pause.

The 10% of objects that survived may be promoted to the old generation immediately, or they may survive for another round or two in young generation (depending on various factors). The important overall behavior to understand however, is that objects start off in the young generation, and are promoted to the old generation as a result of surviving in the young generation.

(Astute readers may have noticed that collecting the young generation completely separately is not possible – what if an object in the old generation has a reference to an object in the new generation? This is indeed something a garbage collector must deal with; a future post will talk about this.)

The optimization is quite dependent on the size of the young generation. If the size is too large, it may be so large that the pause times associated with collecting it is a noticeable problem. If the size is too small, it may be that even objects that die young do not die quite quickly enough to still be in the young generation when they die. Recall that the young generation is collected when it becomes full; this means that the smaller it is, the more often it will be collected. Further recall that when objects survive the young generation, they get promoted to the old generation. If most objects, despite dying young, never have a chance to die in the young generation because it is too small – they will get promoted to the old generation and the optimization that the generational garbage collector is trying to make will fail, and you will take the full cost of collecting the object later on in the old generation (plus the up-front cost of having copied it from the young generation).

Parallel collection

The point of having a generational collector is to optimize for throughput; in other words, the total amount of work the application gets to do in a particular amount of time. As a side-effect, most of the pauses incurred due to garbage collection also become shorter. However, no attempt is made to eliminate the periodic full collections which will imply a pause time of whatever is necessary to complete a full collection.

The throughput collector does do one thing which is worth mentioning in order to mitigate this: It is parallel, meaning it uses multiple CPU cores simultaneously to speed up garbage collection. This does lead to shorter pause times, but there is a limit to how far you can go – even in an unrealistic perfect situation of a linear speed-up (meaning, double CPU count -> half collection time) you are limited by the number of CPU cores on your system. If you are collecting a 30 GB heap, that is going to take some significant time even if you do so with 16 parallel threads.

In garbage collection parlance, the word parallel is used to refer to a collector that does work on multiple CPU cores at the same time.

Incremental collection

Incremental in a garbage collection context refers to dividing up the work that needs to be done in smaller chunks, often with the aim of pausing the applications for multiple brief periods instead of a single long pause. The behavior of the generational collector described above is partially incremental in the sense that the young generation collectors constitute incremental work.  However, as a whole, the collection process is not incremental because of the full heap collections incurred when the old generation becomes full.

Other forms of incremental collections are possible; for example, a collector can do a tiny bit of garbage collection work for every allocation performed by the application. The concept is not tied to a particular implementation strategy.

Concurrent collection

Concurrent in a garbage collection context refers to performing garbage collection work concurrently with the application (mutator). For example, on an 8 core system, a garbage collector might keep two background threads that do garbage collection work while the application is running. This allows significant amounts of work to be done without incurring an application pause, usually at some cost of throughput and implementation complexity (for the garbage collector implementor).

Available Hotspot garbage collectors

The default choice of garbage collector in Hotspot is the throughput collector, which is a generational, parallel, compacting collector. It is entirely optimized for throughput; total amount of work achieved by the application in a given time period.

The traditional alternative for situations where latency/pause times are a concern, is the CMS collector. CMS stands for Concurrent Mark & Sweep and refers to the mechanism used by the collector. The purpose of the collector is to minimize or even eliminate long stop-the-world pauses, limiting garbage collection work to shorter stop-the-world (often parallel) pauses, in combination with longer work performed concurrently with the application. An important property of the CMS collector is that it is not compacting, and thus suffers from fragmentation concerns (more on this in a later blog post).

As of later versions of JDK 1.6 and JDK 1.7, there is a new garbage collector available which is called G1 (which stands for Garbage First). It’s aim, like the CMS collector, is to try to mitigate or eliminate the need for long stop-the-world pauses and it does most of it’s work in parallel in short stop-the-world incremental pauses, with some work also being done concurrently with the application. Contrary to CMS, G1 is a compacting collector and does not suffer from fragmentation concerns – but has other trade-offs instead (again, more on this in a later blog post).

Observing garbage collector behavior

I encourage readers to experiment with the behavior of the garbage collector. Use jconsole (comes with the JDK) or VisualVM (which produced the graph earlier on in this post) to visualize behavior on a running JVM. But, in particular, start getting familiar with garbage collection log output, by running your JVM with (updated with jbellis’ feedback – thanks!):

  • -XX:+PrintGC
  • -XX:+PrintGCDetails
  • -XX:+PrintGCDateStamps
  • -XX:+PrintGCApplicationStoppedTime
  • -XX:+PrintPromotionFailure

Also useful but verbose (meaning explained in later posts):

  • -XX:+PrintHeapAtGC
  • -XX:+PrintTenuringDistribution
  • -XX:PrintFLSStatistics=1

The output is pretty easy to read for the throughput collector. For CMS and G1, the output is more opaque to analysis without an introduction. I hope to cover this in a later update.

In the mean time, the take-away is that those options above are probably the first things you want to use whenever you suspect that you have a GC related problem. It is almost always the first thing I tell people when they start to hypothesize GC issues; have you looked at GC logs? If you have not, you are probably wasting your time speculating about GC.

Conclusion

I have tried to produce a crash-course introduction that I hope was enlightening in and of itself, but is primarily intended as background for later posts. I welcome any feedback, particularly if things are unclear or if I am making too many assumptions. I want this series to be approachable by a broad audience as I said in the beginning, though I certainly do assume some level of expertise. But intimate garbage collection knowledge should not be required. If it is, I have failed – please let me know.

Solving the EULA problem

Posted: December 4, 2011 | Author: | Filed under: Uncategorized | Tags: , , , , | 9 Comments

Everyone recognizes the situation. Some piece of software has released an update, or you are installing it for the first time, and you’re asked to accept a new EULA. The EULA is typically 5-15 pages of dense legal text. Almost everyone just tries their best to find the “accept” button as quickly as possible, sometimes giving a sigh as they realize you first have to scroll down before you can hit the accept button.

I think there is something fundamentally flawed here. (In reality this extends to more than just EULA:s and include e.g. contracts signed for a cell phone subscription, and many other things – essentially most legal contracts intended for consumers)

The problem

As the title indicates, I intend to suggest a solution. But first I need to be clear about what I am solving, and make a case for it being a problem to begin with. I will assume the following premises as true (and I think most readers will agree):

  • A very tiny portion of end-users actually read the EULA.
  • If a user does decide to read (not sift) it, it will take, on average,  at least 10 minutes for an average-length EULA to be properly understood.
  • Most EULA:s tend to contain a lot of standard boilerplate that may not be exactly identical to other EULA:s, but is pretty much the same. Some key variation exists depending on the use-case and the desires of the author of the EULA.
  • To the extent that users at large notice the contents of EULA:s, it is typically because a change or a particular part of the EULA is brought to public attention by a blog post, news article or the like.

Given that the average user by far, never reads the EULA, it is essentially still just down to “do you trust the maker of this software to do something sensible?”. The details of the EULA are mostly irrelevant to the user. The user wants to use the software, and either explicitly or implicitly chooses to trust the maker of said software to some extent. If the software does something the user does not like (such as posting information the user thought was private, publicly) the user will become angry regardless of whether the EULA happened to have a clause that directly or indirectly allowed it. The effects (whether positive or negative) are there regardless of the EULA. The individual user has very little chance to do anything about it, regardless of the EULA. A company that changes its EULA will probably do so in response to overall public pressure for PR purposes or for legal/political reasons.

Further, let us take the theoretical legal stance and just say that it is up to a user to read and accept the EULA and that’s the end of discussion. There are at least two huge problems with this:

  • If a piece of software with 100 million users gets updated and prompts for EULA acceptance, given the above premises that means that 1902 human-years are wasted on reading the EULA (100000000*10/60/24/365). Let me repeat that. A couple of thousand human years would be spent reading the EULA. Can you think of something better to do with two thousand human years? Note that no attempt is typically made to optimize this process; for example by presenting a nicely annotated view of the EULA which shows which sections have been changed relative to the last version that was accepted.
  • The argument assumes the user can realistically be expected to opt out of using the application because a part of the EULA is unclear, slightly too allowing or restrictive, etc. Most of the time, the user really isn’t interested in precise interpretation of legal paragraphs and they just want to use the software/service and have it behave decently.

But, despite this, EULA:s still exist. I am not a lawyer, but as far as I can tell, this is because they actually do matter legally. In the rare case that someone does bring suit, the fine print matters. Despite the fact that the original users affected by the alleged violation of the EULA might not ever have read the EULA to begin with. When the fight turns legal, the legal text is the weapon and the battleground. It’s as if we have two different worlds – the real world, and the legal world. And there are some doors in between them. In the real world, we have to deal with EULA:s that are completely useless to us in the real world (and as an end-user one exists in the real world only), because the company behind the service or software exists in both worlds and they must protect themselves in the legal world by subjecting the real world to certain side-effects.

Rejecting non-solutions

I find that there is an overly adversarial view going on whenever EULA:s are discussed. There is plenty of people complaining about companies not caring about privacy and having EULA:s that allow all sorts of things.

Well, I think that is because just like the end-user lives in the real world and not the legal world, the same is true for the developers, marketers and monetizers on the company end. They want to do what they want to do, while maintaining some decent behavior, without legal issues getting in the way. In order to get things done, you thus want an EULA that opens up as much as possible to be flexible so as to not get into a huge legal issue as soon as some tiny change needs to be made.

Pretend you’re writing an EUAL for something like iTunes or Facebook. You  could go into excruciating detail about exactly how each piece of information is allowed to be used. For example, in Facebook’s case, they could specify exactly how information is presented, exactly to whom, and under what circumstances.

But what happens when you make a minor tweak to the UI? Or you introduce new privacy features – even if they are intended to, and legitimately try to, improve privacy features now suddenly an entire EULA needs to be very carefully re-written to match. As a developer, would you want this constantly impeding your progress? As an end-user, would you really want to read new EULA:s every time some little detail is changed?

This is why I think it makes total sense that EULA:s often seem to be very much geared towards letting the company do whatever they want with you. I don’t think this is evilness (usually); I think it’s about being practical in a real world hampered by spill-over from the legal world.

Another potential solution is legislation. Making it illegal to use information in certain ways, or for software to behave in certain ways. I wil immediately reject this notion as unrealistic because no law will ever be flexible enough to allow true innovation and freedom while somehow preventing companies from “doing evil”. We have a public that expects everything to be free (in part I suspect because there is no usable micro payment system, but that’s another blog entry), yet any attempt to monetize on content or statistics is hailed as evil. Let’s say such laws were enacted a few years ago, and that these laws said that software installed on a computer must never send personal user information over the network to servers operated by a third party or the company providing the software. That would probably have made sense to many people at the time, but would make a huge part of today’s software legally impossible. Before the buzzword of “cloud computing” existed, people didn’t really get it. There are plenty of reasons to want your data available centrally and online instead of tying it to a specific physical computer. This was always the case, but before the advent of the cloud computing buzz, it was not commonly realized.

No, we need to have a free system that can grow organically and where innovation is possible and encouraged. We cannot legislate away this problem.

A quick look at open source licenses

Most open source software today is distributed under one of the well-known open sources licenses. I want to mention some things about the situation, in part because some readers may not be familiar with the situation, and in part because even for readers that are I want to highlight certain specific properties.

There is plenty of room for debate concerning which open source license is the better choice for a particular project. But the unique aspect of an open source project that adopts one of them, is that it does not invent its own license. Someone inquiring about the license of a piece of software may receive the response “the MIT license“, or “the Apache license“. While there is some potential for confusion (for example, if you say “The BSD license” are you talking about the 2-clause version or the 3-clause version?), in general the name of the license is enough for the person asking. There is no need to read through the license in detail, because you already know what the license entails by name.

From the point of view of the software developer, adopting a well-known established license is an easy way to get going with actually writing the software, which is often what you care about, without getting a lawyer to draft you a license. (This does not mean that all open source software projects get away without legal assistance, particularly if there are multiple contributors, but it does make the barrier of releasing something much smaller.)

For many end-users, this is probably not a very big deal. Many people probably only care about the fact that it is free-as-in-beer and generally open, rather than each particular detail. But even though it may not be perceived as a big deal, it does allow the individual end-user to make a true decision to accept the license with reasonable knowledge of what it entails, without constantly be reading dense legal text.

More importantly however, look at the typical company. Companies typically have to be very careful in what software they use internally (or as part of released products), mainly for legal reasons. A big company has a lot at stake, and a foul-up with incorrect use of a software can be extremely costly. The difference between the company and the typical private individual, is that the company has the motivation to actually treat the software license the way it is really supposed to be treated; they cannot do the “scroll down and accept” dance that the average person does when he or she gets an iTunes update. And that causes practical problems. If you want to use some piece of software with an arbitrary license, you are likely to have to clear it through legal. That means costs (for legal services), and inconvenience (developers having to go through a bureaucratic process to get to use software), and ultimately less use of the software if the software has a problematic license.

But a company can easily have a policy that anything under a certain set of licenses are always okay to use. Typically licenses like the 2-clause BSD license, the MIT license, the Apache license etc fall in that category (sometimes the GPL).

Solving the EULA problem by generalizing the named license model

My proposed solution is the following.

  1. Establish an authoritative directory of contract clauses where each clause has a globally unique identifier (similarly to how you can link to a license at the Open Source Initiative).
  2. Establish a format and software to express combinations of these clauses.
  3. When constructing a contract or EULA, pick a combination of clauses that exist in this directory.
  4. If needed, add additional clauses – but then also add the clause to said authoritative directory.

Imagine the ecosystem that can be built on top of this. Imagine a database of clauses, which over time are debated and discussed and even tested in court. There can be an accumulation of interpretation and evidence of legal effectiveness and implications for the user. An end-user can decide that a certain set of clauses are acceptable, while others are not.

In this world, whenever I, as a user, am prompted with a new license agreement for iTunes (or whatever), I would (thanks to the above points) see on my screen (presented by my license handling software, not trusting iTunes):
  • Any clauses I have not previously approved as generally acceptable will be very obviously highlighted, and have direct links to evaluation, discussion and legal precedence (if any) for the clause, to give me the best possible chance to evaluate whether I want to accept it.
  • Any notes I have associated with certain clauses (e.g., maybe I have flagged some clauses as dubious and I want to see when a contract contains them).
  • If I so desire, the complete list of clauses if I really want to look at it.

Consider a world where this was standard practice. Not only it is so much easier for the user to make informed decisions, it is also much more difficult for companies to sneak in unreasonable changes in contracts because they will be so obviously flagged – and everyone following directly links will be able to see whatever discussion and evaluation has happened. The user is not alone.

But it is also better for the company – hopefully there will be an ecosystem of clauses that give companies reasonable rights to be flexible and monetize in a reasonable fashion, while also being reasonable to the user. There should be no, or less (only when doing unique things that have not yet had time to establish an ecosystem of contract clauses), reason to construct clauses that are seen as questionable to the user and maybe even are in fact more allowing than they need to be,.

Note that this system assumes that clauses are independent of each other. The way contracts are expressed has to actually change to make this a reality; you can no longer have subtle relations between difference clauses of a contract; each has to stand on its own. That said, some context is probably going to be necessary, such as identifying different parties to the contract in a way which allows automatic treatment of it (for example, I might consider a certain clause acceptable if I am party A, but not if I am party B).

I do not pretend to have all the details worked out, but it seems very much plausible that something along these lines would be possible. I would appreciate comments, especially by lawyers, as to the feasibility of such a system.

 

EC2 AMI creation without magic

Posted: October 28, 2011 | Author: | Filed under: Uncategorized | Tags: , , , , | Leave a comment

Magic

While I enjoy the fact that there are people out there maintaining EC2 AMI:s for other people to use, I was faced with two problems. First, there were no AMI:s maintained for the Linux distribution I wanted to use (Arch). Second, I don’t like the idea of relying on something magical out of my hands that I don’t understand and cannot affect – in this case, I am referring to the kernel AKI:s that were traditionally not under the control of an average EC2 user (I believe Amazon itself and select partners were able to provide these kernels). Using one of those AKI:s I would essentially be relying on someone else release engineering kernels that are compatible with my userland.

In short, given a Linux kernel I have built, and a userland I know how to prepare, I want to create an EC2 bootable disk image/AMI.

As it turns out, this is possible nowadays, but the details were a bit hard to find (for me anyway). So, here is a short guide on how to go create an AMI from scratch, relying only on the kernel, your distribution and a host system on which to create the image (can be virtualized, such as with VirtualBox). It is assumed that you’re already familiar with things like boot loaders, building a kernel and such.

arch4ec2

If you wish, you can look at arch4ec2 as an example of the process briefly described below. arch4ec2 is a small tool that automates the creation of an Arch Linux system (with a btrfs root fs). It must be run from a host Arch Linux system, such as one installed using the Arch Linux installation ISO onto a VirtualBox. Alternatively if you want to play, you can use one of the AMI:s I built and list in README.

Doing without the magic (almost)

EC2 supports something called user specified kernels. Without it, as mentioned above, you choose which kernel to boot by selecting a so-called AKI to boot your image with. The AKI was provided by Amazon or (I believe) one of a few select partners, and you had to run an image that was compatible with that kernel.

With user specified kernels, the AKI you choose is instead pv-grub (which I assume stands for “paravirtualized grub”). As a result, all you have to do is create a disk image which is accessible by grub (i.e., correct partitioning/filesystem layout) and which has a grub configuration that points to a kernel which is compiled with the necessary support for paravirtualization (i.e., it has to be Xen compatible). The only significant difference from installing grub locally is that grub itself is provided by Amazon (through the AKI chosen) rather than being installed in the boot record of your image (this is where there is still a small bit of magic).

Step 1: Selecting an AKI

In the user specified kernel documentation (NOTE: do not cut’n’paste the “hyphen” from this PDF as it is not actually a hyphen, and ec2-register will fail) there is a list of AKI:s to use depending on whether you intend to run a 32 bit or a 64 bit kernel, which region you intend to run in, and whether or not your image will be based on EBS or S3. I have only tested EBS, and I don’t know what might be different for S3 based images. I have also only tested 32 bit as of this writing.

Step 2: Paravirtualization support in the kernel

In order to enable the appropriate support (for a 32 bit kernel, 64 bit not yet tested by me), these are needed:

CONFIG_HIGHMEM64G=y
CONFIG_HIGHMEM=y
CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_CLOCK=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y

(It is possible some variation is acceptable.)

Step 3: Use pv-grub compatible kernel compression

If you’re using a sufficiently new kernel, the kernel build might produce a kernel compress with XZ/LZMA2 instead of GZIP. Such a kernel will not boot on EC2 and you need to use GZIP instead:

CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set

Step 4: Populate a boot partition and root file system

Your disk image should be partitioned and file system initialized (in the case of arch4ec2 I use a small ext3fs boot partition and a btrfs root partition). If you have a separate boot partition mounted under /boot, do not forget to put a boot directory in it and symlink grub to boot/grub.

How to best populate a system is mostly up to which distribution you use. In the case of Arch Linux, the mkarchroot tool is helpful for scripting it (this is what arch4ec2 uses). But, in most cases if you are doing this manually as a one-off, you can just install your system as you would normally in a virtualized environment and take whatever steps necessary to switch to a properly configured kernel.

Step 5: Make an EBS snapshot of your disk image

In order to register an AMI, you must have an EBS snapshot which contains the contents to be used when spawning an instance using the AMI. If you did the original setup on ec2 maybe you already have an EBS volume and can just snapshot it. Otherwise, you are going to have to get your disk image onto EC2 in some way. For example, you can use the alesic AMI:s I mentioned before and boot a system, then mount an EBS volume you’ve created and ‘dd’ your device image to it over ssh.

In any case, once you have an EBS volume containing the image as you want it to appear in the AMI, snapshot it:

ec2-create-snapshot -d 'my-ami-snapshot' vol-XXXXXXX

Step 6: Register an AMI based on your snapshot

You will first have to recall the AKI you chose in step 1, and the snapshot id that was emitted by ec2-create-snapshot in step 5. Then, register the AMI:

ec2-register --debug -s snap-XXXXXXXX --root-device-name /dev/sda -n my-arch-ami --kernel AKI

The AMI registration will not succeed until the EBS snapshot has completed, so you will have to wait for that first.

Done

At this point your AMI is ready and you should be able to spawn instances with your AMI.

Some resources

Pipeline stalls and the human brain

Posted: October 23, 2011 | Author: | Filed under: Uncategorized | Tags: , , | 3 Comments

It’s amazing how often I am stricken by how closely, in some ways, the human brain mimics the behavior of a CPU in day-to-day work and interaction with other people or organizations. The analogy works particularly well for optimizing productivity in the workplace (at least I find that to be the case as a developer).

Consider a productive day programming, being in the zone. Very likely, this involves little interaction with others, and few interruptions. In fact, it seems most programmers feel the most productive when they are able to sit down and just work continuously, with focus, for an extended period. In my analogy, this is executing a tight loop with all instructions and memory fetches in cache.

An example of the opposite is when you are blocking on an external entity. Having to interrupt your work to wait for something else (outside of your control) to complete is analogous to a pipeline stall. Examples include:

  • Waiting for the compiler to build your software.
  • Waiting for a deployment step to complete.
  • Waiting for a slow web page to load when searching for documentation.
  • Waiting for someone to answer a question for you.
  • Waiting for someone to give you access to something so that you can complete testing or deployment.
  • Waiting for someone to review something.

It is worth noting that blockages can be either technical or human or a combination thereof.

Given a fixed amount of time spent working, and assuming you wish to be productive, clearly you do not want to twiddle your thumbs while waiting for various things that you’re blocking on. For example, if you have to wait 3 hours for someone to review your code prior to merging to the production branch, you do not want to sit there idling for 3 hours. The solution to this is typically to context switch to something else (in other words, you are engaged in concurrent execution of multiple tasks). Whenever you reach a blocking point, you context switch to an appropriately non-blocked (runnable) task.

What happens when everything you are working on is blocking on something? You start working on something else, increasing concurrency. As is generally recognized, context switching is expensive for the human mind (just as it is for an operating system on a CPU). In this case, as you increase concurrency, you will either be context switching more often (each context switch carrying with it some overhead) or you’re context switching to tasks that you, on average, worked on a longer time ago. In other words, all the caches relevant to the switched-to task will be colder.

Either way, all forms of blocks are expensive in the sense that overall productivity is decreased.

A way to mitigate the problem is to predict future choices or over-commit on options (i.e., branch prediction), thereby causing would-be blocking points to block for less time or not at all once you reach them (i.e., pre-fetching).

While the analogy is fun to draw, the main point here is that blocking on external entities is very expensive and in order to remain productive such blockages should be minimized. It follows that whenever you do absolutely have to block, you want to do so for a short a time as possible. In other words, the latency of external dependencies is crucial to productivity.

Interestingly, this implies that while a single individual or team may be considered most productive when they focus only on the task at hand, reality complicates matters. Suppose you have people or teams that have dependencies on other people or teams in a working environment. Every time you interrupt your work to help someone else, you lose some productivity. But someone else is gaining because you are decreasing latency for them.

I believe that in reality, a reasonable balance has to be kept in terms of quickly servicing external requests vs. focusing for productivity. I also believe that typically, this is ignored completely when arguments are put forth that you should e.g. only check your E-Mail once per day or refuse to do anything other than your main task for an entire day.

People, scrum teams, whatever it is – don’t exist in a vacuum and the greater good has to be considered in order to maximize global productivity.

There and back again – frustrations of a would-be music consumer

Posted: July 7, 2007 | Author: | Filed under: Uncategorized | Tags: | Leave a comment

(originally posted on blogspot; re-posted and back-dated on wordpress)

I have been waiting a long time for a clueful and usable online music store.

I have no affinity whatsoever for purchasing music in any kind of physical form. Even ignoring ideological issues such as the DRM used on CD:s, I find no pleasure in heaving a physical collection. In fact I specifically do not want it to be physical, so as to obviate the possibility of loosing music as a result of scratches or physical loss of the CD.

Buying music on CD just means that I have to spend time ripping it and tweaking tags to be consistent and correct (abcde in all honor, but the free CDDB databases are not perfect, and ripping still requires rotating discs). In other words, it does nothing but waste my time.

While there are some clueful stores in existence, they are limited in terms of the availability of music. Given a random artist whose music I want to purchase, the probability of it being available in such a store is very close to zero.

Because I recently purchased a MacBook, I suddenly had the possibility to use iTunes. I had very much resisted doing so for a long time because of their use of DRM and because the store requires you to run MacOS or Windows (for no particular reason I can detect). However, when I discovered that they had begun offering content without DRM I finally relented.

I was so tired of the music purchasing situation that I was going to give them a chance, despite the non-portability of the store and despite the lack of lossless downloads. I was going to drop my ideological reservations and just go ahead and start buying music.

And I did – for a while. Initially all was as expected; no particular problems encountered beyond the fact that only a subset of the music was available without DRM, which was frustrating in and of itself. But I was content to limit myself to that, and to lobby for the inclusion of additional DRM-free content (by submitting pleas to iTunes, and by buying DRM-free exclusively, thus voting with my money).

Then, while researching something unrelated, I came across forum threads where people were talking about how they had lost all their music due to hard drive crashes, and had gotten iTunes customer service to enable them to re-download their music as a one-off favor to the customer. Sure enough, looking carefully at the iTunes FAQ there are indications that they expect you to safely backup your music yourself.

I was shocked.

Not only are we talking about an online electronic music store. We are talking about one which aims for a level of integration and easy and use that they are apparently above simply providing a portable web interface. Yet they have failed to provide the most obvious and basic of services – the separation between delivery and the right to obtain a particular piece of music.

There are obvious reasons why you cannot walk into a store and ask for another CD because you lost the previous one, but with electronic delivery there is simply no reason to limit the customer to a single download. At worst the customer could be required to pay a small charge to cover the bandwidth/serving expenses associated with repeated downloads, but to require the re-purchase of the entire collection?

What happened to easy of use and practicality? What happened to simply signing into your account on another computer and have instant access to your music? Certainly – I can host my own music on my colocated server. But I am clearly in the minority to have that option. What about the average user?

One possible excuse is DRM. With DRM, any kind of re-download from scratch entails loss of whatever content tracking is supposed to be done via DRM. I can understand the limitation for DRM:ed music, but not for non-DRM:ed music. Could it in fact be that they do not want to allow re-downloads of non-DRM:ed music because it would highlight to the general public the limitations imposed onto them by the use of DRM? That it would make it much clearer to the average consumer that they cannot treat music purchased on iTunes as “safely theirs”, as they can with physical CD:s? (We all know that the average consumer does not take regular backups.)

I was sufficiently angry at this act of totally missing the point with online content delivery that I gave up. Never mind giving them a chance. That was it. I was not going to be buying any music online that is not lossless, entirely without drm and available to me at any time.

This only meant my frustration rose. I had finally started buying some music that I had wanted to purchase for a long time. I was finally going to contribute to the artists I liked. As a result I decided that – screw it, I was going to start buying CD:s again, despite their use of so-called copy protection that tries its best to prevent me from actually listening to the music I have payed for. After all, in practice the copy protection will not be preventing me from ripping the music for storage into usable form.

At this point one might imagine that I would be happy. Having taken the step to ignore my ideological oppositions, I could get on with buying music.

Unfortunately, the situation on the CD market is as bad as the online purchasing one, except in a different regard: availability.

For example, I want to purchase all Enigma albums. Enigma is sufficiently well-known that the major albums are at least available somewhere. Browsing some of the local online shops here in Sweden I see partial coverage; some albums are offered, some others are not. Some are offered in multiple incarnations, each differently misspelled (but all presumably the same actual album). I could get them all from Amazon.com with some effort. There used to be a “15 Years Later” collector’s release containing all albums; this seems to have vanished now for some unknown reason. Bottom line: I can get it, but with much more effort than I would have liked.

A worse case is Deep Forest which is presumably not as widely popular as Enigma. Some albums are available on Amazon.com, others are not. Most local online shops have none of them, or just a single album or two. Checking out their discography on Allmusic.com, it seems all albums are in fact available – but spread out over various different stores. I refuse to go and cherry pick albums one at a time in different stores, all with their own shipping issues, payment issues and customer services issues. I want to stick with a select few trusted stores (preferably only one).

So I am now once again stuck back where I was before embarking on this entire mess of a trip in a despeerate attempt to buy some music.

Why can I not go to the website of my favorite artist and simply buy it directly from them in lossless, DRM-free FLAC?

Why can I not go to amazon.com or some similar place and have complete sets of albums available?

RIAA – are you listening? Can you please tell me how to buy your music? Or would you rather put me in jail for piracy? It is up to you. Take my money, or spend money on prosecuting me for trying to enjoy good music.

I cannot wait for artists to escape the claws of the major record labels.