Let’s start with a very simple demo. One form and one button with its OnClick event assigned.
procedure TForm11.btnDeadlockClick(Sender: TObject);
lock := TCriticalSection.Create;
thread := TThread.CreateAnonymousThread(
thread.FreeOnTerminate := false;
The code first initializes and acquires a critical section. Then it creates a background thread, starts the thread and waits for it to terminate. At the end, the form’s Caption is changed to “OK”.
What about the background thread? It is implemented using “anonymous threads” – a new approach in Delphi XE, similar to AsyncCalls and (parts of) OmniThreadLibrary. CreateAnonymousThread allows you to implement the threaded code in an anonymous function. This thread first changes its name to “AnonymousThread” (we’ll come back to that in a moment) and tries to acquire the lock.
Let’s put aside the threading and locking, look at the code again and ask ourselves – which lock? There’s no lock declared in the anonymous function so this must clearly be the lock variable from the btnDeadlockClick method. If we were to implement this background thread with a “classical” approach (using a TThread-based object), we would have to pass this lock to the threaded code somehow, possibly as a parameter in its constructor. Because here we’re using an anonymous function, the compiler does all that work for us. It “captures” the lock variable and allows the anonymous function to use it.
OK, back to the code. Did you already spot a problem? A lock is acquired in the main thread before the background thread starts execution. Because of that, the background thread cannot re-acquire it and will block on the lock.Acquire call. And because the background thread never terminates, the thread.WaitFor call never returns. When you run the program and click on the button, the background thread will stay blocked in lock.Acquire and the main thread in thread.WaitFor. A classical deadlock.
If you now press the “pause” debugger button (), Delphi will stop the program execution and display information on all running threads. (At least if you have the appropriate debugging windows visible. Be sure to enable debug windows (View, Debug Windows) Call Stack and Threads.
In the Threads window you’ll see three threads (at least on Delphi XE) – one is the main program thread, one is the background anonymous thread and the third thread was created temporarily by the debugger. You should ignore it. Since Delphi 2009 (and when running on Vista or Windows 7) this window also displays a “wait chain”. In our example we can directly read that the background thread is blocked on critical section, owned by the thread 5500. (By the way, this number will not be the same if you’re repeating the experiment. Thread IDs are randomly allocated and will be different on each execution.) We also know that this is the main thread since it is the first thread listed in that window. Interestingly, the debugger doesn’t know that the main thread is also blocked waiting on the background thread. This is one of the weird features of the wait chain detection (which is, by the way, implemented by Windows itself, not by Delphi) – sometimes deadlocks are detected, sometimes not.
Before we proceed with the debugging, let’s return briefly to the “AnonymousThread” name. While other threads have only numeric “names”, the background thread is properly named. That’s hardly surprising as we’ve seen that the code calls NameThreadForDebugging and assigns a name to the ThreadID (which is the number visible in the Threads window if the thread doesn’t have a name). NameThreadForDebugging is fairly new, but you can use the same functionality in older Delphis too. For example, the OmniThreadLibrary contains a function SetThreadName which performs exactly the same task.
FType : LongWord;
FName : PAnsiChar;
FFlags : LongWord;
ansiName : AnsiString;
if DebugHook <> 0
ansiName := AnsiString(name);
threadNameInfo.FType := $1000;
threadNameInfo.FName := PAnsiChar(ansiName);
threadNameInfo.FThreadID := $FFFFFFFF;
threadNameInfo.FFlags := 0;
RaiseException($406D1388, 0, SizeOf(threadNameInfo)
div SizeOf(LongWord), @threadNameInfo);
This function raises a special exception, which is caught by the debugger. The debugger then updates its internal tables and throws away the exception so that the program can continue its execution. As we’ll see later, you can also change a thread name “manually” in the debugger, at least in more recent Delphi versions.
Let’s return to the debugger and see what more we can learn. Double-click on the line starting with ‘5500’ in the debugger, and it will show the currently executing instruction and the call stack for that thread. At the top of the call stack we see many uninteresting WaitFor instructions which indicate only that the thread is waiting for something. The first interesting item is Classes.TThread.WaitFor. Click it and the debugger will display the WaitFor method and mark the currently executing statement.
We can also see that the WaitFor was called by Controls.TControl.Click, a short method in the Controls unit that calls the OnClick event handler. We would also expect the btnDeadlockClick to be visible in the call stack but somehow it is not. No idea why.
Just a short side notice – if you want to see the source for internal Delphi units as Classes in this example, make sure that you build your app with the “Use debug .dcus” setting.
Since Delphi 2010 the context menu in the Threads window contains some very useful options. You can suspend (block) threads with Freeze and Freeze All Other Threads and resume (unblock) them with Thaw and Thaw All Threads. This feature can be immensely helpful while debugging although it can also introduce deadlocks (if you're trying to run a thread that is waiting on a synchronisation object owned by a suspended thread). Use these options with caution!
Another interesting item here is Name Thread. Use it to temporarily assign a name to a thread (i. e. for the duration of the debugging session). This name will then be visible in the Threads window.
The last debugger option I want to emphasize is the thread-specific breakpoint. This is a type of breakpoint that will break the execution only when triggered from a specific thread. They were first implemented in Delphi 2010 and are incredibly useful when more than one thread is executing the same code.
When you’re writing multithreaded applications a proper approach to testing will (and please note that I’m not saying “may” or “can”!) mean a difference between working and crashing code.
Always write automated stress tests for your multithreaded code. Write a testing app that will run a variable number of threads to execute your code for some prolonged time. Then check the results, the status of internal data structures, etc. – check whatever your multithreaded code is depending upon. Run those tests whenever you change the code. Run them for long time – overnight is good.
Always test multithreaded code using both a small and a large number of threads. Always test your apps with a minimum number of required threads (even one, if it makes sense) on only one core, and then increase number of threads and cores until you’re running many more threads than you have cores. I’ve discovered that most problems occur when threads are blocked at “interesting” points in the execution and the simplest way to simulate this is to overload the system by running more threads than there are cores.
When you find a problem in the application that the automated test didn’t find, make sure that you first understand how to replicate the problem. Include it in the automated test next and only then start to fix it.
In other words – unit testing is your friend. Use it!
Most bugs in multithreaded programs spring from over-complicated designs. Complicated architecture translates into complicated and hard-to-find (and even harder-to-fix) problems. Keep it simple!
Instead of inventing your own multithreaded solutions, use as many well-tested tools as possible. More users means that more bugs will have been found and dealt with. Of course, you should make sure that your tools are regularly upgraded and that you don’t use obsolete code that everybody else has abandoned.
Keep the interaction points between threads simple, small and well defined. That will reduce the possibility of conflicts and will simplify the creation of automated tests.
Share data as little as possible. Global state (shared data) requires locking and is therefore bad by definition. Message queues will reduce possibilities for deadlocking. However, don’t expect message-based solutions to be magically correct – they can still lead to locking.
And besides everything else – have fun! Multithreaded programming is immensely hard but it is also extremely satisfying.