Send Message: Race Conditions
One of the most common folkloristic methods (and one I used for years, alas)
is to use EnumWindows, do a SendMessage to each window, and look
at the return result from SendMessage. What you send is a Registered Window
Message (see my essay on Message Management), and if
you receive this message you return TRUE. All other windows will not understand
this and return FALSE. This turns out to be deeply flawed in a variety
of ways.
Note that this method always worked in Win16 because it used cooperative
multitasking. It is the preemptive multitasking of Win32 that makes this method
fail. And it does.
The SendMessage can hang indefinitely. But if the thread that owns the
handle is blocked, on a Semaphore, a Mutex, an Event, an I/O operation, or some
other manner, the SendMessage will block until that thread frees up and
runs. But this may never happen.. So you haven't gained any reliability.
You can solve this by using SendMessageTimeout. This more-or-less works.
You will typically choose a short timeout, for example about 200ms.
But it gets worse.
Microsoft, in violation of all known Windows specifications, has created an
application that does not pass messages it doesn't understand to DefWindowProc,
which would return 0 for any message it does not understand. Instead, they have
a component, apparently associated with Personal Web Server, which has the truly
antisocial property of returning 1 for every message sent to its top-level window,
whether it understands it or not. So you can't rely on a zero meaning it is not
your app.
Well, this could be solved. When I needed to do this for another reason, I
ended up having to return the registered window message value, which avoids the
Microsoft blunder.
So we've solved the problem of timeouts and bogus messages. We know not to
depend on the caption contents, and probably want to avoid worrying about the
Window class name. So why doesn't this work?
Because of a much more fundamental problem: a race condition. Just like the
one described in the previous section.
The code did an EnumWindows loop and for each HWND it did a SendMessage.
So what happened was that application instance 1 searched for another instance
of itself, didn't find one, and proceeded to come up. Meanwhile, application
instance 2 searched for an instance of itself, but since instance 1 had not yet
come up and created its own main window, instance 2 did not find a conflicting
instance, and it proceeded to come up. Using a scheme similar to the table
I used to demonstrate why the basic FindWindow method doesn't work, you
can show that this method will fail for the same reason.
This is the most fundamental failure of this mechanism, and a reason it cannot
be used.