System.Threading.SynchronizationLockException

October 20, 2010

I got this exception a while ago when implementing a simple publisher/consumer scheme with 2 threads in C#. What this exception says is that you’re trying to use some operations without a lock held. In my case it was Monitor.Pulse(object).
These sort of operations conceptually need to be done with the lock on object held, for the same reason old Unix condition variable had to do certain operations with the lock held: to avoid race conditions where one thread can miss the signal and get blocked forever on a synchronization primitive.

So the first really easy way to catch this is check if you hold the lock, if not, easy, just add it.

The other situation is to lock one object and operate on another:

lock(Object1)
{
    if(condition)
    {
        Monitor.Wait(Object2)
    }
}

These scenarios are easy to catch and fix because they throw all the time. But this exception can occur in race conditions where it’s more difficult to catch. Let’s look at the following code:

Thread 1 (publisher):

lock(_queue)
{
     _queue.Add(new_element);
     Monitor.Pulse(_queue);
}

Thread 2 (consumer):

List queue;
lock(_queue)
{
     queue = _queue;
     _queue = new List();
}
ProcessQueue(queue);

The idea here is to try to optimize the way the list of type T is emptied and then processed. In particular, the optimization is that it doesn’t create a copy. The copy can be re-written, without this optimization as:

     queue = new List(_queue);
     _queue.Clear();

This has the problem that it allocates memory and copies data, all with the lock held.

But unfortunately, while the optimization works, it’s not entirely correct, as it created a hidden race condition. To illustrate the race condition, I’ll present it here, step by step, with the code running on both threads interlaced below:

1: Thread2:  lock(_queue) <- acquires lock on _queue
2: Thread1:  lock(_queue) <- lock is held by thread2, so thread1 gets blocked here
3: Thread2:  queue = _queue;
4: Thread2:  _queue = new List();
5: Thread2:  release lock on _queue
6: Thread1:  _queue.Add(new_element);
7: Thread1:  Monitor.Pulse(_queue);

At this step 7, Monitor.Pulse throws a System.Threading.SynchronizationLockException.
The reason is that thread1 acquired a lock on _queue but then thread2 changed the “meaning” of the reference _queue, so when thread1 pulses the _queue, it pulses a different object than the one it acquired the lock on. Thus the exception.

Once the problem is understood, the solution is simple: synchronize on an immutable object. You can create a dummy object and sync on that, rather than the mutable _queue.
Or you can implement the more inefficient version, which doesn’t actually change the object _queue refers to.

PS: To reproduce this relatively consistently, insert a Thread.Sleep(5000) in thread2, right after it acquired the lock.

Advertisements

Excel: UDF not available in automation mode

October 19, 2010

One problem I was having lately is that some UDF in an Automation addin was not available when Excel was started in automation mode, via a C# application. Checking the Automation addin in Tools/Addins showed it was enabled, ticked and should have been working.
But when trying to call the UDF, I was getting #NAME?

This can be easily replicated by starting Excel in automation mode:

Excel /automation spreadsheet_name.xls

Using the UDF in the automation addin has the same problem, it shows #NAME?

After a bit of googling, it turns out that in automation mode, you have to re-enable the plugin as it’s not loaded by default. So the solution is to un-tick the addin in the Tools/Addins dialog and tick it back on.

Or programatically you need to do this:

set oAddin = Application.Addins(“YourAddin.ProgId”)
oAddin.Installed = False
oAddin.Installed = True

If it’s not in the list, you will have to add it first with:

Application.Addins.Add(“YourAddin.ProgId”)

Once you do that, the UDF starts working fine.

If your UDF is in an XLL rather than an Automation Addin, then you can also use the RegisterXLL API.

Excel

October 9, 2010

I know it’s been a long time since my last post and I apologize.

In the mean time, I was doing quite a bit of Excel programming and that’s the experience I want to talk about. I didn’t know much about Excel before starting. I have done VB, VBScript before so that came as no surprise. I also knew Excel is used quite a lot in the financial community, so I was keen to get some exposure to it.
After almost a year of development, ranging from VBA, plugin development (COM addin + Automation addin – DDE) and RTD, I can say, I have mixed feelings.

The overall experience I’m taking from this is that Excel is not a mature environment for development and the experience can be quite frustrating. It is almost decent at VBA, but when you start with addins, RTD, it all leaves a slightly bitter taste in your mouth. And here’s why, all the reasons listed one after the other:

1. Excel is single threaded. This has several implications:
a. You cannot call into Excel when it’s busy, RPC calls fail, for this you have to retry, or sometimes you get the dreaded dialog telling you that the STA is busy
b. It’s slower than it should be. Cannot take advantage of multi-processors
c. Can get easily into a “frozen” state, when long operations have to be done on the UI thread
d. Operations need to be marshalled into the STA, which is extremely slow and I do mean extremely

In Excel 2007 they have introduced multi-threading. However, they have also changed the interface, introducing the ribbon interface and alienating the entire financial community. No one is moving to Excel 2007 for that reason.

2. Error handling is extremely poor

When something happens, you don’t get a clear error, but instead a general error of what happens. Many APIs return the same errors and the errors are not documented.
This is truly a killer, and it’s true with the whole Excel object model. A very very big issue.

3. The addin frameworks are all a bit of mess

There are 3 main types of addins:
a. COM addin, you can implement UI elements, but no UDFs
b. Automation addins, you can implement UDFs, but no UI elements
c. RTD servers, you can implement a very limited interface to update data, a pull-push hybrid, but no access to the Excel object model at all

So, let’s say you want to implement an addin which has a UI menu with some functionality, provides a UDF, which operates asynchronously. Then you need a combination of potentially all three addins in one, maybe add VBA xla on top of that.

4. Some operations don’t work from addins, only from VBA, or only from some context, or only if the VBA_E_IGNORE is not set, or if other black box conditions are set

Again, lack of documentation makes it really difficult to develop. It’s a trial & error approach which works most of the time.
You always face the problem where your addin works most of the time, unless the user does something which you didn’t expect and tested.

5. Pushing data into Excel is very difficult

Either Excel is busy, or sets itself into VBA_E_IGNORE when a dialog is present, or operations fail because you’re in edit mode.
This problem was addressed by the introduction of the RTD framework

6. Excel is still buggy and it can crash

If you do certain operations, you might hit a bug in Excel and crash it. It crashes because bugs in Excel, but your addin/VBA are triggering, so you look like a culprit.

7. Excel doesn’t perform very well when you have big workbooks with a lot of data

8. Very difficult to debug problems in Excel, or weirdness like above, because Excel is a black box

Still, Excel is loved in the financial community, because it makes non-programmers more productive by its operations. It’s an easy way to present, format data which can be refreshed and kept up to date.
Yet, for developers, it can be a nightmare.

Photographing the Earth from above

September 25, 2009

I was impressed by this: http://www.wired.com/gadgetlab/2009/09/the-150-space-camera-mit-students-beat-nasa-on-beer-money-budget/

Two MIT students have managed to take pictures or the Earth curvature by launching a photo camera + mobile phone attached to a balloon. I was really impressed. If I was a kid, this would be very motivating stuff to get me into science. I would probably try to replicate it too…

Being a software engineer, I’m thinking … what if you get a “smartphone” and you write some software for it … what software would you write? Well, let’s see:

1. Software to control the camera integrated in the smartphone, so you don’t need a camera and a phone
2. Software to connect via gprs and submit live images (or hey, a live streaming) … question is, would you have any signal at 28km above the earth?
3. Software to monitor the GPS trajectory and altitude, so that you can plot it on a map and you can draw all sorts of graphs
4.  A IM chat client to tell your friends you’ve done it 🙂

MQL

September 22, 2009

I’ve added recently another language to my repertoire: MQL. It’s the native language in the MetaTrader, which is a FX trading application. I’ve started FX trading only recently but I’m really attracted by the algo trading and trading in general.
MQL is quite basic, it doesn’t support complex data types, I could really use with a dictionary or an object of some sort.

What can you do with it? You can write an Expert Advisor, this is a algo trading application.
It has a 3 methods: init, deinit and start. The first 2 act as constructor/destructor, whereas the second is called on every tick, which you can process and then decide whether to open a trade, close a trade or do nothing.

Outsourcing

September 7, 2009

Does outsourcing works? It’s a question I had to revisit recently because my company is thinking about outsourcing. The main reason is cost and I think that’s the wrong reason to outsource for. And because you start outsourcing for the wrong reasons, it ultimately ends up in a failure.

Most companies look at it as a purely tactical move, although they call it strategic. The attraction is cost. However, because the move is tactical, the integration is also at a very tactical level. Usually the team assigns unattractive tasks like defect fixing or other uninteresting work to the offshore teams. Which ultimately leads to other problems like lack of knowledge, specialization and ultimately productivity (which is labelled as failure, and rightly so). Also, because the work is so un-attractive, it only appeals to mediocre employees, the best ones leave for greener pastures. In the end, it becomes a problem of loyalty.

An important problem is communication and the fact that there’s a bigger disconnect betwen the onshore and offshore (or nearshore) teams. Specs have to be more detailed and knowledge needs to be transferred more formally (which in IT doesn’t work that well – a lot of knowledge is tacit and not necessarily easy to capture in documents and diagrams).
This in the end results in more tech lead and PM time, which is quite expensive. Tech leadership and PM become full time jobs dedicated to the offshore team. If you add the cost of lower productivity to do with poorer communication and disconnection from the business/front office and you add also the travel costs, I’m not sure it’s such a good deal anymore.

Nevertheless, waves and waves of management professionals and consultants will fall for it, because on paper it represents such a good deal. You can boast about cutting cost which is music to the ear for the upper management, especially in recessions. And then they can cash in their bonuses and claim success in the short term, whereas the company (and shareholders ultimately) are left to pick up the pieces.

I’m not against outsourcing though, I’m just against it when it’s done for the wrong reasons. I also think that doing it for the right reasons would also mean doing less of it overall. Just because it’s not a universal panacea.

After a long time…

August 31, 2009

It’s been a long time since I’ve posted here and it’s all my fault. A lot happened but the markets now are stabilizing, jobs are safer and we can go back to writing blogs and sipping coffee … I suppose.
I saw recently one comment to a previous problem I posted here and I’m trying to contact the poster to provide more explanation, as I’m intrigued. I didn’t have to think more about the problem and solutions, but if he comes back to me with a detailed answer, I will try to digest it and publish it.
I will also try to write more here and resurrect this blog…

Discount yield to money market yield conversion

January 26, 2009

I haven’t written in a long time. On top of that my next entry is technical. It’s about the conversion between discount yield to money market yield. Reading one of Fabozzi’s book, I was puzzled about the equation. It was presented without the proper deduction or explanation.

The equation is:

m = (360 * Yd) / (360 – t * Yd)

where m is the Money Market equivalent yield, Yd is the discount yield, t is the number of days between the settlement and maturity.

This equation is required when you want to compare the return of a tresury bill (quoted in discount yield) and an equivalent CD (certificate of deposit). The day count convension for both is Act/360.

The discount yield for a t-bill is calculated with the following:

Yd = D * 360 / (t * F)

where D is the cash discount (actual amount of money that you get after t days per face value unit) and F is the face value.

The price of the t-bill is then P = F – D, assuming usually a face value of 100$.

The T-bill cash flow after t days is D (the discount). For a CD, the cash flow after t days is m * P * t / 360 where m is the money market yield and P is the price (or money invested).

To calculate equivalent yields, the cashflows must match. So, we have:

D = m * P * t / 360

Resolving for m, results into:

m = D * 360 / (t*P)

We extract D from the definition of the discount yield:

D = Yd * F * t / 360 

which gives:

m = Yd * F / P

P = F – D = F – Yd * F * t / 360 = F ( 1 – Yd * t / 360) = F * ( 360 – Yd * t) / 360

We replace the price in the equation for m:

m = Yd * 360 / (360 – Yd * t)

which is the equation in Fabozzi.

This conversion is used to compare T-bill investments with money market investments because the discount yield cannot be directly compared with the money market yield.

Bloomberg by Bloomberg

October 14, 2008

A while ago I finished reading Bloomberg by Bloomberg. It’s a book about the history of Bloomberg narrated by Michael Bloomberg himself.
It’s a frustrating book, sometime I felt I don’t want to read it anymore. The main problem is Mike’s style that seems a hard sell of his company and himself ultimately. It’s also full of pride reflected in the style, well deserved pride, but still, frustrating when reading it in a book. People want real content not marketing blah blah.
Anyway, he’s making some good points about software development in the financial services. And one thing he mentions is the “agile” methodology that they used at Bloomberg, emphasizing shipping products on time, even if they’re not perfect, even if they have to cut down features, and from there on, working in quick iterations.
It’s how the banking industry in general works. First because the business wants tangible results. Second because bonuses are paid annually.

It’s the model to be followed by all entrepreneurs I think. It was the same model followed by Microsoft.

OO in Javascript

October 10, 2008
framework.graphics.object = function(id)
{
   this.name = "object";
   this.id = id;
};

That’s a “class”. To create an instance of that, you do the usual:

  var myObject = new framework.graphics.object(1);

In order to emulate an inheritance hierarchy, Javascript has this special property called prototype. You can set the prototype of an object to any other object. Object, not class. That’s a bit unusual from an OO perspective.
So it’s not proper OO, it’s an emulation of OO. It relies on the way the Javascript engine resolves properties and methods.
When you do: object.property1 or object.method1() Javascript looks in the object and sees if there is a local property and local method. If there isn’t one, it checks the prototype, then the prototype’s prototype until either the name is resolved or the chain ends.
It takes a bit of getting used to, but it’s still a very useful way of reusing code.