Knight Trading Software Problem

This story is interesting.  Knight Trading had a high-profile software problem yesterday.  They flooded the market with orders.  They moved the price of some stocks by 30% or more, buying some and selling others.  Knight will probably be stuck with a huge loss.  Allegedly, a large volume of orders triggered a previously-unnoticed software bug.

The 2nd link had an video explaining exactly what happened.  Knight sent a lot of 100 share orders, buying some stocks and selling others.

Suppose a customer wants to sell 1M shares of a stock.  Rather than sending one order for 1M shares, the order is broken up into 10000 100 share orders.  This minimizes the minimizes the amount your order moves the stock.  It also makes it harder for other traders to figure out what you’re doing.  In this case, the program for “break up one big order into lots of small orders” was misbehaving.

The algorithm ran for more than 30 minutes, before someone shut it down.  That’s gross negligence by the risk managers at Knight Trading.  At one place I worked, the risk managers had a program that displayed all positions.  If a risk manager had a program that showed “live mark-to-market gain/loss”, he would have immediately noticed a problem.

The NYSE announced they are busting some, but not all, of the bad trades.  This causes problems.  Suppose you thought you bought stock at a favorable price, and then hedged or sold.  One trade is busted but not the other, leaving you with a loss.

Whenever a business sends a lot of defective orders, the only correct solution is “Too bad.  You’re stuck with the loss, even if you’re bankrupt now.”

This is a variation of “Too big too fail.”  If a small broker makes $1M of bad trades, they go broke.  If a big broker makes $100M+ bad trades, then they get their trades busted.  It’s a type of bailout.

It encourages sloppy software, if trades are busted whenever someone makes a mistake.

When the stock started moving, some people came on the other side to buy or sell, minimizing the price movement.  If those trades are busted, it isn’t fair to the people who came in to buy or sell.  What incentive is there to buy during a price crash, if you know you’re going to get busted?  If it was an algorithm error, you get busted.  If it was a legitimate price decrease, now you’re stuck with the loss.

It isn’t fair to everyone else, if the trades are busted.

Also, some customers may have had “stop” orders in those stocks.  Those stop orders were triggered, when the price moved.  As a retail customer, you should never use stop orders.

Most financial software is pretty lousy.  Most financial institutions have very large teams of mediocre people.  There’s lots of bureaucracy.  This way, nobody can be blamed for a problem.  Some programmer at Knight will be scapegoated, but he probably isn’t the person who caused it.

There’s a simple common solution, to prevent this type of problem.  You have another program monitoring your trades.  If there’s a large volume of trades in a short period of time, then the circuit-breaker program takes over, shutting things down.  Knight Trading lacked the obvious risk-management program.

If the risk manager had a “live mark-to-market gain/loss” program, he would have know something was wrong.  Superficially, the program was making a huge mark-to-market gain, as it was buying and pushing up the price.

There were some NYSE circuit-breaker programs that shut down Knight trading.  That wasn’t until after a lot of bad trades were made.

This is the third high-profile financial software glitch in the past year.  There was the failed BATS IPONASDAQ had a software glitch during the Facebook IPO.  (That is a separate issue from “Facebook’s IPO was overpriced.  There was improper disclosure of financial information, prior to the IPO.”)

I was surprised that some trolls ridiculed me when I pointed this out.  If BATS/NASDAQ/Knight hired me or some other highly skilled programmers, they might have avoided this embarrassment.  For large financial institutions, their hiring process favors mediocre people rather than top performers.

Here is a conspiracy theory, regarding the Knight Trading software problem.  Did some programmers at Knight sabotage their software on purpose?  If people knew in advance that Knight Trading was going to have a glitch, they could have made a lot of money at Knight Trading’s expense.  I’ve seen suspicious things at some of the places I worked.

Overall, financial software is lousy.  By busting trades, this rewards sloppy software, by reducing the risk of failure.  When a software glitch moves prices by 30% or more, that raises an important issue.  Is the stock market based on genuine value, or is it all hype and one big Ponzi?

5 Responses to Knight Trading Software Problem

  1. This link is interesting. Knight says they lost $440M due to the software problem. If Knight lost $440M, someone else made $440M!

    However, Knight’s excuse is fishy. They said “We released new software that day, which caused the glitch.” If you release new trading software, you should be watching it closely and be ready to pull the plug if there’s a problem.

  2. Anonymous Coward August 2, 2012 at 6:15 pm

    I started programming from a very young age as a hobby long before I started work in the software industry.

    My first job was working for a software company producing risk management software for banks.

    At first I was so pleased to finally work with PROFESSIONALS, as I thought of myself as only a hobbyist amateur.

    Within a few months, I realized their software was buggy and the company was sitting on a couple of software disasters. I realized that no matter how hard I worked and how much good software I wrote, I would only be subsiding all the other crap.

    There was one software developer there with a reputation for shoddy work. He got promoted and put on one of the new whizzy products.

    I got stuck in maintenance of an old software product, which I thought was stupid as at that time I had a reputation for very high quality work. Clients often recommended my work. I left that company.

    I didn’t think they made good use of their resources.

    • I noticed another common misconception cited in many articles. “Knight has until settlement, T+3, to come up with the money for the bad trades.”

      That is false. For clearing and settlement, positions are marked-to-market at the end of each day. In fact, they now do intraday mark-to-market.

      Knight doesn’t get the perk of waiting. They had an immediate margin call. They would have already been shut down, if they couldn’t come up with the money.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>