How One Piece of Hardware Took Down a $6 Trillion Stock Market, , on October 5, 2020 at 12:48 am

Financial News
By
On October 5, 2020
In Financial News
Tags:

(Bloomberg) — At 7:04 a.m. on an autumn Thursday in Tokyo, the stewards of the world’s third-largest equity market realized they had a problem.A data device critical to the Tokyo Stock Exchange’s trading system had malfunctioned, and the automatic backup had failed to kick in. It was less than an hour before the system, called Arrowhead, was due to start processing orders in the $6 trillion equity market. Exchange officials could see no solution.The full-day shutdown that ensued was the longest since the exchange switched to a fully electronic trading system in 1999. It drew criticism from market participants and authorities and shone a spotlight on a lesser-discussed vulnerability in the world’s financial plumbing — not software or security risks but the danger when one of hundreds of pieces of hardware that make up a trading system decides to give up the ghost.“Exchanges are a crucial part of market infrastructure and it’s unacceptable that trading opportunities were denied,” Finance Minister Taro Aso told reporters in Tokyo. “You’re dealing with machines so it’s always possible they will break. They need to create the infrastructure with that possibility of a breakdown in mind.”The TSE’s Arrowhead system launched to much fanfare in 2010, billed as a modern-day solution after a series of outages on an older system embarrassed the exchange in the 2000s. The “arrow” symbolizes speed of order processing, while the “head” suggests robustness and reliability, according to the exchange. The system of roughly 350 servers that process buy and sell orders had had a few hiccups but no major outages in its first decade.Read: Tokyo Stock Exchange to Resume Trading Friday After OutageThat all changed on Thursday, when a piece of hardware called the No. 1 shared disk device, one of two square-shaped data-storage boxes, detected a memory error. These devices store management data used across the servers, and distribute information such as commands and ID and password combinations for terminals that monitor trades.When the error happened, the system should have carried out what’s called a failover — an automatic switching to the No. 2 device. But for reasons the exchange’s executives couldn’t explain, that process also failed. That had a knock-on effect on servers called information distribution gateways that are meant to send market information to traders.Disappearing DataAt 8 a.m., traders preparing at their desks for the market open an hour later should have been seeing indicative prices on their terminals as orders were processed. But many saw nothing, while others reported seeing data appearing and disappearing. They had no idea if the information was accurate.A minute later, the bourse made its first communication, informing systems administrators at securities firms that there had been an issue. At some brokerages, that didn’t immediately filter down to befuddled trading desks.At about 8:05 a.m., Twitter — often used by traders to communicate outside of more official communication channels monitored by compliance — began to buzz with rumors of an issue. Traders described a growing sense of confusion as few answers came from the bourse.“We didn’t know if it was our system or the exchange,” said Masaya Akiba, a broker at Marusan Securities Co.’s stock-trading department. “We only confirmed it when the exchange put out a release.”At 8:36 a.m., the bourse finally informed securities firms that trading would be halted. Three minutes later, it issued a press release on its public website — although only in Japanese. A confusingly translated English release wouldn’t follow for more than 90 minutes.It was the first time in almost fifteen years that the exchange had suffered a complete trading outage. The Tokyo bourse has a policy of not shutting even during natural disasters, so for many on trading floors in the capital, this experience was a first.Historic DecisionSome market participants fumed at the closure. Others, with nothing to do, occupied their time by reading research notes or trading commodities.“I didn’t think much of it at first,” said Kiyoshi Ishigane, the chief fund manager at Mitsubishi UFJ Kokusai Asset Management Co. in Tokyo. “Previous outages were quickly resolved so I assumed orders would just be delayed.”In 2012, after the switchover to Arrowhead, the exchange had quickly resolved limited issues. Many expected the bourse to do the same this time, too.But as the hours passed, Hajime Sakai, the chief fund manager at Mito Securities Co., grew increasingly uneasy.“I really couldn’t pay attention to much else,” he said. “I wasn’t like, ‘Open the market!’ It was more like, ‘whichever it is, make your call on it, fast.’”The call was a daunting one. After the failed switch to the backup, the exchange had manually forced a switchover to the No. 2 shared disk device. At this point, the administrators had a choice: they could seek to restart trading, but this would have entailed a full reset of the system — shutting down the power and rebooting.Data for orders already received from securities firms would have been lost, without having been canceled. That would have led to anarchy, securities firms told the exchange. After speaking with market participants, the exchange made its decision: trading would be called off for the entire day.Many in the market say they were relieved. A call to resume trading would have been chaotic, said one worker at a Tokyo-based brokerage, with no way to tell which existing client orders remained active, while also trying to process new asks and bids.Technical DiscussionAt 4:30 p.m. local time, four TSE executives, including Chief Executive Officer Koichiro Miyahara and Chief Information Officer Ryusuke Yokoyama, faced journalists at the exchange to explain the outage. In a briefing that lasted about 100 minutes, they bowed in apology in front of the crowded room before going into a detailed technical discussion of the breakdown.If the bourse was criticized for its communications earlier in the day, it won praise for how it handled the press conference. The executives answered questions from the media with relative ease, discussing areas such as systems architecture in highly technical terms. They also squarely accepted responsibility for the incident, rather than trying to deflect blame onto the system vendor Fujitsu Ltd. It bore little resemblance to gaffe-filled briefings by other Japanese firms in the past. On Twitter, the Japanese public voiced its approval.“Management explained very clearly during the briefing last night,” said Megumi Takarada, a senior analyst at Toyo Securities Co. in Tokyo. “The briefing provided some reassurance that management clearly understands the issue.”Later in the evening, the announcement came that the bourse would restart trading Friday. While that passed without issue, many questions remain unanswered. The Financial Services Agency has ordered the exchange to issue a report on the outage, according to local media, which may give further insight on some of the issues. But one of the biggest is whether the same kind of hardware-driven failure could happen in other stock markets. For one strategist, it almost certainly could — but that’s not something to worry too much about.“There’s nothing uniquely Japanese about this,” said Nicholas Smith of CLSA Ltd. in Tokyo. “I think we’ve just got to put that in the box of ‘stuff happens.’ These things happen. They shouldn’t, but they do.”(Updates with FSA ordering report on the outage in 25th paragraph)For more articles like this, please visit us at bloomberg.comSubscribe now to stay ahead with the most trusted business news source.©2020 Bloomberg L.P.,

How One Piece of Hardware Took Down a $6 Trillion Stock Market(Bloomberg) — At 7:04 a.m. on an autumn Thursday in Tokyo, the stewards of the world’s third-largest equity market realized they had a problem.A data device critical to the Tokyo Stock Exchange’s trading system had malfunctioned, and the automatic backup had failed to kick in. It was less than an hour before the system, called Arrowhead, was due to start processing orders in the $6 trillion equity market. Exchange officials could see no solution.The full-day shutdown that ensued was the longest since the exchange switched to a fully electronic trading system in 1999. It drew criticism from market participants and authorities and shone a spotlight on a lesser-discussed vulnerability in the world’s financial plumbing — not software or security risks but the danger when one of hundreds of pieces of hardware that make up a trading system decides to give up the ghost.“Exchanges are a crucial part of market infrastructure and it’s unacceptable that trading opportunities were denied,” Finance Minister Taro Aso told reporters in Tokyo. “You’re dealing with machines so it’s always possible they will break. They need to create the infrastructure with that possibility of a breakdown in mind.”The TSE’s Arrowhead system launched to much fanfare in 2010, billed as a modern-day solution after a series of outages on an older system embarrassed the exchange in the 2000s. The “arrow” symbolizes speed of order processing, while the “head” suggests robustness and reliability, according to the exchange. The system of roughly 350 servers that process buy and sell orders had had a few hiccups but no major outages in its first decade.Read: Tokyo Stock Exchange to Resume Trading Friday After OutageThat all changed on Thursday, when a piece of hardware called the No. 1 shared disk device, one of two square-shaped data-storage boxes, detected a memory error. These devices store management data used across the servers, and distribute information such as commands and ID and password combinations for terminals that monitor trades.When the error happened, the system should have carried out what’s called a failover — an automatic switching to the No. 2 device. But for reasons the exchange’s executives couldn’t explain, that process also failed. That had a knock-on effect on servers called information distribution gateways that are meant to send market information to traders.Disappearing DataAt 8 a.m., traders preparing at their desks for the market open an hour later should have been seeing indicative prices on their terminals as orders were processed. But many saw nothing, while others reported seeing data appearing and disappearing. They had no idea if the information was accurate.A minute later, the bourse made its first communication, informing systems administrators at securities firms that there had been an issue. At some brokerages, that didn’t immediately filter down to befuddled trading desks.At about 8:05 a.m., Twitter — often used by traders to communicate outside of more official communication channels monitored by compliance — began to buzz with rumors of an issue. Traders described a growing sense of confusion as few answers came from the bourse.“We didn’t know if it was our system or the exchange,” said Masaya Akiba, a broker at Marusan Securities Co.’s stock-trading department. “We only confirmed it when the exchange put out a release.”At 8:36 a.m., the bourse finally informed securities firms that trading would be halted. Three minutes later, it issued a press release on its public website — although only in Japanese. A confusingly translated English release wouldn’t follow for more than 90 minutes.It was the first time in almost fifteen years that the exchange had suffered a complete trading outage. The Tokyo bourse has a policy of not shutting even during natural disasters, so for many on trading floors in the capital, this experience was a first.Historic DecisionSome market participants fumed at the closure. Others, with nothing to do, occupied their time by reading research notes or trading commodities.“I didn’t think much of it at first,” said Kiyoshi Ishigane, the chief fund manager at Mitsubishi UFJ Kokusai Asset Management Co. in Tokyo. “Previous outages were quickly resolved so I assumed orders would just be delayed.”In 2012, after the switchover to Arrowhead, the exchange had quickly resolved limited issues. Many expected the bourse to do the same this time, too.But as the hours passed, Hajime Sakai, the chief fund manager at Mito Securities Co., grew increasingly uneasy.“I really couldn’t pay attention to much else,” he said. “I wasn’t like, ‘Open the market!’ It was more like, ‘whichever it is, make your call on it, fast.’”The call was a daunting one. After the failed switch to the backup, the exchange had manually forced a switchover to the No. 2 shared disk device. At this point, the administrators had a choice: they could seek to restart trading, but this would have entailed a full reset of the system — shutting down the power and rebooting.Data for orders already received from securities firms would have been lost, without having been canceled. That would have led to anarchy, securities firms told the exchange. After speaking with market participants, the exchange made its decision: trading would be called off for the entire day.Many in the market say they were relieved. A call to resume trading would have been chaotic, said one worker at a Tokyo-based brokerage, with no way to tell which existing client orders remained active, while also trying to process new asks and bids.Technical DiscussionAt 4:30 p.m. local time, four TSE executives, including Chief Executive Officer Koichiro Miyahara and Chief Information Officer Ryusuke Yokoyama, faced journalists at the exchange to explain the outage. In a briefing that lasted about 100 minutes, they bowed in apology in front of the crowded room before going into a detailed technical discussion of the breakdown.If the bourse was criticized for its communications earlier in the day, it won praise for how it handled the press conference. The executives answered questions from the media with relative ease, discussing areas such as systems architecture in highly technical terms. They also squarely accepted responsibility for the incident, rather than trying to deflect blame onto the system vendor Fujitsu Ltd. It bore little resemblance to gaffe-filled briefings by other Japanese firms in the past. On Twitter, the Japanese public voiced its approval.“Management explained very clearly during the briefing last night,” said Megumi Takarada, a senior analyst at Toyo Securities Co. in Tokyo. “The briefing provided some reassurance that management clearly understands the issue.”Later in the evening, the announcement came that the bourse would restart trading Friday. While that passed without issue, many questions remain unanswered. The Financial Services Agency has ordered the exchange to issue a report on the outage, according to local media, which may give further insight on some of the issues. But one of the biggest is whether the same kind of hardware-driven failure could happen in other stock markets. For one strategist, it almost certainly could — but that’s not something to worry too much about.“There’s nothing uniquely Japanese about this,” said Nicholas Smith of CLSA Ltd. in Tokyo. “I think we’ve just got to put that in the box of ‘stuff happens.’ These things happen. They shouldn’t, but they do.”(Updates with FSA ordering report on the outage in 25th paragraph)For more articles like this, please visit us at bloomberg.comSubscribe now to stay ahead with the most trusted business news source.©2020 Bloomberg L.P.

,

Instant Quote

Enter the Stock Symbol.

Select the Exchange.

Select the Type of Security.

Please enter your First Name.

Please enter your Last Name.

Please enter your phone number.

Please enter your Email Address.

Please enter or select the Total Number of Shares you own.

Please enter or select the Desired Loan Amount you are seeking.

Please select the Loan Purpose.

Please select if you are an Officer/Director.

High West Capital Partners, LLC may only offer certain information to persons who are “Accredited Investors” and/or “Qualified Clients” as those terms are defined under applicable Federal Securities Laws. In order to be an “Accredited Investor” and/or a “Qualified Client”, you must meet the criteria identified in ONE OR MORE of the following categories/paragraphs numbered 1-20 below.

High West Capital Partners, LLC cannot provide you with any information regarding its Loan Programs or Investment Products unless you meet one or more of the following criteria. Furthermore, Foreign nationals who may be exempt from qualifying as a U.S. Accredited Investor are still required to meet the established criteria, in accordance with High West Capital Partners, LLC’s internal lending policies. High West Capital Partners, LLC will not provide information or lend to any individual and/or entity that does not meet one or more of the following criteria:

1) Individual with Net Worth in excess of $1.0 million. A natural person (not an entity) whose net worth, or joint net worth with his or her spouse, at the time of purchase exceeds $1,000,000 USD. (In calculating net worth, you may include your equity in personal property and real estate, including your principal residence, cash, short-term investments, stock and securities. Your inclusion of equity in personal property and real estate should be based on the fair market value of such property less debt secured by such property.)

2) Individual with $200,000 individual Annual Income. A natural person (not an entity) who had individual income of more than $200,000 in each of the preceding two calendar years, and has a reasonable expectation of reaching the same income level in the current year.

3) Individual with $300,000 Joint Annual Income. A natural person (not an entity) who had joint income with his or her spouse in excess of $300,000 in each of the preceding two calendar years, and has a reasonable expectation of reaching the same income level in the current year.

4) Corporations or Partnerships. A corporation, partnership, or similar entity that has in excess of $5 million of assets and was not formed for the specific purpose of acquiring an interest in the Corporation or Partnership.

5) Revocable Trust. A trust that is revocable by its grantors and each of whose grantors is an Accredited Investor as defined in one or more of the other categories/paragraphs numbered herein.

6) Irrevocable Trust. A trust (other than an ERISA plan) that (a)is not revocable by its grantors, (b) has in excess of $5 million of assets, (c) was not formed for the specific purpose of acquiring an interest, and (d) is directed by a person who has such knowledge and experience in financial and business matters that such person is capable of evaluating the merits and risks of an investment in the Trust.

7) IRA or Similar Benefit Plan. An IRA, Keogh or similar benefit plan that covers only a single natural person who is an Accredited Investor, as defined in one or more of the other categories/paragraphs numbered herein.

8) Participant-Directed Employee Benefit Plan Account. A participant-directed employee benefit plan investing at the direction of, and for the account of, a participant who is an Accredited Investor, as that term is defined in one or more of the other categories/paragraphs numbered herein.

9) Other ERISA Plan. An employee benefit plan within the meaning of Title I of the ERISA Act other than a participant-directed plan with total assets in excess of $5 million or for which investment decisions (including the decision to purchase an interest) are made by a bank, registered investment adviser, savings and loan association, or insurance company.

10) Government Benefit Plan. A plan established and maintained by a state, municipality, or any agency of a state or municipality, for the benefit of its employees, with total assets in excess of $5 million.

11) Non-Profit Entity. An organization described in Section 501(c)(3) of the Internal Revenue Code, as amended, with total assets in excess of $5 million (including endowment, annuity and life income funds), as shown by the organization’s most recent audited financial statements.

12) A bank, as defined in Section 3(a)(2) of the Securities Act (whether acting for its own account or in a fiduciary capacity).

13) A savings and loan association or similar institution, as defined in Section 3(a)(5)(A) of the Securities Act (whether acting for its own account or in a fiduciary capacity).

14) A broker-dealer registered under the Exchange Act.

15) An insurance company, as defined in Section 2(13) of the Securities Act.

16) A “business development company,” as defined in Section 2(a)(48) of the Investment Company Act.

17) A small business investment company licensed under Section 301 (c) or (d) of the Small Business Investment Act of 1958.

18) A “private business development company” as defined in Section 202(a)(22) of the Advisers Act.

19) Executive Officer or Director. A natural person who is an executive officer, director or general partner of the Partnership or the General Partner, and is an Accredited Investor as that term is defined in one or more of the categories/paragraphs numbered herein.

20) Entity Owned Entirely By Accredited Investors. A corporation, partnership, private investment company or similar entity each of whose equity owners is a natural person who is an Accredited Investor, as that term is defined in one or more of the categories/paragraphs numbered herein.

Please read the notice above and check the box below to continue.

Singapore

168 Robinson Road,
Capital Tower, Singapore 068912
+65 3105 1295

Taiwan

5th Floor, No. 1-8, Section 5, Zhongxiao East Road, Taipei

Hong Kong

R91, 3rd Floor,
Eton Tower, 8 Hysan Ave.
Causeway Bay, Hong Kong
+852 3002 4462

Australia

44 Martin Place, Sydney 2000 Australia
+02 8319 3232

Indonesia

Millennium Centennial Center, 38th Floor, Jl. Jend. Sudirman Kav. 25
Jakarta 12920, Indonesia

Market Coverage