>> that they are just as guilty of human-trafficking Cambodian brides, that some Koreans who were 'rescued' from a scam center don't actually want to go home
They don't want to go home because they fear the Chinese criminal gangs would follow them home in South Korea where they were recruited in the first place by Korean-speaking Chinese for debts they accrued while they were held captive. Others also fear they could also be held liable for participating in their fraud -- even if they were forced -- not because they love getting beaten.
Yes of course, but this claim is a response to Korea's claim that they are 'missing', presumed trafficked to scam compounds.
Cambodia is claiming they are 'detained' by immigration, and refuse to leave, making it sound like the Koreans themselves are the problem, trying to deflect from the real issue.
too invested in LazyVim.. just learned to use "edgy" with multiple windows: neo-tests, fs browser, diagnostic-trouble, Outline, etc.. and they are just too nice.
> Allowing in Chinese EVs into markets where there are important domestic auto manufacturers will be very bad for those domestic manufacturers.
I want a level playing field/market competition. Allowing China's illegal subsidies and anti-market tactics to dominate the global EV industry is very dangerous, which is why they are already countervailed in many developed countries.
Subsidies aren't necessarily bad, but it's become China's choice of blunt instrument to price out/drive out foreign competition.
China's EV subsidies are basically identical to US and EU subsidies. They offer tax credits and other perks to customers, just like in Western countries.
Subsidies are not the reason why China's EVs are cheap. The reason is that China has a much more competitive EV market than the US or EU. There are many manufacturers that are competing with one another, the charging infrastructure is much better than in the West, and Chinese cities heavily discourage internal combustion engine vehicles.
> Subsidies aren't necessarily bad, but it's become China's choice of blunt instrument to price out/drive out foreign competition.
China specifically encouraged foreign car companies to enter its market, most recently Tesla (which has done very well in China). Allowing foreign car companies to compete in the Chinese market was a major part of China's strategy to improve its own domestic manufacturing.
Subsidies are part of the reason. Definitely not the whole reason, but part of it The Chinese provinces pump a bunch of money in to try to make their region's car manufacturers succeed. And every province is doing it. So you have a bunch of carmakers competing for market while being kept alive, so they do the most rational thing possible to gain share: lower prices.
Of course, they're also just good at building things since they do so much of it. And cheaper labor. Much better supply chain.
Do you have any figures on the degree of subsidy? The impression I’ve gotten over the last year or so is that China has been phasing out their subsidies, which also went to companies like Tesla and Stellantis, and the main shift recently was the Make Smog Great Again bill over the summer setting the US back a generation which isn’t really a criticism of Chinese industrial policy.
> China's EV subsidies are basically identical to US and EU subsidies.
Not really, not much similarities between China and EU subsidies past 15 years. China's NEV subsidies are illegal because they are either conditioned on illegal tech transfer, local content requirement, or restrict market access. To give a high level view of the problems:
1) forced technology transfer/IP theft -- all foreign automakers/EV battery producers forced to give up IP to access China's market (and subsidies). This was litigated before the WTO by the EU in 2018 (see WT/DS549):
Hybrid in a Trade Squeeze, Keith Bradsher, Sept 5, 2011, NYT
... The Chinese government is refusing to let the Volt qualify for subsidies totaling up to $19,300 a car unless G.M. agrees to transfer the engineering secrets for one of the Volt’s three main technologies to a joint venture in China with a Chinese automaker, G.M. officials said.
2) Once foreign battery producers made IPR/IP concessions to access China's growing EV market and significant investment in battery production in China, they were effectively banned. All domestic, foreign automakers were likewise forced to switch to local champions, namely CATL/BYD, promoted by the gov't under MIIT's 2015 Regulation on Power Standard:
Power Play, Trefor Moss, May 17, 2018, WSJ
... China requires auto makers to use batteries from one of its approved suppliers if they want to be cleared to mass-produce electric cars and plug-in hybrids and to qualify for subsidies. These suppliers are all Chinese, so such global leaders as South Korea’s LG Chem Ltd and Japan’s Panasonic Corp. are excluded.
... Foreign batteries aren’t officially banned in China, but auto executives say that since 2016 they have been warned by government officials that they must use Chinese batteries in their China-built cars, or face repercussions. That has forced them to spend millions of dollars to redesign cars to work with inferior Chinese batteries, they say.
... “We want to comply, and we have to comply,” said one executive with a foreign car maker. “There’s no other option.”
3) China also made sure no Chinese consumers had access to EVs with batteries from foreign EV battery producers effectively creating a captive market of buyers for CATL/BYD.
Why a Chinese Company Dominates Electric Car Batteries. Keith Bradsher and Michael Forsythe, Dec 22, 2021, NYT
The government soon said electric car buyers could get subsidies only if the battery was made by a Chinese company. G.M., which had not been notified of the rule, started shipping Buick Velite electric cars in 2016 with batteries made in China by LG, a South Korean company.
Angry consumers and dealers complained that local officials were denying them subsidies, people familiar with the episode said. G.M. switched heavily to CATL for the huge Chinese market.
4) another fairly recent example of China's arbitrary regulatory barriers to keep out foreign competition, which was later dropped after the gov't found out their local "champion," CATL, couldn't pass the EV battery safety test:
Why a Chinese Company Dominates Electric Car Batteries.
... A rival had released a video suggesting that a technology used by the company, CATL, and other manufacturers could cause car fires. Imitating a Chinese government safety test, the rival had driven a nail through a battery cell, one of many in a typical electric car battery. The cell exploded in a fireball.
Chinese officials took swift action — by dropping the nail test, according to documents reviewed by The New York Times. The new regulation, released two months later, listed who had drafted it: First on the list, ahead of the government’s own vehicle testing agency, was CATL.
So these are very deliberately orchestrated mercantile policies to gain advantages with forced tech transfer, limited foreign competition, and subsidized overcapacity and export subsidies. It's just too bad that the existing global trade/subsidies regulation regime, aka, the WTO, doesn't have much effective enforcement tool to discourage/punish such behavior. EU's shortcoming IMO is their blind faith in the market and their belief that the market would autocorrect.
As of this week, EU has over 100+ countervailing measures (anti-dumping/anti-subsidy) in force against Chinese imports, ranging from ceramic tiles (AD560), to decor paper (AD712), to polyester yarn (AD690); in addition to few more dozens of on-going investigations from candles (AD726) to hardwood plywood (AD717).
Your example of this is from 2011. Chinese joint venture / technology transfer requirements in the automobile sector were eliminated several years ago.
This was a policy that was enacted when China first opened up. It was a fair deal: foreign companies got to exploit cheap Chinese labor, and in return, they transferred some IP to China. However, that IP transfer was never enough to make Chinese cars internationally competitive. Only the development of EVs - where China is the biggest R&D spender in the world - allowed China to leapfrog foreign manufacturers.
You also raise domestic component requirements to qualify for subsidies. The US does exactly the same thing.
> So these are very deliberately orchestrated mercantile policies to gain advantages with forced tech transfer, limited foreign competition, and subsidized overcapacity and export subsidies.
The problems with this explanation are:
1. China leads in EV R&D. Chalking up its dominance to theft of foreign IP doesn't make any sense.
2. China specifically invited Tesla to enter the country, and showered it with subsidies. As a result, Tesla has done very well in China. The foreign companies that are losing market share in China are the ones that missed the EV transition. VW dominated the Chinese auto market until just a few years ago. Now, it's heading to 0% market share. Why? It didn't focus on EVs.
3. China is not dumping its "excess capacity." Chinese companies are selling their cars in foreign markets at a substantial markup, and netting large profit margins in foreign markets. That's the opposite of how dumping works.
> As of this week, EU has over 100+ countervailing measures
This was a purely political decision. Automobile manufacturers in France were scared of Chinese competition and demanded protectionist measures. The Germans opposed the measures, because they sell lots of things in China and don't want to get into a trade war. The French won that fight at the EU level.
Sure, I'm giving you a chronological high level view of China's illegal practices past 15 years when China's NEV subsidies programs started.
> Chinese joint venture / technology transfer requirements in the automobile sector were eliminated several years ago.
This was never allowed and China upon China's 2001 Accession were required to phase them out 15 years ago, which China never did.
> You also raise domestic component requirements to qualify for subsidies. The US does exactly the same thing.
Sure, Biden's IRA passed in 2022 is a counter measure against China's domestic sourcing requirement since 2015.
> 1. China leads in EV R&D. Chalking up its dominance to theft of foreign IP doesn't make any sense.
False. Most, or close to 80% of all ACTIVE lithium ion battery patents are held by Japan and South Korea. The lithium ion battery industry was single-handledly created by Sony in Japan back in the early 1990's; quickly followed by South Korea. China was very late to the game and so far behind, which is why China forced tech transfer from Japan and South Korea since 2011 (see example #1) and effectively banned them in 2015 (example #2) -- still refuses to enforce IPR of foreigners, which isn't anything new. Japan + Korea in fact started going after the Chinese infringers only this year and in Europe -- already scored significant legal victories and sales injunctions in Germany. Many more coming and CATL isn't far in their legal pipelines.
> 2. China specifically invited Tesla to enter the country, and showered it with subsidies.
Sure, again Tesla is the only foreign automaker operating fully independently without forced tech transfer and other jazz in China. Tesla is an exception, not the norm. After EU filed WT/DS549, China promised to reform FIL and supposedly implemented in 2020/2021, but Tesla still remains the only foreign automaker without forced JV/tech transfer today.
> 3. That's the opposite of how dumping works.
Wrong again. That's exactly how dumping works and why there are over 6-7 dozens of anti-dumping measures against China in EU. Dumping doesn't depend on a markup or profit/loss, but on the undistorted "normal-value" born by market without gov't interference -- eg, price fixing or illegal subsidies.
> This was a purely political decision.
Again there are over 100+ ACTIVE anti-subsidies/dumping measures in force against China. It's just one of many and has been on EU's radar for 15 years.
> Sure, I'm giving you a chronological high level view of China's illegal practices past 15 years
The Chinese automobile industry in 2011 is hardly relevant to the EV industry in China today. The EV industry was not built by technology transfer requirements.
> This was never allowed
It was not only allowed, but actually viewed as a legitimate way for underdeveloped economies to develop.
> Sure, Biden's IRA passed in 2022 is a counter measure against China's domestic sourcing since 2015.
The US has all sorts of "Buy American" provisions and subsidies, going way back before 2022.
> False. Most, or close to 80% of all ACTIVE lithium ion battery patents are held by Japan and South Korea.
You're talking about the 1990s. I'm talking about now, 30 years later. The Chinese lead in battery technology and spend massive amounts of money on R&D.
> Sure, again Tesla is the only foreign automaker operating fully independently without forced tech transfer
Not true. First off, just as a footnote, there never was "forced technology transfer." Foreign companies knew what the regulations in China were and made a rational business decision to trade some amount of IP for access to cheap labor. Both sides benefited. But beyond that, nowadays, any foreign car company can operate in China without a local joint venture partner. Tesla was the first, but it's not the only one, and other companies are free to leave their joint ventures if they want to. Most of the large foreign automobile manufacturers in China have either acquired majority stakes in their China operations or have bought out their JV partners completely. This is the norm now.
> Wrong again. That's exactly how dumping works
No, dumping involves selling your products below the cost of manufacture in foreign markets. When you instead sell them at a substantial markup, that's called "making bank."
> The Chinese automobile industry in 2011 is hardly relevant to the EV industry in China today. The EV industry was not built by technology transfer requirements.
of course they are relevant and are built on tech transfer.
> It was not only allowed, but actually viewed as a legitimate way for underdeveloped economies to develop.
Again, this was illegal then and China was taken to the WTO in 2018 (WT/DS549 China — Certain Measures on the Transfer of Technology ) as I'd already explained (example #1).
> The US has all sorts of "Buy American" provisions and subsidies, going way back before 2022.
Sure, which EV or batteries before 2022?
> You're talking about the 1990s. I'm talking about now, 30 years later. The Chinese lead in battery technology and spend massive amounts of money on R&D.
Again, "ACTIVE" patents. Patents last just 20 years. Korea in particular have already dominated most automotive lithium ion battery patents 10-15 years. Again, this why is China forced tech transfer and "effectively" banned Japan + Korea battery makers when they realized their local "champions" still couldn't catch up or compete in 2015. China's obsession with LFP whose core patents expired last 3-4 years is likewise no coincidence and their "RECENT" investment in "post"-lithium ion batteries, such as sodium.
> Not true. First off, just as a footnote, there never was "forced technology transfer."
Again, see the WTO case WT/DS549 China — Certain Measures on the Transfer of Technology
> No, dumping involves selling your products below the cost of manufacture in foreign markets. When you instead sell them at a substantial markup, that's called "making bank."
Again this doesn't apply to China. China's "local price" or "cost of manufacture" is not considered a "normal value" as their entire supply-chain is distorted by gov't subsidies.
Europe's Basic Regulation (EU) 2016/1036 (anti-subsidy regulation) has specific provisions for non-market-economy countries -- ie China, Article 2(7a)
In the case of imports from non-market-economy countries (6), the normal value shall be determined on the basis of the price or constructed value in a market economy third country, or the price from such a third country to other countries, including the Union, or, where those are not possible, on any other reasonable basis, including the price actually paid or payable in the Union for the like product, duly adjusted if necessary to include a reasonable profit margin.
> of course they are relevant and are built on tech transfer.
The small amount of technology transfer that happened in 2011 for internal combustion engines is not relevant to the EV industry in 2025. It wasn't even enough back then to make China competitive in internal combustion engines.
> Again, this was illegal then and China was taken to the WTO in 2018
No, technology transfer is not blanket banned by the WTO. It's actually encouraged for developing countries.
> Again, see the WTO case WT/DS549 China — Certain Measures on the Transfer of Technology
I don't think the WTO has ruled on that complaint.
> Sure, which EV or batteries before 2022?
The US faced virtually no competition for EV vehicles from China before 2022. The protectionist measures came up as soon as the competition appeared.
> Again, "ACTIVE" patents. Patents last just 20 years
You're really going to claim that China does not lead in current-day research? You're reaching back to decades ago, when that wasn't the case, to dismiss the massive Chinese R&D on batteries today.
> China's "local price" or "cost of manufacture" is not considered a "normal value" as their entire supply-chain is distorted by gov't subsidies.
Government subsidies are fairly small, and are paid to the consumer (not the producer), so they don't affect the cost of exported goods. Chinese companies are selling EVs in Europe at far, far higher prices than in China.
As I said, some EU countries are afraid of legitimate competition in EVs, because their own EV industry is hopelessly backwards. They're raising protectionist barriers, and coming up with a fig leaf to justify it.
> The small amount of technology transfer that happened in 2011..
Sure, China's tech transfer in BEV/hybrid/battery tech had been going on since 2011 and continued until fairly recently, not just in 2011 (read NYT article cited in example #1).
> No, technology transfer is not blanket banned by the WTO. It's actually encouraged for developing countries.
Wrong. That only applies to LDC, or Least Developed Countries under the TRIPS Agreement, GATT 1994. And we aren't exactly talking about high-tech EV/battery or semi-manufacturing tech, but better farming, irrigating, fertilizer techniques in countries like Bukina Faso, Angola, or Haiti. The rules aren't for China to exploit.
China's "developing" status allows "additional transition time," in implementing necessary local IP regulatory regime. China's WTO Accession was in 2001 and this transition arrangement/allowance expired about 15 years ago and China is still inconsistent with the global standard (see for instance EU's recent anti-injunction suit). Nothing under the WTO allows China's illegal forced tech transfer/IP theft otherwise.
> I don't think the WTO has ruled on that complaint.
Sure, there was hardly anything for China to deny or defend. China instead agreed to reform their foreign investment laws (FIL): no further market restriction or force tech transfer, but took another 3 years to implement in 2020/2021 and many are still afraid to pull out some 4-5 years later; again Tesla being the only foreign automaker operating fully independently without a JV in China.
> The US faced virtually no competition for EV vehicles from China before 2022. The protectionist measures came up as soon as the competition appeared.
BYD's electric bus business in California has been around since the early 2010s. The Japanese + Korean battery producers, such as LG, Panasonic, Samsung, etc banned in China since 2015 under Xi's protectionism (aka, Made-In-China 2025), have been in the US without any restriction for well over a decade. The last American battery producer, A123, collapsed in 2012 and the foreign battery producers have dominated the US market without any restriction. Unlike China, America has no problem collaborating with foreign trading partners.
> You're really going to claim that China does not lead in current-day research?
Not really. Many are awe'ed as China's illegally subsidized overcapacity floods their local market, but often conflate the two: market domination vs. tech innovation. China's competitive edge is a function of China's illegal subsidies and protectionism. No evidence to believe Chinese EV/battery producers can compete without daddy Xi's big wallet or baton to keep away foreign competition.
> You're reaching back to decades ago, ..
Sure, most lithium ion battery tech used in today's EVs were developed over 20+ years ago and most relevant EV battery patents developed past 10-15 years are by Japanese/Koreans and they will dominate for quite some time. Again, this is why China has focused on LFP (effectively royalty free after 2022/2023 for export) or post-lithium batteries instead. In other word, China is likely to benefit from their R&D in post-lithium once they are commercialized and mass-produced at scale years down the road, if ever.
> Government subsidies are fairly small, and are paid to the consumer (not the producer), ...
Already cited an article showing that China's consumer direct subsidy was significant (see example #1, $19+K per EV which more or less continued until 2019) and another showing that the consumer subsidy was anything, but pro-consumer (see example #3) -- consumer's choice was limited to EVs with Chinese batteries to funnel subsidies back to their local battery "champions" only -- ie, anti-consumer. China's neo-mercantile economy prioritizes national "champions," not consumers.
> so they don't affect the cost of exported goods.
Of course they do. That's what EU's recent probe (2024/1866 and 2024/27) revealed and also why China hand them out like Halloween candies.
> Chinese companies are selling EVs in Europe at far, far higher prices than in China.
Again, China's local price or cost of production don't mean jack -- China is a non-market-economy and their local price/cost products are artificially deflated by the Chinese gov't's illegal subsidies. Also cited EU's Anti-dumping Regulation (2016/1036) explaining how the "normal value" is determined in such a case (see Article 2, Determination of dumping; A. NORMAL VALUE). It's well to remember however that Chinese EVs are countervailed under EU's Anti-Subsidy Regulation (2016/1037) where the price level/normal value is NOT a major consideration.
Thanks for playing, but I don't like talking in circles. Good bye.
> Wrong. That only applies to LDC, or Least Developed Countries under the TRIPS Agreement, GATT 1994.
No, it's strongly encouraged for least developed countries.
> Nothing under the WTO allows China's illegal forced tech transfer/IP theft otherwise.
There is no "forced tech transfer," and we're not discussing IP theft, which is not allowed by China's IP laws. Companies are making decisions to enter into IP sharing agreements with joint-venture partners, out of their own economic calculus. They're free to refuse.
> again Tesla being the only foreign automaker operating fully independently without a JV in China.
Not true. You need to update take a refresher on this issue. Toyota is setting up its own wholly owned operations in China, for example: [0].
> BYD's electric bus business in California has been around since the early 2010s.
This is a tiny niche in the US market. Meanwhile, US auto manufacturers have had large market shares in China for decades.
> Again, this is why China has focused on LFP (effectively royalty free after 2022/2023 for export) or post-lithium batteries instead. In other word, China is likely to benefit from their R&D in post-lithium once they are commercialized and mass-produced at scale years down the road, if ever.
So you're admitting that Chinese companies are respecting foreign IP and responding by innovating in areas not covered by that IP.
> Of course they do. That's what EU's recent probe (2024/1866 and 2024/27) revealed and also why China hand them out like Halloween candies.
No, tax rebates to customers in China do not reduce the cost of cars exported to Europe. China "hands [tax rebates to customers] out like Halloween candies" because it wants to rapidly transition to EVs (and has been incredibly successful in doing so).
> China's neo-mercantile economy prioritizes national "champions," not consumers.
Yet somehow, EVs are way cheaper in China than in the EU and US, and all the local companies are engaged in continuous price wars.
> Again, China's local price or cost of production don't mean jack -- China is a non-market-economy and their local price/cost products are artificially deflated by the Chinese gov't's illegal subsidies.
Those "illegal subsidies" (a loaded term) are overwhelmingly tax rebates to consumers - the exact same mechanism that the US and many EU countries use.
> Not really. Many are awe'ed as China's illegally subsidized overcapacity floods their local market, but often conflate the two: market domination vs. tech innovation. China's competitive edge is a function of China's illegal subsidies and protectionism. No evidence to believe Chinese EV/battery producers can compete without daddy Xi's big wallet or baton to keep away foreign competition.
This is just completely out of touch with reality. I'll just say two words: BYD, CATL. It's getting to the point that European countries are talking about encouraging battery technology transfer from China.
What do you mean by illegal? According to who? Like WTO or something? Genuinely asking, I don't know. I don't see how what they're doing is more illegal than paying corn farmers to make dumbass biofuel. But I don't know international trade law, hence the question.
the international legal standard for subsidies is based on the WTO's Subsidies and Countervailing Measures (SCM) Agreement and related anti-dumping regulation under GATT 1994.
In the US, it's implemented thru 19 U.S.C § 3571. The EU's foreign subsidies are regulated by Regulation (EU) 2016/1037 of the European Parliament and of the Council of 8 June 2016 based on the same WTO SCM Agreement.
While the WTO's regulations don't preclude local subsidy regulation, they must be consistent with the WTO's Agreement. In other word, any gov't subsidy favoring a "specific" company(ies) over domestic, foreign competitors, or distort market competition is an "actionable" offense and can be litigated before the WTO -- agriculture (quota based) and national security are however exempted. Others, such as export subsidies or local content requirement are prohibited under Article 3, "Prohibition" of the SCM.
Yeah, those evil corn subsidies. It's well to remember however that there is no international law against your gov't pissing away your tax payers hard-earned money -- so long as the product remains domestic and doesn't cause injuries to other trading partners.
Tesla's US subsidies mainly came from a ATVM loan from the DOE back in 2008/2009, repaid back in 2013 -- Ford ($6B) was the largest recipient. The DOE had/have other auto/EV/battery subsidies, equally available to foreign domestic producers/recipients, which is what the global subsidies standard requires. No special, or "specific" subsidy favoring Tesla over others (or local companies over foreign competitors). America's consumer purchase EV subsidies prior to the IRA were also neutral without preference to any "specific" company, industry or country of origin.
China's NEV subsidies were/are illegal and Chinese EVs are countervailed in many developed countries (eg, the EU, Turkiye, Canada, US) because they are misapplied in three key ways.
i. forced tech transfer: no subsidies or market access unless hybrid/BEV/batteries tech transfer to China since 2011; violates China's 2001 Accession Protocal (see Section 7, Non-Tariff Measures); litigated before the WTO by the EU (WT/DS549)
ii, local content requirement: no subsidies or license/permit to operate in China unless automakers' EVs used Chinese batteries made by Chinese local "champions" only, namely CATL, since 2015; all foreign battery producers effectively banned and all EV producers forced to switch to local battery suppliers; violates Article 3(a) Prohibition of the Subsidies and Countervailing Measures (SCM) Agreement
iii, export subsidies: subsidies given to MIC exporters to under price/under cut foreign competitors in markets abroad; EU Commission's counter measures: 2024/1866 and 2024/2754.
Tesla China benefited significantly from China's NEV subsidies and is likewise also countervailed in the EU, at the import tariff rate of 7+%. Tesla no longer imports EV from China in the US/Canada.
Some foolishly believed that the twin towers were invincible after the 1993 WTC bombing.
Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule.
Jersey City still was fine and 50 miles can be problematic for certain types of backup (failover) protocols. Regular tape backups would be fine but secondary databases can't be that far away (at least not at the time). I remember my boss at WFC saying that the most traffic over the data lines was in the middle of the night due to backups - not when everybody was in the office.
Companies big enough will lay the fibre. 50-100 miles of fibre isn't much if you are a billion dollar business. Even companies like BlackRock who had their own datacenters have since taken up Azure. 50 miles latency is negligible, even for databases.
The WTC attacks were in the 90s and early 00s and back then, 50 miles of latency was anything but negligible and Azure didn’t exist.
I know this because I was working on online systems back then.
I also vividly remember 9/11 and the days that followed. We had a satellite dish with multiple receivers (which wasn’t common back then) so had to run a 3rd party Linux box to descramble the single. We watch 24/7 global news on a crappy 5:4 CRT running Windows ME during the attack. Even in the UK, it was a somber and sobering experience.
For backups, latency is far less an issue than bandwidth.
Latency is defined by physics (speed of light, through specific conductors or fibres).
Bandwidth is determined by technology, which has advanced markedly in the past 25 years.
Even a quarter century ago, the bandwidth of a station wagon full of tapes was pretty good, even if the latency was high. Physical media transfer to multiple distant points remains a viable back-up strategy should you happen to be bandwidth-constrained in realtime links. The media themselves can be rotated / reused multiple times.
I’ve covered those points already in other responses. It’s probably worth reading them before assuming I don’t know the differences between the most basic of networking terms.
I was also specifically responding to the GPs point about latency for DB replication. For backups, one wouldn’t have used live replication back then (nor even now, outside of a few enterprise edge cases).
Snowmobile and its ilk was a hugely expensive service by the way. I’ve spent a fair amount of time migrating broadcasters and movie studios to AWS and it was always cheaper and less risky to upload petabytes from the data centre than it was to ship HDDs to AWS. So after conversations with our AWS account manager and running the numbers, we always ended up just uploading the stuff ourselves.
I’m sure there was a customer who benefited from such a service, but we had petabytes and it wasn’t us. And anyone I worked with who had larger storage requirements didn’t use vanilla S3, so I can’t see how Snowmobile would have worked for them either.
Switching gear was slower and laying new fibre wasn't an option for your average company. Particularly not point-to-point between your DB server and your replica.
So if real-time synchronization isn't practical, you are then left to do out-of-hours backups and there you start running into bandwidth issues of the time.
Plus long distance was mostly fibre already. And even regular electrical wires aren’t really much slower than fibre in term of latency. Parent probably meant bandwidth.
Copper doesn't work over these kinds of distances without powered switches, which adds latency. And laying fibre over several miles would be massively expensive. Well outside the realm of all but the largest of corporations. There's a reason buildings with high bandwidth constraints huddle near internet backbones.
What used to happen (and still does as far as I know, but I've been out of the networking game for a while now) is you'd get fibre laid between yourself and your ISP. So you're then subject to the latency of their networking stack. And that becomes a huge problem if you want to do any real-time work like DB replicas.
The only way to do automated off-site backups was via overnight snapshots. And you're then running into the bandwidth constraints of the era.
What most businesses ended up doing was tape backups and then physically driving it to another site -- ideally then storing it an fireproof safe. Only the largest companies could afford to push it over fibre.
To be fair, tape backups are very much ok as a disaster recovery solution. It's cheap once you have the tape drive. Bandwith is mostly fine if you want to read them sequentially. It's easy to store and handle and fairly resistant.
It's "only" poor if you need to restore some files in the middle or want your backup to act as a failover solution to minimise unavailability. But as a last resort solution in case of total destruction, it's pretty much unbeatable cost-wise.
G-Drive was apparently storing less than 1PB of data. That's less than 100 tapes. I guess some files were fairly stable so completely manageable with a dozen of tape drives, delta storage and proper rotation. We are talking of a budget of what 50k$ to 100k$. That's peanuts for a project of this size. Plus the tech has existed for ages and I guess you can find plenty of former data center employees with experience handling this kind of setup. They really have no excuse.
The suits are stingy when it's not an active emergency. A former employer declined my request for $2K for a second NAS to replicate our company's main data store. This was just days after a harrowing data recovery of critical from a failing WD Green that was never backed up. Once the data was on a RAID mirror and accessible to employees again, there was no active emergency, and the budget dried up.
I don't know. I guess that for all intents and purposes I'm what you would call a suit nowadays. I'm far from a big shot at my admittedly big company but 50k$ is pretty much pocket change on this kind of project. My cloud bill has more yearly fluctuation than that. Next to the cost of employees, it's nothing.
> There's a reason buildings with high bandwidth constraints huddle near internet backbones.
Yeah because interaction latency matters and legacy/already buried fiber is expensive to rent so you might as well put the facility in range of (not-yet-expensive) 20km optics.
> Copper doesn't work over these kinds of distances without powered switches, which adds latency.
You need a retimer, which adds on the order of 5~20 bits of latency.
> And that becomes a huge problem if you want to do any real-time work like DB replicas.
Almost no application would actually require "zero lost data", so you could get away with streaming a WAL or other form of reliably-replayable transaction log and cap it to an acceptable number of milliseconds of data loss window before applying blocking back pressure.
Usually it'd be easy to tolerate enough for the around 3 RTTs you'd really want to keep to cover all usual packet loss without triggering back pressure.
Sure, such a setup isn't cheap, but it's (for a long while now) cheaper than manually fixing the data from the day your primary burned down.
Yes but good luck trying to get funding approval. There is a funny saying that wealthy people don't become wealthy by giving their wealth away. I think it applies to companies even more.
In the US, dark fiber will run you around 100k / mile. Thats expensive for anyone even if they can afford it. I worked in HFT for 15 years and we had tons of it.
DWDM per-wavelength costs are way, way lower than that, and, with the optional addition of encryption, perfectly secure and fast enough for disk replication for most storage farms. I've been there and done it.
Assuming that dark fiber is actually dark (without amplifiers/repeaters), I'd wonder how they'd justify the 4 orders of magnitude (99.99%!) profit margin on said fiber.
That already includes one order of magnitude between the 12th-of-a-ribbon clad-fiber and opportunistically (when someone already digs the ground up) buried speed pipe with 144-core cable.
So that's 5 million bucks for 50 miles? If there are other costs not being accounted for, like paying for the right-of-way that's one thing, but I would think big companies or in this case, a national government, could afford that bill.
Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule.
IIRC, multiple IBM mainframes can be setup so they run and are administered as a single system for DR, but there are distance limits.
A Geographically-Dispersed Parallel Sysplex for z\OS mainframes, which IBM has been selling since the '90s, can have redundancy out to about 120 miles.
At a former employer, we used a datacenter in East Brunswick NJ that had mainframes in sysplex with partners in lower manhattan.
If you have to mirror synchronously the _maximum_ distances for other systems (e.g. storage mirroring with NetApp SnapMirror Synchronous, IBM PPRC, EMC SRDF/S) are all in this range.
But an important factor is, that performance will degrade with every microsecond latency added as the active node for the transaction will have to wait for the acknowledgement of the mirror node (~2*RTT).
You can mirror synchronously that distance, but the question is if you can accept the impact.
That's not to say that one shouldn't create a replica in this case. If necessary, synchronize synchronous to a nearby DC and asynchrone to a remote one.
>Some foolishly believed that the twin towers were invincible after the 1993 WTC bombing.
I was told right after the bombing, by someone with a large engineering firm (Schlumberger or Bechtel), that the bombers could have brought the building down had they done it right.
It's mostly about cost and market access to China.
Most smartphone supply-chain for Samsung and Apple exist outside China -- primarily in Japan (camera, sensors), South Korea (DRAM/NAND, OLED), and the US (various ICs fabbed at TSMC in Taiwan). There are quite a few reliable estimates/teardowns showing that these three countries account for close to about 90% of iPhone BOM (bill of materials). That's one reason why Samsung's smartphone unit was able to pull out of China without much disruption back in 2019 -- ie, low dependence on China.
I feel that Apple has pushed this misleading narrative a bit too long to defend their massive China outsourcing.
Occam's razor: the wall is most probably the cause of the unnecessary explosion that killed 179 people. The airport built ILS, or localizer, on unnecessarily over-engineered concrete structure where there shouldn't have been any obstruction. The ILS are supposed to be built on level surface or "frangile" structure so they can be easily destroyed when there is an overrun. There are reportedly at least 4 other airports with such obstructions in South Korea -- at Yeosu, it's 4 meters high (also concrete foundation)[1].
<strikethrough>There was a similar accident in Hiroshima, Japan several years ago: Airbus A320-200 skidded past the end of the runway at similar speed and struck down ILS. It eventually stopped -- damaged the airplane but no fatality.</strikethrough>
1. Localizer at Yeosu Airport, similar to Muan's, raises safety concerns, 2025.01.02 (23:58), KBS News.
> There was a similar accident in Hiroshima, Japan several years ago: Airbus A320-200 skidded past the end of the runway at similar speed and struck down ILS. It eventually stopped -- damaged the airplane but no fatality.
Are you talking about Asiana Airlines 162? It hit localizer on its way to the runway because it came in too low. It then hit the runway, skidded on the runway, and stopped about halfway (after veering off the runway at the last moment).
If the same thing happened in Muan, the plane would have hit the localizer and then touched down, stopping in the runway. The fact that the localizer's base was concrete wouldn't have mattered because that's not where the plane would hit it.
If the wheels drop off my car at 100km/h and I lose control and hit a wall, is the wall the cause of the accident?
The barrier was 250m away from the end of the runway, the extra 50m if following regulations wouldn’t have changed the outcome. And if the wall wasn’t there, the plane would dive right into a highway anyway. That’s the point.
Technically correct yet missing the point. The plane could have disintegrated in a dozen other ways if the wall wasn’t there, or something else could be in the way.
I think the fact we have dramatic footage of the crash makes it a very attractive topic for engagement. News are spinning it into something it is not. A handful of massive errors happened and we know nothing yet, the focus on the structure for being in the way just makes for good clickbait, gives people an easy target to blame - the airport, engineers, regulators (doesn’t really matter who), and something to get riled up about.
bad analogy. a better one would be: a wheel fell of an F1 car, car hit perimeter wall, driver dies. should we maybe put a crash barrier in front of the wall?
This airport is basically already complying with the highest possible standard [1]; so in your analogy the crash barrier already exists, the equivalent argument would be for the track to have a 200m run-off area all around.
It's not feasible to plan for every possible freak occurrence, an accident like this is only possible after a long list of other safety procedures have failed (as is often the case for aviation).
Planes sometimes overshoot the runway, building an unecessary wall at the end of it might comply with a standard, but that doesn't make it a good idea.
Maybe that section of highway could easily be cleared in time with a warning system, like when we warn for crossing trains. A wall doesn't have to be the only solution.
I suppose we have to get into the engineering thought process of these Asian cultures.
Where did the idea of fortifying Localizer (LOC) come from, was there prior art specifying any degree of fortification (or lack) ? Perhaps these kind of radar (adjacent) installations are traditionally fortified, like maybe the civil air services were influenced from militry air services were things tend to be overengineered?
Perhaps saving the LOC is more important than saving a single aircraft? Some idea like this piece of infrastructure being more important that any single aircraft's safety?
The point is we might not see the reason why fortifying the LOC was obvious or straight forward to the Korean engineers.
Also, I've seen no credible evidence to support the idea the LOC is supposed to be built on a level surface, or built in a destructable way. Addressing the first point, the LOC makes sense to be slightly elevated given how they operate with near field low power radio waves. That said, these kind of slightly elevated instrument might not require an earthen mound, sourounded by concreate walls. Which goes to the second point, I can think of reasons to have these kind of LOC runways homing antenas destructable, and I can see them being robustly durrable. This thing was way past the end of the runway, and as unpo0pular as it might seems... it's a very bad thing to run out of runway. Somebody else wrote that some airports have a lack of open field beyond the end of the runway, and so it's a persuasive argument stuff could be built out there.
The pilots on 9/11 were intentionally directing their aircraft toward specific structures to maximise destruction, casualties, and terrorist impact. That was their express intent and mission for which they had specifically trained.
In all likelihood the pilot of Jeju 2216 was not specifically directing his aircraft to the nonfrangible ILS Muan Murder Wall. There is no way to charitably argue otherwise.
China is quite anti-competitive. BYD/CATL's success, which comes from their battery market dominance, is a largely outcome of the CCP's protectionism past ~10 years.
They don't want to go home because they fear the Chinese criminal gangs would follow them home in South Korea where they were recruited in the first place by Korean-speaking Chinese for debts they accrued while they were held captive. Others also fear they could also be held liable for participating in their fraud -- even if they were forced -- not because they love getting beaten.