This is speculation, of course, but: there are several large Internet Exchanges in NYC, and US/EU peering points. It's probable that a fiber cut could have cut several other links, and the failover links are now completely overwhelmed.
I'm not knowledgeable enough to answer that question. However, this is the only Twitter comment that mentions this on their page, and I don't have access to a Verizon account to verify their outage status memo. We had a flurry of employees saying their connections were going haywire to the business and this is the only thing I've been able to gather outside of downdetector.
Apparently, Verizon's outage page doesn't actually say if there's an outage. You have to log in to see if there's an outage. Too bad logins are failing after ~10 minutes for me.
When houses are created there are building codes to adhere to. For example, when there is a water pipe behind the walls a plumber (a good one) would also put a plate in front of it. In this way, once the walls are in place, drilling away into the wall, or nailing, would hit the metal plate, thus provides some safety mechanism. As another example, underground, in public areas are markings (or at least procedures) to identify such areas gas pipe areas before excavating.
How are the fiber optics being cut? Are there no procedures for this or is there some challenges how the infra is setup today?
I'm probably the only person reading this thread who owns a backhoe, and it's only a small one. There's really no material that will resist being dug through and is also practical to bury over miles and miles of fiber. There obviously are processes and measures in place to prevent "backhoe-fade" events, including armored cable, tracer wires and tape above the main cable but humans being humans things sometimes don't work out. Contractors can be liable for the cost to spice repairs which will be very costly so they're incented to not cut fibers.
Backhoes also can't be blamed for every outage. Cars hitting poles that carry aerial fiber, people ripping up fiber thinking it is copper they can steal, boat anchors at landing sites, etc.
I thought they were supposed to lay that "cable/pipe/Jimmy Hoffa buried here" warning tape somewhere above the fragile thing that's buried, so if digging you'd catch the tape, see it, and stop?
Used to work for a few ISPs. 90% of the fiber cuts that were experience while I worked there were from municipal construction crews who either weren't fully paying attention when working around fiber lines, or didn't know they were there in the first place.
The lines are supposed to be known/marked to some extent but you can't avoid the human factor.
In our worst case the same city cut the fiber line in multiple places over the course of a few weeks. Our crews would finish patching one fiber break only for the same (city) crew to cut it again a few hours later.
I wasn't on the ground so I don't know exactly how they managed it, but it was a real headache for us to manage on the customer end.
Scale: wrapping all fiber optic cables in a protective metal sheath would be cost prohibitive. Whereas adding a metal sheet to prevent a house from flooding or someone from getting electrocuted makes sense.
One idea, borrowed from some chainsaw safety pants, would be to wrap the cable in a fiber web sheath that tangles in whatever cutting equipment
Fibre isn’t usually cut by something like a chainsaw but rather by something like a backhoe, rodent’s teeth (squirrels on overhead lines and rats if they run through utility tunnels), or a gunshot. I doubt these super-long chainsaw chaps would even stand up to a determined rodent.
Literally took off the pants testing that! I'm unsure how much is worth getting paid to run a self-test. This product looks pretty neat. Half joking, anyone ever, or witnessed getting eletrocuted trying to get the internet / computer working again?
This spun threads reminds me of nature. Spiders, silk works spinning web. Wonder if that's where the inspiration came from.
Old Joke: How do you get rescued from a desert island if all you have is some networking gear?
Simple! Bury a network cable and wait for the backhoe to show up.
Backhoes are VERY strong and will easily go through gas pipe let alone anything that might protect fiber. And if the fiber is not where the plans say it is, even calling 811 won't save you.
Even worse is that (here in Canada at least) we have infrastructure surveyed, its GPS coords recorded, then manually entered into CAD (or similar) drawings using a lot of guess work. Things like metal detectors are used to more or less guess that a detected object is in fact a gas line or something, GPS location is recorded, then someone plays connect the dots in CAD. It's unreal. So many opportunities to introduce error, and the initial locating of the underground object could easily be incorrect.
I know newer infrastructure is more diligently recorded, but there is a TON of old crap in the ground ready to be ripped up.
We did some work on the foundation right next to the gas meter which had a pipe going down - which unexpectedly went off direct into the yard to who knows where. They hand dug until they were confident (still took out the power line to the garage. Twice).
The vast majority of fiber optic cuts are someone digging through the cable underground.
You can't realistically prevent this with additional armoring, because it's quite a different failure to someone drilling in the wrong place - the forces are a lot higher, or heavy machinery is involved etc. (see also: the rate at which workers go through water mains in footpaths).
Fiber optics rely on obscurity to remain secure. Not far from my home, there’s at least 2-3 significant fiber optics buried in a trench next to the street in the outskirts of the city, probably 30-40 inches under the ground.
The dig safe people can and do mark them, but people tend to do stupid things.
Some highways have fiber rights of way too. You can see fiber conduit on bridges on the mass pike, for example.
I've heard of such a thing but never seen it. I was a satellite installer and drilled into a lot of walls. I'd just drill thru the drywall then feel around with a drill bit.
I've never heard of it for plumbing, but kinda in a similar vein most electrical codes require a certain setback for all cabling running through studs and protection where it can't be maintained.
If the cable is passing through a stud less than 1.25" from either face of the stud (giving you ~1.75" once drywall is installed), then a metal plate has to be installed to stop someone trying to hang a picture or shelf or something innocuous up from driving a screw into the cable.
Lots of ways, unfortunately. But generally a quality network will be built to have physical redundancy. Yet, the ISP game at the top is all about cost cutting. So we'll never get what we paid for back in the 90s.
Plates are generally only placed when the pipe or wire is within drywall-screw depth once you add drywall to the studs. They aren't infallible -- For example, they don't protect against carpenters with nailguns patching sheathing on the outside who miss the stud entirely and hit a water pipe instead (you may wonder how I know this..)
As for fiber cuts, one case I'm aware of involved a large diameter concrete drill/hole saw with a diamond-tipped blade. Not much you can do to armor against that.
Liability is pretty good protection -- pretty much everyone that uses heavy equipment to dig is aware of the need to use the "call before you dig" numbers to have someone come out and mark underground utilities.
There's always the person with a rented ditch digger or over confidence "Well I dug out here last year and didn't hit anything, so I can dig again" who doesn't call or notice the trace cable, but overall that service works pretty well at preventing disruptions.
In a most states, there are blue stake laws requiring you to call and set an appointment to have existing utilities marked with spray paint before you dig. Even though this service is usually free to the contractor, it's sometimes viewed as one more hassle. Usually they didn't plan in advance and so their trying to cut corners to save time.
When half of my coworkers on a Zoom call (including myself) got kicked within ten seconds of each other, I figured something major had happened. Western PA. Was somewhere around 11:24 EST.
Network never went down completely, bitrate just took a nosedive.
Issues with FiOS in the Boston area. Traceroute seems to point to an issue originating in NY but that's just speculation. Apparently a lot of other services are down. Might be a Tier 1 or 2 ISP.
Imagine a world where all our enemies DIDN'T know how fragile our infrastructure is. False security, surely, but still would be somewhat comforting. :)
Firstly, America should strive to not have enemies. We fail at this very badly.
Secondly, ZERO countries in the world have non-fragile infrastructure. America's infrastructure is better than most countries and definitely above average even among developed countries, except public transportation.
> Firstly, America should strive to not have enemies. We fail at this very badly.
This will never be possible, even if America does nothing 'wrong' against anyone else. As long as America (flaws and all) is a better place to live than other countries, and can be used as an example of how things can be better (though not perfect), and so will be a target of powers that don't want their own people thinking about improving things.
There are also plenty of bullies out there, who are only checked by American power, and wouldn't mind the US being more isolationist.
There are all sorts of problems of how America interacts with the world, but there are many other countries who would do worse things IMHO. America simply existing generates enemies.
I disagree it's our fault for having enemies if we want to keep our values but regardless we have enemies now and I don't see them going away for a while.
The problem isn't our enemies knowing how fragile our infrastructure is as much as it Americans not knowing (or not caring) and being unwilling to spend money to make it less fragile.
> Americans not knowing (or not caring) and being unwilling to spend money to make it less fragile.
And of course the fact that for the most part those most impacted by these outages aren't the ones making the decisions about what money to spend in the first place.
I have no meaningful influence on my ISP's network backbone. I don't even have a real choice in ISP so it's not like I can go to someone else, and they know it.
I currently have 500mbit/sec service from my cable company and the only other options are ~50mbit/sec 4G or Frontier DSL that claims 24mbit/sec while rarely syncing beyond 8. Those aren't actual alternatives. If I want to have a connection that can properly support a household full of gamers and streamers, I'm stuck with this unless I want to add another zero or two to my monthly bill for a dedicated fiber line.
This is the best case scenario for the majority of the US. One cable company that's mostly decent, one shitty DSL provider, and a few wireless services that have very limited spectrum and data caps. Almost no one in this country has true competition in wired broadband unless they have a municipal fiber service or are lucky enough to be in one of the few cities that Google dumped money in to.
Honestly, at this point I've found HN to be the best "Status Page" for this sort of thing. I came here first when I found issues and fiddling around with my custom DNS lookup hierarchy didn't help.
downdetector.com did well today. Front page showed a generalized web issue, and any one of the individual services' status pages showed it was geographically localized to the midatlantic and northeast US.
It set off the flamewar detector. We monitor that list and eventually cancel the penalty for threads that aren't actual flamewars, but it takes a while. If anyone wants to make it faster, emailing hn@ycombinator.com is usually a forcing function.
Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.
That's answered in my sibling (well, cousin) comment here: https://news.ycombinator.com/item?id=25921757. It can't test for flamewars directly so it ends up guessing wrong sometimes.
> How does one programmatically detect a flamewar? :)
An NLP/Machine learning model would be a great way! Probably also taking in other features as input such as the number of comments per second, and how deeply nested those comments are.
I'm glad I'm not the only one noticing this. I could usually rely on HN to not be so heavily "filtered" for a lack of a better term but I have noticed the slide into a more controlled/gamed PR outlet.
My corporate laptop is affected. My roommate’s corporate laptop is totally unaffected. We’re on the same router/gateway.
Any theories out there to explain this phenomenon? Fascinating.
(My theory is a network partition, where my roommate’s IP addresses happen to be in regions that are on “our side” of the partition, whereas my IP Addresses are on the “other”, unreachable side).
Are you using different DNS servers? My work laptop was using 8.8.8.8 and is getting name resolution errors, but my personal computer is using 1.1.1.1 and working fine.
Completely unable to load anything today from 11:30sh to 12:30ish EST in Brooklyn, all seems to be smooth now. First ever issue with FiOS in two years, would still fully recommend them to anyone when comparing to most options, especially Comcast.
I have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to see what the experts think.
So, for clarity, it's never one fiber line being cut. It's a big massive bundle of fiber carrying different circuits for different carriers different places that were run in the same trench and all got cut at once.
Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one.
Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.
And when it's a large bundle cut, it's going to take a bit before service is restored. I have been told that a highly skilled tech can splice a single strand in a matter of minutes. Looking it up, I find figures of an average of 30 minutes per joint for 4-strand bundles, and slowing down from there. So if you have a 144-strand bundle to repair, or multiple 144-strand bundles, it's going to take several hours to effect the repair, and you're probably going to have to rotate through techs to avoid fatigue-induced errors slowing things down even more.
It is also, as a matter of practicality, going to be very difficult to find 144 colors that can‘t be mistaken for one another, particularly in conditions that i would imagine wouldn‘t have great lighting.
If they’re all in use and it’s point to point, might be easier connecting whatever to whatever. Then simultaneously swap line cards around as they get lit up or letting IP do its thing.
Not as feasible as it sounds initially. I don't know if you've ever looked inside a cat-5/6/whatever networking cable[1], but if you do, you'll see that it is actually a bundle of 8 individual wires, in 4 twisted-together pairs, color coded. Those different wires carry different electrical signals, and if the various signals are not supposed to be on the various wires they are supposed to be on, the connection won't work.
So imagine if you cut a Cat6 cable in half, and then spliced the individual wires back together without regard to which wires you connected. Very likely, Pair 1 negative is connected Pair 3 negative, Pair 2 has its polarity reversed, etc, and so the network link never comes online.
At the individual device level, fiber cables are connected in pairs, in a crossover fashion. Device A Tx connects to Device B Rx, and vice versa. So with a random splicing, you would end up with pairs getting crossed or broken up, invalidating the connections and labeling of every patch panel downstream, making for yet more work than the initial work of splicing the cut bundle.
1. Not impugning your intelligence, knowledge, or character here, some people never have because they've never needed or wanted to, and that's totally okay.
This. Even if you have redundant lines with multiple providers there is a good chance they are part of the same fiber bundle. It can be really difficult to get providers to admit who they are leasing from and what the physical runs are.
While I'm also not qualified to diagnose the problem, I would imagine it's something like this...
The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic.
The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)
The odd thing about this situation though is the geographic location of Brooklyn: on an island. Why would a significant amount of traffic from outside Long Island be going across the harbor to Brooklyn, especially given all the major switching facilities in Manhattan (closer to shore).
What makes you think it is only connected to Manhattan? Long Island has submarine cables connected to New England, various points along the East Coast south of NYC, and several transatlantic cables.
My naive explanation: fiber networks handle incredible loads, and a single fiber line might correspond to 10+ Tbs (terabytes per second). So all that data needs to go somewhere else, and if it gets shunted, packets get backed up, resulting in a cascading failure.
Note: I am not a fiber technician nor do I work for a large ISP.
If a 100 Gbps link goes down, and all the other links that could carry that route are only 10 Gbps... it goes downhill quickly.
Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
Just to add to that, sometimes the redundancy doesn't work out as planned, maybe another weird set of circumstances, or some routers can't handle spikes in load as the network reconverges, causing other reconverges, confusing bgp routing peers, etc. Fun stuff!
Same, also fios in Brooklyn. Not getting my full speeds but I am getting 100Mbps, which is far better than the 60 Kbps I was getting.
Ironically things recovered within 30 seconds me of messaging our systems team to ask if the development VPN was having an outage. That's like a magic rule of internet outages, isn't it? Nothing works until you try to show someone else the issue, then it all works.
Small world, I'm in the same place. I noticed long periods of dropped packets, it looks like existing connections are ok but establishing a new one is a problem. My wife working over a VPN doesn't notice a issue.
Too much is failing from single point outages. The FCC may have to establish standards to prevent over-centralization.
In the entire history of the Bell System, no electromechanical central office was ever down for more than 30 minutes for any reason than a natural disaster or, one occasion, a major file. That record has not been maintained in the Internet era.
The Bell System's job was to operate a nationwide telephone system for voice, not high-speed data. Even then, you only had one path to the telephone CO and only one CO it could go to. Outages were rare because voice-grade wire is cheap and all the complexity is handled by the CO.
Also, while my Internet may have outages from time to time, I can't recall the last time I couldn't use my cell phone other than being in a bad signal area. For voice, cellular is definitely the way to go as capacity is easily added by adding new towers; not something easy to do with landlines.
What do you mean by saying the wire is cheap? Obviously you can’t send much high frequency data down the wire but each wire could only be used for one call at a time for much of the company’s history, so more wires were needed. What effect does that have on the cost? And surely the complexity is still in the CO with the internet, or am I missing something?
> And surely the complexity is still in the CO with the internet
Your cable modem (which has a CPU and does high-speed signaling) or your FIOS box is much more complicated than an analog phone.
The carrier-grade router on the other end of your cable modem is really just a bigger version of routers you buy off the shelf or for enterprises. It's literally routing IP packets in the same way.
Whereas you have to have a whole switching system at the other end of an analog voice CO. (Or an operator making physical connections if we're still in the 1800's). The best you can do for yourself in that world today is get an PBX system like an old NEC NEAX box or similar, but you're still not going to be on the same level as the CO.
They can get temporary cellular devices to fill in on generator power pretty quickly. I know most if not all phone lines are underground, but something like a 2000-pair "wet pulp" repair (necessary in the first place due to negligence) takes a week. I don't know the name of these devices but something of that nature was deployed in Nashville on Dec 25 when the AT&T building was bombed and I think most cell service downtown was restored the next day.
> Maybe a new administration will put as the head of the FCC someone that is "good of citizens" minded and uses a normal size coffee cup
Someone who is interested in anything beyond personal profit, status, and influence, would be nice for a change. Also someone who is smart would be very nice.
We have FiOS in Boston and noticed it. Nothing terrible but for ~15 seconds at a time, recurring sporadically, we lost connection. Happened for a few hours, maybe 10am-1pm EST.
Based on other comments we probably just didn't notice the big actual outage that happened all at once.
Is this still ongoing? I'm having issues that I can confirm that as a result of this, and now three days later things are still spotty. Could be conflating multiple problems, of course.
No problems in Jersey City all day. Bleeping Computer article cites someone as saying it's a specific AS that's affected. JC routes through an AS in Newark, NJ and not NYC so, for me at least, there hasn't been an outage.
Hey neighbor, I disagree, though switching my VPN gateway to our DR one in NJ vs our primary in Chicago cleared things up. For about an hour and a half though, remote desktop was struggling, even though my speed tests were fine. I'm on FIOS.
I saw some disruptions in services that people in my household rely on, but I think that is because of disruptions in DCs located in NYC and not FiOS itself.
Issues here too, I'm on Verizon Fios. Right when starting class, what a mess. (As a colleague joked, Google going down (or Verizon I guess) is like the modern snow day)
I thought the whole point of the internet and protocols like TCP/IP is that it's supposed to be fault tolerant and traffic is supposed to automatically re-route. Seems kind of weird how a single cable being cut can cause half the country to lose internet...
TCP doesn't know anything about route failure or rerouting. All it does is handle a few dropped or reordered packets here and there.
In principle, you can have systems that very quickly route around damage like this, but in practice it happens so rarely that it's not worth it for consumer ISPs. I think it should exist, but the ISP space is so burdened with red tape that the only way anyone could compete with it was literally to start a rocket ship company and launch a bunch of satellites.
The Internet is not as redundant as you might think, and in fact there is less spare capacity to go around as a result of the pandemic (a lot more time spent streaming and on video chats by a larger number of people). Internet routing is also non-trivial (convergence has not been reached in years), and when a major link goes down it can take time for routes to stabilize. Also keep in mind that the Internet itself was not down for most people who were affected by this; rather, connectivity between users and various services was affected. So people could have continued torrenting, but their Zoom sessions would get dropped because connectivity to some data center was impacted.
I wonder how much has to do with the cost-optimization pressure on a private carrier. Building robust infrastructure is only valuable to them insofar as it enables them to continue to earn money. Reliable service for the sake of reliable service--especially to residential regions--is a non-starter.
Less pessimistically, it's possible that the regions that're affected have natural "choke points" for infrastructure, either moving through the mountains or along highways, etc., which of course leads to a single-point-of-failure, though it may be unlikely.
The ability for the internet to reroute after N path failures is there but someone has to pay for N paths worth of unused network to be ready for it. Most would rather just pay low cost and utilize all the bandwidth that can get them 364/365 days a year. For others (usually businesses) there are more expensive options.
FIOS is WAY better, they will fix their broken cable. I get (so far unlimited) 30gb up/down for $40/mo but it typically runs closer to 60 down 70 up. Fiber optic is baller. Though I do feel like they throttle me (web browsing becomes slow) if I am watching a streaming video service or mining crypto without a VPN on, could be an illusion but it always tends to clear up as soon as I activate the VPN.
60-70 gigabits per second for $40/month? Guessing you mean megabits. Not worth being packet inspected. 1gbit on comcast seems to give 42 mbps upload, was 50 a few months ago. ~$80/mo
Comcast upload is a farce. They don't bother stating it anywhere, I presume because they modify it as needed to satisfy their needs for providing TV channels or their "burst" download speeds.
They never advertise upload bandwidth anywhere, and it varies hugely when I'm at other people's houses that have cable internet. I assumed it's because they might provision 200mbps down / 10mbps up for a neighborhood, and then throttle you down as necessary.
i think so, but i had to upgrade to a docsis 3.0 modem, but told the physical infrastructure in place here cannot exceed 35mbit up. im not even going to pretend to know enough about cable to postulate how possible that is
The download isn't a lie and it's what they advertise. I was getting 800mbps before I updated my home router from a 2014 model to a 2020 one, now I hit 950-960. I wish the upload was symmetric, but I understand that's not always possible on copper, getting about 50 now. Not bad for a ~50-year-old apartment.
The problem isn't with copper, it's with the cost of building out amplifiers. It's a lot more cost effective to amplify downstream than upstream, especially when most use cases are heavily asymmetric.
(I'd expect that only in the past year or so has upstream been a lot more heavily utilized, what with massively increased work-from-home due to the pandemic. Before then, the primary stressors of the network were things like streaming 4k60 video.
And I'll be very surprised if any ISPs thought torrent uploads to be a use case worth optimizing for.)
I love these posts. They remind me that there are places in the world where technology is moving forward. Where we live in Tennessee I have to run 2 internet connections to be able to have a service when I need it. 3gb DSL and a 5gb satellite connection. The sat connection has the additional bonus of being capped at 150Gb per month. All these new streaming services may as well not even exist for those of us in rural areas.
3 gigabits per second DSL and 5 gigabits per second to a satellite? You (like the other commenter...) probably mean megabits. And also Gb is gigabits, but your cap is probably in bytes, gB or GB.
I’ve got Comcast 2 gig fiber and Verizon 1 fig fiber to my house, plus 1 gig cable. I live 10 minutes from horse farms in MD. (The horse farms also have fiber.)
I've never seen Comcast offer fiber, except to businesses at astronomical costs. I can't even find it on their website. Do they advertise upload bandwidth for their fiber?
It’s expensive, $300/month. I try to justify it to myself by noting it’s on their business Metro-E network so it’s completely reliable and has amazing support. Also an entire /48 of static IPv6. It was $150/month for the first two years, which was much more palatable.
Their model is expensive because they don’t wire while neighborhoods. For me they had to run fiber a quarter mile, down my street, and trench under my driveway. I imagine all these one-offs make the per house investment quite high.
That's not bad, I'd be willing to pay that and even more. I was simply unable to contact anyone at Comcast who was willing to entertain running fiber to a home. So I simply had to buy a home that had an option other than Comcast.
In another instance, I needed symmetric fiber at one of my businesses. I contacted Comcast, the business/commercial departments multiple times, got numerous peoples' names, left voice and email messages left and right and got nowhere after weeks. I ended up contacting a different ISP, signed up with them, and then on the install date, Comcast showed up to install the fiber modem (who were, of course, reselling it to the ISP I contracted with). I ended up paying the other ISP $1,700 per month.
The Gigabit Pro support story is very odd. The initial contact goes through the consumer support folks, who are clueless the service even exists. The preferred solution is actually getting on the Comcast Xfinity reddit site to reach a support tech who knows what's going on. After that you're handed off to a Comcast Small Business support agent, who is a real person who maintains contact with you throughout the process (which is a bit involved because it can require permitting, etc.). After that you get a support number that goes to the Metro-E NOC, which is phenomenal. But the billing website still goes through the customer online portal, which doesn't quite understand what's going on.
This section is an amazing indictment of the company that tens of millions of Americans have to deal with to get access to the internet.
>The Important Part
Because so many people had this question, and I finally talked to someone at Comcast who knew what Gigabit Pro actually is, here is how to get in touch with them:
I'm pretty sure I have Comcast fiber, its 1Gbps down and around 750Mbps upload and there's a wire coming out of the wall into the router so... ¯\_(ツ)_/¯
I priced having a 10m fiber connection brought out here. I was quoted just over $5k/month with a minimum 3 year contract. I assume they were expecting me to just pay for the new service to the entire area for them to resell.
Wife wanted to live out here. So we traded my being able to work easily for amazing views and lots of land. Not entirely unhappy with the trade, but there are days when I think about just renting an office in town. (30 minutes away).
I was basically restricted to buying homes in new developments since 2010. Those all tended to have fiber, and anything before did not, and they had no plans to install it. I pay $65 per month for 940mbps up and down from Centurylink, and it's been as advertised so far.
Unfortunately, the cable company usually offers some ludicrous 500mbps (burst) download / 5mbps upload connection that people are willing to accept for cheaper, so if the phone company comes in and pays to install fiber and charge accordingly for the new fiber runs, insufficient people would switch to the fiber to make the numbers work since they are okay with the shitty cable company connection.
Haha. I have a multi generational household. 4 adults, 2 children, and we're heavy Facetime/Apple TV users. 4 to 8 HD simultaneous video streams is normal, plus the security cameras. I also wanted to be able to stream from my server at home anywhere I am. And with 2 people working from home, it was an amazing decision.
I bought a house 1+ year ago and had "good internet" connection on my must-haves. I remember visiting a house during the search and asking the listing agent about it. The guy was super candid, he basically told me no, there was no cable internet and that was the reason he knew it would be sitting on the market for a year at least.
To be fair, I've had FiOS for more than a year now, and this is the first incident I've had with them. I'm pretty comfortable saying they have a much better reputation than Comcast...
I was a long-term Fios customer in SoCal, and they were indeed very reliable up until Verizon sold the local service to Frontier. Glad to hear they're not a problem for everyone at least.
Fios service is fantastic, and this morning's outage barely impacted me in Jersey City (only some disruption that seemed to be related to NYC area DCs more than Fios service itself). I have found their service to be very reliable for years now, with throughput that does not fluctuate (I get what I pay for regardless of the time of day) and lower latency than I had with Comcast. No regrets at all.
The only real complaint I have is that Verizon still seems to be unable to deploy IPv6. They have promised it for years and never seem to get past the testing phase.
To be fair, that was just Ajit Pai, who also thought net neutrality was not important. Now that he's gone, I hope he gets subjected to 3mbps upload speeds for the rest of his days.
> “We find that the current speed benchmark of 25/3 Mbps remains an appropriate measure by which to assess whether a fixed service is providing advanced telecommunications capability,” the report reads.
Not "3mbps upload speed is all internet users need".
If 100 Gbps of capacity suddenly went away, then BGP attempted to route around it, and sent that traffic down 10 Gbps routers instead, which overloaded, or were not ready for the deluge of traffic, then those BGP routes could start flapping, at which point those routes would get pulled, forcing it onto yet other circuits, which may be 1 Gbps and that's how the outage continues to grow.
Too much capacity is centralized in certain providers, and if they go down there simply is no backup route that can handle the same traffic patterns.
Yep, and it's possible that running low on ipv4 address space exacerbated the situation, so many more small routes than there used to be, maybe not all getting summarized during an outage they way they used to be, causing problems on peer bgp routers like you said.
Could be that. Also could be too much complexity, as opposed to capacity. Could have been timing didn't help, sometimes you take calculated risks or have to deal with additional complexity during migrations, etc.
I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm...
I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system.
As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two.
It might sound conspiratorial, but when the WTC was attacked in 2001, the Internet slowed down a few days after. We'll probably never know, but I assumed it was deeper surveillance being implemented.
Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.
Why not take the simpler interpretation that millions of dollars of equipment was in the buildings that were destroyed, and it took a while for the network's self-healing algorithms to discover a new steady-state configuration that approximated the performance of what was suddenly lost?
One thing they always fall down is the idea that government bureaucracy is so quick and competent that they could do something like installing mass surveillance at a moments notice while the country is in chaos without it leaking due the next 20 years
It didn't have to be installed, it was already, although in a pre-Snowden era, governments doing Internet surveillance was the conspiracy theory. My speculation is that more international traffic got routed through the existing surveillance infrastructure, causing a bottleneck that slowly got better.
Occam's razor is a guide to start with the simpler explanation. If the simpler explanation is not sufficient, you add more elements until it is.
My point is, the Internet rerouting around damage doesn't explain the slowdown, mostly because it happened days after, not immediately, meaning the damaged infrastructure was not critical for the global internet.
The attacks unleashed a huge intelligence operation, and it's reasonable to expect a parallel intelligence operation online. If the latter caused a traffic bottleneck or not, we can't know because it was never addressed, it's speculation on my part.
Because it happened a few days after, not immediately, and there was nothing on the news about that. It was working fine immediately after, although people then mostly got their news from TV; a major event didn't increase bandwidth usage as drastically as today.
There is a fiber cut in Brooklyn. We have no ETR, as of yet. You can use the MY Fios app for updates. *EAG