Most testing has been on a Lan or on the local machine for local VFS use cases.
This layer eschews responsibility for multiple concurrent clients for simplicity. No promises you'll get either decent performance or proper cache invalidation when using it that way. It tends to be conservative in not filling in all the opportunistic caches.
There's a hook the backing application can use to tune reads and write sizes that becomes much more relevant in a network case. How those should be set in practice isn't something I've spent time on.
This turned out easier than i was expecting it to be. It's nice to be able to mount a VFS without needing privileges on the server side.
The main intention for this code is to eventually use it to replace fuse on mac, since nfs is a valid mount time for mac clients to consume.
This is great! SOOO useful to be able to do mounts w/o fuse, w/o kernel extension, w/o root. This is a HUGE UX thing, specially for macs.
Fuse is a really great idea & project, but sadly today impls require a lot of install steps in some platforms, and some have painful bugs/UX. Amazing if we can mount VFSes from Go without those hurdles.
- Under Linux and FreeBSD, bazil fuse offers a nice low dependency path to supporting FUSE. It is all pure Go code.
- Under Windows, you can use WinFSP, which is a rock solid usermode filesystem driver with FUSE support. Despite its name, you can use this with Cgofuse to get FUSE on Windows without CGo.
Combining the two above gets you pretty far already (though if you want to actually avoid a CGo dependency you do need to explicitly disable it.)
I think the next level would be to have a general filesystem interface (there are several options in the ecosystem already) that can be used as a common ground between different usermode filesystem and network filesystem server implementations. Then ideally, a magic Go library can exist that gives you the best possible setup with switchable backends depending on the platform and build configuration.
Right now Go itself is possibly standardizing a read-only filesystem interface, an extension of this with write support might serve as a decent jumping point for getting toward an idealized pluggable filesystem library.
It won't be enough for the likes of FUSE, though. Some extensions might be enough to bridge that gap, it's too early to tell. FUSE is pretty particular about things like rename(2) preserving node identity.
The fuse replacement is super interesting. So you’d basically have a user space utility that provides NFS<->fs bridging? Would you support the existing fuse API? What does NFS not allow that Fuse would, if anything?
one of the major attractions of the fuse abstraction layer is that once you have fuse you can interact with more exotic file system types without having to deal with new kernel drivers each time.
Instead though, you can translate the exotic file system types to nfs, and use the existing kernel drivers that already exist for mounting an NFS mount.
Very cool! With "only" 3.2k lines of code for a NFS implementation I was indeed surprised that it was so easy. As for the privileges, how do you deal with the issue of port 111 being a privileged port?
For BGP partitions / Significant network outage conditions:
You may not find content partitioned on the other side of a netsplit from you, but the degraded condition is that you can still query and find content present in the same part of the network as you. This is better than centralized solutions face at present - e.g. google not being available in China, and isn't too far off from what you might hope.
For Sybils:
You've left the attack you're worried about pretty vague. IPFS itself doesn't need to tackle many of the sybil-related issues by being content addressable (so only worrying about availability - not integrity) and not being a discovery platform - so not worrying about spam / influence. For the remaining degradation attacks - someone overwhelming the DHT with misbehaving nodes - there's been a bunch of work in this release looking at how to score peers and figure out which ones aren't worth keeping in the DHT.
> You may not find content partitioned on the other side of a netsplit from you, but the degraded condition is that you can still query and find content present in the same part of the network as you.
This becomes hugely problematic the minute you start using IPNS. On one side of the split, the name `foo` can resolve to `bar`, but on the other side, it resolves to `baz`. If you're trying to impersonate someone, then a netsplit would make it easy for you to do so (barring an out-of-band content authentication protocol, of course).
> You've left the attack you're worried about pretty vague. IPFS itself doesn't need to tackle many of the sybil-related issues by being content addressable (so only worrying about availability - not integrity) and not being a discovery platform - so not worrying about spam / influence.
On the contrary, a Sybil node operator can censor arbitrary content in a DHT by inserting their own nodes into the network that are all "closer" to the targeted key range in the key space than the honest nodes. This can be done by crawling the DHT, identifying the honest nodes that route to the targeted key range, and generating node IDs that correspond to key ranges closer than them. Honest nodes will (correctly) proceed to direct lookup requests to the attacker nodes, thereby leading to content censorship.
Honest nodes can employ countermeasures to probe the network in order to try and see if/when this is happening, but an attacker node can be adapted to behave like an honest node when another honest node is talking to it.
> For the remaining degradation attacks - someone overwhelming the DHT with misbehaving nodes - there's been a bunch of work in this release looking at how to score peers and figure out which ones aren't worth keeping in the DHT.
Sure, and while this is a good thing, it's ultimately an arms race between the IPFS developers and network attackers who can fool the automated countermeasures. I'm not confident in its long-term ability to fend off attacks on the routing system.
Censorship / availability is the issue. As you lay out, Sybils can be used to target a specific address / piece of content in an attempt to prevent it from being found. The good news is that there are some pretty mechanical things - like maintaining a consensus of known-trusted nodes that can be used to validate a node/piece of content isn't under attack - that should be sufficient until IFPS is quite a bit larger than it is today
I don't think I see the impersonation / bad resolution problem though. IPNS records are content addressed to the key. Having control of a portion of the network isn't sufficient to compromise that (you can prevent availability though).
> The good news is that there are some pretty mechanical things - like maintaining a consensus of known-trusted nodes that can be used to validate a node/piece of content isn't under attack - that should be sufficient until IFPS is quite a bit larger than it is today
What would I be trusting the nodes for? If I'm trusting them to just keep my data available, then why not just put it into S3? What role is IPFS playing at all, then, if I find myself having to pick trusted nodes to defend against low-cost route-censorship attacks?
Also, the size of the network doesn't really seem to make large DHTs resilient to Sybils. BitTorrent in 2010 had over 2 million peers [1], but north of 300,000 of them were Sybils [2]. That's pretty bad.
> I don't think I see the impersonation / bad resolution problem though. IPNS records are content addressed to the key. Having control of a portion of the network isn't sufficient to compromise that (you can prevent availability though).
Correct me if I'm wrong, but IPNS resolves a human-readable name to a content hash, right? If all I'm going off of is the name and the DHT (no DNS), then having a network that can return two (or more!) different content addresses for that name can lead to problems for users, no? If IPNS/IPFS is supposed to be a hypermedia protocol bent on replacing HTTP/DNS, then its inability to handle the case where `google.com` can resolve to either the legitimate Google or a phishing website sounds like a showstopping design flaw that just begging to be abused.
IPNS names are based on public keys. A sybil attack can't make an IPNS name resolve to an arbitrary attacker-chosen value because the attacker can't make signatures for the public key.
A sybil attack could be used to cause part of the network to see a recent old value for an IPNS name, but clients keeping recent values cached for a while is already something that happens naturally, so it just seems like a more minor example of the general problem that a sybil attack could do a denial-of-service.
The release notes contain a pretty extensive list of what APIs have changed.
The path to upgrade the dependencies is probably to run `go get -u <dep>` for the direct dependencies you're including, and then fixing errors that pop up from doing so.
Depends what exactly you are importing and for what. I suggest you ask in the forums and post a link to your project. go get -u is dangerous as it updates all subdependencies used to latest.
Usually take go-ipfs's go.mod as a guide on what versions to use.
>go get -u is dangerous as it updates all subdependencies used to latest
Yes, this is the problem I initially had; I would end up with a host of incompatible deps (leading me to wonder what the point of go.mod even was ... but I digress...).
Sounds like this is going to be one of those super fun dependency hack-jobs :C
Ah well, this is the price to pay for playing with beta software!
IPFS is actually a better fit for this than it might appear.
It doesn't pro-actively replicate content, so it isn't like bit torrent where everyone is uploading.
Instead, it provides resilience, where the actual location of sci-hub could move around, but requesters could still get access to the content. The rest of the network acts to help with routing, but won't be serving data they haven't accessed themselves.
https://repeatr.io/welcome/what-is-timeless/ explains timeless as a set of tooling for reproducible builds with a broader and more hash-based mandate than what's out there currently. While early, it seems like it's a reasonable path towards getting a reproducible system that won't fall apart as fast over time, by vendoring the dependency system in the same way modern language-specific package managers do, but at a language-agnostic/distribution level.
Is there any particular difference with https://nixos.org/nix/ which is backed by academia and has years of experience? Never heard about Timeless though.
FWIW, I'm using Nix to package Android app builds in a reproducible way (for React Native apps).
I've gone through their website and I think I have a reasonable grasp of the difference. Unlike NixOS and Guix which enforce a store, all builds are performed in a container[0] and produce a self contained executable[1].
I speculate that the container approach will make porting slightly easier as dependancies can be placed where they are expected, thus no patches are required for project build systems. On the other hand producing a self contained executable seems more difficult, and I wonder if they'll go with an AppImage like approach to minimise patches required.
Overall the vision has overlap but the end result is very different from that of NixOS and Guix.
[0]:
> Typically, we use an executor which uses “linux containers”. However, for the most part, we try to consider that an implementation detail. Executors might also be simple chroots; or virtual machines; or other interesting kinds of sandboxing. Nor are executors strictly limited to the Linux operating system; it's simply the most common place to work.
https://repeatr.io/design/executors/
[1]:
> Radix “Packages” are similar to packages from other linux distributions… however, we're aiming to do something not like linux distributions: our vision for Radix Packages is that they go anywhere, with or without the rest of a distro associated with them.
https://repeatr.io/welcome/what-is-radix/
VLDB also just published a paper on a similar, improved bloom-like data structure
"Vacuum Filters: More Space-Efficient and Faster Replacement for Bloom and Cuckoo Filters"
http://www.vldb.org/pvldb/vol13/p197-wang.pdf
It would be interesting to get a comparison between Xor and Vacuum.
To me it looks like a version of the cuckoo filter that isn't restricted to a 2^n size. We made the same change in the benchmark of the xor filter against the cuckoo filter (and others made the same change). There are some more changes, and the paper claims the vacuum filter is faster than the cuckoo filter. I will try to add the vacuum filter to our benchmark suite at https://github.com/FastFilter/fastfilter_cpp and see what results I get.
The patent restrictions make Apache 2.0 less permissive (even if you think they're a good thing.) They cause practical problems, even for well-meaning free software developers, because they make the licence incompatible with GPLv2.0, which means, if you use an Apache 2.0 library in your program, you can never use any GPLv2.0 licensed libraries or vice versa.
This layer eschews responsibility for multiple concurrent clients for simplicity. No promises you'll get either decent performance or proper cache invalidation when using it that way. It tends to be conservative in not filling in all the opportunistic caches.
There's a hook the backing application can use to tune reads and write sizes that becomes much more relevant in a network case. How those should be set in practice isn't something I've spent time on.